Splunk's api how we built it
•
4 likes•1,439 views
These are the slides for my September API Craft SF talk on how we built / design Splunk's API. You can find the video here: https://www.youtube.com/watch?v=vHXcDKK4eGY. My talk starts at about 60 mins. The other two are on Uber and Sift Science and you should watch them as well!
Report
Share
Splunk's api how we built it
- 1. Copyright © 2014 Splunk, Inc. Splunk’s API How we built it!
- 2. Agenda Why we built Splunk’s API How we built Splunk’s API
- 3. What is Splunk A product for handling REALLY large and varied sets of evented data
- 4. What is Splunk From megabytes to hundreds of terabytes daily
- 5. What is Splunk It is highly scalable and distributed
- 6. What is Splunk Useful in many domains: IT/Ops/DevOps, security, healthcare, financial,IOT/Devices
- 7. 7 Splunk storage Other Big Data stores Developer Pla6orm Data collecUon and indexing Report and analyze Custom dashboards Monitor and alert Ad hoc search
- 8. Why – Product need Need to build a Splunk UI that surfaces all of Splunk’s capabiliUes
- 9. Why – Customer need Need to allow customers to integrate their applicaUons and scripts with Splunk
- 10. Why – Cost Less code to maintain, a single source of truth
- 11. Why – Reach HTTP is ubiquitous, every plaZorm has a client
- 12. The arch 12
- 13. Server Tech Stack 13 C/C++ Python – Cherry PI Python – Django Javascript Node
- 14. Client Tech Stack 14 Javascript Node Python Ruby PHP C#
- 15. The API 15
- 16. Log directly to Splunk via HTTP Run historical and real-‐Ume searches What can you do with Splunk’s API? 16 Search Manage Add/Delete Users ReporUng/Alerts Manage Inputs ConfiguraUon Index Login to a Splunk instance and get a session token Auth
- 17. The API design 17 Service Categories Endpoints Endpoints
- 18. The API design -‐ Categories 18
- 19. The API design – Endpoints 19
- 20. The API design – Endpoints 20
- 21. Responses -‐ Feeds and Hypermedia <entry xmlns="hjp://www.w3.org/2005/Atom" xmlns:s="hjp://dev.splunk.com/ns/rest" xmlns:opensearch="hjp://a9.com/-‐/spec/opensearch/1.1/"> <Utle>search index</Utle> <id>hjps://localhost:8089/services/search/jobs/mysearch_02151949</id> <updated>2011-‐07-‐07T20:49:58.000-‐07:00</updated> <link href="/services/search/jobs/mysearch_02151949" rel="alternate"/> <published>2011-‐07-‐07T20:49:57.000-‐07:00</published> <link href="/services/search/jobs/mysearch_02151949/search.log" rel="search.log"/> <link href="/services/search/jobs/mysearch_02151949/events" rel="events"/> <link href="/services/search/jobs/mysearch_02151949/results" rel="results"/> <link href="/services/search/jobs/mysearch_02151949/results_preview" rel="results_preview"/> <link href="/services/search/jobs/mysearch_02151949/Umeline" rel="Umeline"/> <link href="/services/search/jobs/mysearch_02151949/summary" rel="summary"/> <link href="/services/search/jobs/mysearch_02151949/control" rel="control"/> </entry> 21
- 22. Auth 22 HTTP Basic Token based LDAP/AD Cookie based
- 23. Auth – HTTP Basic 23 curl -‐k -‐u admin:changeme hjps://localhost:8089/services/auth/login -‐ d username="admin" -‐d password="changeme"
- 24. Auth – Splunk Token 24 curl -‐k -‐H "AuthorizaUon: Splunk SfH2D^zvPyLu^mO61C9kWtB7TOuQs0i9oSzh4lD7ho7Gvw26I61VYRjXkgj LQlJDJ0hER^q^A6v0BHYiKNba^CMbOmC63frGCrDqr2Zt" hjps:// localhost:8089/services/search/jobs -‐d output_mode="json" -‐-‐get
- 25. Search – Oneshot – Get me results! 25 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot"
- 26. Search – Oneshot – Get me results in json 26 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json"
- 27. Search – Oneshot – Get me json columns 27 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"
- 28. Search – Oneshot – Get me json rows 28 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"
- 29. Search – Blocking – Wait Ull done! 29 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5” -‐d exec_mode=”blocking” output_mode="json” | python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool
- 30. Search – List search jobs 30 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d output_mode="json" -‐-‐get | python -‐mjson.tool
- 31. Search – Normal– Run in the background 31 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 10000" -‐d exec_mode="normal" output_mode="json"| python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool
- 32. Search -‐ Export 32 curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/ search/search/jobs/export -‐d search="search index%3D_internal | head 100000" -‐d output_mode="raw"
- 33. Search – Export REALTIME 33 curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/ search/search/jobs/export -‐d search="search index%3D_internal" -‐d output_mode="raw" earliest_Ume="rt-‐1m" latest_Ume="rt"
- 34. Copyright © 2014 Splunk, Inc. Splunk’s API How we built it! dev.splunk.com splunk.com/jobs