Surviving A DDoS Attack: Securing CDN Traffic at CloudFlare

Surviving a DDoS Attack:
Securing CDN traffic at CloudFlare
Martin J. Levy, Network Strategy
CloudFlare, Inc.
MSK-IX Moscow Russia
December 4, 2014

24 December 2014 MSK-IX Moscow Russia - CloudFlare - Surviving a DDoS Attack - Securing CDN traffic at CloudFlare
DDoS Attacks are
becoming massive,
and easier to initiate
Today:

Major Attacks against
CloudFlare Customers
Mar 2013
309
Gbps
Feb 2014
400
Gbps
Feb 2012
65 Gbps
Next?
???
Gbps

CloudFlare

CloudFlare network locations
• Global: Deployed into 28 data centers in 20 countries
• Secure: Built into every layer and every protocol
• Robust: Every node can perform any task. Anycast HTTP routing
• Reliable: Built-in redundancy, load balancing, and high-availability

CloudFlare works at the network level

CloudFlare works globally

CloudFlare sample customers
Nearly two million websites

Chronology
of the major attacks

March 2013

The Spamhaus attack

Monday, March 18th thru 21st
“Annoyance” attacks, 10-80Gbps

Wednesday, March 20th
~75Gbps attack
“Instant on”

Then, it got real …

Sunday, March 24th thru 25th
Peaks of the attack reached 309Gbps
“Instant on”

February 2014

Monday, February 10th, 2014
400Gbps, Globally Distributed Attacks

The Evolving Landscape

The Evolving Landscape of DDoS Attacks
DNS amplification
Up to 300 Gbps
NTP reflection
Up to 400+ Gbps
(35% up from DNS amplification)
DNS infrastructure
100s Gbps
HTTP Application
100s Gbps
Sophistication
ATTACK TYPE TREND
• Volumetric Layer 3 / 4
• DNS Infrastructure
• HTTPS application
• Origin: 100s of countries
More sophisticated DDoS mitigation and larger surface area to
block volumetric attacks has forced hackers to change tactics.
New DNS infrastructure and HTTP layer 7 attack signatures that
mimic human-like behavior are increasing in frequency.
20142013

Layer 7 Attacks

Layer 7 Attack methodology
Exhausts CPU
Attackers use millions of
compromised machines to launch
a sophisticated attack that mimics
real users and overloads the slow
points in your web property.
A highly advanced attack that
mimics real users is detected and
blocked by CloudFlare before it
can overload the slow parts of
web server software.

DNS Infrastructure Attacks

Layer 7 – Malicious Payload
• Request sent to exploit vulnerability on server
• WAF on CloudFlare blocks 1.2 billion request per day
• Shellshock
• 10 to 15 attacks per second during the first week
• Top countries: France (80%), US (7%), Netherlands (7%)

Breaking Down the Attacks

DDoS mechanics

IP Spoofing
Attacker
Network Device
1.2.3.4
Hi, I’m 1.2.3.4,
and I need some info
Here’s your info
213.4.99.70
Target Website

IP Spoofing
http://spoofer.cmand.org/
25.5% of networks allow spoofing

Attack #1 – Spamhaus Attack:
309Gbps
UDP DNS

Getting to 309Gbps
10 Mbps 30,900X 309,000 Mbps=

An easier way: DNS
Amplification
Attacker
Open DNS Resolver
Target Website
64 Bytes
3,363 Bytes
~50x

How easy is it to
create a DNS query packet?

64 bytes becomes 3,363 bytes
$ dig ANY isc.org @63.21*.**.** +edns=0 +notcp +bufsize=4096
64 byte query
Sorryforthesmallprint
;; ANSWER SECTION:
isc.org. 7147 IN SOA ns-int.isc.org. hostmaster.isc.org. 2013073000 7200 3600 24796800 3600
isc.org. 7147 IN NS ns.isc.afilias-nst.info.
isc.org. 7147 IN NS ord.sns-pb.isc.org.
isc.org. 7147 IN NS ams.sns-pb.isc.org.
isc.org. 7147 IN NS sfba.sns-pb.isc.org.
isc.org. 7 IN A 149.20.64.69
isc.org. 7147 IN MX 10 mx.pao1.isc.org.
isc.org. 7147 IN TXT "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all"
isc.org. 7147 IN TXT "$Id: isc.org,v 1.1835 2013-07-24 00:15:22 dmahoney Exp $"
isc.org. 7 IN AAAA 2001:4f8:0:2::69
isc.org. 7147 IN NAPTR 20 0 "S" "SIP+D2U" "" _sip._udp.isc.org.
isc.org. 3547 IN NSEC _adsp._domainkey.isc.org. A NS SOA MX TXT AAAA NAPTR RRSIG NSEC DNSKEY SPF
isc.org. 7147 IN DNSKEY 256 3 5 BQEAAAABwuHz9Cem0BJ0JQTO7C/a3McR6hMaufljs1dfG/inaJpYv7vH XTrAOm/MeKp+/x6eT4QLru0KoZkvZJnqTI8JyaFTw2OM/ItBfh/hL2lm Cft2O7n3MfeqYtvjPnY7dWghYW4sVfH7VVEGm958o9nfi79532Qeklxh x8pXWdeAaRU=
isc.org. 7147 IN DNSKEY 257 3 5 BEAAAAOhHQDBrhQbtphgq2wQUpEQ5t4DtUHxoMVFu2hWLDMvoOMRXjGr hhCeFvAZih7yJHf8ZGfW6hd38hXG/xylYCO6Krpbdojwx8YMXLA5/kA+ u50WIL8ZR1R6KTbsYVMf/Qx5RiNbPClw+vT+U8eXEJmO20jIS1ULgqy3 47cBB1zMnnz/4LJpA0da9CbKj3A254T515sNIMcwsB8/2+2E63/zZrQz
Bkj0BrN/9Bexjpiks3jRhZatEsXn3dTy47R09Uix5WcJt+xzqZ7+ysyL KOOedS39Z7SDmsn2eA0FKtQpwA6LXeG2w+jxmw3oA8lVUgEf/rzeC/bB yBNsO70aEFTd
isc.org. 7147 IN SPF "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all"
isc.org. 7147 IN RRSIG SPF 5 2 7200 20130828233259 20130729233259 50012 isc.org. XDoOYzkTHEV1W1V4TT50SsqXn4cxNhPvEuz3iFjq/oskLY9UOaK4GYDO GqHAjwNT0B6pUakKTQ3GvBjUBufPcEauCOl7L7kb8/cC6zYifUCoW0pS moiQxmyqfrPDTzyVA894myUONGgMmB6QW68HGPVvc6HzGWx9bOmjvFyX uOs=
isc.org. 7147 IN RRSIG DNSKEY 5 2 7200 20130828230128 20130729230128 12892 isc.org. COfF8fU6a8TBUG97SI/X+u2eKv7/mw+ixD3IWBnr3d3cWZmzF1sV8bWT YbuJebwnJMgN5OfB9PLsN+4QT617OjBe1dUF2M9jZeBiWsSsvvrdHnHM P8KwX6eayByDUoFsYe9VAH6C94XmOVXTQ7h7Cr0ytaVSXUytqFZV+DGn
v3kqSi50V3YkFNPAJDqqs0treWjwV/SPlqWVqEAoU/KMZMtYpCEMbHVP 8nbRP3jj/10WLccPtjHhiw4Ka9Sk4o+b7BMYCgXGXlhaap21SUqkytHt 8RJdVxxd5Cj7Bi+O4LXODlS4bAZEDG7UoHR27MzXtvMZogsNyyNUKHSs FtUvBg==
isc.org. 7147 IN RRSIG DNSKEY 5 2 7200 20130828230128 20130729230128 50012 isc.org. Mbr/QqJPoIuf3K5jCuABUIG0/zSHQ8iWZpqvHx7olVBEmTxhi3/vW+IE DW6VCE1EpZclSIlMMRZUnbBnVSpe0rZ13BxoLlRQqvsbC3jD15Se41WB DscD0S63C0GLqI9IhSyVugtlpqhA3CaluSqtABHbAktPP05Rm00tST2Y A5Q=
isc.org. 3547 IN RRSIG NSEC 5 2 3600 20130828233259 20130729233259 50012 isc.org. V7G42xY7TY9wF1vsBlRFuJ2ror/QjftLoRrDCMfqFW6kb5ZswjKt5zho 4o2sIrylTqad68O+lMxrDcg+7c2D8Hdh84SC0DEkjunBXkGBtLtaJvO5 zMn+d/OgUY5O7wtkerybJwZeiHcFxIkMRIcvsPKJYZWKCdaaCWibne7c w1s=
isc.org. 7147 IN RRSIG NAPTR 5 2 7200 20130828233259 20130729233259 50012 isc.org. gWDvD0KACaYgsCgtRS4iKkHBBidfJfqS4drUf4kuPX2Etl9fj1YrqOQK QFB5kBrzJLKh1IF4YpV+KYVUF82l3AtpsohpUH5Uyc3yD3r1CUDVyVvc T9qUrIuRpZLInD2kBLmDaG76MRz4Fz+NAkdXmwxZJhgTrfMLy+Uw/Ktk H7w=
isc.org. 7 IN RRSIG AAAA 5 2 60 20130828233259 20130729233259 50012 isc.org. dfzIo0VGT0MptTaPoua3tFwDxSpeuOg127QedlqLGTxKGN1ppV/bd6R0 WktMagZY9rSqmjfXNPlF3Q+7YeTpMssQhHqjE/tDoj9q9r8RXuBLJ1+a VRq3+xMbxb5EXAyQVZw24LIuloqNprXePRUGCXNINSWd7VZEIDNqhu9C g7U=
isc.org. 7147 IN RRSIG TXT 5 2 7200 20130828233259 20130729233259 50012 isc.org. WtB3SYzcOKpNbOtBlnmtsI0DCbDB4Kiv/HBY24PTZyWF/3tI8l+wZ+/p MfJ/SblbAzT67DO5RfxlOhr8UlRKVa70oqinQp5+rqiS67lv1hGO6ArO k+J0jLTis9Uz32653dgAxlgjEdWDKAg4F12TaHirAXxyI8fos5WNl/h4 GLo=
isc.org. 7147 IN RRSIG MX 5 2 7200 20130828233259 20130729233259 50012 isc.org. BSXC42oV6MCF0dX2icyxnvyijhy569BJCoanm5VrIIuiNeTeo261FQJx 7ofFCWa4fKOoa+EZ0qloNPfDiczStr8MmK8Lznu6+8IRfdmcG/kURuSi JdvDa0swxjmCm9aYu2nhoyHs+jqbJ+9+fneI0iDUX1fiM+9G2K9BjLru NxU=
isc.org. 7 IN RRSIG A 5 2 60 20130828233259 20130729233259 50012 isc.org. Gmb8tt8d7kxx4HsA8L6IdFYGGSJCA8PTWexUP3CBLna39e4a6gVzjoNd dEI7B5mySAujZBEXNx3dSagpjiTJYfMML8AY0uO0tgyjqaTyzwPPV5lW xQKVC092BPJx9IeKw+DC57f3m9LOaHJlMIh7wYFn8jxqeg1lSwJN0e35 Qvc=
isc.org. 7147 IN RRSIG NS 5 2 7200 20130828233259 20130729233259 50012 isc.org. RBvXLeTH0726iKvElmBZYUE+AWG3s2YRxKxuCnrhg7o9qIQGKXvEXrb3 wJeC/74KY2FW+RRz4F0QxODnPm+frpWIPbCpRf0SUFDQ82opQDwAb2CM 0D9N95y1t9hYfSeHEsEEk2yWgLymd9/S24XCmwuVVZ7ZeYQmVEVkF7Jt V3A=
isc.org. 7147 IN RRSIG SOA 5 2 7200 20130828233259 20130729233259 50012 isc.org. iiDnH6tvmap0h2cdULI8Ihme+zbtQ2+D3ycKRqBc9TRfA0poNaaZ97aF 15EIKyIpjiVybkP2DNLm5nkpNsgA+Ur+YQ6pr0hZKzbDkBllBIW4C0LV DsjzPX3qLPH4G3x/20M+TeGe4uzPB5ImPuw0VxB8g8ZP5znvdiZG6qen jas=
;; AUTHORITY SECTION:
isc.org. 7147 IN NS ns.isc.afilias-nst.info.
isc.org. 7147 IN NS ord.sns-pb.isc.org.
isc.org. 7147 IN NS ams.sns-pb.isc.org.
isc.org. 7147 IN NS sfba.sns-pb.isc.org.
;; ADDITIONAL SECTION:
ns.isc.afilias-nst.info. 56648 IN A 199.254.63.254
ns.isc.afilias-nst.info. 56652 IN AAAA 2001:500:2c::254
ord.sns-pb.isc.org. 31018 IN AAAA 2001:500:71::30
ord.sns-pb.isc.org. 31018 IN A 199.6.0.30
ams.sns-pb.isc.org. 31018 IN AAAA 2001:500:60::30
ams.sns-pb.isc.org. 31018 IN A 199.6.1.30
sfba.sns-pb.isc.org. 31018 IN AAAA 2001:4f8:0:2::19
sfba.sns-pb.isc.org. 31018 IN A 149.20.64.3
mx.pao1.isc.org. 3547 IN AAAA 2001:4f8:0:2::2b
mx.pao1.isc.org. 3547 IN A 149.20.64.53
_sip._udp.isc.org. 7147 IN SRV 0 1 5060 asterisk.isc.org.
3,363 byte response

300Gbps+ of DDoS attack traffic
1 laptop
+ 5-7 compromised servers
+ 3 networks which allow spoofing
+ 9Gbps of DNS requests to
+ 0.1% of all open resolvers
----------------------------------------
= 300Gbps of DDoS traffic

Attack #2 – The NTP Attack:
400Gbps
UDP NTP

Tweets report attack issues

An EVEN easier way: NTP
Amplification
Attacker
NTP Server with
MONLIST
Target Website
64 Bytes
13,184 Bytes
~206x

1 laptop
+ 1 compromised server
+ 1 network which allowed spoofing
+ 1.94Gbps of MONLIST to
----------------------------------------------------
= 400Gbps+ of DDoS attack traffic
400Gbps+ of DDoS attack traffic

What’s Next?
~206x
650x
~50x
8xDNS
EDNS
NTP
SNMP
→
→
→
→

Something extra …

Largest Attack in History
• Hong Kong
• Peaked at ~500Gbps
• 7 days
• Reflection attack (DNS, NTP)
• DNS flood - 250 million DNS
requests per second
• HTTP(S) attack

Protecting your network

28 Million Open DNS Resolvers
http://OpenResolverProject.org/
Lock your DNS server (recursive & authoritative) down

28 Million Open DNS Resolvers
http://team-cymru.org/
Confirm that the resolver is a closed resolver
options {
recursion no;
additional-from-cache no;
};
UNIX bind configuration examples
acl "trusted" {
10.42.0.0/16;
192.0.2.0/24;
192.0.6.0/24;
};
options {
recursion no;
additional-from-cache no;
allow-query { none; };
};
view "trusted" in {
match-clients { trusted; };
allow-query { trusted; };
recursion yes;
additional-from-cache yes;
};

NTP Amplification Attacks
http://OpenNTPProject.org/
Turn off MONLIST on your NTP servers

“noquery” is required to disable MONLIST
# by default act only as a basic NTP client
restrict -4 default nomodify nopeer noquery notrap
restrict -6 default nomodify nopeer noquery notrap
# allow NTP messages from the loopback address, useful for debugging
restrict 127.0.0.1
restrict -6 ::1
# server(s) we time sync to
server 192.0.2.1
server 2001:db8::1
server time.example.net
UNIX ntpd configuration example

Prevent IP Spoofing (network hygiene)
• BCP38 / RFC2827 (ingress filtering) – May, 2000:
• http://bcp38.info/
• http://www.ietf.org/rfc/bcp/bcp38.txt
• http://www.ietf.org/rfc/rfc2827.txt
• BCP84 / RFC3704 (for multihomed) – March, 2004:
• http://www.ietf.org/rfc/bcp/bcp84.txt
• http://www.ietf.org/rfc/rfc3704.txt
… and yet still an issue today!

Securing CDN traffic at
CloudFlare

CloudFlare security

CloudFlare – a global network
Attack traffic is global and hence a global edge is valuable

Anycast Dilutes Attacks
300Gbps of attack traffic
/ 28 locations
----------------------------------------------------
= ~10.7Gbps average per location
Reality is that some locations are much larger
than others. However every location is vital.

Anycast Dilutes Attacks

Solution: Hide Origin IPs
• Use separate IPs for HTTP, DNS, SMTP, etc
• Public DNS should route to your EDGE’s public IPs
• Keep actual/origin web device IPs protected

Filter traffic by IP and protocol
• No UDP packets should be able to hit your HTTP
server
• UDP is IP protocol 17 vs. TCP for HTTP is IP protocol 6
• No HTTP packets should be able to hit your SMTP
server
• HTTP is TCP port 80 & 443 vs. SMTP is port 25 & 587

Filter traffic by IP and protocol
Allow only HTTP& HTTPS via TCP protocol to a specific IP
!
hostname router-www
!
interface ethernet0
ip access-group 102 in
!
access-list 102 permit tcp any host 10.0.0.100 eq 80
access-list 102 permit tcp any host 10.0.0.100 eq 443
access-list 102 deny all
!
Simple Cisco filter configuration example

Protect your infrastructure
• Internal switches, routers, and other devices should be
locked down from any external access
• All traffic should flow through EDGE devices which
handle attacks
• CloudFlare Web Application Firewall (WAF) service

Build relationships upstream
• Understand what your data center and bandwidth
providers do about DDoS
• Know who to call when trouble strikes
• Share your IP/Protocol architecture with them

Communicate about attacks

Summary

Summary
• Volumetric DDoS evolving (NTP came and went)
• Larger botnets / Cloud services user in botnets
• DNS flood on the rise / Application-level on the rise
• Politically motivated attacks
• First, make sure you’re not part of the problem …
• Second, practice good protocol hygiene …
• Third, implement infrastructure ACLs …
• Fourth, know your upstreams

Questions?
Martin J. Levy, Network Strategy
@mahtin
@cloudflare
http://www.cloudflare.com/
AS13335

Surviving A DDoS Attack: Securing CDN Traffic at CloudFlare

Related slideshows

More Related Content

Surviving A DDoS Attack: Securing CDN Traffic at CloudFlare

Editor's Notes