Outage analysis: BGP Routing Errors Ripple Across the Internet
•
1 like•302 views
How did Cloudflare, AWS, Discord and a metals manufacturer all end up in the same Internet pile-up? For nearly two hours on June 24, 2019, a BGP routing error from an ISP in Pennsylvania rippled outward across the global Internet, creating havoc and causing Cloudflare, AWS and related services like Discord and Nintendo Life to become unavailable to many users. The worst part is that it was entirely preventable.
Report
Share
Outage analysis: BGP Routing Errors Ripple Across the Internet
- 1. 1© 2018 ThousandEyes Inc. All Rights Reserved. @ThousandEyes
- 2. Offices Austin New York Dublin London Tokyo HQ San Francisco2010 Founded We empower enterprises to see, understand and improve digital experiences over the Internet. Global 2000 Companies 100+ 6 of the 7 Top US Banks 20 of the 25 Top SaaS Companies 8 of the 10 Top Global Software Companies 65+ Fortune 500 Companies 2Confidential ©2019 ThousandEyes Inc. All Rights Reserved.
- 3. 3Copyright ©2019 ThousandEyes Inc. @ThousandEyes Cloud & Internet Outages are Not Unusual • 31% of businesses with outages said they were due to failures at third-party service cloud, hosting or colocation providers (Uptime Institute, 2018) • 14,000 routing outages, leaks and hijacks in 2017 (Internet Society) • Macro, major, minor outages
- 5. 5Copyright ©2019 ThousandEyes Inc. @ThousandEyes ThousandEyes Detects Digital Experience Impact Many locations can’t load page
- 6. 6Copyright ©2019 ThousandEyes Inc. @ThousandEyes Evidence of Massive Congestion Very high packet loss Loss across multiple Tier 1 ISPs, particularly Verizon Cloudflare Edge Server
- 7. 7Copyright ©2019 ThousandEyes Inc. @ThousandEyes Odd AS-PATH Indicates Route Leak Path shifted from Tier 1 ISP Level 3 to enterprise network Allegheny Technologies. Allegheny is a metals manufacturer with two transit providers: • Verizon and DQE (AS 33154—shown above as MetNet) • ~20K routes leaked
- 8. 8Copyright ©2019 ThousandEyes Inc. @ThousandEyes Route Leak Included More Specific Prefixes Cloudflare announces /20s. This /21 was one of many generated most likely from DQE’s route optimization
- 9. 9Copyright ©2019 ThousandEyes Inc. @ThousandEyes DQE is Likely Original Source of Route Leak
- 10. 10Copyright ©2019 ThousandEyes Inc. @ThousandEyes Critique of Verizon’s Role Applies to Whole Industry Routing hygiene is a major issue on the Internet and a vulnerability point.
- 11. How to Reduce Business Risk from Outages
- 12. 12Copyright ©2019 ThousandEyes Inc. @ThousandEyes It Used to Be That You Controlled Everything On-premises, IT-controlled monolithic Branch Office Branch Office Branch Office Branch Office Siebel HRMS Flow PCAP SNMP Exchange Data Center
- 13. 13Copyright ©2019 ThousandEyes Inc. @ThousandEyes External Dependencies Are the Norm for IT Branch Office Branch Office Branch Office Branch Office Siebel HRMS Flow PCAP SNMP Exchange Data CenterData Center Data Center Data Center Data Center New performance, security and continuity risks Data Center Data Center Data Center Data Center Customer SaaS IaaS CDN DNS DNS Provider Security Provider API IaaS
- 14. 14Copyright ©2019 ThousandEyes Inc. @ThousandEyes Operations Processes Change in the Cloud IT Assets You Don’t Control Evidence Escalate?? IT Assets You Control Find Fix
- 15. 15Copyright ©2019 ThousandEyes Inc. @ThousandEyes You Can’t Afford to Wait for Insight • Status pages – May be slow – Often sparse – Hard to integrate into Ops process – Not very actionable • Time is of the essence – Your brand: informing customers – Your revenue: restoring service • Independent visibility is key to taking action
- 16. 16Copyright ©2019 ThousandEyes Inc. @ThousandEyes Pre-Cloud Monitoring Stack Monitoring Category IT Management Domain Presentation Layer All IT Service Management All APM Internally Developed Apps & Services IT Infrastructure Mgmt (SNMP) Internal Data Center & WAN infrastructure Network Perf Mgmt & Diagnostics (Flow/PCAP) Network Capacity Mgmt (Flow) Log Mgmt All
- 17. 17Copyright ©2019 ThousandEyes Inc. @ThousandEyes Fill the Cloud-Sized Hole In Your Monitoring Stack Monitoring Category IT Management Domain Presentation Layer All IT Service Management All APM Internally Developed Apps & Services IT Infrastructure Mgmt (SNMP) Internal Data Center & WAN infrastructure Network Perf Mgmt & Diagnostics (Flow/PCAP) Network Capacity Mgmt (Flow) Log Mgmt All Internal Data Center & WAN infrastructure Cloud-Specific Mgmt (CloudWatch) Cloud Infrastructure Missing Visibility Digital Experience ISP, DNS, CDN, DDoS. CASB providers SaaS providers SD-WAN Internet transport Internet Routing IT Infrastructure Mgmt (SNMP) Network Perf Mgmt & Diagnostics (Flow/PCAP) Network Capacity Mgmt (Flow)
- 18. 18Copyright ©2019 ThousandEyes Inc. @ThousandEyes Understand Experience for Any User and App Lightweight software-based agents easily installed on your own network, in data centers, branch offices & VPCs. Enterprise Agent Endpoint Agent Browser-based plugins installed on end-user laptops and desktops. End User ExperienceInternal Vantage Points Cloud Agent Globally distributed agents installed and managed by ThousandEyes in 175+ POPs around the world. External Vantage Points
- 19. 19Copyright ©2019 ThousandEyes Inc. @ThousandEyes 3 Boston, MA Dallas, TX Hong Kong 182.50.78.169 182.50.78.41 San Francisco, CA See What You Don’t Own
- 20. 20Copyright ©2019 ThousandEyes Inc. @ThousandEyes Manage Through Your Providers Your Network ISP Cloud Provider SnapshotsDashboards / ReportsAlerts
- 21. 21Copyright ©2019 ThousandEyes Inc. @ThousandEyes What to Do Next • Get more outage analyses: sign up for our blog at www.blog.thousandeyes.com • Get immediate visibility into your critical customer or employee- facing digital experiences https://www.thousandeyes.com/r equest-demo • Get more BGP learning. https://www.thousandeyes.com/r esources 21