Preparing for CDN failure: Why and how
•
4 likes•1,643 views
This document discusses preparing for content delivery network (CDN) failures and how to monitor CDN performance. It provides examples of past CDN outages and failures. It then covers different methods for monitoring CDN performance, including synthetic monitoring and real user monitoring. It emphasizes the importance of measuring failure rates not just speeds. The document also discusses mitigating CDN failures through a multi-CDN approach with dynamic traffic steering based on performance data. It notes some challenges in decision making and with low volume data. Finally, it shares a story about responding to an outage at a company.
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (8)
Similar to Preparing for CDN failure: Why and how
Similar to Preparing for CDN failure: Why and how (20)
Recently uploaded
Recently uploaded (20)
Preparing for CDN failure: Why and how
- 1. Preparing for CDN Failure: Why and How Aaron Peters TurboBytes @aaronpeters & @ksgyoung Kyle Young Mobify
- 2. Examples of CDN Failure https://www.flickr.com/photos/phobia/2308371224/
- 3. Multi-continent outage for 17 minutes @aaronpeters & @ksgyoung 100% broken in North-America and Europe.
- 4. Up and down for days, some ASNs only @aaronpeters & @ksgyoung
- 5. Zoomed in on May 4 and May 5 … @aaronpeters & @ksgyoung OMG!
- 6. Japan, all networks: 5 hours of bad Three days later, it happened again … @aaronpeters & @ksgyoung
- 7. #1 eyeballs network in Germany Single country, single ASN perf degradations are not uncommon in CDN land … @aaronpeters & @ksgyoung
- 8. “System Maintenance” @aaronpeters & @ksgyoung
- 9. Take-Aways @aaronpeters & @ksgyoung Expect your CDN to fail on you They won’t tell you about it
- 11. Synthetic monitoring @aaronpeters & @ksgyoung Pingdom, cron+curl, ThousandEyes, Interns, etc.
- 12. Synthetic monitoring @aaronpeters & @ksgyoung The Pros Very Specific Known Intervals Controlled Locations Lots of Information
- 13. Synthetic monitoring @aaronpeters & @ksgyoung The Cons Limited Locations Datacenters Hyper-idealized Client Model
- 14. Synthetic monitoring @aaronpeters & @ksgyoung Still better than nothing.
- 15. RUM for Page Load Time @aaronpeters & @ksgyoung CDN must serve HTML + resources Third party content adds noise Can’t capture CDN failing Not very useful
- 16. RUM with Test Object @aaronpeters & @ksgyoung Fetch small object from CDN(s) after onload Beacon timings, or a Fail Use Nav Timing for best insight Challenge get your JS on other sites too, to capture CDN failing
- 17. RUM for Page Resources @aaronpeters & @ksgyoung Resource Timing API Send assets with TAO header Onload => beacon timings Easy, right? Not so fast …
- 18. Starting Points @aaronpeters & @ksgyoung Did the asset come from CDN? Did it load fast enough? Was it a good response (200/304)? Fetched from network? if (total time > 20 ms) { // from network } DOES NOT WORK !
- 19. RT API has many quirks @aaronpeters & @ksgyoung DNS time and Connect time always zero in IE No data for 4xx/5xx responses, except in IE Nothing in RT API until asset fully loaded, except in IE FF doesn’t tell the truth
- 20. How measure CDN perf with RT API @aaronpeters & @ksgyoung main.css inline JS, high in HEAD, exec only if window.chrome setTimeOut(checkRTAPI, 5000); at onload or when timer ended: if ( main.css in RT API && connectTime > 3 ) { loaded fine from network } else { meh } Too slow: Fail
- 21. transferSize attribute, FTW! @aaronpeters & @ksgyoung transferSize = byte size that came over the wire if ( transferSize != 0 ) { // from network } Status: no browser is implementing this yet Background/discussion: https://github.com/w3c/navigation-timing/issues/3 Future
- 22. Take-Aways @aaronpeters & @ksgyoung Measure Fail Ratio too, not just Speed Use RUM for real-world performance insight And Synthetic monitoring for deep visibility Beware of the many bugs in Res Timing API
- 24. Selecting your CDNs Feature & Behaviour Parity Performance Costs @aaronpeters & @ksgyoung
- 25. Doing Multi-CDN @aaronpeters & @ksgyoung Perf data Decision making Targeting Time-to-Switch High volume, high quality Make good sense of the data, quickly Very granular, very accurate Asap!
- 26. Where switch CDNs? @aaronpeters & @ksgyoung cs109.wac.edgecastcdn.net. cds.z4b9c4e6.hwcdn.net //cdn1.mydomain.com/main.css //cdn2.mydomain.com/main.css in DNS in HTML OR
- 27. Traffic management in DNS @aaronpeters & @ksgyoung resolver authoritativeclient I see the request comes from NL, based on resolver IP address … … so I’ll handout the CNAME to the CDN configured for NL CDN BCDN A
- 28. Static Geo @aaronpeters & @ksgyoung Always route to that CDN in that geo Easy: no need to monitor perf But what if CDN has boo boo?
- 29. Dynamic Geo @aaronpeters & @ksgyoung Always route to best CDN per geo Needs solid perf data (RUM !) Geo targeting accuracy important
- 30. Dynamic Geo + ASN @aaronpeters & @ksgyoung Holy grail: gives best results Really needs RUM data, and lots of it Targeting accuracy even more important
- 31. Geo targeting gone wrong @aaronpeters & @ksgyoung 8.8.8.8 authoritativeclient I see the request comes from MY, based on resolver IP address … … so I’ll handout the CNAME to the CDN configured for MY CDN B - best in India !CDN A
- 32. EDNS0 to the rescue ! @aaronpeters & @ksgyoung 8.8.8.8 authoritativeclient I see the request comes from IN, based on client IP address /24 … … so I’ll handout the CNAME to the CDN configured for IN CDN B - best in India !CDN A
- 33. Decision Making @aaronpeters & @ksgyoung Look at everything, not just ‘Response Time’ Use multiple statistics, not just median Make your ‘decider’ sensitive to Fail Ratio ! Tuning your logic takes time
- 34. Coping with low volume data @aaronpeters & @ksgyoung Don’t make changes Make decisions with lower confidence Have a dynamic targeting granularity
- 35. Experiment: do a Pat Meenan ! @aaronpeters & @ksgyoung http://www.slideshare.net/patrickmeenan/service-workers-for-performance Hi, I’m Pat
- 36. Example CDN Perf Program http://bit.ly/1KppCfo @aaronpeters & @ksgyoung
- 37. Example CDN Perf Program @aaronpeters & @ksgyoung
- 38. Example CDN Perf Program Limitations Not Practical for Monitoring Humans are Required Misses Important Factors (EG SSL) Hard to Commit to Bandwidth
- 39. War Story
- 40. Seattle Down @aaronpeters & @ksgyoung or How we Started Caring About MultiCDN
- 42. @aaronpeters & @ksgyoung Yay, Monitoring!
- 43. @aaronpeters & @ksgyoung Support Lines - A Tale of Lowered Expectations IT Crowd - Fremantle Media Batman - 20th Century Fox
- 44. @aaronpeters & @ksgyoung SLA Reminder 99.9% 8 hours 99.99% 52 minutes 99.999% 5 minutes
- 45. @aaronpeters & @ksgyoung Best Effort Switch over to an alternate CDN for the entire service, across the globe, as per the “Holy S#!t Handbook”.
- 46. @aaronpeters & @ksgyoung This Isn’t Pretty Cold Cache Backend Thrashing 3 to 4 hours of Intermittent Failure
- 47. @aaronpeters & @ksgyoung Lessons Learned Hot Standby Geographic DNS Control, and Optimization More Monitoring
- 48. @aaronpeters & @ksgyoung Thank you. @aaronpeters - aaron@turbobytes.com @ksgyoung - kyoung@mobify.com Questions?