Walking Through Cloud Serving at Yahoo!
•
19 likes•2,759 views
This document summarizes a presentation by Surendra Reddy of Yahoo on Yahoo's cloud serving platform. The presentation discusses the challenges of traditional infrastructure management including outages, inflexibility, and complexity. It then introduces Yahoo's principles for cloud serving including declarative application structure, continuous integration/deployment, virtualized hardware, and extreme automation. Key aspects of Yahoo's platform are described like abstracting services, standardized software stacks, and an open cloud access protocol submitted to IETF for resource management. The cloud is portrayed as enabling innovation by cleaning up infrastructure messes.
Report
Share
Walking Through Cloud Serving at Yahoo!
- 1. Yahoo Cloud ServingSurendra ReddyCloud Computing Conference & ExpoNovember 3, 2009
- 2. Imagining the Future of Cloud!
- 3. Cloud is a Promise, But …
- 4. Reference: Tussle in Cyberspace: Defining Tomorrow’s Internet http://conferences.sigcomm.org/sigcomm/2002/papers/tussle.pdf
- 5. Customer Tussles:Outages: 76% outages caused by change management
- 6. Unmet Needs: Rapidly changing needsManagement Tussles:Growing Costs and Complexity
- 7. Long Deployment Cycles: Inflexible, peak-provisioned infrastructureOperations Tussles:Entangled in many day-to-day failures & troubleshooting
- 9. Building on weak foundationDevelopers Tussles:Bloated Operating System, App Stacks
- 10. Staggering Complexities of Dependencies
- 11. No time for QA – Bugs Creeping in all overSecurity Tussles:Whose problem is it?
- 13. Governance
- 14. Compliance
- 15. As a Result: More Friction, Waste, and Frustration
- 16. Less Innovation So, What is Cloud?C – Customer Driven
- 17. L – Lean
- 18. O – Optimized
- 19. U – Utility
- 20. D – Demand-DrivenCloudis NOT about Cost cutting
- 21. It is about Value Creation enabling InnovationCloud is an opportunity to clean up the Infrastructure mess!
- 22. Yahoo! is Perfect for Cloud Computing600MUNIQUE USERS / MONTH300M+YAHOO! MAIL USERS / MONTHHUNDREDSOF PROPERTIES / PRODUCTSBILLIONSOF OBJECTS STOREDHUNDREDSOF PETABYTES OF STORAGEPETABYTESOF TRAFFIC DAILY
- 23. Declarative language for specifying the full, end-to-end structure of a service
- 24. Includes multiple environments: Dev, QA, Staging, and production
- 25. Extreme AutomationYahoo! Cloud Serving - PrinciplesMetastructure: Abstracts concerns of the underlying infrastructure and the network communication
- 27. Continuous Integration, Continuous Deployment, & Continuous Compliance*
- 28. End-to-End Security* Rich Miller @RHM2K first coined “Continuous Compliance”
- 29. Yahoo! Cloud Serving - PrinciplesVirtualized hardware
- 30. Standardized software stacks and packaging
- 32. Containers vs. MachinesYahoo! Cloud Serving - PrinciplesElastic serving of changing workloads
- 36. Service Associations, Bindings, and Access ControlsYahoo! Cloud: Tiers without TearsEntrypointNamingPresentationTierApplication LogicTierBindingsConnectionsDatabase Tier
- 37. Yahoo! Cloud: Tiers without TearsEntrypointNamingLoadBalancerPresentationTierApplication LogicTiernodenodenodeBindingsConnectionsDatabase Tier
- 38. Yahoo! Cloud: Extreme AutomationAbstracting ServicesComponent AnswersCore { Entry root;Tier pres { app = “answers_web_tier”; Exit applogic; nsize = small; nnodes = 3; };Tier applogic { app = “answers_al_tier”; Exit data; nsize = medium; nnodes = 3; };Tier data; Connections { root -> pres; pres.applogic -> applogic; applogic.data -> data; }} PresentationTierApplication LogicTierDatabase Tier
- 39. Yahoo! Cloud: Extreme AutomationQA EnvironmentEnvironment qa(ne1) { TrafficDirector root { name = “qa.answers.yahoo.com”; access = testing; }; AnswersCore core; Connections { root -> core; }} PresentationTierPresentationTierApplication LogicTierApplication LogicTierTesting TestingEnvironment production(ac4,sk1) { TeeTrafficDirector root { name = “answers.yahoo.com”; access = public; split_percent = 10; split_destination = qa.root; }; AnswersCore core; Connections { root -> core; }} Database TierDatabase Tier
- 40. Yahoo! Cloud: Extreme AutomationApps written against rich stacksIncludes core infra and platformsInfrastructure: Security, monitoring, metering, abuse prevention
- 41. Platforms: data, user info, ad systems, configuration bundles
- 42. All “auto configured”Serving Cloud Serving for Y!’s Cloud
- 46. Tears to Tiers
- 48. Costs to Revenue StreamsCloud is a Promise, Service is the fulfillment!
- 49. OPEN CLOUD ACCESS PROTOCOL
- 50. OPEN CLOUD ACCESS PROTOCOLDefinition, Deployment, and Life Cycle Management of Cloud Resources
- 51. Allocation, Provisioning, and Metering of Cloud Resources
- 52. Metadata/Registry for Cloud Resources
- 53. Virtual Infrastructure ManagementYahoo! Submitted OpenCAP to IETFWhy IETF?
- 54. Specification behind Yahoo Implementation
- 55. Plan to open source implementationOpen Cloud Access ProtocolResource ModelAll Infrastructure and Functional Components are modeled as Resources
- 56. Nodes, Networks, Storage Devices, Databases, Load Balancers, Identities, Key Value Store, Object Stores, Caching ServicesResource PropertiesModeled as JSON objects
- 57. Standard Catalog of attributes, extensibleResource OperationsOperation, Control, and NotificationOpen Cloud Access ProtocolManagement Operation ServicesCREATE - Create an instance of a managed resource
- 58. DELETE - Delete an instance of a managed resource
- 59. GET - Request managed resource properties (for one resource or a set of resource)
- 60. CANCEL - Cancel an outstanding request
- 61. SET - Set managed resource properties
- 62. ACTION - Request an action to be performed on a managed resourceOpen Cloud Access ProtocolManagement Notification ServicesEVENT- Send events occurring on managed resources (VM events, VM stats etc, SNMP events etc)
- 63. ALERT- Send alerts for resource failures and exceptionsLife Cycle Management ServicesSTART- Starts the resource
- 64. TERMINATE – Stop the resource
- 65. SUSPEND – Suspend the resource
- 66. RESUME – Resume the resource
- 67. ARCHIVE – Archive the resourceYahoo! Open Source & ResearchOpen SourceCloud Serving Platform, Yahoo Implementation of OpenCAP
- 68. Traffic ServerResearch ActivitiesOpen Cloud Consortium
- 69. OpenCirrus & M45THANK YOUSurendra Reddy@sureddy twittersureddy@yahoo-inc.com emailyCloudBlog.com blog
- 70. TUESDAY, 11/34:50pm – 5:35 pmHadoop @ Yahoo! – InternetScale Data ProcessingEric BaldeschwielerVP, Hadoop Software DevelopmentWEDNESDAY, 11/49:10 am - 9:55 amYahoo! Scalable Storage andDelivery ServicesChuck NeerdaelsVP, Storage and Edge ServicesVISIT BOOTH #103 TO TALK WITH YAHOO! ENGINEERS AND LEARN MORE ABOUT YAHOO!’S VISION FOR CLOUD COMPUTING.
Editor's Notes
- _____We believe, Cloud computing is about Driving Innovation.At Yahoo!, we are developing and deploying Cloud Services to help us drive innovationBy providing a set of horizontal Cloud services that are broadly applicable to building many Yahoo! products We are able to reallocate many developers to building products and features from building product-centric infrastructure. We are providing a stable base of Cloud Services that enable product developers to iterate quickly and innovate without risking product quality We are providing the ability to process and analyze enormous amounts of data easily and quickly for purposes of improving consumer experience with our products.
- Hundreds of properties/productsEnormous Scale 500M+ unique visitors monthly 300M+ Mail users 100’s of petabytes of storage 100’s of billions of objects stored Billions of daily requestsGlobal footprint