More Related Content
What's hot
What's hot (20)
Similar to Horizontal Scaling for Millions of Customers!
Similar to Horizontal Scaling for Millions of Customers! (20)
Recently uploaded
Recently uploaded (20)
Horizontal Scaling for Millions of Customers!
- 1. 28th January 2020 Elangovan Shanmugam HORIZONTAL SCALING FOR MILLIONS OF CUSTOMERS
- 2. Intuit Confidential and Proprietary 1 Name: Elangovan Shanmugam Title: Distinguished Engineer @ Intuit Inc. EcoSystem Integration. Experience: 25+ Years in Software Development Work Highlights: - Designed Resilient systems to help importing Tax documents in (TP50) less than 2 seconds to our 45M tax filers. - Mint, Chief Architect. Designed & Deployed First ever Intuit Offering to Public Cloud. Serving for 35M customers and processing billions of bank transactions. - Event based notification systems processing millions of events and delivering 10+ millions notifications per day. - 11 Software Patents, 30 pending. - Scott Cook Innovation award winner. ABOUT ME … Twitter @elangovans LinkedIn https://www.linkedin.com/in/elangovans Blogs https://medium.com/@elangovan_shanmugam Patents https://patents.justia.com/inventor/elangovan-shanmugam
- 3. Intuit Confidential and Proprietary 2 We understand … - What is considered in the software architecture - Strategies for delivering Microservices that can scale - How Mint scales for millions of customers THIS MEETING WILL BE GOOD IF …
- 5. Intuit Confidential and Proprietary 4 TECHNICAL REQUIREMENTS Services Stds – Resiliency – Scalability – Reliability HOSTING REQUIREMENTS CICD – HADR – Cloud DEV OPS REQUIREMENTS Monitoring – Config mgmt. – Env. mgmt. BIZ OPS REQUIREMENTS Admin – Care – Risk – Fraud Mgmt. FUNCTIONAL REQUIREMENTS Product – Experience – Integration BUSINESS REQUIREMENTS Monetization– Metrics – Growth SECURITY REQUIREMENTS Security Stds – Data Stds ENGINEERING EXCELLENCE FMEA – Quality – Testability COMPLIANCE REQUIREMENTS HIPAA – PCI – Accessibility – Patriot act - Privacy OPERATIONAL EXCELLENCE SLA – Performance - Availability Customers Developers Partners BizOps Customer Success ARCHITECTURE HOW I SEE THE SOFTWARE ARCHITECTURE
- 6. Intuit Confidential and Proprietary 5 HOW DO WE STRUCTURE OURSELVES TO SOLVE THIS PROBLEM PEOPLE TECHNOLOGY PROCESS COST (TCO)P T Pr C Engineers (Dev, DevOps, QE) Analyst / Data Science Other actors Application / Services / Database AI/ML & Warehouse Cloud & Hosting technologies Agile, Kanban … Development Build / Deploy Process Development Cost Maintenance Cost Hosting Cost
- 7. Intuit Confidential and Proprietary 6 ARCHITECTURE PRINCIPLES Keep compute tier stateless1 Resilient to failures and work in a dynamic environment and No Single point of failures (reduce number of FCI) 2 Low latency and high performing systems 3 Security is number one concern 4 Fault Isolation by decreasing the blast radius 5 Zero downtime deployment - using Canary, Blue/Green Deployment 6
- 8. Intuit Confidential and Proprietary 7 SLA’S THAT WE ARE SOLVING FOR … AVAILABILITY 4 9’s ~ 52 mins downtime per year. LATENCY TP50: 1 Sec. TP90: 2 Secs. TP99: 5 Secs. SITE RECYCLE FREQUENCY 15 days FAULT TOLERANCE FCI: 0.02% RPO: 5 mins RTO: 20 mins SECURITY SLA ZERO SECURITY DEFECTS P1 DEFECTS MUST BE FIXED & DEPLOYED IN 48 HOURS TEST COVERAGE 100% AUTOMATION DEPLOYMENT TIME 45 Mins after code is committed Blast Radius No more than 10% user - Understand Traffic Patterns (daily, weekly, monthly, …) - Measure Daily, Weekly, Monthly, and Yearly Active users - Identify most used features, How much time the users are spending in a feature, Hotspots in the system - Build Operational rigour to support the users and system behaviours - Be Cost aware. How much each request costs
- 10. Intuit Confidential and Proprietary 9 API & MicroServices Data Lake Applications Hosting Logging Instrumentation Monitoring Alerting Content Source Control Analyzer Build Artifact & Packaging Deploy ABSTRACT TECHNOLOGY STACK Technology Stack ● Typical stack contains many micro services, many databases, many applications along with several utility functions, hosting and monitoring. ● In Mint, ○ AWS is the hosting platform ○ Java and Restful services for the middle layer ○ Single page application for the browser app ○ Native applications for the mobile devices (iOS and Android). DEVELOPMENT RUNTIME
- 11. Intuit Confidential and Proprietary 10 API & MicroServices Data Lake Applications Hosting Logging Instrumentation Monitoring Alerting Content Source Control Analyzer Build Artifact & Packaging Deploy ABSTRACT TECHNOLOGY STACK Application Stack ● User facing features ● Typically running in the internet ● Built for multiple platforms (Web Browser, Desktop, Mobile devices, and so on)
- 12. Intuit Confidential and Proprietary 11 API & MicroServices Data Lake Applications Hosting Logging Instrumentation Monitoring Alerting Content Source Control Analyzer Build Artifact & Packaging Deploy ABSTRACT TECHNOLOGY STACK Services Stack ● Capabilities built and deployed in the server side ● Typically runs in the data center behind Firewall ● Should be horizontally scalable ● Solves for lots compliance requirements
- 13. Intuit Confidential and Proprietary 12 API & MicroServices Data Lake Applications Hosting Logging Instrumentation Monitoring Alerting Content Source Control Analyzer Build Artifact & Packaging Deploy ABSTRACT TECHNOLOGY STACK Data Lake ● Ingest click-stream data to understand user behaviors ● Ingest all data and analyse to gain business insights ● Develop & Train ML models
- 15. Intuit Confidential and Proprietary 14 MicroService - 1 Gateway authentication Microservices Strategy Modern systems cannot be built as a single monolithic service. Split the concerns into functional areas. In Mint, the splits are as follows, Budgets, Credit Reports, Investments, Goals, Notifications, Insights and so on. Common Capabilities: - Within the services there are horizontal concerns like how to communicate between Databases, API’s, Brokers, etc. Scale per functionality: - This allows us to design them independent of each other, based on their scale. For example, Users will have 1000’s of transactions, but there will be only one credit report per month. Separation of concerns: - Delivery of services per their own schedule. Credit report team’s cadence can be different from Budget service cadence. - In case of any issues in Budget service must not impact Credit service or other services. This is also known as control the blast radius. ORIENTATION SCALING FUNCTIONALITIES MicroService - 2 MicroService - 3 MicroService - 4 MicroService - 5
- 16. Intuit Confidential and Proprietary 15 Business Logic Proxy Layer (Routing & Stack Picking) API Broker Database Layer Offline Process Gateway authentication Authenticate any requests to microservices. This is critical security layer. Restful API layer exposes various CRUD resources operations to it clients. This defines endpoints and resource models. Service layer where all the business logic is implemented. There are two different aspects, online and offline process. Databases, Caches and Data Access Layer. Platform Developer Role Technologies API Backend Developer JaxRS, Jersey Broker ActiveMQ, Kafka Cache Redis, MemCache Business Logic Java Database MySQL, C*, DynamoDB Service Template A representation of a Microservice. Though each service can customize per its needs a typical template would be like this so that there is uniform approach and easy to understand, move talents between services, and easy to troubleshoot, etc. MICROSERVICES ARCHITECTURE Cache ORIENTATION
- 17. Intuit Confidential and Proprietary 16 RUN TIME & NETWORK COMMUNICATIONS ELB Proxy API Broker Business Database Sharded RDS MySQL:: Each DB contains set of users data. Business logic, Caching and Data access logic. REST API, Data Transform to Resource model, Distributed Caching for sessions EC2 Instance RDS MySQL
- 18. Intuit Confidential and Proprietary 17 RUN TIME & NETWORK COMMUNICATIONS ELB Proxy API Broker Business Database Sharded RDS MySQL:: Each DB contains set of users data. Typically compute, I/O intensive and scale this layer based on the load. REST API instances can be scaled horizontally. Add new EC2 nodes to support load. ActiveMQ Brokers are typically will work as single Master instance and Slave instance. This layer is used as RMI over JMS. Alternate would be - using Kafka Cluster. - using Vert.x infrastructure. Can be a separate topics!!
- 19. Intuit Confidential and Proprietary 18 Availability Zone 1 Proxy Layer (Routing & Stack picking) Web, Broker, and Business logic Layer Database Layer Availability Zone 1 Proxy Layer (Routing & Stack picking) S1 S2 S3 S4 WHAT IS SL? Availability Zone 1 Proxy Layer (Routing & Stack picking) S1 S2 S3 S4 WHAT IS INSIDE A SL? EC2 RDS for MySQL HORIZONTAL SCALING & DEPLOYMENT ARCHITECTURE Strategies for Service deployment Pick a strategy for partitioning the data. For example, one DB shard could hold data for n users. For a given service, all the data will be available in that shard. Another service could use very different partitioning strategy. Alternate: Cassandra or DynamoDB could be an alternate, but choose proper persistence strategy for the problem. Stacks (or Swimlanes) where the services are deployed can communicate to within the boundary. Avoid Cross Stack Communication. Central Stack (not shown in the diagram) contains the configuration of which user data belongs to which shard and which shard belongs to which stack. AWS RegionElastic Load Balancer
- 20. Intuit Confidential and Proprietary 19 Data Center 2 Availability Zone 1 Proxy Layer (Routing & Stack picking) S1 S2 S3 S4 Data Center 1 HIGH AVAILABILITY AWS RegionElastic Load Balancer HA Strategies Services are deployed in Multiple Availability Zones (AZ) so that they are highly available. Each Stack is distributed in multiple AZs. Primary database is one AZ. Slaves (real-time replication) of the data is distributed in the 2nd and 3rd AZ. Compute tier will reach out to the Primary database. Benefits of this model: - If there is an outage on an AZ other two AZs will still serve the requests for our customers. - Reduce the blast radius. If there is an issues in the system it could only affect smallest scope of the users. Operations procedures: Stack rebalancing, Data model rollouts, Blue/Green Deployments, Canary deployments. Availability Zone 2 Proxy Layer (Routing & Stack picking) S1 S2 S3 S4 Availability Zone 3 Proxy Layer (Routing & Stack picking) S1 S2 S3 S4 Data Center 3
- 22. Intuit Confidential and Proprietary 21 Plugins Utilities (Logging, Instrumentation …) Sandbox API App Shell Shell Service Plugin Reg + Personalization Plugin - 1 Plugin - 3 Platform Developer Role Technologies Web Application Web app developer Java Script, ReactJS, HTML5, CSS3. iOS (phone, tablet, wearables) iOS Developer Swift, XCode Android (phone, tablet, wearables) Android Developer Kotlin, Android Studio Experience Designers, Customer research, Product Managers Accessibility experts. Plugin - 2 Plugin - 4 A Software infrastructure that helps register and render Plugins, packages the common libraries avoid loading them multiple times, etc. Plugins are set of customer facing capabilities that can be reused in multiple apps. Think of Login, Shopping cart, Payment, etc. as plugins. APPLICATION ARCHITECTURE Common utilities can be used across all the plugins. Uniform method of logging and collecting metrics and more. Scalability in App Architecture What are the strategies for multiple platforms, Web, iOS, Android, Tablets, Wearables, IoT devices? What are the customer base? Do they expect native experience? Does your product demand native experience? For example, does your product need any of the device capabilities, like activating camera? ORIENTATION
- 23. Intuit Confidential and Proprietary 22 Q & A
- 24. THANK YOU!