SlideShare a Scribd company logo
Twitter!  Architecture and Scalability Aditya B 05IT04
WHAT IS TWITTER ?
Its addictive micro-blogging platform text-based posts 140 characters in length followers receive updates
 

Recommended for you

Introduction to Node.js
Introduction to Node.jsIntroduction to Node.js
Introduction to Node.js

This is the slide deck from the popular "Introduction to Node.js" webinar with AMD and DevelopIntelligence, presented by Joshua McNeese. Watch our AMD Developer Central YouTube channel for the replay at https://www.youtube.com/user/AMDDevCentral.

java webinar nodejs programming
Spring boot
Spring bootSpring boot
Spring boot

This document contains an agenda and slides for a presentation on Spring Boot. The presentation introduces Spring Boot, which allows developers to rapidly build production-grade Spring applications with minimal configuration. It demonstrates how to quickly create a "Hello World" application using Spring Boot and discusses some of the features it provides out-of-the-box like embedded servers and externalized configuration. The presentation also shows how to add additional functionality like Thymeleaf templates and actuator endpoints to monitor and manage applications.

springspring-bootjava
Backend Programming
Backend ProgrammingBackend Programming
Backend Programming

Backend development focuses on the server-side of web applications, including databases, data storage and retrieval, security, and APIs. The key skills required for backend development are proficiency in server-side programming languages like JavaScript, Python, Java, and C#, as well as knowledge of database technologies like MySQL, MongoDB, and SQL Server. Backend developers are responsible for designing effective backend solutions, storing and securing user data access across devices through APIs and version control systems.

 
Who uses twitter
Web Traffic Twitter’s web-based traffic  Plus Twitter's API Traffic which is 10x the Site’s
As it often happens..

Recommended for you

고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들
고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들
고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들

고려대학교 컴퓨터학과에서 진행한 특강 "대학생 때 알았더라면 좋았을 것들"의 발표 자료입니다. 한참 고민이 많을 시기인 대학생 여러분들에게 도움이 되었으면 합니다.

universitytime managementcommunication
Introduction to ReactJS
Introduction to ReactJSIntroduction to ReactJS
Introduction to ReactJS

React (or React Js) is a declarative, component-based JS library to build SPA(single page applications) which was created by Jordan Walke, a software engineer at Facebook. It is flexible and can be used in a variety of projects.

reactjsweb developmentknoldus
MERN PPT
MERN PPTMERN PPT
MERN PPT

This document presents information on the MERN stack and how it can be used to build a Twitter clone application. It defines each component of the MERN stack: MongoDB for the database, ExpressJS for the backend framework, ReactJS for the frontend framework, and NodeJS as the runtime environment. It explains that MongoDB is a flexible NoSQL database, ExpressJS simplifies backend coding in NodeJS, ReactJS allows building user interfaces with JavaScript, and NodeJS enables running JavaScript on the server. The document outlines the main benefits of using the MERN stack, such as having a single coding language across front- and backend and the ability to build dynamic web apps quickly. It concludes by describing how to start the server and client for

web development
 
Downtimes!! 2008
 
So why the problem ? Over 350,000 users. The actual numbers are as always, very super super top secret. 600 requests per second. Average 200-300 connections per second. Spiking to 800 connections per second. MySQL handled 2,400 requests per second.

Recommended for you

[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유

NDC18에서 발표하였습니다. 현재 보고 계신 슬라이드는 1부 입니다.(총 2부) - 1부 링크: https://goo.gl/3v4DAa - 2부 링크: https://goo.gl/wpoZpY (SlideShare에 슬라이드 300장 제한으로 2부로 나누어 올렸습니다. 불편하시더라도 양해 부탁드립니다.)

awsdatapipelineelasticmapreduce
Reactjs
Reactjs Reactjs
Reactjs

Getting started with the reactjs, basics of reactjs, introduction of reactjs, core concepts of reactjs and comparison with the other libraries/frameworks

javascriptreactjsintroduction of reactjs
ClassLoader Leaks
ClassLoader LeaksClassLoader Leaks
ClassLoader Leaks

Are you tired of java.lang.OutOfMemoryError: PermGen space? Then this talk is for you! We'll begin with a crash course in the Java memory model in order to understand what the error message means. Then we'll look at different causes of the error and how to avoid them. We may glance at a few interesting mistakes from the Open Source world. Last but not least you'll learn how you can get rid of java.lang.OutOfMemoryError: PermGen space once and for all.

classloaderpermgenleaks
What? Why so?? When a user Abhinav writes a simple “I’m hanging out with…” message, Twitter has two choices – PUSH the message to the queue’s of each of his 6,864 followers, or Wait for the 6,864 followers to log in, then PULL the message. It is not as easy as it looks.
A 6000x multiplication factor D o you see a scaling problem with this scenario? Scoble  writes something  boom  6,800 writes  are kicked off. 1 for each follower. Michael Arrington  replies  boom  another  6,600 writes . Jason   Calacanis  jumps in  boom  another  6,500 writes .
कितनॆ आदमी थे  ? ~ 350,000  सरकार् और् तुम्  ? 1 database  सरकार् बहुत् नाइन्साफी है !
Bottlenecks Single MySQL database no monitoring, no graphs, no statistics Abuses Plan to partition in the future

Recommended for you

Introduction to GraphQL
Introduction to GraphQLIntroduction to GraphQL
Introduction to GraphQL

GraphQL is a query language for APIs and a runtime for fulfilling those queries. It gives clients the power to ask for exactly what they need, which makes it a great fit for modern web and mobile apps. In this talk, we explain why GraphQL was created, introduce you to the syntax and behavior, and then show how to use it to build powerful APIs for your data. We will also introduce you to AWS AppSync, a GraphQL-powered serverless backend for apps, which you can use to host GraphQL APIs and also add real-time and offline capabilities to your web and mobile apps. You can follow along if you have an AWS account – no GraphQL experience required! Level: Beginner Speaker: Rohan Deshpande - Sr. Software Dev Engineer, AWS Mobile Applications

awsamazon-web-servicescloud
React js
React jsReact js
React js

ReactJS for Beginners provides an overview of ReactJS including what it is, advantages, disadvantages, typical setup tools, and examples of basic React code. Key points covered include: - ReactJS is a JavaScript library for building user interfaces and is component-based. - Advantages include high efficiency, easier JavaScript via JSX, good developer tools and SEO, and easy testing. - Disadvantages include React only handling the view layer and requiring other libraries for full MVC functionality. - Examples demonstrate basic components, properties, events, conditional rendering, and lists in ReactJS.

reduxreact nativereactjs
Introduction to the Web API
Introduction to the Web APIIntroduction to the Web API
Introduction to the Web API

The document provides an introduction to web APIs and REST. It defines APIs as methods to access data and workflows from an application without using the application itself. It describes REST as an architectural style for APIs that uses a client-server model with stateless operations and a uniform interface. The document outlines best practices for REST APIs, including using HTTP verbs like GET, POST, PUT and DELETE to perform CRUD operations on resources identified by URIs. It also discusses authentication, authorization, security concerns and gives examples of popular REST APIs from Facebook, Twitter and other services.

restapi
SOLUTION ?
Caching Getting your friends status is complicated.  There are security and other issues.  So rather than doing a query, a friend's status is updated in cache instead.  It never touches the database.  This gives a predictable response time frame (upper bound 20 ms)
Partitioning Plan to partition in the future. Currently they don't. The partition scheme will be based on time, not users Because most requests are very temporally local.
Abuse Prevention Bots crawl the site and add everyone as friends.  9000 friends in 24 hours. It would take down the site. Saraha Be ruthless. Delete them as users. 9000 14 2 Following Followers  Updates

Recommended for you

Intro to React
Intro to ReactIntro to React
Intro to React

The document provides an introduction to React, a JavaScript library for building user interfaces. It discusses key React concepts like components, properties, state, one-way data flow, and JSX syntax. It also covers setting up a development environment with Create React App and shows how to create a basic React component with state. The target audience appears to be people new to React who want to learn the fundamentals.

reactjavascriptopen source
검색엔진이 데이터를 다루는 법 김종민
검색엔진이 데이터를 다루는 법 김종민검색엔진이 데이터를 다루는 법 김종민
검색엔진이 데이터를 다루는 법 김종민

데이터야 놀자 2017 발표 자료. https://datayanolja.github.io/program-2017.html#jongmin.kim 검색엔진이 데이터를 다루는 법

검색엔진엘라스틱서치elasticsearch
로그 기깔나게 잘 디자인하는 법
로그 기깔나게 잘 디자인하는 법로그 기깔나게 잘 디자인하는 법
로그 기깔나게 잘 디자인하는 법

흔히 만나는 실제 사례를 중심으로

business intelligencestatisticsbig data
Scalability -- Doing It Right Asynchronous event-driven design Partitioning/Shards Parallel execution Replication (read-mostly)
Are we all doomed to go through this painful process when we are successful? Time-To-Market Vs Architecture  Good, Fast, Cheap - pick two  :P
LESSONS LEARNED
Talk to the community. Treat your scaling plan like a business plan Build it yourself Build in user limits Don't make the database the central bottleneck of doom Make your application easily partitionable from the start

Recommended for you

[Final] ReactJS presentation
[Final] ReactJS presentation[Final] ReactJS presentation
[Final] ReactJS presentation

React JS is a JavaScript library for building user interfaces. It uses a virtual DOM to efficiently update the real DOM and render user interfaces from components. Components are reusable pieces of UI that accept input data via properties but maintain private state data. The lifecycle of a component involves initialization, updating due to state/prop changes, and unmounting. React uses a single-directional data flow and the concept of components makes code modular and reusable.

webservice scaling for newbie
webservice scaling for newbiewebservice scaling for newbie
webservice scaling for newbie

elastic, resiliency, sharding, service discovery

Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018
Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018 Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018
Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018

En nuestro equipo enfocamos nuestros esfuerzos en recopilar datos para desarrollar una tabla de comparación que muestra el verdadero coste al mover dinero entre diferentes divisas. En su desarrollo, se nos ha presentado el reto de abstraernos de la propia tecnología elegida, en este caso React, para hacerlo compatible con cualquier cliente. Con esta charla quiero compartir las lecciones aprendidas convirtiendo la tabla en un componente web con el propósito de facilitar su distribución y la propagación de nuevas versiones con la mínima fricción para el cliente. Find out more presentations at https://madrid2018.codemotionworld.com/speakers/

conferencecodemotion madrid 2018technology
Optimize the database Cache the hell out of everything Most performance comes not from the language, but from application design Turn your website into an open service by creating an API.  Their API is the single most powerful reason for Twitter's success.
References http://twitter.com/ http://highscalability.com/ scaling-twitter-making-twitter-10000-percent-faster http://www.slideshare.net/ Blaine/scaling-twitter http://dev.twitter.com/ 2008/05/twittering-about-architecture.html http://www.danga.com/memcached/ http://geekandpoke.com/
QUESTIONS
ADITYA http://twitter.com/arbitya [email_address]

Recommended for you

Stress Test as a Culture
Stress Test as a CultureStress Test as a Culture
Stress Test as a Culture

If you are working on a serious project, you want it to scale. The thing about scale is, you only focus on it once you really need it. I’m the CTO of an soccer social network based in Brazil. To put it mildly, soccer is big in my country. This summer, we focused our marketing on the World Cup, preparing our application to support as many users as possible. To do that, we had to benchmark and improve, but how could we load test? What tool should we use? Those are just some questions that I'll go through in this talk, that will show youhot to address this challenge so stress test you app.

web design and developmentstressloas test
Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...
Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...
Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...

Slide deck for talk at the 2013 HTML5 Developers Conference in San Francisco. Covers the main BaaS critical success factors: SMART (Scalable, Mobile-ready, Available, Real-time enabled and Truly secure)

realtimecloud storagebaas
Fronteers 20131205 the realtime web
Fronteers 20131205   the realtime webFronteers 20131205   the realtime web
Fronteers 20131205 the realtime web

The document discusses how to build real-time web applications. It emphasizes storing data client-side and pushing changes to clients in real-time using technologies like websockets to avoid slow AJAX calls. Examples are given of building an auto-complete search and news ticker in a real-time manner by initially loading all data and then querying client-side or pushing updates via websockets. The key is avoiding roundtrips to databases by doing work client-side and using fast databases like Redis that support publishing changes.

 
 
 
 

Recommended for you

Facebook, Robert Johnson
Facebook, Robert JohnsonFacebook, Robert Johnson
Facebook, Robert Johnson

Facebook has over 500 million active users, with half logging in every day. It processes over 4 trillion feed actions per day and caches over 2 trillion objects. Facebook has scaled to over 1 million active users per engineer, significantly more efficient than other large tech companies. To achieve this scale, Facebook relies on techniques like frequent small releases, dark launching of major changes, and shedding load during outages to maintain reliability as the site grows enormously.

highload-2010
Дмитро Волошин "High[Page]load"
Дмитро Волошин "High[Page]load"Дмитро Волошин "High[Page]load"
Дмитро Волошин "High[Page]load"

We at Preply do our best to ensure that our website loads quicky as it has huge impact on business. In my talk I will explain: - why pageload metric is important from business standpoint and how to measure its impact. - how we evolved with our speed optimization technics starting from very basic ones(caching, orm optimizations) to more advanced(replicas, load-balancing) and the level where we are now(CDN optmization, microservices etc.) - I will talk about both front-end and backend optimization with focus on the stack we use: AWS, Django/Python, Postgres, Docker.

architecturefwdayshighloaded systems
A Day Building Fast, Responsive, Extensible Single Page Applications
A Day Building Fast, Responsive, Extensible Single Page ApplicationsA Day Building Fast, Responsive, Extensible Single Page Applications
A Day Building Fast, Responsive, Extensible Single Page Applications

This is an older slide deck I realized I never uploaded. It is a slightly longer deck than the Night at the SPA deck. This features many concepts that are forerunners to the modern progressive web application. There are slides related to web performance best practices, JavaScript architecture, responsive web design, touch and much more.

single page appresponsive web designweb performance

More Related Content

What's hot

React + Redux Introduction
React + Redux IntroductionReact + Redux Introduction
React + Redux Introduction
Nikolaus Graf
 
React Hooks
React HooksReact Hooks
React Hooks
Joao Marins
 
Introduction to MERN Stack
Introduction to MERN StackIntroduction to MERN Stack
Introduction to MERN Stack
Surya937648
 
Introduction to Node.js
Introduction to Node.jsIntroduction to Node.js
Introduction to Node.js
AMD Developer Central
 
Spring boot
Spring bootSpring boot
Spring boot
sdeeg
 
Backend Programming
Backend ProgrammingBackend Programming
Backend Programming
Ruwandi Madhunamali
 
고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들
고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들
고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들
Chris Ohk
 
Introduction to ReactJS
Introduction to ReactJSIntroduction to ReactJS
Introduction to ReactJS
Knoldus Inc.
 
MERN PPT
MERN PPTMERN PPT
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유
Hyojun Jeon
 
Reactjs
Reactjs Reactjs
Reactjs
Neha Sharma
 
ClassLoader Leaks
ClassLoader LeaksClassLoader Leaks
ClassLoader Leaks
Mattias Jiderhamn
 
Introduction to GraphQL
Introduction to GraphQLIntroduction to GraphQL
Introduction to GraphQL
Amazon Web Services
 
React js
React jsReact js
Introduction to the Web API
Introduction to the Web APIIntroduction to the Web API
Introduction to the Web API
Brad Genereaux
 
Intro to React
Intro to ReactIntro to React
Intro to React
Justin Reock
 
검색엔진이 데이터를 다루는 법 김종민
검색엔진이 데이터를 다루는 법 김종민검색엔진이 데이터를 다루는 법 김종민
검색엔진이 데이터를 다루는 법 김종민
종민 김
 
로그 기깔나게 잘 디자인하는 법
로그 기깔나게 잘 디자인하는 법로그 기깔나게 잘 디자인하는 법
로그 기깔나게 잘 디자인하는 법
Jeongsang Baek
 
[Final] ReactJS presentation
[Final] ReactJS presentation[Final] ReactJS presentation
[Final] ReactJS presentation
洪 鹏发
 
webservice scaling for newbie
webservice scaling for newbiewebservice scaling for newbie
webservice scaling for newbie
DaeMyung Kang
 

What's hot (20)

React + Redux Introduction
React + Redux IntroductionReact + Redux Introduction
React + Redux Introduction
 
React Hooks
React HooksReact Hooks
React Hooks
 
Introduction to MERN Stack
Introduction to MERN StackIntroduction to MERN Stack
Introduction to MERN Stack
 
Introduction to Node.js
Introduction to Node.jsIntroduction to Node.js
Introduction to Node.js
 
Spring boot
Spring bootSpring boot
Spring boot
 
Backend Programming
Backend ProgrammingBackend Programming
Backend Programming
 
고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들
고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들
고려대학교 컴퓨터학과 특강 - 대학생 때 알았더라면 좋았을 것들
 
Introduction to ReactJS
Introduction to ReactJSIntroduction to ReactJS
Introduction to ReactJS
 
MERN PPT
MERN PPTMERN PPT
MERN PPT
 
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유
 
Reactjs
Reactjs Reactjs
Reactjs
 
ClassLoader Leaks
ClassLoader LeaksClassLoader Leaks
ClassLoader Leaks
 
Introduction to GraphQL
Introduction to GraphQLIntroduction to GraphQL
Introduction to GraphQL
 
React js
React jsReact js
React js
 
Introduction to the Web API
Introduction to the Web APIIntroduction to the Web API
Introduction to the Web API
 
Intro to React
Intro to ReactIntro to React
Intro to React
 
검색엔진이 데이터를 다루는 법 김종민
검색엔진이 데이터를 다루는 법 김종민검색엔진이 데이터를 다루는 법 김종민
검색엔진이 데이터를 다루는 법 김종민
 
로그 기깔나게 잘 디자인하는 법
로그 기깔나게 잘 디자인하는 법로그 기깔나게 잘 디자인하는 법
로그 기깔나게 잘 디자인하는 법
 
[Final] ReactJS presentation
[Final] ReactJS presentation[Final] ReactJS presentation
[Final] ReactJS presentation
 
webservice scaling for newbie
webservice scaling for newbiewebservice scaling for newbie
webservice scaling for newbie
 

Similar to Twitter - Architecture and Scalability lessons

Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018
Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018 Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018
Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018
Codemotion
 
Stress Test as a Culture
Stress Test as a CultureStress Test as a Culture
Stress Test as a Culture
João Moura
 
Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...
Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...
Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...
João Parreira
 
Fronteers 20131205 the realtime web
Fronteers 20131205   the realtime webFronteers 20131205   the realtime web
Fronteers 20131205 the realtime web
Bert Wijnants
 
Facebook, Robert Johnson
Facebook, Robert JohnsonFacebook, Robert Johnson
Facebook, Robert Johnson
Fuenteovejuna
 
Дмитро Волошин "High[Page]load"
Дмитро Волошин "High[Page]load"Дмитро Волошин "High[Page]load"
Дмитро Волошин "High[Page]load"
Fwdays
 
A Day Building Fast, Responsive, Extensible Single Page Applications
A Day Building Fast, Responsive, Extensible Single Page ApplicationsA Day Building Fast, Responsive, Extensible Single Page Applications
A Day Building Fast, Responsive, Extensible Single Page Applications
Chris Love
 
Expecto Performa! The Magic and Reality of Performance Tuning
Expecto Performa! The Magic and Reality of Performance TuningExpecto Performa! The Magic and Reality of Performance Tuning
Expecto Performa! The Magic and Reality of Performance Tuning
Atlassian
 
MeasureWorks - Design for Fast Experiences (Startup session).key
MeasureWorks  - Design for Fast Experiences (Startup session).keyMeasureWorks  - Design for Fast Experiences (Startup session).key
MeasureWorks - Design for Fast Experiences (Startup session).key
MeasureWorks
 
AWS Cloud Kata | Manila - Opening Keynote
AWS Cloud Kata | Manila - Opening KeynoteAWS Cloud Kata | Manila - Opening Keynote
AWS Cloud Kata | Manila - Opening Keynote
Amazon Web Services
 
Best practices with Microsoft Graph: Making your applications more performant...
Best practices with Microsoft Graph: Making your applications more performant...Best practices with Microsoft Graph: Making your applications more performant...
Best practices with Microsoft Graph: Making your applications more performant...
Microsoft Tech Community
 
EEDC 2010. Scaling Web Applications
EEDC 2010. Scaling Web ApplicationsEEDC 2010. Scaling Web Applications
EEDC 2010. Scaling Web Applications
Expertos en TI
 
Apply best parts of microservices to serverless
Apply best parts of microservices to serverlessApply best parts of microservices to serverless
Apply best parts of microservices to serverless
Yan Cui
 
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
Nati Shalom
 
Large-Scale Web Scraping: An Ultimate Guide
Large-Scale Web Scraping: An Ultimate GuideLarge-Scale Web Scraping: An Ultimate Guide
Large-Scale Web Scraping: An Ultimate Guide
Data Scraping and Data Extraction
 
Synchronous Reads Asynchronous Writes RubyConf 2009
Synchronous Reads Asynchronous Writes RubyConf 2009Synchronous Reads Asynchronous Writes RubyConf 2009
Synchronous Reads Asynchronous Writes RubyConf 2009
pauldix
 
Best Practices to Mitigate from the Emerging Vectors of Network Attack
Best Practices to Mitigate from the Emerging Vectors of Network AttackBest Practices to Mitigate from the Emerging Vectors of Network Attack
Best Practices to Mitigate from the Emerging Vectors of Network Attack
Amazon Web Services
 
Douglas Crockford - Ajax Security
Douglas Crockford - Ajax SecurityDouglas Crockford - Ajax Security
Douglas Crockford - Ajax Security
Web Directions
 
Starting Your DevOps Journey – Practical Tips for Ops
Starting Your DevOps Journey – Practical Tips for OpsStarting Your DevOps Journey – Practical Tips for Ops
Starting Your DevOps Journey – Practical Tips for Ops
Dynatrace
 
Best And Worst Practices Building Ria with Adobe and Microsoft
Best And Worst Practices Building Ria with Adobe and MicrosoftBest And Worst Practices Building Ria with Adobe and Microsoft
Best And Worst Practices Building Ria with Adobe and Microsoft
Josh Holmes
 

Similar to Twitter - Architecture and Scalability lessons (20)

Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018
Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018 Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018
Catalina Oyaneder | Ultimate Stack Compilation | Codemotion Madrid 2018
 
Stress Test as a Culture
Stress Test as a CultureStress Test as a Culture
Stress Test as a Culture
 
Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...
Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...
Building collaborative HTML5 apps using a backend-as-a-service (HTML5DevConf ...
 
Fronteers 20131205 the realtime web
Fronteers 20131205   the realtime webFronteers 20131205   the realtime web
Fronteers 20131205 the realtime web
 
Facebook, Robert Johnson
Facebook, Robert JohnsonFacebook, Robert Johnson
Facebook, Robert Johnson
 
Дмитро Волошин "High[Page]load"
Дмитро Волошин "High[Page]load"Дмитро Волошин "High[Page]load"
Дмитро Волошин "High[Page]load"
 
A Day Building Fast, Responsive, Extensible Single Page Applications
A Day Building Fast, Responsive, Extensible Single Page ApplicationsA Day Building Fast, Responsive, Extensible Single Page Applications
A Day Building Fast, Responsive, Extensible Single Page Applications
 
Expecto Performa! The Magic and Reality of Performance Tuning
Expecto Performa! The Magic and Reality of Performance TuningExpecto Performa! The Magic and Reality of Performance Tuning
Expecto Performa! The Magic and Reality of Performance Tuning
 
MeasureWorks - Design for Fast Experiences (Startup session).key
MeasureWorks  - Design for Fast Experiences (Startup session).keyMeasureWorks  - Design for Fast Experiences (Startup session).key
MeasureWorks - Design for Fast Experiences (Startup session).key
 
AWS Cloud Kata | Manila - Opening Keynote
AWS Cloud Kata | Manila - Opening KeynoteAWS Cloud Kata | Manila - Opening Keynote
AWS Cloud Kata | Manila - Opening Keynote
 
Best practices with Microsoft Graph: Making your applications more performant...
Best practices with Microsoft Graph: Making your applications more performant...Best practices with Microsoft Graph: Making your applications more performant...
Best practices with Microsoft Graph: Making your applications more performant...
 
EEDC 2010. Scaling Web Applications
EEDC 2010. Scaling Web ApplicationsEEDC 2010. Scaling Web Applications
EEDC 2010. Scaling Web Applications
 
Apply best parts of microservices to serverless
Apply best parts of microservices to serverlessApply best parts of microservices to serverless
Apply best parts of microservices to serverless
 
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
 
Large-Scale Web Scraping: An Ultimate Guide
Large-Scale Web Scraping: An Ultimate GuideLarge-Scale Web Scraping: An Ultimate Guide
Large-Scale Web Scraping: An Ultimate Guide
 
Synchronous Reads Asynchronous Writes RubyConf 2009
Synchronous Reads Asynchronous Writes RubyConf 2009Synchronous Reads Asynchronous Writes RubyConf 2009
Synchronous Reads Asynchronous Writes RubyConf 2009
 
Best Practices to Mitigate from the Emerging Vectors of Network Attack
Best Practices to Mitigate from the Emerging Vectors of Network AttackBest Practices to Mitigate from the Emerging Vectors of Network Attack
Best Practices to Mitigate from the Emerging Vectors of Network Attack
 
Douglas Crockford - Ajax Security
Douglas Crockford - Ajax SecurityDouglas Crockford - Ajax Security
Douglas Crockford - Ajax Security
 
Starting Your DevOps Journey – Practical Tips for Ops
Starting Your DevOps Journey – Practical Tips for OpsStarting Your DevOps Journey – Practical Tips for Ops
Starting Your DevOps Journey – Practical Tips for Ops
 
Best And Worst Practices Building Ria with Adobe and Microsoft
Best And Worst Practices Building Ria with Adobe and MicrosoftBest And Worst Practices Building Ria with Adobe and Microsoft
Best And Worst Practices Building Ria with Adobe and Microsoft
 

Recently uploaded

Quantum Communications Q&A with Gemini LLM
Quantum Communications Q&A with Gemini LLMQuantum Communications Q&A with Gemini LLM
Quantum Communications Q&A with Gemini LLM
Vijayananda Mohire
 
Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024
BookNet Canada
 
Pigging Solutions Sustainability brochure.pdf
Pigging Solutions Sustainability brochure.pdfPigging Solutions Sustainability brochure.pdf
Pigging Solutions Sustainability brochure.pdf
Pigging Solutions
 
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Erasmo Purificato
 
Manual | Product | Research Presentation
Manual | Product | Research PresentationManual | Product | Research Presentation
Manual | Product | Research Presentation
welrejdoall
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 
WPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide DeckWPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide Deck
Lidia A.
 
Transcript: Details of description part II: Describing images in practice - T...
Transcript: Details of description part II: Describing images in practice - T...Transcript: Details of description part II: Describing images in practice - T...
Transcript: Details of description part II: Describing images in practice - T...
BookNet Canada
 
20240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 202420240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 2024
Matthew Sinclair
 
INDIAN AIR FORCE FIGHTER PLANES LIST.pdf
INDIAN AIR FORCE FIGHTER PLANES LIST.pdfINDIAN AIR FORCE FIGHTER PLANES LIST.pdf
INDIAN AIR FORCE FIGHTER PLANES LIST.pdf
jackson110191
 
20240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 202420240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 2024
Matthew Sinclair
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 
Mitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing SystemsMitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing Systems
ScyllaDB
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
Eric D. Schabell
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
Emerging Tech
 
Cookies program to display the information though cookie creation
Cookies program to display the information though cookie creationCookies program to display the information though cookie creation
Cookies program to display the information though cookie creation
shanthidl1
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Bert Blevins
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
Mark Billinghurst
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
ishalveerrandhawa1
 

Recently uploaded (20)

Quantum Communications Q&A with Gemini LLM
Quantum Communications Q&A with Gemini LLMQuantum Communications Q&A with Gemini LLM
Quantum Communications Q&A with Gemini LLM
 
Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024
 
Pigging Solutions Sustainability brochure.pdf
Pigging Solutions Sustainability brochure.pdfPigging Solutions Sustainability brochure.pdf
Pigging Solutions Sustainability brochure.pdf
 
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
 
Manual | Product | Research Presentation
Manual | Product | Research PresentationManual | Product | Research Presentation
Manual | Product | Research Presentation
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 
WPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide DeckWPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide Deck
 
Transcript: Details of description part II: Describing images in practice - T...
Transcript: Details of description part II: Describing images in practice - T...Transcript: Details of description part II: Describing images in practice - T...
Transcript: Details of description part II: Describing images in practice - T...
 
20240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 202420240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 2024
 
INDIAN AIR FORCE FIGHTER PLANES LIST.pdf
INDIAN AIR FORCE FIGHTER PLANES LIST.pdfINDIAN AIR FORCE FIGHTER PLANES LIST.pdf
INDIAN AIR FORCE FIGHTER PLANES LIST.pdf
 
20240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 202420240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 2024
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 
Mitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing SystemsMitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing Systems
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
 
Cookies program to display the information though cookie creation
Cookies program to display the information though cookie creationCookies program to display the information though cookie creation
Cookies program to display the information though cookie creation
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
 

Twitter - Architecture and Scalability lessons

  • 1. Twitter! Architecture and Scalability Aditya B 05IT04
  • 3. Its addictive micro-blogging platform text-based posts 140 characters in length followers receive updates
  • 4.  
  • 5.  
  • 7. Web Traffic Twitter’s web-based traffic Plus Twitter's API Traffic which is 10x the Site’s
  • 8. As it often happens..
  • 9.  
  • 11.  
  • 12. So why the problem ? Over 350,000 users. The actual numbers are as always, very super super top secret. 600 requests per second. Average 200-300 connections per second. Spiking to 800 connections per second. MySQL handled 2,400 requests per second.
  • 13. What? Why so?? When a user Abhinav writes a simple “I’m hanging out with…” message, Twitter has two choices – PUSH the message to the queue’s of each of his 6,864 followers, or Wait for the 6,864 followers to log in, then PULL the message. It is not as easy as it looks.
  • 14. A 6000x multiplication factor D o you see a scaling problem with this scenario? Scoble writes something boom 6,800 writes are kicked off. 1 for each follower. Michael Arrington replies boom another 6,600 writes . Jason Calacanis jumps in boom another 6,500 writes .
  • 15. कितनॆ आदमी थे ? ~ 350,000 सरकार् और् तुम् ? 1 database सरकार् बहुत् नाइन्साफी है !
  • 16. Bottlenecks Single MySQL database no monitoring, no graphs, no statistics Abuses Plan to partition in the future
  • 18. Caching Getting your friends status is complicated. There are security and other issues. So rather than doing a query, a friend's status is updated in cache instead. It never touches the database. This gives a predictable response time frame (upper bound 20 ms)
  • 19. Partitioning Plan to partition in the future. Currently they don't. The partition scheme will be based on time, not users Because most requests are very temporally local.
  • 20. Abuse Prevention Bots crawl the site and add everyone as friends. 9000 friends in 24 hours. It would take down the site. Saraha Be ruthless. Delete them as users. 9000 14 2 Following Followers Updates
  • 21. Scalability -- Doing It Right Asynchronous event-driven design Partitioning/Shards Parallel execution Replication (read-mostly)
  • 22. Are we all doomed to go through this painful process when we are successful? Time-To-Market Vs Architecture Good, Fast, Cheap - pick two :P
  • 24. Talk to the community. Treat your scaling plan like a business plan Build it yourself Build in user limits Don't make the database the central bottleneck of doom Make your application easily partitionable from the start
  • 25. Optimize the database Cache the hell out of everything Most performance comes not from the language, but from application design Turn your website into an open service by creating an API. Their API is the single most powerful reason for Twitter's success.
  • 26. References http://twitter.com/ http://highscalability.com/ scaling-twitter-making-twitter-10000-percent-faster http://www.slideshare.net/ Blaine/scaling-twitter http://dev.twitter.com/ 2008/05/twittering-about-architecture.html http://www.danga.com/memcached/ http://geekandpoke.com/
  • 29.  
  • 30.  
  • 31.  
  • 32.