SlideShare a Scribd company logo
The Stack Exchange 
Infrastructure 
Vroom Vroom
inet.perf.profile 
• SRE Generalist @ Stack Exchange 
• @GABeech 
• http://brokenhaze.com 
• http://stackexchange.com
A brief Overview 
• 560 Million Page Views a Month 
• 34TB of Data transfered a Month 
• 1665 rps (2250 peak) Across web Farm 
• WISC(HER)
Our First Priority is 
Performance 
Nobody likes a slow site, least of all us. 
When your site is slow people leave. 
Make your site fast, and the people will stay 
Good write up on moz.com: 
http://moz.com/blog/site-speed-are-you-fast-does- 
it-matter

Recommended for you

Lessons from the Trenches - Building Enterprise Applications with RavenDB
Lessons from the Trenches - Building Enterprise Applications with RavenDBLessons from the Trenches - Building Enterprise Applications with RavenDB
Lessons from the Trenches - Building Enterprise Applications with RavenDB

It's easy, fun, and simple to get a prototype application built with RavenDB, but what happens when you get to the point of shipping v1.0 into Production? Many of the subtle decisions made during development can have undesirable consequences in Production. In this session, Dan Bishop will explore some of the pain points that arise when building, deploying, and supporting enterprise-grade applications with RavenDB.

big datatechnologydatabase
Queryable State for Kafka Streamsを使ってみた
Queryable State for Kafka Streamsを使ってみたQueryable State for Kafka Streamsを使ってみた
Queryable State for Kafka Streamsを使ってみた

The document discusses queryable state for Apache Kafka Streams. It introduces Kafka Streams and stateful transformations. It then describes state for Kafka Streams, including how state is stored in RocksDB and tracked with a changelog in Kafka. Finally, it covers the new queryable state feature in Kafka Streams 0.10.1, which provides APIs to access state stores and retrieve values by key for windowed state.

kafkajpkafka
Mobile 3: Launch Like a Boss!
Mobile 3: Launch Like a Boss!Mobile 3: Launch Like a Boss!
Mobile 3: Launch Like a Boss!

Our Application development is nearing completion. It's time to prepare our cluster for production, but are we sure the system is capable of handing the load? Have we achieved high availability? What preflight checks should we be running. Learn how Dev & Ops work together to achieve production readiness and plan for scale, availability, monitoring.

mongodb worldmongodb
The Performance 
toolkit 
• Mini Profiler 
• OpServer 
(https://github.com/opserver/Opserver) 
• Client Timings 
(http://teststackoverflow.com/)
Mini Profiler
OpServer
OpServer HAproxy

Recommended for you

Velocity - Edge UG
Velocity - Edge UGVelocity - Edge UG
Velocity - Edge UG

This document discusses Microsoft's AppFabric distributed caching technology. It provides an overview of AppFabric, why distributed caching is useful, how to configure AppFabric clients and servers, and how to manage data in an AppFabric cache, including concurrency and high availability. While version 1 has some limitations, it is suitable as a session state provider, and the author expects version 2 to improve the product.

CFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful CodeCFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful Code

This is my presentation about CFWheels at CFObjective ANZ, November 2010, Melbourne, Australia. ColdFusion on Wheels (CFWheels), is an elegant framework inspired by Ruby on Rails.

mvc frameworkcfwheelscoldfusion on wheels
Solving Enterprise Integration with Apache Camel
Solving Enterprise Integration with Apache CamelSolving Enterprise Integration with Apache Camel
Solving Enterprise Integration with Apache Camel

Talk given at Phoenix Desert Code Camp on enterprise integration, enterprise service bus, and Apache Camel

integrationenterprise architectureopensource
OpServer Redis
OpServer SQL
Client Timings
You can’t be fast if 
you are not up 
• Highly Redundant network 
• Datacenter, ISP, Edge, Core, Server, 
Port

Recommended for you

Building big data pipelines with Kafka and Kubernetes
Building big data pipelines with Kafka and KubernetesBuilding big data pipelines with Kafka and Kubernetes
Building big data pipelines with Kafka and Kubernetes

This document discusses setting up a streaming platform using Apache Kafka and Kubernetes. It describes containerizing Kafka and Kafka Streams applications and deploying them on Kubernetes for scalability, fault tolerance, and easy upgrades. It also covers performance tuning of the platform, including optimizations for RocksDB, state stores, and network traffic. Troubleshooting performance issues within containers is discussed, such as installing profiling tools in separate containers. The goal is to provide a modern, scalable platform for data pipelines and microservices.

kafkaapachekafkakubernetes
Big Data DC - BenchPress
Big Data DC - BenchPressBig Data DC - BenchPress
Big Data DC - BenchPress

Talk at Big Data DC (http://www.meetup.com/bigdatadc/) about BenchPress (https://github.com/palominolabs/benchpress)

javaprogrammingbenchmarking
Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016

This document provides a 3 sentence summary of an Elasticsearch meetup in Mexico City in April 2016. It introduces Elasticsearch as a open-source search and analytics engine, discusses its features including real-time data, being massively distributed and schema-free, and provides examples of creating an index, mapping types, indexing documents, and searching in Elasticsearch using HTTP requests.

elasticsearchclusterlucene
Stack Exchange Infrastructure - LISA 14
Load Balencers 
• HAProxy 
• 2 Servers (Hot/Standby) 
• Multiple Tiers (HAProxy Processes)
Stack Exchange Infrastructure - LISA 14
SSL Termination 
• Terminated at LB 
• Feature added to HAProxy 1.5 
• See: 
http://brokenhaze.com/blog/2014/03/25/ 
how-stack-exchange-gets-the-most-out-of- 
haproxy/

Recommended for you

Redis Day Keynote Salvatore Sanfillipo Redis Labs
Redis Day Keynote Salvatore Sanfillipo Redis LabsRedis Day Keynote Salvatore Sanfillipo Redis Labs
Redis Day Keynote Salvatore Sanfillipo Redis Labs

Redis' seventh birthday was recently celebrated with the community, several contributors and users. This is Salvatore's keynote as he kicked off Redis Day in Tel Aviv.

redisredislabscommunity
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit ChopraI3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra

In this session we are going to discuss the entire process of planning and successfully running SharePoint 2016 in the cloud. We will cover topics related to Security, Performance, Scalability, High Availability, Backup & Restore and Disaster Recovery. We will also cover the IaC (Infrastructure as Code) use case scenario in this deployment.

azuresharepointspsparis
Velocity - NxtGen Oxford
Velocity - NxtGen OxfordVelocity - NxtGen Oxford
Velocity - NxtGen Oxford

Velocity is a distributed cache that allows sharing of cached data across multiple servers. In version 1, it is best suited for session state caching due to limitations in handling dependencies between cached objects. Future versions will expand its capabilities to support full output caching and read-write operations. Currently, Velocity provides a basic set of cache operations and management functionality through its client and server configuration.

Web Servers 
• IIS 
• 9 Production (2 Test/Dev) 
• Dell R610’s 
• 32GB Memory 
• 2xE5-5640
Data Tier 
• MS SQL Server 
• 4 Servers 
• 2 Always-On Clusters 
• Each Cluster 1 RW, 1 RO
Caching Tier 
• Redis 
• 2 Servers 
• Hot / Standby configuration
Tag Engine 
• Our Special index of SO 
• Tagging is hard 
• Written by Marc Gravell 
• http://blog.marcgravell.com/2014/04/technica 
l-debt-case-study-tags.html

Recommended for you

Building Ext JS Using HATEOAS - Jeff Stano
Building Ext JS Using HATEOAS - Jeff StanoBuilding Ext JS Using HATEOAS - Jeff Stano
Building Ext JS Using HATEOAS - Jeff Stano

This document discusses using HATEOAS (Hypermedia as the Engine of Application State) with the ExtJS JavaScript framework. It defines HATEOAS and common media types like HAL and Siren. HATEOAS reduces client-server coupling by having the client discover available state transitions from resource representations instead of having hardcoded URLs. The document explains how to extend ExtJS models and proxies to consume HAL responses and handle state transitions based on the link relations in HAL documents. Live coding examples demonstrate implementing this with the Sencha HAL package.

ext jsweb app develomenthateoas
SenchaCon 2016 - How to Auto Generate a Back-end in Minutes
SenchaCon 2016 - How to Auto Generate a Back-end in MinutesSenchaCon 2016 - How to Auto Generate a Back-end in Minutes
SenchaCon 2016 - How to Auto Generate a Back-end in Minutes

Connecting your JavaScript application to a database is tedious. Back-end developers spend hours modeling the database, securing connections, writing SQL, optimizing queries, deploying to a server, and fixing bugs. In this session, you'll learn how Ext Speeder gives your front-end team a tool to automatically generate a full back-end. In minutes, a REST API between a Sencha Ext JS Grid application and a relational database is created. This will save you a huge amount of time and also minimizes the risk of human error. Application time-to-market has never been shorter.

backendextspeederbig data
Ansible for large scale deployment
Ansible for large scale deploymentAnsible for large scale deployment
Ansible for large scale deployment

This is my presentation on MySQL user camp on 26-06-2015. It gives basic introduction to Ansible and how it can be benefited for MySQL deployment and configuration.

mysqlansible
Elastic Search 
• 203GB Index 
• 3 Machines 
• 42M searches/day
Deployment 
• Git 
• TeamCity 
• Custom Powershell Scripts
So what does this get 
you 
• 52 ms homepage render time 
• 33 ms questions page render time
Always See our 
Performance 
• http://stackexchange.com/performance

Recommended for you

SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...
SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...
SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...

LinkRest is an active project commissioned and open sourced by NHL under Apache 2.0 license. It was conceived specifically as the server-side counterpart to Ext JS, so it natively supports Ext JS rest proxy, including CRUD, grouping, sorting, filtering, and more. In this session, we'll review Ext JS protocol support. You'll also see extensions that allow the client to safely query the server for specific data and shape the response. We'll cover advanced features, including security, constraints, idempotent methods, and metadata service. We'll show demos and try to allow time for Q&A.

ext jsweb app developmentlinkrest
Optimising for Performance
Optimising for PerformanceOptimising for Performance
Optimising for Performance

This document discusses various techniques for optimizing website performance in order to improve search engine optimization (SEO) rankings. It outlines strategies for reducing page load time such as compressing files, lazy loading images, minimizing requests, caching content, and using modules like Apache's mod_pagespeed. The document stresses that the primary goal should be improving the user experience rather than solely focusing on achieving full scores on Google's Pagespeed Insights tool.

wordpresshostingperformance
SharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 PerformanceSharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 Performance

Is your farm struggling to server your organization? How long is it taking between page requests? Where is your bottleneck in your farm? Is your SQL Server tuned properly? Worried about upgrading due to poor performance? We will look at various tools for analyzing and measuring performance of your farm. We will look at simple SharePoint and IIS configuration options to instantly improve performance. I will discuss advanced approaches for analyzing, measuring and implementing optimizations in your farm.

sharepoint 2010performancemicrosoft sharepoint
Thank YOU! 
Contact: 
@GABeech 
george@stackoverflow.com 
Office Hours: 
Wednesday, November 12th 
(today…) 
2:00pm - 3:30pm 
LISA Lab

More Related Content

What's hot

Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDBZapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Oren Eini
 
Redis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs TalksRedis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs Talks
Redis Labs
 
SQL Azure for ISUG(SQL Server Israeli User Group)
SQL Azure for ISUG(SQL Server Israeli User Group)SQL Azure for ISUG(SQL Server Israeli User Group)
SQL Azure for ISUG(SQL Server Israeli User Group)
Pini Krisher
 
Lessons from the Trenches - Building Enterprise Applications with RavenDB
Lessons from the Trenches - Building Enterprise Applications with RavenDBLessons from the Trenches - Building Enterprise Applications with RavenDB
Lessons from the Trenches - Building Enterprise Applications with RavenDB
Oren Eini
 
Queryable State for Kafka Streamsを使ってみた
Queryable State for Kafka Streamsを使ってみたQueryable State for Kafka Streamsを使ってみた
Queryable State for Kafka Streamsを使ってみた
Yoshiyasu SAEKI
 
Mobile 3: Launch Like a Boss!
Mobile 3: Launch Like a Boss!Mobile 3: Launch Like a Boss!
Mobile 3: Launch Like a Boss!
MongoDB
 
Velocity - Edge UG
Velocity - Edge UGVelocity - Edge UG
Velocity - Edge UG
Phil Pursglove
 
CFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful CodeCFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful Code
indiver
 
Solving Enterprise Integration with Apache Camel
Solving Enterprise Integration with Apache CamelSolving Enterprise Integration with Apache Camel
Solving Enterprise Integration with Apache Camel
Christian Posta
 
Building big data pipelines with Kafka and Kubernetes
Building big data pipelines with Kafka and KubernetesBuilding big data pipelines with Kafka and Kubernetes
Building big data pipelines with Kafka and Kubernetes
Venu Ryali
 
Big Data DC - BenchPress
Big Data DC - BenchPressBig Data DC - BenchPress
Big Data DC - BenchPress
Drew Stephens
 
Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016
Domingo Suarez Torres
 
Redis Day Keynote Salvatore Sanfillipo Redis Labs
Redis Day Keynote Salvatore Sanfillipo Redis LabsRedis Day Keynote Salvatore Sanfillipo Redis Labs
Redis Day Keynote Salvatore Sanfillipo Redis Labs
Redis Labs
 
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit ChopraI3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
SPS Paris
 
Velocity - NxtGen Oxford
Velocity - NxtGen OxfordVelocity - NxtGen Oxford
Velocity - NxtGen Oxford
Phil Pursglove
 
Building Ext JS Using HATEOAS - Jeff Stano
Building Ext JS Using HATEOAS - Jeff StanoBuilding Ext JS Using HATEOAS - Jeff Stano
Building Ext JS Using HATEOAS - Jeff Stano
Sencha
 
SenchaCon 2016 - How to Auto Generate a Back-end in Minutes
SenchaCon 2016 - How to Auto Generate a Back-end in MinutesSenchaCon 2016 - How to Auto Generate a Back-end in Minutes
SenchaCon 2016 - How to Auto Generate a Back-end in Minutes
Malin Weiss
 
Ansible for large scale deployment
Ansible for large scale deploymentAnsible for large scale deployment
Ansible for large scale deployment
Karthik .P.R
 
SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...
SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...
SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...
Sencha
 
Optimising for Performance
Optimising for PerformanceOptimising for Performance
Optimising for Performance
thomas_mb
 

What's hot (20)

Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDBZapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
 
Redis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs TalksRedis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs Talks
 
SQL Azure for ISUG(SQL Server Israeli User Group)
SQL Azure for ISUG(SQL Server Israeli User Group)SQL Azure for ISUG(SQL Server Israeli User Group)
SQL Azure for ISUG(SQL Server Israeli User Group)
 
Lessons from the Trenches - Building Enterprise Applications with RavenDB
Lessons from the Trenches - Building Enterprise Applications with RavenDBLessons from the Trenches - Building Enterprise Applications with RavenDB
Lessons from the Trenches - Building Enterprise Applications with RavenDB
 
Queryable State for Kafka Streamsを使ってみた
Queryable State for Kafka Streamsを使ってみたQueryable State for Kafka Streamsを使ってみた
Queryable State for Kafka Streamsを使ってみた
 
Mobile 3: Launch Like a Boss!
Mobile 3: Launch Like a Boss!Mobile 3: Launch Like a Boss!
Mobile 3: Launch Like a Boss!
 
Velocity - Edge UG
Velocity - Edge UGVelocity - Edge UG
Velocity - Edge UG
 
CFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful CodeCFWheels - Pragmatic, Beautiful Code
CFWheels - Pragmatic, Beautiful Code
 
Solving Enterprise Integration with Apache Camel
Solving Enterprise Integration with Apache CamelSolving Enterprise Integration with Apache Camel
Solving Enterprise Integration with Apache Camel
 
Building big data pipelines with Kafka and Kubernetes
Building big data pipelines with Kafka and KubernetesBuilding big data pipelines with Kafka and Kubernetes
Building big data pipelines with Kafka and Kubernetes
 
Big Data DC - BenchPress
Big Data DC - BenchPressBig Data DC - BenchPress
Big Data DC - BenchPress
 
Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016
 
Redis Day Keynote Salvatore Sanfillipo Redis Labs
Redis Day Keynote Salvatore Sanfillipo Redis LabsRedis Day Keynote Salvatore Sanfillipo Redis Labs
Redis Day Keynote Salvatore Sanfillipo Redis Labs
 
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit ChopraI3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
 
Velocity - NxtGen Oxford
Velocity - NxtGen OxfordVelocity - NxtGen Oxford
Velocity - NxtGen Oxford
 
Building Ext JS Using HATEOAS - Jeff Stano
Building Ext JS Using HATEOAS - Jeff StanoBuilding Ext JS Using HATEOAS - Jeff Stano
Building Ext JS Using HATEOAS - Jeff Stano
 
SenchaCon 2016 - How to Auto Generate a Back-end in Minutes
SenchaCon 2016 - How to Auto Generate a Back-end in MinutesSenchaCon 2016 - How to Auto Generate a Back-end in Minutes
SenchaCon 2016 - How to Auto Generate a Back-end in Minutes
 
Ansible for large scale deployment
Ansible for large scale deploymentAnsible for large scale deployment
Ansible for large scale deployment
 
SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...
SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...
SenchaCon 2016: LinkRest - Modern RESTful API Framework for Ext JS Apps - Rou...
 
Optimising for Performance
Optimising for PerformanceOptimising for Performance
Optimising for Performance
 

Similar to Stack Exchange Infrastructure - LISA 14

SharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 PerformanceSharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 Performance
Brian Culver
 
Scaling with swagger
Scaling with swaggerScaling with swagger
Scaling with swagger
Tony Tam
 
Boost the Performance of SharePoint Today!
Boost the Performance of SharePoint Today!Boost the Performance of SharePoint Today!
Boost the Performance of SharePoint Today!
Brian Culver
 
SharePoint Saturday The Conference 2011 - SP2010 Performance
SharePoint Saturday The Conference 2011 - SP2010 PerformanceSharePoint Saturday The Conference 2011 - SP2010 Performance
SharePoint Saturday The Conference 2011 - SP2010 Performance
Brian Culver
 
My Site is slow - Drupal Camp London 2013
My Site is slow - Drupal Camp London 2013My Site is slow - Drupal Camp London 2013
My Site is slow - Drupal Camp London 2013
hernanibf
 
Be faster then rabbits
Be faster then rabbitsBe faster then rabbits
Be faster then rabbits
Vladislav Bauer
 
ActiveMQ 5.9.x new features
ActiveMQ 5.9.x new featuresActiveMQ 5.9.x new features
ActiveMQ 5.9.x new features
Christian Posta
 
My site is slow
My site is slowMy site is slow
My site is slow
hernanibf
 
Scaling Social Games
Scaling Social GamesScaling Social Games
Scaling Social Games
Paolo Negri
 
Tuning Your SharePoint Environment
Tuning Your SharePoint EnvironmentTuning Your SharePoint Environment
Tuning Your SharePoint Environment
vmaximiuk
 
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
MariaDB 10.1   what's new and what's coming in 10.2 - Tokyo MariaDB MeetupMariaDB 10.1   what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
Colin Charles
 
Drupal performance
Drupal performanceDrupal performance
Drupal performance
Piyuesh Kumar
 
Operationalizing MongoDB at AOL
Operationalizing MongoDB at AOLOperationalizing MongoDB at AOL
Operationalizing MongoDB at AOL
radiocats
 
MongoDC 2012: "Operationalizing" MongoDB@AOL
MongoDC 2012: "Operationalizing" MongoDB@AOLMongoDC 2012: "Operationalizing" MongoDB@AOL
MongoDC 2012: "Operationalizing" MongoDB@AOL
MongoDB
 
Restful风格ž„web服务架构
Restful风格ž„web服务架构Restful风格ž„web服务架构
Restful风格ž„web服务架构
Benjamin Tan
 
mtl_rubykaigi
mtl_rubykaigimtl_rubykaigi
mtl_rubykaigi
Hirotomo Oi
 
Apache Geode Meetup, London
Apache Geode Meetup, LondonApache Geode Meetup, London
Apache Geode Meetup, London
Apache Geode
 
Pascal benois performance_troubleshooting-spsbe18
Pascal benois performance_troubleshooting-spsbe18Pascal benois performance_troubleshooting-spsbe18
Pascal benois performance_troubleshooting-spsbe18
BIWUG
 
SharePoint 2013 Performance Analysis - Robi Vončina
SharePoint 2013 Performance Analysis - Robi VončinaSharePoint 2013 Performance Analysis - Robi Vončina
SharePoint 2013 Performance Analysis - Robi Vončina
SPC Adriatics
 
Rack
RackRack
Rack
shaokun
 

Similar to Stack Exchange Infrastructure - LISA 14 (20)

SharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 PerformanceSharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 Performance
 
Scaling with swagger
Scaling with swaggerScaling with swagger
Scaling with swagger
 
Boost the Performance of SharePoint Today!
Boost the Performance of SharePoint Today!Boost the Performance of SharePoint Today!
Boost the Performance of SharePoint Today!
 
SharePoint Saturday The Conference 2011 - SP2010 Performance
SharePoint Saturday The Conference 2011 - SP2010 PerformanceSharePoint Saturday The Conference 2011 - SP2010 Performance
SharePoint Saturday The Conference 2011 - SP2010 Performance
 
My Site is slow - Drupal Camp London 2013
My Site is slow - Drupal Camp London 2013My Site is slow - Drupal Camp London 2013
My Site is slow - Drupal Camp London 2013
 
Be faster then rabbits
Be faster then rabbitsBe faster then rabbits
Be faster then rabbits
 
ActiveMQ 5.9.x new features
ActiveMQ 5.9.x new featuresActiveMQ 5.9.x new features
ActiveMQ 5.9.x new features
 
My site is slow
My site is slowMy site is slow
My site is slow
 
Scaling Social Games
Scaling Social GamesScaling Social Games
Scaling Social Games
 
Tuning Your SharePoint Environment
Tuning Your SharePoint EnvironmentTuning Your SharePoint Environment
Tuning Your SharePoint Environment
 
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
MariaDB 10.1   what's new and what's coming in 10.2 - Tokyo MariaDB MeetupMariaDB 10.1   what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
 
Drupal performance
Drupal performanceDrupal performance
Drupal performance
 
Operationalizing MongoDB at AOL
Operationalizing MongoDB at AOLOperationalizing MongoDB at AOL
Operationalizing MongoDB at AOL
 
MongoDC 2012: "Operationalizing" MongoDB@AOL
MongoDC 2012: "Operationalizing" MongoDB@AOLMongoDC 2012: "Operationalizing" MongoDB@AOL
MongoDC 2012: "Operationalizing" MongoDB@AOL
 
Restful风格ž„web服务架构
Restful风格ž„web服务架构Restful风格ž„web服务架构
Restful风格ž„web服务架构
 
mtl_rubykaigi
mtl_rubykaigimtl_rubykaigi
mtl_rubykaigi
 
Apache Geode Meetup, London
Apache Geode Meetup, LondonApache Geode Meetup, London
Apache Geode Meetup, London
 
Pascal benois performance_troubleshooting-spsbe18
Pascal benois performance_troubleshooting-spsbe18Pascal benois performance_troubleshooting-spsbe18
Pascal benois performance_troubleshooting-spsbe18
 
SharePoint 2013 Performance Analysis - Robi Vončina
SharePoint 2013 Performance Analysis - Robi VončinaSharePoint 2013 Performance Analysis - Robi Vončina
SharePoint 2013 Performance Analysis - Robi Vončina
 
Rack
RackRack
Rack
 

Recently uploaded

The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
Larry Smarr
 
Comparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdfComparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdf
Andrey Yasko
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Mydbops
 
Password Rotation in 2024 is still Relevant
Password Rotation in 2024 is still RelevantPassword Rotation in 2024 is still Relevant
Password Rotation in 2024 is still Relevant
Bert Blevins
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Chris Swan
 
WPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide DeckWPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide Deck
Lidia A.
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
Quantum Communications Q&A with Gemini LLM
Quantum Communications Q&A with Gemini LLMQuantum Communications Q&A with Gemini LLM
Quantum Communications Q&A with Gemini LLM
Vijayananda Mohire
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
ishalveerrandhawa1
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
The Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU CampusesThe Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU Campuses
Larry Smarr
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
rajancomputerfbd
 
Coordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar SlidesCoordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar Slides
Safe Software
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Bert Blevins
 
論文紹介:A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...
論文紹介:A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...論文紹介:A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...
論文紹介:A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...
Toru Tamaki
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
Manual | Product | Research Presentation
Manual | Product | Research PresentationManual | Product | Research Presentation
Manual | Product | Research Presentation
welrejdoall
 
Pigging Solutions Sustainability brochure.pdf
Pigging Solutions Sustainability brochure.pdfPigging Solutions Sustainability brochure.pdf
Pigging Solutions Sustainability brochure.pdf
Pigging Solutions
 
Measuring the Impact of Network Latency at Twitter
Measuring the Impact of Network Latency at TwitterMeasuring the Impact of Network Latency at Twitter
Measuring the Impact of Network Latency at Twitter
ScyllaDB
 

Recently uploaded (20)

The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
 
Comparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdfComparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdf
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
 
Password Rotation in 2024 is still Relevant
Password Rotation in 2024 is still RelevantPassword Rotation in 2024 is still Relevant
Password Rotation in 2024 is still Relevant
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
 
WPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide DeckWPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide Deck
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
Quantum Communications Q&A with Gemini LLM
Quantum Communications Q&A with Gemini LLMQuantum Communications Q&A with Gemini LLM
Quantum Communications Q&A with Gemini LLM
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
The Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU CampusesThe Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU Campuses
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
 
Coordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar SlidesCoordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar Slides
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
 
論文紹介:A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...
論文紹介:A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...論文紹介:A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...
論文紹介:A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
Manual | Product | Research Presentation
Manual | Product | Research PresentationManual | Product | Research Presentation
Manual | Product | Research Presentation
 
Pigging Solutions Sustainability brochure.pdf
Pigging Solutions Sustainability brochure.pdfPigging Solutions Sustainability brochure.pdf
Pigging Solutions Sustainability brochure.pdf
 
Measuring the Impact of Network Latency at Twitter
Measuring the Impact of Network Latency at TwitterMeasuring the Impact of Network Latency at Twitter
Measuring the Impact of Network Latency at Twitter
 

Stack Exchange Infrastructure - LISA 14

  • 1. The Stack Exchange Infrastructure Vroom Vroom
  • 2. inet.perf.profile • SRE Generalist @ Stack Exchange • @GABeech • http://brokenhaze.com • http://stackexchange.com
  • 3. A brief Overview • 560 Million Page Views a Month • 34TB of Data transfered a Month • 1665 rps (2250 peak) Across web Farm • WISC(HER)
  • 4. Our First Priority is Performance Nobody likes a slow site, least of all us. When your site is slow people leave. Make your site fast, and the people will stay Good write up on moz.com: http://moz.com/blog/site-speed-are-you-fast-does- it-matter
  • 5. The Performance toolkit • Mini Profiler • OpServer (https://github.com/opserver/Opserver) • Client Timings (http://teststackoverflow.com/)
  • 12. You can’t be fast if you are not up • Highly Redundant network • Datacenter, ISP, Edge, Core, Server, Port
  • 14. Load Balencers • HAProxy • 2 Servers (Hot/Standby) • Multiple Tiers (HAProxy Processes)
  • 16. SSL Termination • Terminated at LB • Feature added to HAProxy 1.5 • See: http://brokenhaze.com/blog/2014/03/25/ how-stack-exchange-gets-the-most-out-of- haproxy/
  • 17. Web Servers • IIS • 9 Production (2 Test/Dev) • Dell R610’s • 32GB Memory • 2xE5-5640
  • 18. Data Tier • MS SQL Server • 4 Servers • 2 Always-On Clusters • Each Cluster 1 RW, 1 RO
  • 19. Caching Tier • Redis • 2 Servers • Hot / Standby configuration
  • 20. Tag Engine • Our Special index of SO • Tagging is hard • Written by Marc Gravell • http://blog.marcgravell.com/2014/04/technica l-debt-case-study-tags.html
  • 21. Elastic Search • 203GB Index • 3 Machines • 42M searches/day
  • 22. Deployment • Git • TeamCity • Custom Powershell Scripts
  • 23. So what does this get you • 52 ms homepage render time • 33 ms questions page render time
  • 24. Always See our Performance • http://stackexchange.com/performance
  • 25. Thank YOU! Contact: @GABeech george@stackoverflow.com Office Hours: Wednesday, November 12th (today…) 2:00pm - 3:30pm LISA Lab

Editor's Notes

  1. Windows IIS SQL Server C# HAProxy Elastic Search Redis
  2. Why do I bring up performance in an infra talk? simple. It drives our design decisions.
  3. Shown to every Dev/SRE on every page Oneboxed in our chat system
  4. Bubbles up problems
  5. How well are we actually doing when _you_ load the page
  6. The actual design starts now.
  7. 4 Different providers Selected for different characteristics Router Redundancy Hot/Standby HSRP/BGP on “T2” Full BGP tables and HSRP on T1
  8. 4B requests/month 3000 req/sec peak 10% CPU 18% peak Between 600k and 700k concurrent connections (EST, TIME_WAIT, ETC) Multiple Processes Allow for granular restarts and segregation of faults SSL Termination done on the LB Websockets: The weird connection Long lived TCP not HTTP
  9. Request flow In, is http? yes, servers: no term https, is http
  10. Source Port Exhaustion use 127.0.0.0/8 to resolve Server only running at ~12% cpu We don’t run full SSL everywhere yet
  11. 185 req/s 250 peak 15% CPU usage 20% peak
  12. (SO) 343 M Queries per day (SO) Peak of 7500 queries / second (SE) 216M Queries per day (SE) Peak 3200 queries / second CPU Use: SO 8% Peak 15% — SE 10% Peak 20%
  13. 3.65 B operations a day Peak 60,000/s 3% cpu usage
  14. 3 Servers, 32 GB RAM 3644 req/s 3% CPU 10% peak Replaced Full Text search in SQL Server Spins up a full copy of SO/SE Cool thing can be upgraded with 0 downtime
  15. 2 others/ not prod Machine learning Log stash (300TB)
  16. Team City monitors our Development Git repository Dev Auto builds (Deploy to Meta) When the build is verified Dev triggers Prod Build Copy Artifacts from Dev Build