SlideShare a Scribd company logo

Carlos Conde : AWS Game Days - TIAD Paris

•
•
The Incredible Automation Day
The Incredible Automation Day

This document discusses using "Game Days" to simulate crisis situations and test systems, applications, and teams. It recommends creating a simulated environment to practice deployments, failures, and responses. Key aspects include role playing different scenarios, recording results, and debriefing to identify issues and improve procedures. The goal is to validate architectures and assumptions, prove resilience, and train teams through a fun but realistic learning experience.

Related slideshows

Go Faster with Ansible (PHP meetup)
Go Faster with Ansible (PHP meetup)Go Faster with Ansible (PHP meetup)
Go Faster with Ansible (PHP meetup)
 
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesReusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modules
 
(WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014
(WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014(WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014
(WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014
 
1 of 82
@caarlco GAME DAYS CRASH TEST YOUR APPLICATION… AND YOUR TEAM
2008 CONTINUOUS DEPLOYMENT
Carlos Conde : AWS Game Days - TIAD Paris
THEORY OF CONSTRAINTS 1.  THROUGHPUT 2.  INVENTORY 3.  OPERATIONAL EXPENSE
“At Amazon, the goal is to transform new ideas into high quality production systems that provide customer value”
1.  THROUGHPUT Number of ideas successfully transformed into high quality prod systems providing customer value over a unit of time
2. SOFTWARE INVENTORY Code you created, time and money invested. If not deployed into prod, no return on investment.
3. OPERATIONAL EXPENSE Get lean: eliminate waste. Every process that is not automated is waste.
FASTER IS ALWAYS BETTER
11.6s Mean time between deployments (weekday) 1,079 Max number of deployments in a single hour 10,000 Mean # of hosts simultaneously receiving a deployment 30,000 Max # of hosts simultaneously receiving a deployment DEPLOYMENTS AT AMAZON.COM (2011)
Carlos Conde : AWS Game Days - TIAD Paris
MODULES & LIBRARIES LANGUAGE INTERPRETERS OBJECT-ORIENTED PROGRAMMING
MODULES & LIBRARIES LANGUAGE INTERPRETERS OBJECT-ORIENTED PROGRAMMING CONTAINERS VIRTUAL MACHINES MICRO SERVICES ARCHITECTURES
SCALABILITY MAINTAINABILITY RELIABILITY DURABILITY CONFIGURABILITY ...
RESILIENCE Ability to provide and maintain an acceptable level of service in the face of faults and challenges to normal operation
RESILIENCE Capacity to anticipate disruptions, adapt to events, and deliver value
RESILIENCE Ability to cope with change
ADAPTABILITY
HUMAN OPS ARE PART OF THE SYSTEM
BEST ARCHITECTURES HAVE THE BEST OPS
HOW TO TRAIN THE BEST OPS?
PLAY IN A SIMULATED ENVIRONMENT
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
IF SOMETHING IS HARD REPETITION MAKES IT EASIER
SIMULATION ENVIRONMENT FOR CRISIS SITUATIONS
GOOD WEATHER DOESN’T MAKE GOOD SAILORS
2000 GAME DAYS
1. PREPARE SETTING THE STAGE
PROCTORS & PLAYERS ROLE PLAY
MAKE A PLAN WHAT ARE YOU GOING TO TEST?
PRE-MORTEM BRAINSTORMING TECHNIQUE IMAGINE HOW THINGS CAN FAIL
CREATE THE SIMULATION ENVIRONMENT
ON-DEMAND PAY AS YOU GO ELASTIC
+ = PROGRAMMABLE PLATFORM AWS APIs
AWS CLOUDFORMATION STACK-BASED PROVISIONING SERVICE
CLOUDFORMATION TEMPLATE
{      "Description"  :  "Create  RDS  with  username  and  password",      "Resources"  :  {            "MyDB"  :  {              "Type"  :  "AWS::RDS::DBInstance",              "Properties"  :  {                  "AllocatedStorage"  :  "500",                  "DBInstanceClass"  :  "db.m1.small",                  "Engine"  :  "MySQL",                  "EngineVersion"  :  "5.5",                  "MasterUsername"  :  "MyName",                  "MasterUserPassword"  :  "MyPassword"              }          }}}  
"AWS::CloudFormation::Init"  :  {  "config"  :  {            "packages"  :  {              "yum"  :  {                  "mysql"                :  [],                  "mysql-­‐server"  :  [],                  "httpd"                :  [],                  "php"                    :  [],                  "php-­‐mysql"        :  []              }},            "sources"  :  {              "/var/www/html"  :                    "https://s3.amazonaws.com/my-­‐builds/build-­‐v4.zip"          }}}  
INFRASTRUCTURE AS CODE
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
Carlos Conde : AWS Game Days - TIAD Paris
SIMULATE ACTIVITY
REPLAY PRE-RECORDED NETWORK ACTIVITY
“PRODUCTION IS TRULY THE ONLY PLACE YOU CAN VALIDATE YOUR CODE”
(THROTTLED) LOAD-TEST USE AMAZON EC2 SPOT INSTANCES
APACHE BENCH – AB APACHE JMETER – JAVA, DBs LOCUST.IO – PYTHON THE GRINDER – JAVA
Carlos Conde : AWS Game Days - TIAD Paris
USING AMAZON EC2 TO SIMULATE 2.4 MILLION PLAYERS
MAKE IT FUN SET A DEADLINE
2. PLAY! THE FUN PART
SIMULATE FAILURES
REMINDER COMPARED TO REAL LIFE, YOU’RE NOT SADISTIC ENOUGH
•  TERMINATE RESOURCES •  CHANGE SECURITY GROUPS •  CHANGE IAM ROLES •  DISABLE IAM USER •  CHANGE /ETC/HOSTS FILE •  AMAZON RDS FAIL-OVER TEST
Carlos Conde : AWS Game Days - TIAD Paris
IF YOU DON’T MEASURE IT IT DIDN’T HAPPEN
RECORD & ANNOTATE PLAYER’S ACTIONS
AWS CLOUDTRAIL GAMEPLAY RECORDING & AUDITING
Carlos Conde : AWS Game Days - TIAD Paris
3. DEBRIEF WHERE THE MAGIC HAPPENS
Carlos Conde : AWS Game Days - TIAD Paris
HOW LONG DOES IT TAKE TO DETECT AN EVENT? TO TAKE ACTION?
WHICH LATENT DEFECTS DID YOU EXPOSE?
COMMUNICATION CHANNELS ONLINE CHAT, CRIS ROOM, SHARED DOCS, ETC…
ROOKIE DAY GAME DAY FOR NEW HIRES
STAR-LESS DAY GAME DAY WITHOUT YOUR ROCK-STAR TEAM MEMBERS
VALIDATE YOUR ASSUMPTIONS PROVE YOUR ARCHITECTURE KNOW YOUR PROCEDURES LEARN FROM YOUR FAILURES
SUCCESS MAKES PEOPLE FEEL GOOD FAILURE MAKES PEOPLE BETTER
LET ME KNOW ABOUT YOUR GAME DAYS @caarlco
@aws_actus

More Related Content

Carlos Conde : AWS Game Days - TIAD Paris