SlideShare a Scribd company logo

Lights, Camera, Docker: Streaming Video at DramaFever

•
•
B
bridgetkromhout

Bridget Kromhout gave a presentation on how her company DramaFever uses Docker for streaming video. They run Docker in production and have seen peak loads of tens of thousands of requests per second. They use Docker, AWS, and other technologies to build and deploy images automatically and scale their infrastructure elastically. Bridget discussed challenges they faced with Docker and solutions like changing the storage driver and registry configuration to address issues with builds and image pulls failing.

1 of 44
Download to read offline
Lights, Camera, Docker: Streaming Video at_______________ Bridget Kromhout
@bridgetkromhout Bridget Kromhout Operations Engineer @DramaFever Minneapolis, Minnesota @devopsdays @devopsdaysMSP @arresteddevops bridgetkromhout.com
dramafever.com: streaming international content @bridgetkromhout starting in 2009; Docker in prod since Oct 2013
peak load: tens of thousands of requests per second traffic variance: swings 10-20x throughout the week @bridgetkromhout
@bridgetkromhout docclub.com starting in Fall 2014
@bridgetkromhout shudder.com starting in Spring 2015
autoscaling in AWS @bridgetkromhout streaming delivery via Akamai
Python (Django) main website Go microservices (video analytics ingest, on-the-fly image processing, bookmarking…) Upstreams routed via nginx Celery + SQS for async tasks @bridgetkromhout Architecture
distributed private S3-backed Docker registry: for effective scaling @bridgetkromhout
docker options # this goes in /etc/default/docker to control docker's upstart config DOCKER_OPTS="--graph=/mnt/docker --insecure- registry=localhost-alias.com:5000" ● localhost-alias.com in DNS with A record to 127.0.0.1 ● OS X /etc/hosts: use the boot2docker host-only network IP @bridgetkromhout
registry upstart docker pull public_registry_image docker run -p 5000:5000 --name registry -v /etc/docker-reg:/registry-conf -e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml public_registry_image @bridgetkromhout
config.yml s3_region: us-east-1 s3_access_key: <aws-accesskey> s3_secret_key: <aws-secretkey> s3_bucket: <bucketname> standalone: true storage: s3 storage_path: /registry @bridgetkromhout
@bridgetkromhout Jenkins-driven image builds
weekly base builds FROM local-repo-alias.com:5000/www-base ● include infrequently-changing dependencies ○ ubuntu packages ○ pip requirements ○ wheels ● other builds can start from these images (so they’re faster): @bridgetkromhout
sudo docker build -t="a12fbdc" . sudo docker run -i -t -w /var/www -e DJANGO_TEST=1 -- name test.a12fbdc a12fbdc py.test -s sudo docker tag a12fbdc local-repo-alias.com: 5000/www:'dev' sudo docker push local-repo-alias.com:5000/www:'dev' @bridgetkromhout www-master build
2014/10/30 21:35:31 Error getting container init rootfs b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f 3108f86386aab67 from driver devicemapper: open /dev/mapper/docker-9:127-14024705- b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f 3108f86386aab67-init: no such file or directory make: *** [build-django] Error 1 Build step 'Execute shell' marked build as failure @bridgetkromhout breaking builds
@bridgetkromhout
@bridgetkromhout useful for unattended base builds, but... ...seeing this in Slack got old
@bridgetkromhout DOCKER_OPTS="--graph=/mnt/docker --insecure-registry=local-repo- alias.com:5000 --storage- driver=aufs" replace docker storage driver on jenkins instance
@bridgetkromhout bash 'install kernel extras for aufs' do code <<-EOH apt-get -y install linux-image- extra-$(uname -r) EOH end ubuntu 14.04 provides aufs in kernel extras
@bridgetkromhout
@bridgetkromhout
#!/bin/bash cat <<EOF > /etc/init/django.conf description "Run Django containers for www" start on started docker-reg stop on runlevel [!2345] or stopped docker respawn limit 5 30 [...] replacing 100s of lines of userdata... @bridgetkromhout
#!/bin/bash # upstart configs are now created by chef rm /etc/chef/client.pem mkdir -p /var/log/chef chef-client -r 'role[rolename]' -E 'environment' -L /var/log/chef/chef-client.log ...with a chef-client run. @bridgetkromhout
upstart config docker run -e DJANGO_ENVIRON=PROD -e HAPROXY=df/haproxy-prod.cfg -p 8000:8000 -v /var/log/containers:/var/log --name django localhost-alias.com:5000/www:prod /var/www/bin/start-django @bridgetkromhout
docker run <% if @docker_rm == true -%> --rm <% end %> <% @docker_env.each do |k, v| -%> -e <%= k %>=<%= v %> <% end %> <% @docker_port.each do |p| -%> -p <%= p %> <% end %> upstart template @bridgetkromhout
<% @docker_volume.each do |v| -%> -v <%= v %> <% end %> --name <%= @application_name %> localhost-alias.com:<%= @registry_port %>/<%= @docker_image %>:<%= @docker_tag %> <%= @docker_command %> upstart template (cont) @bridgetkromhout
attribute :command, :kind_of => String, :required => true attribute :env, :kind_of => Hash, :default => {} attribute :port, :kind_of => Array, :default => [] attribute :volume, :kind_of => Array, :default => ['/var/log/containers:/var/log'] attribute :rm, :kind_of => [TrueClass, FalseClass], :default => false attribute :image, :kind_of => String, :required => true attribute :tag, :kind_of => String, :required => true attribute :type, :kind_of => String, :required => true attribute :cron, :kind_of => [TrueClass, FalseClass], :default => false using attributes @bridgetkromhout
recipe using LWRP base_docker node['www']['django']['name'] do command node['www']['django']['command'] env node['www'][service]['django'][env]['env'] image node['www']['django']['image'] port node['www'][service]['django'][env]['port'] tag node['www'][service]['django'][env]['tag'] type node['www']['django']['type'] end @bridgetkromhout
ami factory Autoscaling Packer AMI EC2 Instances Jenkins GitHub Chef @bridgetkromhout
packer for ami building { "type": "chef-client", "server_url": "https://api.opscode. com/organizations/dramafever", "run_list": [ "base::ami" ], "validation_key_path": "{{user `chef_validation`}}", "validation_client_name": "dramafever-validator", "node_name": "packer-ami" } @bridgetkromhout
packer run$HOME/packer/packer build -var "account_id=$AWS_ACCOUNT_ID" -var "aws_access_key_id=$AWS_ACCESS_KEY_ID" -var "aws_secret_key=$AWS_SECRET_ACCESS_KEY" -var "x509_cert_path=$AWS_X509_CERT_PATH" -var "x509_key_path=$AWS_X509_KEY_PATH" -var "s3_bucket=bucketname" -var "ami_name=$AMI_NAME" -var "source_ami=$SOURCE_AMI" -var "chef_validation=$CHEF_VAL" -var "chef_client=$HOME/packer/client.rb" -only=amazon-instance $HOME/packer/prod.json @bridgetkromhout
limiting packer IAM permissions "Action":[ "ec2:TerminateInstances", "ec2:StopInstances", "ec2:DeleteSnapshot", "ec2:DetachVolume", "ec2:DeleteVolume", "ec2:ModifyImageAttribute" ], "Effect":"Allow", "Resource":"*", "Condition":{ "StringEquals":{ "ec2:ResourceTag/name":"Packer Builder" } } @bridgetkromhout
@bridgetkromhout uses fabric: tag for staging tag for prod out of ELB restart upstart back in ELB Ship it!
@bridgetkromhout obligatory picture of container disaster (protip: does not represent reality)
S3 requires clock sync $ docker pull local-repo-alias.com:5000/mysql Pulling repository local-repo-alias.com:5000/mysql 2014/11/24 19:44:31 HTTP code: 500 $ boot2docker ssh sudo date --set "$(env TZ=UTC date '+%F %H:%M:%S')" @bridgetkromhout
for persistent instances # remove stopped containers @daily docker rm `docker ps -aq` # remove images tagged "none" @daily docker rmi `sudo docker images | grep none | awk -F' +' '{print $3}'` @bridgetkromhout
what even is flate?! Pulling repository local-repo-alias.com:5000/www 4dda2b433370: Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671 flate: corrupt input before offset 54393671 d497ad3926c8: Error downloading dependent layers 2014/12/07 02:34:54 Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671 @bridgetkromhout
registry rewrite just landed! DOCKER_OPTS="--graph=/mnt/docker -- insecure-registry=local-repo-alias. com:5000 -e STORAGE_REDIRECT=true" until we upgrade to 2.0, our “flate” workaround: @bridgetkromhout
isolation? -v /var/log/containers:/var/log <Input containers_in> Module im_file Recursive False File '/var/log/containers/*.log' Exec $FileName = file_name(); Exec $raw_event = $FileName + ' ' + $raw_event ; Exec $Message = $raw_event ; </Input> @bridgetkromhout
(un-)containerize some of the things @bridgetkromhout
container-building containers easier with statically linked binaries Go microservices Android APK @bridgetkromhout
$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE local-repo-alias.com:5000/mysql dev b0dc5885f767 2 days ago 905.9 MB local-repo-alias.com:5000/www dev 82cda604a4f1 2 days ago 1.092 GB local-repo-alias.com:5000/micro local bed20dc84ea1 4 days ago 10.08 MB google/golang 1.3 e3934c44b8e4 2 weeks ago 514.3 MB public_registry_image 0.6.9 11299d377a9e 6 months ago 454.5 MB scratch latest 511136ea3c5a 18 months ago 0 B $ ever-smaller images @bridgetkromhout
@bridgetkromhout www.dramafever.com/company/careers.html Thank you! (and we’re hiring!)

More Related Content

Lights, Camera, Docker: Streaming Video at DramaFever

  • 1. Lights, Camera, Docker: Streaming Video at_______________ Bridget Kromhout
  • 2. @bridgetkromhout Bridget Kromhout Operations Engineer @DramaFever Minneapolis, Minnesota @devopsdays @devopsdaysMSP @arresteddevops bridgetkromhout.com
  • 3. dramafever.com: streaming international content @bridgetkromhout starting in 2009; Docker in prod since Oct 2013
  • 4. peak load: tens of thousands of requests per second traffic variance: swings 10-20x throughout the week @bridgetkromhout
  • 8. Python (Django) main website Go microservices (video analytics ingest, on-the-fly image processing, bookmarking…) Upstreams routed via nginx Celery + SQS for async tasks @bridgetkromhout Architecture
  • 10. docker options # this goes in /etc/default/docker to control docker's upstart config DOCKER_OPTS="--graph=/mnt/docker --insecure- registry=localhost-alias.com:5000" ● localhost-alias.com in DNS with A record to 127.0.0.1 ● OS X /etc/hosts: use the boot2docker host-only network IP @bridgetkromhout
  • 11. registry upstart docker pull public_registry_image docker run -p 5000:5000 --name registry -v /etc/docker-reg:/registry-conf -e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml public_registry_image @bridgetkromhout
  • 12. config.yml s3_region: us-east-1 s3_access_key: <aws-accesskey> s3_secret_key: <aws-secretkey> s3_bucket: <bucketname> standalone: true storage: s3 storage_path: /registry @bridgetkromhout
  • 14. weekly base builds FROM local-repo-alias.com:5000/www-base ● include infrequently-changing dependencies ○ ubuntu packages ○ pip requirements ○ wheels ● other builds can start from these images (so they’re faster): @bridgetkromhout
  • 15. sudo docker build -t="a12fbdc" . sudo docker run -i -t -w /var/www -e DJANGO_TEST=1 -- name test.a12fbdc a12fbdc py.test -s sudo docker tag a12fbdc local-repo-alias.com: 5000/www:'dev' sudo docker push local-repo-alias.com:5000/www:'dev' @bridgetkromhout www-master build
  • 16. 2014/10/30 21:35:31 Error getting container init rootfs b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f 3108f86386aab67 from driver devicemapper: open /dev/mapper/docker-9:127-14024705- b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f 3108f86386aab67-init: no such file or directory make: *** [build-django] Error 1 Build step 'Execute shell' marked build as failure @bridgetkromhout breaking builds
  • 18. @bridgetkromhout useful for unattended base builds, but... ...seeing this in Slack got old
  • 20. @bridgetkromhout bash 'install kernel extras for aufs' do code <<-EOH apt-get -y install linux-image- extra-$(uname -r) EOH end ubuntu 14.04 provides aufs in kernel extras
  • 23. #!/bin/bash cat <<EOF > /etc/init/django.conf description "Run Django containers for www" start on started docker-reg stop on runlevel [!2345] or stopped docker respawn limit 5 30 [...] replacing 100s of lines of userdata... @bridgetkromhout
  • 24. #!/bin/bash # upstart configs are now created by chef rm /etc/chef/client.pem mkdir -p /var/log/chef chef-client -r 'role[rolename]' -E 'environment' -L /var/log/chef/chef-client.log ...with a chef-client run. @bridgetkromhout
  • 25. upstart config docker run -e DJANGO_ENVIRON=PROD -e HAPROXY=df/haproxy-prod.cfg -p 8000:8000 -v /var/log/containers:/var/log --name django localhost-alias.com:5000/www:prod /var/www/bin/start-django @bridgetkromhout
  • 26. docker run <% if @docker_rm == true -%> --rm <% end %> <% @docker_env.each do |k, v| -%> -e <%= k %>=<%= v %> <% end %> <% @docker_port.each do |p| -%> -p <%= p %> <% end %> upstart template @bridgetkromhout
  • 27. <% @docker_volume.each do |v| -%> -v <%= v %> <% end %> --name <%= @application_name %> localhost-alias.com:<%= @registry_port %>/<%= @docker_image %>:<%= @docker_tag %> <%= @docker_command %> upstart template (cont) @bridgetkromhout
  • 28. attribute :command, :kind_of => String, :required => true attribute :env, :kind_of => Hash, :default => {} attribute :port, :kind_of => Array, :default => [] attribute :volume, :kind_of => Array, :default => ['/var/log/containers:/var/log'] attribute :rm, :kind_of => [TrueClass, FalseClass], :default => false attribute :image, :kind_of => String, :required => true attribute :tag, :kind_of => String, :required => true attribute :type, :kind_of => String, :required => true attribute :cron, :kind_of => [TrueClass, FalseClass], :default => false using attributes @bridgetkromhout
  • 29. recipe using LWRP base_docker node['www']['django']['name'] do command node['www']['django']['command'] env node['www'][service]['django'][env]['env'] image node['www']['django']['image'] port node['www'][service]['django'][env]['port'] tag node['www'][service]['django'][env]['tag'] type node['www']['django']['type'] end @bridgetkromhout
  • 31. packer for ami building { "type": "chef-client", "server_url": "https://api.opscode. com/organizations/dramafever", "run_list": [ "base::ami" ], "validation_key_path": "{{user `chef_validation`}}", "validation_client_name": "dramafever-validator", "node_name": "packer-ami" } @bridgetkromhout
  • 32. packer run$HOME/packer/packer build -var "account_id=$AWS_ACCOUNT_ID" -var "aws_access_key_id=$AWS_ACCESS_KEY_ID" -var "aws_secret_key=$AWS_SECRET_ACCESS_KEY" -var "x509_cert_path=$AWS_X509_CERT_PATH" -var "x509_key_path=$AWS_X509_KEY_PATH" -var "s3_bucket=bucketname" -var "ami_name=$AMI_NAME" -var "source_ami=$SOURCE_AMI" -var "chef_validation=$CHEF_VAL" -var "chef_client=$HOME/packer/client.rb" -only=amazon-instance $HOME/packer/prod.json @bridgetkromhout
  • 33. limiting packer IAM permissions "Action":[ "ec2:TerminateInstances", "ec2:StopInstances", "ec2:DeleteSnapshot", "ec2:DetachVolume", "ec2:DeleteVolume", "ec2:ModifyImageAttribute" ], "Effect":"Allow", "Resource":"*", "Condition":{ "StringEquals":{ "ec2:ResourceTag/name":"Packer Builder" } } @bridgetkromhout
  • 34. @bridgetkromhout uses fabric: tag for staging tag for prod out of ELB restart upstart back in ELB Ship it!
  • 36. S3 requires clock sync $ docker pull local-repo-alias.com:5000/mysql Pulling repository local-repo-alias.com:5000/mysql 2014/11/24 19:44:31 HTTP code: 500 $ boot2docker ssh sudo date --set "$(env TZ=UTC date '+%F %H:%M:%S')" @bridgetkromhout
  • 37. for persistent instances # remove stopped containers @daily docker rm `docker ps -aq` # remove images tagged "none" @daily docker rmi `sudo docker images | grep none | awk -F' +' '{print $3}'` @bridgetkromhout
  • 38. what even is flate?! Pulling repository local-repo-alias.com:5000/www 4dda2b433370: Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671 flate: corrupt input before offset 54393671 d497ad3926c8: Error downloading dependent layers 2014/12/07 02:34:54 Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671 @bridgetkromhout
  • 39. registry rewrite just landed! DOCKER_OPTS="--graph=/mnt/docker -- insecure-registry=local-repo-alias. com:5000 -e STORAGE_REDIRECT=true" until we upgrade to 2.0, our “flate” workaround: @bridgetkromhout
  • 40. isolation? -v /var/log/containers:/var/log <Input containers_in> Module im_file Recursive False File '/var/log/containers/*.log' Exec $FileName = file_name(); Exec $raw_event = $FileName + ' ' + $raw_event ; Exec $Message = $raw_event ; </Input> @bridgetkromhout
  • 41. (un-)containerize some of the things @bridgetkromhout
  • 42. container-building containers easier with statically linked binaries Go microservices Android APK @bridgetkromhout
  • 43. $ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE local-repo-alias.com:5000/mysql dev b0dc5885f767 2 days ago 905.9 MB local-repo-alias.com:5000/www dev 82cda604a4f1 2 days ago 1.092 GB local-repo-alias.com:5000/micro local bed20dc84ea1 4 days ago 10.08 MB google/golang 1.3 e3934c44b8e4 2 weeks ago 514.3 MB public_registry_image 0.6.9 11299d377a9e 6 months ago 454.5 MB scratch latest 511136ea3c5a 18 months ago 0 B $ ever-smaller images @bridgetkromhout