PDXPortland - Dockerize Django
•
1 like•1,186 views
This document summarizes Dockerizing a Django application. It describes the speakers' experiences moving from a non-Dockerized setup with many issues, like outdated images and long recovery times, to a Dockerized setup with improved scalability, documentation, and development workflows. Key aspects of the new setup include using Docker Compose to run multiple services, Docker Machine to provision environments, and Docker Swarm for production deployments across multiple instances.
Report
Share
PDXPortland - Dockerize Django
- 1. Dockerizing Django PDX Portland October 27th, 2016
- 2. Who are we? Michael Dougherty @maackle Senior Front-end Engineer CrowdStreet, Inc. Hannes Hapke @hanneshapke Software Engineer Talentpair, Inc.
- 3. Our pre-Docker World … • Single instance world (e.g. celery ran on the web server) • Outdated Amazon machine image • No documentation about the setup, consultancy work • Live data monkey patching • Scaling/Recovery time > 8 hours • Clunky QA setup > bottleneck
- 4. Our post-Docker World … • Single instance world (e.g. celery ran on the web server) • Outdated Amazon machine image • No documentation about the setup, consultancy work • Live data monkey patching • Scaling/Recovery time > 8 hours • Clunky QA setup > bottleneck • One service per container, redundancy of instances • One common base image shared across all instances • Explicit, declarative server setup • Immutable infrastructure (mostly) • Scaling/Recovery time ~ 20min • As many QA instances as we want
- 5. What is Docker? Docker • Compose • Machine • Swarm
- 6. Docker containers … … wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment. * Basically a virtual env for your operating system. * from https://www.docker.com/what-docker
- 8. Where is the difference? Images from https://www.docker.com/what-docker VM includes OS No OS needed
- 10. • Create a Dockerfile • Build the Docker image and push it to the docker registry Plain Docker FROM ubuntu:16.04 RUN apt-get update && apt-get upgrade -y RUN pip install Django COPY requirements.txt . WORKDIR /dev $ docker build -t your_project/your-whale . $ docker images $ docker tag {image_hash} your_project/your-whale:latest $ docker push
- 11. Plain Docker • Run a shell in a docker container -i start an interactive container -t creates “Pseudo interface” with stdin and stdout • Run the Django server in a container -d run container in detached mode -P maps all ports to the host machine $ docker run -i -t ubuntu /bin/bash $ docker run -d -P my-container python manage.py run server
- 12. What if we need multiple services? Docker • Compose • Machine • Swarm
- 13. Docker Compose Compose is a tool for orchestrating the building, running, and intercommunication of multi-container Docker applications.
- 14. How does it work? 1) Define a Dockerfile for every service 2) Define a Docker compose description of the environment 3) Use docker-compose build/up to start all services version: '2' services: web: build: . ports: - "5000:5000" volumes: - .:/code redis: image: redis
- 15. How can I easily provision a server with the containers? Docker • Compose • Machine • Swarm
- 16. Docker Machine … is a great tool which creates Docker hosts anywhere. Yes, anywhere. Locally, AWS EC2, Digital Ocean, MS Azure, you name it. No Ansible, Puppet, Chef, fabric, etc. required.
- 17. What if I need multiple instances with multiple services? Docker • Compose • Machine • Swarm
- 18. Docker Swarm
- 19. Dockerize for real …
- 20. If you start from scratch … • Docker documentation includes a great Django setup • Too much work? The Django Cookie Cutter template includes a great Docker setup Other projects: • django-docker on github
- 21. If you convert a project like us …
- 23. Normalize folders • Create folders for every service • docker-compose-{env}.yml go into the project root • Dockerfiles go into every service folder • startup.sh scripts go into the service folders • Keep your local folder structure similar to the folder structure within the container(s) - for sanity
- 24. Reorganized folders Project Root |-apps |-settings |-static |-templates |-manage.py |-fabfile.py |-urls.py requirements.txt Project Root |-django | |-apps | |- … | |-Dockerfile | |-startup-django.sh | -manage.py |-nginx |-webpack |-docker-compose.yml |-urls.py requirements.txt
- 25. Build a base image
- 26. Base Image • Create one (or more) base Dockerfile(s) with all common packages • Service containers can use this base image - this will increase build speed • If you store the base image(s) in a separate git repo, the docker registry will build them automatically for you
- 27. Base Image FROM ubuntu:16.04 RUN apt-get update && apt-get upgrade -y RUN apt-get install -y vim # Install some useful editor RUN apt-get install -y build-essential git software-properties-common RUN apt-get install -y python python-dev python-setuptools build-essential RUN apt-get install -y nodejs npm RUN npm install -g n # upgrading the npm version RUN n stable ...
- 28. Base Image Add image of the Docker registry
- 29. Set up Docker compose for the different environments
- 30. Docker Compose • For every environment, local, QA, staging, production, define a docker-compose-{env}.yml file • The files describe the environment stack • Each service within the docker-compose file can have it’s own Dockerfile
- 31. Docker Composeversion: '2' volumes: postgres_data_dev: {} redisdata: {} webpack_data: {} services: postgres: image: postgres:9.5 volumes: - postgres_data_dev:/var/lib/postgresql/data restart: always environment: - POSTGRES_USER=postgres_user - POSTGRES_DB=my_fancy_db - POSTGRES_PASSWORD= webpack: image: crowdstreet/crowdstreet-whale:latest command: npm run watch environment: - NODE_PATH=/node_modules volumes: - ./webpack/frontend-src:/frontend-src - ./django:/crowdstreet-src - webpack_data:/webpack_data/ ports: - "3000:3000" restart: always
- 32. Docker Composedjango: build: context: . dockerfile: ./django/Dockerfile-dev command: python /crowdstreet-src/manage.py runserver 0.0.0.0:8000 --settings=settings.dev depends_on: - postgres environment: - ENV=dev - DJANGO_SETTINGS_MODULE=settings volumes: - ./django:/crowdstreet-src - ./webpack/frontend-src:/frontend-src - webpack_data:/webpack_data/ ports: - "8000:8000" - "80:8000" links: - postgres - redis - webpack - memcached redis: restart: always image: redis:latest volumes: - redisdata:/data restart: always
- 33. Docker Compose • Build your service stack with • Start the container stack with • Access a single container with $ docker-compose -f docker-compose-{env}.yml build $ docker-compose -f docker-compose-{env}.yml up $ docker-compose -f docker-compose-{env}.yml run django bash $ docker-compose -f docker-compose-{env}.yml run container name command
- 35. Set up Docker machine and deploy to the world
- 38. Docker Machine • With will provision you an AWS instance • “Activate” the instance with • Afterwards, any docker-compose command will be executed on the active machine • Easy to start/stop/terminate machines $ docker-machine create --driver amazonec2 --amazonec2-region [e.g. us-west-2] --amazonec2-vpc-id [YOUR_VPC_ID vpc-xxxxxx] --amazonec2-instance-type [e.g. t2.small] [INSTANCE_NAME] $ docker-machine env [INSTANCE_NAME]
- 39. Lessons Learned
- 41. Or... how to cowboy code with Docker • Sometimes you just need to manually change something • Docker provides ways to get a shell inside a running instance and copy files back and forth • Your changes will of course be lost next time you spin up a new container
- 42. The Disciplined Way: The Cowboy Way: $ docker-compose run django bash $ docker exec -it {container_id} bash
- 43. How does QA work with Docker? • No QA bottleneck anymore • No database gridlock anymore • Each feature branch gets its own instance • Once feature is tested, instance gets terminated
- 44. How can I access the manage.py shell/migrate? • Access the bash of the django container with • Continue as usual with Some for migrations, make_migrations, etc. • Or run it from outside of the container stack with $ docker-compose -f docker-compose-{env}.yml run django bash # ./manage.py shell docker-compose -f … run django python manage.py migrate
- 45. Help, ipdb doesn’t work anymore … • Start the Django container with the service ports enabled • If no command is specified, then Docker will default to the command in the docker-compose.yml file $ docker-compose -f dev.yml run --service-ports django
- 46. How to run tests? • Start the Django container with your test command $ docker-compose -f … run django manage.py test
- 47. CI Testing is convenient • Setup for Circle CI machine: pre: - curl -sSL https://s3.amazonaws.com/circle-downloads/install-circleci-docker.sh | bash -s -- 1.10.0 services: - docker dependencies: override: - sudo pip install docker-compose - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS - docker-compose -f docker-compose-circle.yml build - npm install -g jshint test: pre: - sudo killall postgres # not sure why, but port 5432 is already taken up sometimes! - docker-compose -f docker-compose-circle.yml up -d postgres override: - jshint ~/your_project/django/static/js/your_project* - docker-compose -f docker-compose-circle.yml run django /your_project/manage.py test --verbosity=2
- 48. WTF, the files I copied into my container are missing?? • If a volume is mounted at the same directory where you copied other files, you will essentially overwrite those files
- 49. Sharing Docker Machine credentials • Docker machine is great, but there is no concept of sharing credentials • All credentials are simple text files, no magic • npm tool `machine-share` solved the problem • Let’s you export and import machine credentials
- 50. General Troubleshooting • Confirm that the correct docker-machine environment is active • Rebuild your container stack • Rebuild with the --pull and/or --no-cache options • Restart the docker daemon • Restart your docker machine with docker-machine restart [INSTANCE NAME] • Restart your docker machine VirtualBox VM • Remove and recreate your docker machine (essentially recreates your dev environment from scratch)
- 51. So, what does our setup look like now?
- 52. Dev Environment • You can use the same image as in your production builds • All services run at once, all output piped to a single log stream (which we saw earlier) • You can still have live reloading via Docker Volumes (but be careful!)
- 53. How does the deployment work now? • Create AWS instance with docker-machine • Activate the docker machine • Use docker-compose to build the stack • Use docker-compose up -d • Switch the load balancer
- 54. Summary of technologies • Learned about Docker • How to use docker to define images and containers • Learned about Docker-compose to define relationships between containers • Learned about Docker-machine to seamlessly work with containers on local/remote machines
- 55. Summary of benefits • Explicit, declarative server setup • Zero down time deployments • All dev services in one "window" and start with one command • Easy provisioning of multiple QA instances • Quick onboarding for new devs
- 56. Thank you!
- 57. Q&A