Part 2: Django on AWS Elastic Beanstalk with Docker

Dec. 20, 2015
django docker AWS

Docker Logo

Please Note:
This post appears to be over 6 months old.

Note: This is a continuation of a previous tutorial. If you are new to Beanstalk and/or Docker in general it's advisable to read part 1. Thanks.

So far you've learned some of the fundamentals of deploying a Django site to Amazon Web Services (AWS) using a Dockerfile and Beanstalk. However, we've found there are some disadvantages with this approach. For example, Beanstalk had to build the image during deployment which leads to longer deployment times and makes debugging complicated. The main advantages of using Docker is that you can build, run/test then ship it anywhere, so our last approach is somewhat lame. Let us fix that!

In this tutorial you're going to learn about the Docker command line by creating your image from a boilerplate template, pushing your image to a public repository (Docker Hub) and have Beanstalk run your pre-built image as a container on an EC2 instance.

Prerequisites 

This tutorial assumes you have a basic understanding of supervisor, Django/Python, git, running on a Linux/UNIX type system and have an AWS account. If you don't have an AWS account you can signup for their free tier.

Installing Docker on OS X

Docker relies heavily on kernel namespace isolation and control groups (cgroups). In Windows and OS X where operating system-level virtualisation, like containers in Linux is not natively supported you’ll need to run Linux on a VM. In the past, you had to install a number of different applications to get Docker running on a Mac. While this is still the case, thankfully it's all provided in one harmonious package called the Docker Toolbox. Docker Toolbox combines, Docker Client, Machine, Compose, Kitematic and VirtualBox in one installer. [UPDATE] Docker Beta Announcement 

Start by downloading the Docker Toolkit and running the installation file. Your Mac must be running OS X 10.8 “Mountain Lion” or newer. The installer launches an introductory dialog, follow each step as prompted. You shouldn't need to change any of the defaults:

Docker Toolbox install screenshot

Once installed, open the Docker Quick Start Terminal. It may take several seconds to start a new 'default' virtual machine. 

The Docker Test

All interactions with Docker are done through the command line and the Docker Quick Start Terminal is a nice and easy way to run Docker commands. Our first 'Docker' command will verify everything is running correctly and help us learn some more of the basics. Enter the following in the Docker Quick Start Terminal:

$ docker run -p 80:8002 -d glynjackson/django-beanstalk-tutorial:latest

Your output should look similar to below:

Unable to find image 'glynjackson/django-beanstalk-tutorial:latest' locally
latest: Pulling from glynjackson/django-beanstalk-tutorial
9703cd5af9a0: Pull complete
9f137edabe72: Pull complete
afb2d6a0d5f6: Pull complete
3012e0899802: Pull complete
5010cdc20db7: Pull complete
Digest: sha256:d2fbb04c0f096e5929db8fb3f34583af5fdfd6582b32140c56c8a82b772e1235
Status: Downloaded newer image for glynjackson/django-beanstalk-tutorial:latest
72949165f0177e41eb8094904204f2049be9985af3b2d05035062aee2a09992c

Note: You don't actually need to append the 'latest' tag, but it's a good habit!

The 'run' command takes a number of different arguments but, by default a new container based on the image is created and run inside a VM. The first time running this command can take a few minutes to complete.

So what just happened? The 'run' command looked for a local image called 'django-beanstalk-tutorial', but, as the image did not exist it proceeded to 'pull' it from the Docker registry. Behind the scenes, it runs a second command 'docker pull django-beanstalk-tutorial'. 

Next, check that the image exists on your filesystem:

$ docker images

If you don't see any image you may need to wait for the 'run' command to complete first, then try again. 

Let's look at the what we did in more detail. 

The '-d' argument used in the 'run' command make sure the container starts in 'detached' mode, in the background. To view your running containers, enter:

$ docker ps --no-trunc

There should be one running container listed. The no truncate argument is optional and just allows us to view the full names, without truncate the container ID uses a short UUID identifier.

The last argument used in the 'run' command was '-p' i.e. '-p 80:8002' which exposed the container on an internal port. This maps any required network ports inside our container to our host. This lets you view the web application! To find out what IP the VM is running on type:

$ docker-machine ip default 

Enter the outputted IP address in your browser and you should see the Django application running on port 80.

Django start page

Cool! Now we know some of the basics we can start to build our own image ready to deploy.

The Docker Hub

The 'pull' command you used actually searched the Docker Hub Registry, think of it like a package manager providing a centralised distribution resource you can search and install from. 

Information: Docker Hub provides free public and paid private repositories.

To continue you'll need a Docker Hub account. Sign-up for free at Docker Hub (if you haven’t already). The username you chose will become the public namespace for your repositories i.e. myname/repo so pick something sensible!

Docker Hub Screenshot

Docker Hub repositories can be setup in one of two ways. With Automated Builds, which allow you to configure GitHub or Bitbucket to trigger the build process, or pushing images at will from or your local Docker daemon. This tutorial uses the latter method, where we build locally and push directly.

Begin, by creating a public repository you can push to later. 

 Docker tutorial screenshot

For now make the repository public, enter a name for your image (recommended: django-beanstalk-tutorial) and a simple description for the repo. 

Next, head back into terminal and enter:

$ docker login

You will be promoted for your Docker Hub authentication credentials. These credentials will be stored in your '~/.docker/config.json'  authentication file in your home directory (on OS X).

Download boilerplate Docker template

Right, onto the good stuff, let's build our very own image and push to the repository you created! Just like in the previous tutorial we're going to be using a boilerplate template from GitHub, only this template has been customised significantly from our first example, we'll touch on each of these differences as we progress.

As you already have the Quick Start Terminal opened start by downloading the template and initiating git: 

$ django-admin.py startproject --template=https://github.com/glynjackson/django-docker-template2/zipball/master mysite
$ cd mysite
$ git init

If preferred, you can download the boilerplate repository directly here.

Important: The rest of this tutorial is based on the boilerplate template.

Amongst other files the template should contain a default Django project, folder called DockerDockerrun.aws.json and a Makefile. Let's look at each of these in detail.

The Docker folder

You will have noticed the location of the Dockerfile has moved. This is best practice, but more importantly we moved the file to prevent Beanstalk automatically building the Dockerfile when the project is deploy. This is because Beanstalk by default looks for a Dockerfile in the project root and we don't want that!

The Dockerfile has changed very little apart from the ADD command which now adds the contents of a '.rar' archive.

Output of Dockerfile for reference:

# Base python 3.4 build, inspired by https://github.com/Pakebo/eb-docker-django-simple
# Python 3.4 | Django
FROM python:3.4
MAINTAINER Glyn Jackson (me@glynjackson.org)
####################################################
# Environment variables
####################################################
# Get noninteractive frontend for Debian to avoid some problems:
# debconf: unable to initialize frontend: Dialog
ENV DEBIAN_FRONTEND noninteractive
####################################################
# OS Updates and Python packages
####################################################
RUN apt-get update \
 && apt-get upgrade -y \
 && apt-get install -y
RUN apt-get install -y apt-utils
# Libs required for geospatial libraries on Debian...
RUN apt-get -y install binutils libproj-dev gdal-bin
####################################################
# A Few pip installs not commonly in requirements.txt
####################################################
RUN apt-get install -y nano wget
# build dependencies for postgres and image bindings
RUN apt-get install -y python-imaging python-psycopg2
####################################################
# setup startup script for gunicord WSGI service
####################################################
RUN groupadd webapps
RUN useradd webapp -G webapps
RUN mkdir -p /var/log/webapp/ && chown -R webapp /var/log/webapp/ && chmod -R u+rX /var/log/webapp/
RUN mkdir -p /var/run/webapp/ && chown -R webapp /var/run/webapp/ && chmod -R u+rX /var/run/webapp/
####################################################
# Install and configure supervisord
####################################################
RUN apt-get install -y supervisor
RUN mkdir -p /var/log/supervisor
ADD ./supervisor_conf.d/webapp.conf /etc/supervisor/conf.d/webapp.conf
#####################################################
# Install dependencies and run scripts.
#####################################################
ADD mysite.tar /var/projects/mysite
WORKDIR /var/projects/mysite
RUN pip install -r requirements.txt
#####################################################
# Run start.sh script when the container starts.
# Note: If you run migrations etc outside CMD, envs won't be available!
#####################################################
CMD ["sh", "./docker/container_start.sh"]
# Expose listen ports
EXPOSE 8002

Other files

  • container_start.sh - Gets executed by the CMD instruction when a new container is created. This script runs any Django migrations and starts the supervisor process.
  • supervisor_conf.d/webapp.conf - Gets added to the supervisor folder.

Building the Docker image

We are now ready to build our image and push it to Docker Hub. Commit the project locally using git, then in the Quick Start Terminal enter the following commands in the root of the project replacing [NAME] with the namespace you created over at Docker Hub:

$ git archive -o docker/mysite.tar HEAD
$ docker build -t [NAME]/django-beanstalk-tutorial:latest --rm docker
$ rm -f docker/mysite.tar 

The `ADD mysite.tar` command in the Dockerfile should now make more sense, `git archive` creates a file in the docker folder and files committed are added to the image project directory.

The 'build' command uses your Dockerfile to create a local image. The '-t' argument refers to the tag which must match your namespace and repo name setup on Docker Hub in order to push it later. The last argument 'docker' simply refers to the folder where the Dockerfile is located. The last command `rm -f docker/mysite.tar` is just a simple clean-up command which is optional.

Once the build was successful you should see a "Successfully built' message followed by the image ID. View the image again using 'docker images' where you should see two images, the test image from earlier and the new image you just created. 

We can now test your image locally to see if it works:

$ docker run --env-file docker/env.conf -p 80:8002 -d [NAME]/django-beanstalk-tutorial:latest

This time, you have started a new container based on your image running on the VM. View the IP of the VM and visit it in your browser.

$ docker-machine ip default 

Again you should see the default Django message "It worked" page!

Pushing to Docker Hub

Our final task before deploying to AWS it to push your release to Docker Hub. If everything has been setup correct this is simple:

docker push [NAME]/django-beanstalk-tutorial:latest

Once completed you should see your the image on Docker Hub. The screenshot below shows glynjackson as the namespace yours should be different.

Docker Hub Tag Screenshot

Deploying to Beanstalk

In the project root, you will find a file called 'Dockerfile.aws.json':

{
 "AWSEBDockerrunVersion": "1",
 "Image": {
 "Name": "[NAME]/django-boilerplate:latest",
 "Update": "true"
 },
 "Ports": [
 {
 "ContainerPort": "8002"
 }
 ]
}

This file describes how to deploy a container in AWS EB however, it's not part of the Docker specification. Replace the `Name' argument with your own image i.e. '[NAME]/django-boilerplate:latest'. This tells Beanstalk to pull your pre-build image directly from Docker Hub. The 'ContainerPort' argument works like our run command and tells Beanstalk passthrough webserver (Nginx) to map port 80 to the exposed port 8002 on the container.

If you already have your environment initialised from the previous tutorial in a few seconds your application will be deploy and running by simply entering:

$ eb deploy
Creating application version archive "aa3c".
Uploading tutorials/aa3c.zip to S3. This may take a while.
Upload Complete.
INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
INFO: Successfully pulled glynjackson/django-beanstalk-tutorial:latest
INFO: Successfully built aws_beanstalk/staging-app
INFO: Docker container e831a04481c9 is running aws_beanstalk/current-app.
INFO: New application version was deployed to running EC2 instances.
INFO: Environment update completed successfully.

If your want to start a new environment you will need to do a `eb init` and `eb create` command. See part 1 for more details on this.

To see the deployed website enter:

$ eb open

Cool, right?

Django start page

MakeFile

The template includes a simple Makefile which comes in very helpful. Change the 'NAME' variable at the top of the MakeFile to match your namespace and repo and it will provide shortcuts to some of the commands we just learned. 

NAME=YOURNAMESPACE/django-beanstalk-tutorial
VERSION=`git describe --abbrev=0 --tags`
BRANCH=`git rev-parse --abbrev-ref HEAD`
CONTAINER_IP=$(shell echo $(docker-machine ip default))
ifeq ($(shell echo $(BRANCH)),master)
TAG='latest'
else
TAG=$(VERSION)
endif
NO_COLOR=\033[0m
OK_COLOR=\033[32;01m
ERROR_COLOR=\033[31;01m
WARN_COLOR=\033[33;01m
OK_STRING=$(OK_COLOR)[OK]$(NO_COLOR)
IP_STRING=$(OK_COLOR)$$(docker-machine ip default):80$(NO_COLOR)
ERROR_STRING=$(ERROR_COLOR)[ERRORS]$(NO_COLOR)
WARN_STRING=$(WARN_COLOR)[WARNINGS]$(NO_COLOR)
AWK_CMD = awk '{ printf "%-30s %-10s\n",$$1, $$2; }'
PRINT_ERROR = printf "$@ $(ERROR_STRING)\n" | $(AWK_CMD) && printf "$(CMD)\n$$LOG\n" && false
PRINT_WARNING = printf "$@ $(WARN_STRING)\n" | $(AWK_CMD) && printf "$(CMD)\n$$LOG\n"
PRINT_OK = printf "$@ $(OK_STRING)\n" | $(AWK_CMD)
PRINT_IP = printf "$@ $(IP_STRING)\n" | $(AWK_CMD)
BUILD_CMD = LOG=$$($(CMD) 2>&1) ; if [ $$? -eq 1 ]; then $(PRINT_ERROR); elif [ "$$LOG" != "" ] ; then $(PRINT_WARNING); else $(PRINT_OK); fi;
help:
@echo "*******************************************************"
@echo "Please use \`make ' where is one of:"
@echo "*******************************************************"
@echo " container to build and start a local container."
@echo " app to prepare and build image."
@echo " pull to pull from Docker Hub."
@echo " run to run a container already pulled."
@echo " prepare to create local git archive."
@echo " clean to remove git archive and containers."
@echo " push to push to Docker Hub."
@echo " ip to display IP:PORT of default container."
@echo " delete_images to deletes all images."
@echo "*******************************************************"
# Pulls a build of the app and starts a local container.
container: pull run ip
@$(PRINT_OK)
# Build and pushes the app to Docker Hub.
app: prepare build clean
@$(PRINT_OK)
pull:
docker pull $(NAME):$(TAG)
@$(PRINT_OK)
run:
docker run --env-file docker/env.conf -p 80:8002 -d $(NAME):$(TAG)
@$(PRINT_OK)
prepare:
git archive -o docker/mysite.tar HEAD
@$(PRINT_OK)
build:
docker build -t $(NAME):$(TAG) --rm docker
@$(PRINT_OK)
clean:
rm -f docker/mysite.tar
docker rm -f $$(docker ps -a -q)
@$(PRINT_OK)
push:
docker push $(NAME):$(TAG)
.PHONY: ip
ip:
 @$(PRINT_IP)
delete_images:
docker rmi -f $$(docker images -q)
@$(PRINT_OK)

For example, to build a new release you can just use:

$ make app

Or to push:

$ make push

You should experiment as much as possible and find what works best for your situation.

Conclusion

We've only really touched the proverbial tip of the iceberg in what Docker and AWS can do together. Next, we will be looking at Multicontainer Docker Environments and deployment strategies

That's all Folks!

Thanks for reading. Let's keep in Touch:
Follow me on GitHub or Twitter @glynjackson


Glyn Jackson is a Python Nerd, consultant and developer.


Find out more