Docker Swarm With AWS VPC

March 7th, 2016Dan Yoder

Running Docker Swarm in an AWS VPC isn’t quite as easy as you might think it would be. But we’ve got you covered. Photo credit: Wikipedia.

Docker offers a feature rich platform for Linux containers. Amazon Web Services offers a feature rich platform for cloud computing. Naturally, we want to use them together. In particular, we want to use Docker Swarm running within an AWS VPC. Unfortunately, this isn’t quite as easy as you might think. Here’s our solution.

Approach

We’re going to create an AWS VPC with no compute instances. We’ll use Docker Machine to create the Swarm machines.

We don’t use AWS ECS. Swarm and ECS are not integrated yet. Specifically, each has their own approach to allocating cluster resources. Swarm’s approach is both simpler and more powerful. We do sacrifice the integration of ECS with AWS, but its worth it. And, hopefully, ECS and Swarm will interoperate in the near future.

We’re also using Docker Hub. In real life, you’ll probably want to use a private registry via AWS’ EC2 Container Registry (ECR).

Create Your VPC

We’re going to create an Amazon VPC using a CloudFormation template. This template doesn’t include anything except what we need to use the VPC. That’s okay because we can update the stack created by the template later. It’s enough to get us started. And by creating a stack instead of doing each step piecemeal, it’s easy to remove everything later.

Here’s our template, formatted as YAML for readability.

AWSTemplateFormatVersion: '2010-09-09'
Description: AWS CloudFormation Template for use with p42.
Parameters:
  SSHLocation:
    Description: Lockdown SSH access to the bastion host (default can be accessed from anywhere)
    Type: String
    MinLength: '9'
    MaxLength: '18'
    Default: 0.0.0.0/0
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: must be a valid CIDR range of the form x.x.x.x/x.
Mappings:
  SubnetConfig:
    VPC:
      CIDR: 10.0.0.0/16
    Public:
      CIDR: 10.0.0.0/24
Resources:
  VPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'
      CidrBlock:
        'Fn::FindInMap':
          - SubnetConfig
          - VPC
          - CIDR
      Tags:
        - Key: Application
          Value:
            Ref: 'AWS::StackName'
        - Key: Network
          Value: Public
  PublicSubnet:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId:
        Ref: VPC
      CidrBlock:
        'Fn::FindInMap':
          - SubnetConfig
          - Public
          - CIDR
      Tags:
        - Key: Application
          Value:
            Ref: 'AWS::StackName'
        - Key: Network
          Value: Public
  InternetGateway:
    Type: 'AWS::EC2::InternetGateway'
    Properties:
      Tags:
        - Key: Application
          Value:
            Ref: 'AWS::StackName'
        - Key: Network
          Value: Public
  GatewayToInternet:
    Type: 'AWS::EC2::VPCGatewayAttachment'
    Properties:
      VpcId:
        Ref: VPC
      InternetGatewayId:
        Ref: InternetGateway
  PublicRouteTable:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId:
        Ref: VPC
      Tags:
        - Key: Application
          Value:
            Ref: 'AWS::StackName'
        - Key: Network
          Value: Public
  PublicRoute:
    Type: 'AWS::EC2::Route'
    DependsOn: GatewayToInternet
    Properties:
      RouteTableId:
        Ref: PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: InternetGateway
  PublicSubnetRouteTableAssociation:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId:
        Ref: PublicSubnet
      RouteTableId:
        Ref: PublicRouteTable
  PublicNetworkAcl:
    Type: 'AWS::EC2::NetworkAcl'
    Properties:
      VpcId:
        Ref: VPC
      Tags:
        - Key: Application
          Value:
            Ref: 'AWS::StackName'
        - Key: Network
          Value: Public
  InboundHTTPPublicNetworkAclEntry:
    Type: 'AWS::EC2::NetworkAclEntry'
    Properties:
      NetworkAclId:
        Ref: PublicNetworkAcl
      RuleNumber: '100'
      Protocol: '6'
      RuleAction: allow
      Egress: 'false'
      CidrBlock: 0.0.0.0/0
      PortRange:
        From: '80'
        To: '80'
  InboundHTTPSPublicNetworkAclEntry:
    Type: 'AWS::EC2::NetworkAclEntry'
    Properties:
      NetworkAclId:
        Ref: PublicNetworkAcl
      RuleNumber: '101'
      Protocol: '6'
      RuleAction: allow
      Egress: 'false'
      CidrBlock: 0.0.0.0/0
      PortRange:
        From: '443'
        To: '443'
  InboundSSHPublicNetworkAclEntry:
    Type: 'AWS::EC2::NetworkAclEntry'
    Properties:
      NetworkAclId:
        Ref: PublicNetworkAcl
      RuleNumber: '102'
      Protocol: '6'
      RuleAction: allow
      Egress: 'false'
      CidrBlock:
        Ref: SSHLocation
      PortRange:
        From: '22'
        To: '22'
  InboundEphemeralPublicNetworkAclEntry:
    Type: 'AWS::EC2::NetworkAclEntry'
    Properties:
      NetworkAclId:
        Ref: PublicNetworkAcl
      RuleNumber: '103'
      Protocol: '6'
      RuleAction: allow
      Egress: 'false'
      CidrBlock: 0.0.0.0/0
      PortRange:
        From: '1024'
        To: '65535'
  OutboundPublicNetworkAclEntry:
    Type: 'AWS::EC2::NetworkAclEntry'
    Properties:
      NetworkAclId:
        Ref: PublicNetworkAcl
      RuleNumber: '100'
      Protocol: '6'
      RuleAction: allow
      Egress: 'true'
      CidrBlock: 0.0.0.0/0
      PortRange:
        From: '0'
        To: '65535'
  PublicSubnetNetworkAclAssociation:
    Type: 'AWS::EC2::SubnetNetworkAclAssociation'
    Properties:
      SubnetId:
        Ref: PublicSubnet
      NetworkAclId:
        Ref: PublicNetworkAcl
Outputs:
  VPCId:
    Description: VPCId of the newly created VPC
    Value:
      Ref: VPC
  PublicSubnet:
    Description: SubnetId of the public subnet
    Value:
      Ref: PublicSubnet
  AvailabilityZone:
    Description: Availability Zone of the public subnet
    Value:
      'Fn::GetAtt':
        - PublicSubnet
        - AvailabilityZone

Creating the VPC from this template is easy. We use our YAML CLI to dump it to JSON and then use the AWS CLI to create the stack.

yaml json write ./cf-template.yaml > ./cf-template.json
aws cloudformation create-stack \
  --stack-name elegant-cat \
  --template-body file:///$(pwd)/cf-template.json \
  > /dev/null

We have to wait for the stack to be created. We check every five seconds using a while loop. We use a JSON CLI to inspect the status of our stack.

while true; do
  sleep 5
  description=$(aws cloudformation describe-stacks --stack-name elegant-cat)
  status=$(json Stacks[0].StackStatus <<<"${description}")
  if [ "$status" == "CREATE_COMPLETE" ]; then
    break
  fi
done

Once that returns, our stack is ready.

Install Docker Machine

Docker Machine simplifies running a Swarm. But, as of this writing, for our approach to work, you need the latest release candidate.

curl -L https://github.com/docker/machine/releases/download/v0.6.0-rc4/docker-machine-`uname -s`-`uname -m` > /usr/local/bin/docker-machine && \\
chmod +x /usr/local/bin/docker-machine

Create A Docker Host

We want to ultimately create a Docker Swarm, but we need a non-Swarm host to bootstrap the Swarm. This also makes it possible for us to bring the Swarm down and then reuse the VPC to create a new one.

We use Docker Machine to do the heavy lifting. We need to grab a bunch of parameters associated with our stack so we can pass them into Docker Machine.

vpc=$(json Stacks[0].Outputs[0].OutputValue <<<"${description}")
subnet=$(json Stacks[0].Outputs[1].OutputValue <<<"${description}")
az=$(json Stacks[0].Outputs[2].OutputValue <<<"${description}")
region="${az%?}"
zone="${az: -1}"

Now we can go ahead and create a Docker machine in our VPC.

docker-machine create elegant-cat \
  --driver amazonec2 \
  --amazonec2-region ${region} \
  --amazonec2-vpc-id ${vpc} \
  --amazonec2-subnet-id ${subnet} \
  --amazonec2-zone ${zone}

Obtain A Swarm Token

We need a token so that we can do the Swarm magic. For that, we just use Docker. We want to make sure we’re pointing at the host we just created. Then we run docker run swarm create to get the token.

eval "$(docker-machine env elegant-cat)"
swarm=$(docker run swarm create)

Create A Swarm Master

We create another Docker host with Docker Machine, only this time, we make it a Swarm master.

docker-machine create \
  --driver amazonec2 \
  --amazonec2-region ${region} \
  --amazonec2-vpc-id ${vpc} \
  --amazonec2-subnet-id ${subnet} \
  --amazonec2-zone ${zone} \
  --swarm --swarm-master \
  --swarm-discovery token://${swarm} \
  elegant-cat-00

Once this returns, we have our vanilla Docker host and our Swarm master.

docker-machine ls

Which will return something like this.

NAME               ACTIVE   DRIVER      STATE     URL                         SWARM                       DOCKER    ERRORS
elegant-cat        -        amazonec2   Running   tcp://54.183.152.21:2376                                v1.10.2
elegant-cat-00     -        amazonec2   Running   tcp://54.67.11.1:2376        elegant-cat-00 (master)    v1.10.2

Add Swarm Nodes

Adding nodes to a Swarm is similar to creating the master, we just don’t pass the --swarm-master flag.

docker-machine create \
  --driver amazonec2 \
  --amazonec2-region ${region} \
  --amazonec2-vpc-id ${vpc} \
  --amazonec2-subnet-id ${subnet} \
  --amazonec2-zone ${zone} \
  --swarm --swarm-discovery ${swarm} \
  elegant-cat-01

Point Your Environment To The Swarm Master

You’re now ready to start deploying containers to the Swarm master. All that remains is to make sure your environment is set up.

eval $(docker-machine env --swarm elegant-cat-00)

Let’s fire up Nginx just to check it out.

docker run -d -p 80:80 nginx

If you run docker ps after that completes, you should see Nginx running.

ONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                                                             NAMES
6c0115c33e2a        nginx                             "nginx -g 'daemon off"   31 seconds ago      Up 2 seconds        54.183.208.41:80->80/tcp, 443/tcp                                 elegant-cat-01/distracted_roentgen

If you curl the given IP, you’ll see the Nginx greeting.

Where To Go From Here

At this point, you’re ready to run applications on top of a Swarm cluster running within an AWS VPC. You can take advantage of all the powerful features provided by AWS, including ECR, Elastic Load Balancers, Route53 DNS, Key Management System, and so on.

When you’re done with the Swarm cluster, cleaning up is easy.

Stop/remove the containers
Stop/remove the Swarm machines and the vanilla Docker host
Use the AWS CLI to delete the stack by name

Epilogue: p42

We’ve written a simple CLI to help automate this process called p42. p42 is under heavy development, but we invite you to kick the tires. and let us know what you think.

Using p42, this entire process is reduced to running two commands.

$ p42 cluster create
Creating VPC [red-ghost]...
[…lots of output…]
$ p42 cluster add red-ghost -n 3
[…more output…]