CI/CD: Automate server builds with Packer and CodeBuild

Ever wondered how to guarantee consistency between multiple environments and servers? Here is how.

At Tigerspike, we thrive on innovation.
#our_inquisitive_nature_brings_insight_and_innovation

One of the tricks our teams have started using a couple of years ago now is what is commonly known as the Golden Image.

What is a Golden Image?

The concept is simple. Launch a server, patch and configure it, deploy your application on it, then take a snapshot or image of it. Once tested and approved, this image becomes the Golden Image which you can use to launch servers, one or hundreds of them if you want, in multiple environments, and they’ll always be the same. Do the same for each release, and you will be able to deploy and rollback to whichever image you prefer.
Reliability and DR ticked!

So, how do you create Golden Images?

There are many ways to actually achieve this, but my favourite is using Hashi Corp’s Packer.

Packer is an open source build tool. It can create images from JSON templates. In a template, you’d define the base OS image and define the deployment scripts and configuration files. yes, that simple!

Packer can be installed on the fly, e.g. on managed build services such as AWS CodeBuild, or permanently, e.g. on on-prem build servers such as Jenkins. Check Packer’s download and install guide here.

So, let’s see an example:

Let’s assume you have a website with a single index.html page. Every time you push an update to the page, you want to build a new image of your web server with Ubuntu 18.04 LTS image and the latest version of Apache.

In summary, besides your website file(s), you’ll need:
1. a Packer server template
2. a CodeBuild build script

Here is a simple Packer template, let’s call it webserver.json:

{
    "builders": [
        {
            "type": "amazon-ebs",
            "region": "ap-southeast-2",
            "source_ami": "ami-02a599eb01e3b3c5b",
            "instance_type": "t3.large",
            "communicator": "ssh",
            "ssh_username": "ubuntu",
            "ami_name": "Medium_Example_Linux_{{timestamp}}"
        }
    ],
    "provisioners": [
        {
            "type": "shell",
            "inline": ["sudo apt update",
            "sudo apt install -y apache2",
            "sudo rm -r /var/www/html/*",
            "sudo echo 'ServerName example.com' >> /etc/apache2/sites-available/000-default.conf"]
        },
        {
            "type": "file",
            "source": "index.html",
            "destination": "/var/www/html/index.html"
        }
    ]
}

In the template above, we first defined the Builder then the Provisioner.

For the Builder, we defined Amazon-EBS type in Sydney region with t3.large instance type. Then, we specified the source AMI, which is the latest Ubuntu 18.04 LTS image provided by Canonical Ltd, and specified how to connect to it and how to name the created image afterwards.

Note, if you wish to always use the latest base AMI, you can use source_ami_filter directive instead of source_ami. Also, if you wish to give your AMIs cleaner names, use isotime [FORMAT]instead of timestamp.

For the Provisioner, first, we execute inline shell commands, then copy a file. The shell Provisioner installs latest Apache2, removes the default site files then adds a server name to the site’s configurations file. The file Provisioner simply copies a file, provided with the artifacts in our scenario, to a destination on the Builder.

The template above doesn’t include Post-Processors but check it out if you want to run processes after the image is created, e.g. produce Manifest for automated deployments, push to Docker registry, etc.

And, here is a simple CodeBuild script, by default it’s buildspec.yml:

version: 0.2
phases:
  install:
    commands:
      - curl -qL -o packer.zip https://releases.hashicorp.com/packer/1.5.5/packer_1.5.5_linux_amd64.zip
      - unzip packer.zip
  build:
    commands:
      - ./packer build webserver.json

The above build script simply instructs CodeBuild to download Packer (v. 1.5.5 is the latest at the time of writing) and unzip Packer package then run packer build command using the template we provided earlier.

You now have an index.html page, a Packer template, and a build script for CodeBuild. Next, push these files to a code repository (or upload to an S3 bucket) and configure a CodeBuild project to read from this source and run a build. Voila, when the build is done, you will see a new AMI created by Packer.

If you’re not familiar with setting up CodeBuild, check this simple tutorial. Also, make sure CodeBuild has enough IAM permissions for EC2.

That’s it folks!

The Golden Image is very useful for launching any number of servers, 1 or 100s. We used this to deploy a huge stack of 60 servers for a big client once as well as stacks of 2 servers only for much smaller clients.

Tip: think of the power you would have using Packer and Auto Scaling Groups were you could define a standard Launch Configuration and let the Auto Scaling Group do the work for you healing instances and scaling up and down when needed.

Tip2: you can extend this magic by automating your deployments. For example, after creating an AMI, CodeBuild can update your CloudFormation Stacks or Auto Scaling Groups and replace your running instances with new ones.