Building Docker Images using Docker Compose and Gitlab CI/CD

August 27, 2019
Shipping Container pattern in Montreal Canada
Share post

In this post, I outline how to use Docker, Docker Compose, GitLab CI/CD and the GitLab Container Registry to build, tag and push docker images. As an example, I'll guide you through setting up a GitLab CI/CD pipeline that builds a simple docker image to serve a static website.

Check out the
example Repo GitLab Repo
gitlab icon

Intro

Prerequisites

Goals

  • Understand how you can build Docker images using GitLab CI/CD
  • Build a Docker image in GitLab CI/CD using the Docker CLI
  • Build a Docker image in GitLab CI/CD using Docker Compose
  • Push the built Docker image to the GitLab Container Registry
  • Build and push multiple images with Docker Compose and GitLab CI/CD
  • Test the built Docker Image in Gitlab CI/CD

Why build with Docker Compose

Docker-compose provides a higher level of abstraction than the Docker-CLI. It not only allows us to consume already-built images, but it's also a great tool for defining how to build new images in a clean and declarative way.

Docker-compose enables us to:

  • Define image builds in a declarative way using the docker-compose.yml syntax.
  • Declare tags in such a way that local builds and pipeline builds run the same sequence of commands.
  • Keep the image tags definition in the docker-compose.yml, not in our heads or the shell history.
  • Build multiple images with a single docker-compose build command.
  • Define build-time dependencies between images to ensure images are built in order.
  • Push multiple images with a single docker-compose push command.

Vanilla Docker in GitLab CI/CD

Before going into how to setup Docker Compose in a pipeline, let's take a quick look at how to use plain Docker-CLI in GitLab CI/CD.

To be able to build Docker images, we need the Docker CLI and a Docker Service/Dameon. Therefore, to build in a GitLab CI/CID pipeline, we'll need two Docker images:

  1. docker/compose:1.21.2.

    We use this as the Gitlab pipeline job's image. This is how we enable the pipeline to issue both Docker commands and docker-compose commands.

  2. docker:dind

    We define a Docker daemon using the services keyword, it receives the commands issued by the script and performs the actual job.

Docker in a Gitlab CI/CD Pipeline

diagram showing high level architecture

💻 Example: Basic Docker setup

View these changes in the example Gitlab CI/CD repo
gitlab icon

Start by creating a minimal .gitlab-ci.yml in a new git repo.

build:
  stage: build 
  image: 
    name: docker/compose:1.21.2
    entrypoint: ["/bin/sh", "-c"]
  variables:
    DOCKER_HOST: tcp://docker:2375
  services:
    - docker:dind
  script:
    - docker version # verify docker cli is there. Also prints server info 
    - docker-compose version # verify the docker-compose cli is there

Commit .gitlab-ci.yml and push your repo to gitlab.

When the pipeline runs, you should see output like this:

$ docker version
Client:
 Version:	17.12.1-ce
 API version:	1.35
 Go version:	go1.9.4
 Git commit:	7390fc6
 Built:	Tue Feb 27 22:13:43 2018
 OS/Arch:	linux/amd64

Server: Docker Engine - Community
 Engine:
  Version:	18.09.7
  API version:	1.39 (minimum version 1.12)
  Go version:	go1.10.8
  Git commit:	2d0083d
  Built:	Thu Jun 27 18:01:17 2019
  OS/Arch:	linux/amd64
  Experimental:	false

💻 Example: Building a Docker image

Let's create a simple website using:

  • Alpine as a minimal OS
  • Node JS to interpret Javascript
  • Serve to serve assets/resources/html using nodejs

View these changes in the example Gitlab CI/CD repo
gitlab icon

Create the following file structure.

.
├── .gitlab-ci.yml #<<< from previous step
├── Dockerfile
├── public
│   └── index.html

Create a simple html to serve

public/index.html

<html>
  <body>
    <h1>Hello</h1>
  </body>
</html>

Create a Dockerfile to serve our content

# minimal linux distribution with official node  image
FROM node:11-alpine 

# Install the 'serve' npm package
RUN npm install -g serve

# Copy 'public' to 'public'
COPY public public

# When the container start, serve the public/ dir
ENTRYPOINT [ "serve", "-n", "public/" ]

Try the build: On your local machine run

docker build . -t "hello:local"

Notice how, when using the Docker CLI, we'll need to tag the image we built using the -t option. I'll explain a bit more about tagging ahead.

Update .gitlab-ci.yml to build the Dockerfile

build:
  stage: build 
  image: 
    name: docker/compose:1.21.2
    entrypoint: ["/bin/sh", "-c"]
  variables:
    DOCKER_HOST: tcp://docker:2375
  services:
    - docker:dind
  script:
    # build the image and tag it to use the GitLab CI docker registry
    - docker build . -t "${CI_REGISTRY_IMAGE}/hello:${CI_COMMIT_SHORT_SHA}"
    - docker images # list the images the docker services knows about

Commit and push to gitlab, when the pipeline runs you'll get output like this:

$ docker build . -t "${CI_REGISTRY_IMAGE}/hello:${CI_COMMIT_SHORT_SHA}"
Sending build context to Docker daemon  81.92kB

Step 1/4 : FROM node:11-alpine
11-alpine: Pulling from library/node
Digest: sha256:8bb56bab197299c8ff820f1a55462890caf08f57ffe3b91f5fa6945a4d505932
Status: Downloaded newer image for node:11-alpine
 ---> f18da2f58c3d
Step 2/4 : RUN npm install -g serve
 ---> Running in 28b55c2963ef
/usr/local/bin/serve -> /usr/local/lib/node_modules/serve/bin/serve.js
+ serve@11.0.2
added 78 packages from 39 contributors in 5.562s
Removing intermediate container 28b55c2963ef
 ---> dc915ce339a5
Step 3/4 : COPY public public
 ---> 572feffd1db7
Step 4/4 : ENTRYPOINT [ "serve", "-n", "public/" ]
 ---> Running in 97ce87ceac8f
Removing intermediate container 97ce87ceac8f
 ---> 4bfd4cee6f50
Successfully built 4bfd4cee6f50
Successfully tagged registry.gitlab.com/portenez/gitlab-ci-and-docker/hello:132138da

About Docker Image tags

Tagging with pushing in mind

Once you've built an image, you'll likely want to push it to a Docker registry. To be able to push to a Docker registry, the structure of the tag should be:

{repositoryHostName}/{repository}/{image}:{version}

Notice how in our example we define the build command as:

docker build . -t "${CI_REGISTRY_IMAGE}/hello:${CI_COMMIT_SHORT_SHA}"- 

Using the pre-defined GitLab CI/CD environment variables we define the tag as:

| image name part                   | defined as             |
| --------------------------------- | ---------------------- |
| {repositoryHostName}/{repository} | ${CI_REGISTRY_IMAGE}}  |
| {image}                           | hello                  |
| {version}                         | ${CI_COMMIT_SHORT_SHA} |

In the end the image name will look something like this:

registry.gitlab.com/vgarcia.dev/gitlab-ci-and-docker/hello:a5530e6b

For the examples in this post I use the GitLab Container Registry, but you could push to other registries such as: Docker Hub, Amazon Elastic Container Registry or Artifactory.

Check the docker tag documentation for more information about tagging.

Docker Compose in Gitlab CI/CD

Now, let's take advantage of the benefits of using Docker-compose.

Better tagging with Docker Compose

Referring back to our example, notice how we issued passed different parameters to the docker build command.

docker build command in Pipeline

docker build . -t "${CI_REGISTRY_IMAGE}/hello:${CI_COMMIT_SHORT_SHA}"

docker build command in local dev machine

docker build . -t hello:local 

This means, that while iterating on an image, you'll have to remain constantly aware of the difference between the docker build command you need to issue locally and in the pipeline. Moreover, If you decide to change the tags, you'll have to update your local commands, and also the pipeline scripts.

We can use Docker-Compose, to be able to run the same command in both the pipeline and locally:

docker-compose build

Better pushing with Docker Compose

Without Docker compose we would have to type the following command to push an image.

docker push $A_TAG_I_HAVE_TO_REMEMBER

With Docker-compose we can simply call docker-compose push over and over again, as the image tag is defined in the docker-compose.yml.

# the compose file knows the details of the tags
docker-compose push 

As with the docker build command. If you decide to change your tags, you'll have to update both the local commands you issue for testing, and also the script you use in the pipeline.

Environment Variables and defaults

Docker-Compose allows you to reference environment variables in the docker-compose.yml file. Docker-Compose will substitute the variables with the values when any of the docker-compose commands are run.

By using the syntax ${ENV_VAR-default} we can enable docker-compose build to work seamlessly both locally and on the pipeline.

In the docker-compose.yml file, we will define the image as:

${CI_REGISTRY_IMAGE-local}/hello:${CI_COMMIT_SHORT_SHA-local}

The values to the variables will be resolved differently depending on whether the docker-compose command is run locally (on a dev's computer), or on GitLab CI/CD. However, we'll be able to use the same docker-compose build and docker-compose push commands in both places.

Resolved Values On Gitlab CI/CD ☁️

The pipeline will populate variables with values like these:

CI_REGISTRY_IMAGE=registry.gitlab.com/yourRepo 
CI_COMMIT_SHORT_SHA=14f6fe6 

docker-compose.yml will end up using the following tag

image: registry.gitlab.com/yourRepo/hello:14f6fe61

The shell script to use (same as locally 🎉):

docker-compose build

Resolved values locally on the devs computer 💻

The GitLab environment variables will not be present, and docker-compose will see something like this

# You don't need to dfine
# Just to compare with what happens in GitLab
CI_REGISTRY_IMAGE=# not defined
CI_COMMIT_SHORT_SHA=# not defined

docker-compose.yml will end up using the following tag

image: local/hello:local

Shell script to run (same as in the pipeline 🎉)

docker-compose build

💻 Example: Building using docker-compose

View these changes in the example Gitlab CI/CD repo
gitlab icon

Create a docker-compose.yml file:

version: '3'
services:
  hello:
    image: "${CI_REGISTRY_IMAGE-local}/hello:${CI_COMMIT_SHORT_SHA-local}"
    build:
      context: "."

Notice the syntax for defining the image tag:

image: ${CI_REGISTRY_IMAGE-local}/hello:${CI_COMMIT_SHORT_SHA-local}

Test locally by running docker-compose build

Building hello
Step 1/4 : FROM node:11-alpine
 ---> f18da2f58c3d
Step 2/4 : RUN npm install -g serve
 ---> Using cache
 ---> 7b8fa3de17a7
Step 3/4 : COPY public public
 ---> Using cache
 ---> 4f699bcbaa08
Step 4/4 : ENTRYPOINT [ "serve", "-n", "public/" ]
 ---> Using cache
 ---> 8f153797be95
Successfully built 8f153797be95
Successfully tagged local/hello:local

Update .gitlab-ci.yml :

build:
  stage: build 
  image: 
    name: docker/compose:1.21.2
    entrypoint: ["/bin/sh", "-c"]
  variables:
    DOCKER_HOST: tcp://docker:2375
  services:
    - docker:dind
  script:
   - docker-compose build

commit and push to GitLab.

The job output should look like this:

$ docker-compose build

Building hello
Step 1/4 : FROM node:11-alpine
11-alpine: Pulling from library/node
Digest: sha256:8bb56bab197299c8ff820f1a55462890caf08f57ffe3b91f5fa6945a4d505932
Status: Downloaded newer image for node:11-alpine
 ---> f18da2f58c3d
Step 2/4 : RUN npm install -g serve
 ---> Running in 6a5d6ec930c3
/usr/local/bin/serve -> /usr/local/lib/node_modules/serve/bin/serve.js
+ serve@11.0.2
added 78 packages from 39 contributors in 5.013s
Removing intermediate container 6a5d6ec930c3
 ---> 90b12949398d
Step 3/4 : COPY public public
 ---> b7a303b33298
Step 4/4 : ENTRYPOINT [ "serve", "-n", "public/" ]
 ---> Running in 514a9da1cf9f
Removing intermediate container 514a9da1cf9f
 ---> 311040293719
Successfully built 311040293719
Successfully tagged registry.gitlab.com/portenez/gitlab-ci-and-docker/hello:47693dca

💻 Example: Pushing to GitLab Container Registry

View these changes in the example Gitlab CI/CD repo
gitlab icon

Update .gitlab-ci.yml to login

Add docker login and docker-compose push to .gitlab-ci.yml

build:
  stage: build 
  image: 
    name: docker/compose:1.21.2
    entrypoint: ["/bin/sh", "-c"]
  variables:
    DOCKER_HOST: tcp://docker:2375
  services:
    - docker:dind
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker-compose build
    - docker-compose push

commit and push to GitLab

The job output will look similar, but close to the end of the output you should see the push

$ docker-compose push
Pushing hello (registry.gitlab.com/portenez/gitlab-ci-and-docker/hello:4cb7e95a)...
The push refers to repository [registry.gitlab.com/portenez/gitlab-ci-and-docker/hello]
4cb7e95a: digest: sha256:f886900467926c63f7899ce274d5b188853140c63017bb4ada2ba32b65af2a3f size: 1576

At this point, if you go to the GitLab Container Registry for your repository you should see the image

Docker Compose and multiple images

Defining Images with dependencies

As you start using Docker, you often end up in a situation in which you build a base image that then is used to build child images using the docker keyword FROM:.

Consider the following scenario. I want to provide re-usable images that provide:

  • a base os (alpine) image as a minimal OS
  • a base nodejs image to run javascript based apps
  • a base serve image to serve static assets

Then, when I build my service (or in this case a static website), I can start by using my custom serve image.

The Dockerfiles would look roughly like these:

Base alpine, tagged as my.company.co/alpine:v1

FROM alpine 
# Install custom company stuff

Base node, tagged as my.company.co/node:v1

FROM my.company.co/alpine:v1

# Install node in a way I like

Base serve, tagged as my.company.co/serve:v1

FROM my.company.co/node:v1

# Install serve which knows how to serve static files via http

The dependency graph would look as follows:

multiple images denpending on each other as outlined above

Docker-Compose allows us to both:

  • Build the whole graph by using a single docker-compose build command
  • Push the whole graph by issuing the single docker-compose push command

💻 Example: Multi-Image builds with Docker-Compose

View these changes in the example Gitlab CI/CD repo
gitlab icon

Continuing with our example project, let's split our build into 2 images:

  • serve: A generic static server using serve
  • hello: The actual static site

dependency diagram dependency example

We will create 2 dirs and 2 Docker files

├── app
│   ├── Dockerfile
│   └── public
├── serve
│   └── Dockerfile
└── docker-compose.yml

Create serve/Dockerfile

FROM node:11-alpine

RUN npm install -g serve

# It serves as documentation only
# see https://docs.docker.com/engine/reference/builder/#expose
EXPOSE 5000

ENTRYPOINT [ "serve", "-n", "public/" ]

This image will be a generic static site server, which expects contents to be inside the /public dir

Create app/Dockerfile

ARG SERVE_IMAGE

FROM ${SERVE_IMAGE}


# It serves as documentation only
# see https://docs.docker.com/engine/reference/builder/#expose
EXPOSE 5000

COPY public public

This Dockerfile uses the serve image we're creating and simply attaches the public dir.

Notice how this Dockefile uses the ARG directive. This allows docker-compose to pass in build-time parameters.

Move static contents to app/public/**

We need to move our static HTML to app dir.

Update docker-compose.yml

version: '3'
services:
  # Our "app" service
  hello:
    image: "${CI_REGISTRY_IMAGE-local}/hello:${CI_COMMIT_SHORT_SHA-local}"
    build:
      context: "app"
      # build time arguments passed to docker
      args:
        SERVE_IMAGE: "${CI_REGISTRY_IMAGE-local}/serve:${CI_COMMIT_SHORT_SHA-local}"
    ports:
     - "5000:5000"
    depends_on: 
      - serve # ensure serve is built first

  # Base image
  serve:
    image: "${CI_REGISTRY_IMAGE-local}/serve:${CI_COMMIT_SHORT_SHA-local}"
    build:
      context: "serve"

commit and push to GitLab

There's no need to change the build script, just push. By updating docker-compose.yml you've done all the configuration you need.

In GitLab CI/CD you should see output like this:

$ docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
$ docker-compose build
Building serve
Step 1/4 : FROM node:11-alpine
11-alpine: Pulling from library/node
Digest: sha256:8bb56bab197299c8ff820f1a55462890caf08f57ffe3b91f5fa6945a4d505932
Status: Downloaded newer image for node:11-alpine
 ---> f18da2f58c3d
Step 2/4 : RUN npm install -g serve
 ---> Running in a3ce0ce529a4
/usr/local/bin/serve -> /usr/local/lib/node_modules/serve/bin/serve.js
+ serve@11.1.0
added 78 packages from 39 contributors in 4.77s
Removing intermediate container a3ce0ce529a4
 ---> e934dabba9f1
Step 3/4 : EXPOSE 5000
 ---> Running in b46244ecd7ed
Removing intermediate container b46244ecd7ed
 ---> ba764ea8b867
Step 4/4 : ENTRYPOINT [ "serve", "-n", "public/" ]
 ---> Running in 81f2d4a5ba91
Removing intermediate container 81f2d4a5ba91
 ---> 5633cdc309d3
Successfully built 5633cdc309d3
Successfully tagged registry.gitlab.com/portenez/gitlab-ci-and-docker/serve:0d12b5b6
Building hello
Step 1/4 : ARG SERVE_IMAGE
Step 2/4 : FROM ${SERVE_IMAGE}
 ---> 5633cdc309d3
Step 3/4 : EXPOSE 5000
 ---> Running in 2229bea4b771
Removing intermediate container 2229bea4b771
 ---> 00a761340e13
Step 4/4 : COPY public public
 ---> 018202de2275
Successfully built 018202de2275
Successfully tagged registry.gitlab.com/portenez/gitlab-ci-and-docker/hello:0d12b5b6
$ docker-compose push
Pushing serve (registry.gitlab.com/portenez/gitlab-ci-and-docker/serve:0d12b5b6)...
The push refers to repository [registry.gitlab.com/portenez/gitlab-ci-and-docker/serve]
0d12b5b6: digest: sha256:8c7a27f5057e428489da8fad7f042bb4df6de2a9c5fd723ef76b3fd21b772994 size: 1369
Pushing hello (registry.gitlab.com/portenez/gitlab-ci-and-docker/hello:0d12b5b6)...
The push refers to repository [registry.gitlab.com/portenez/gitlab-ci-and-docker/hello]
0d12b5b6: digest: sha256:b88568b2fa155e8e8b3607ba5886af4ee4656d456f73a99e9b1c812894d60415 size: 1576

Testing Docker images

We should automate some tests for our example hello image, and a basic way of verifying that a web-server/app is working is by using the curl command.

As part of our example, we are going to create another image, also based on alpine, that knows how to curl. Then, we'll update our pipeline to use this new image to curl against our hello app.

💻 Example: Testing a service/webapp using Gitlab CI/CD

View these changes in the example Gitlab CI/CD repo
gitlab icon

Create curl/Dockerfile to teach to alpine how to curl

# curl/Dockerfile

FROM node:11-alpine

RUN apk --no-cache add curl

Update docker-compose.yml with new image to build

# docker-compose yml

version: '3'
services:
  # Our "app" service
  hello:
    image: "${CI_REGISTRY_IMAGE-local}/hello:${CI_COMMIT_SHORT_SHA-local}"
    build:
      context: "app"
      # build time arguments passed to docker
      args:
        SERVE_IMAGE: "${CI_REGISTRY_IMAGE-local}/serve:${CI_COMMIT_SHORT_SHA-local}"
    ports:
     - "5000:5000"
    depends_on: 
      - serve # ensure serve is built first

  # Base image
  serve:
    image: "${CI_REGISTRY_IMAGE-local}/serve:${CI_COMMIT_SHORT_SHA-local}"
    build:
      context: "serve"

  # We're adding this to test
  curl:
    image: "${CI_REGISTRY_IMAGE-local}/curl:${CI_COMMIT_SHORT_SHA-local}"
    build:
      context: "curl"

Update .gitlab-ci.yml to include a test based on the new container

build:
  stage: build 
  image: 
    name: docker/compose:1.21.2
    entrypoint: ["/bin/sh", "-c"]
  variables:
    DOCKER_HOST: tcp://docker:2375
  services:
    - docker:dind
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker-compose build
    - docker-compose push

test:
  stage: test
  # Use the image built in the build step to run curl
  image: ${CI_REGISTRY_IMAGE}/curl:${CI_COMMIT_SHORT_SHA}
  services:
    # Start a container of our service to test, built by the build step
    - name: ${CI_REGISTRY_IMAGE}/hello:${CI_COMMIT_SHORT_SHA}
      alias: hello 
  script:
    # Simple test
    - curl --fail http://hello:5000

commit and push to Gitlab.

Notice how a new test job is created. This new job uses the images created in the previous build job to run 2 containers: a target hello container and curl container to execute the test.

Starting service registry.gitlab.com/vgarcia.dev/gitlab-ci-and-docker/hello:a5530e6b ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/vgarcia.dev/gitlab-ci-and-docker/hello:a5530e6b ...
Using docker image sha256:e19ba2c01d9cf4ac04de70f9a5e1be47dadf3270ff9cd13570e13c884263bd85 for registry.gitlab.com/vgarcia.dev/gitlab-ci-and-docker/hello:a5530e6b ...
Waiting for services to be up and running...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/vgarcia.dev/gitlab-ci-and-docker/curl:a5530e6b ...
Using docker image sha256:e5518961e9a8d1d135a2f914f152efb27671e8792b02f23f9f6e9de73f8129c4 for registry.gitlab.com/vgarcia.dev/gitlab-ci-and-docker/curl:a5530e6b ...

Skipping Git submodules setup
Authenticating with credentials from job payload (GitLab Registry)
$ curl --fail http://hello:5000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<html>
  <body>
    <h1>Hello world docker</h1>
  </body>
</html>

100    66  100    66    0     0   4125      0 --:--:-- --:--:-- --:--:--  4125
Job succeeded


  

test
Retry

Duration: 42 seconds

Timeout: 1h (from project)

Runner: shared-runners-manager-4.gitlab.com (#44949)

Commit a5530e6b

Example: Test container
Pipeline #76784047 for master
test

Conclusion

Docker-Compose is a great tool for managing Docker builds. It allows defining builds in a way that works both in a pipeline and locally.

I hope you found this post useful. If you did, share on your favorite social media, and/or lemme know on 🐦 twitter.