CircleCI is great at enabling developers to spin up an environment for testing from a set of defined images. When dealing with a website with a database, the usual build process involves downloading a database dump, installing it, and then performing tests. Here is a sample job that follows this approach. Notice where the majority of the time is allocated:
CircleCI spent two minutes downloading and installing a fairly small database (92MB compressed, 366MB uncompressed). This may be acceptable, but what if the database size was 1GB? CircleCI jobs would take several minutes to complete, which could slow down the development team’s performance. In this article, we will discover how moving the database installation into a custom Docker image helps to dramatically reduce build times on CircleCI. Let’s go straight to the results and see the log of a job where this has been already implemented:
Notice the timings: we are down to eight seconds while the previous screenshot showed two minutes. Isn’t that great? Let’s compare how the traditional method works to the suggested way of doing it.
Installing the database at CircleCI
Here is a typical CircleCI config file, which uses the official MariaDB image and then downloads and installs a database:
With the above setup, CircleCI spends two minutes downloading the database and installing it. Now, let’s see what this job looks like if the database is fetched from a custom Docker image.
Now CircleCI takes only eight seconds to install dependencies and mount the database and voilà, the container has started. Notice that we are using a custom Docker image hosted at Docker Hub:
The above repository is in charge of building and hosting the image that contains the database in it. In the next section, we will discover how it works.
Installing the database via a Dockerfile
I created a repository on GitHub that contains both the CircleCI job above and the Dockerfile, which Docker Hub uses to build the custom image. Here is the Dockerfile, inspired by lindycoder’s prepopulated MySQL database. Notice the highlighted section, which downloads and places the database in a directory where MariaDB will find it and import it:
The GitHub repository is connected with Docker Hub, which has a build trigger to build the image that CircleCI uses in the continuous integration jobs:
Benefits
In addition to achieving faster jobs at CircleCI, once we had the process in place we realized that it was making a positive impact in other areas such as:
- Faster build times on Tugboat, our tool for deployment previews.
- Developers started using the database image locally as it gave them an easy way to get a database for their local environment.
Hosting private images
This article’s example uses a public repository at Docker Hub to store the image that contains the database. For a real project, you would host this in a private repository. Here are a couple options to do so:
Quay.io
Quay.io costs $15 per month for five repositories. What I like the most about Quay.io is that it allows the use of robot accounts, which can be used on CI services like CircleCI in order to pull images without having to share your Quay.io credentials as environment variables. Instead, you would create a robot account which has read-only permissions on a single repository on Quay.io and then use the resulting credentials on CircleCI.
Docker hub
Docker hub's free account supports a single private repository. However, it does not support robot accounts like Quay.io so unless you create a Docker account just for the project you're working on, you would have to store your personal Docker Hub passwords—in the form of environment variables—on third-party services like CircleCI.
Try it out
Find out how this implementation approach works for your project. If you run into any issues or have feedback, please post a comment below or in the repository. There is another implementation at https://gitlab.com/juampynr/drupal8-gitlab/tree/master/scripts/database, which is referenced by the article An Installer for Drupal 8 and GitLab CI.
Acknowledgments
This article was possible thanks to the help of the following folks:
- James Sansbury for showing me how they do this at Georgia.gov.
- Andrew Berry for his help with the Dockerfile and general guidance.
- Salvador Molina Moreno for his editorial and technical feedback.