Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Test Driven Containerized Build Pipelines in ConcourseCI

Test Driven Containerized Build Pipelines in ConcourseCI

This item in japanese

A lead developer at Thoughtworks shared his team's experience in rewriting the build pipeline for one of their clients. They switched from Jenkins to ConcourseCI, with a focus on configuration-as-code, pipeline-driven delivery, container support and visibility into the system.

The existing system had a central Jenkins instance with complex configuration. The pipelines were written in an internal DSL, and ownership was spread throughout the team, with no real owner. Changing the setup was hard, and recreating it was seen as almost impossible.

The team listed what they wanted from a new system. To embrace the spirit of infrastructure-as-code, the configuration had to be in version control. The new system had to support pipelines as the building blocks, support containers, make status easily visible, and be a mature and well-maintained tool. The general approach was to use a new CI system that had collective ownership. They settled on ConcourseCI for its simpler UI and easy declarative configuration after evaluating other tools. This was part of a bigger re-architecting exercise, where they moved to a managed database - Amazon RDS - and AWS autoscaling groups for compute.

The team's focus was primarily on achieving speedy releases. Parallelization of pipelines where possible, a battery of linters, unit and infrastructure tests, and containerization of the deliverable were the key drivers to achieving this. Containers were also used as a cheap isolation mechanism, both to run the applications in production and to run the build pipelines themselves.

InfoQ got in touch with Mario Fernández, lead developer at Thoughtworks and author of the article, to understand the details of the migration, and their usage of containers:

We have different goals for these containers. The production containers are going to run the applications, so we want them to have the least amount of software installed on them, to reduce the footprint and the surface area for vulnerabilities. These containers are always based on Alpine. For the build containers, we want to build our software and make sure that the checks we want to implement are passing. We are a bit more lax here since we do not need to optimize them much.

A ConcourseCI pipeline is declared in YAML, and is built with the Job (how the pipeline runs) and the Resource (the objects that jobs use) abstractions. The pipelines in this case consisted mostly of shell scripts which could be tested independently. To avoid duplication, pipeline tasks were parameterized and reused. The initial effort in keeping pipelines consistent, says Fernández, made reuse easier later. However, he cautions against over-abstracting:

We try not to invest too much time fixing problems before they appear, as sometimes you build complexity that will never be needed. So I like to let the problem develop. The first time you duplicate a task maybe it’s not worth investing time abstracting it, but if that becomes common it can become a burden on the team, as you will increase the amount of maintenance work, or make upgrades quite difficult.

An examples of reuse is a task library in its own repository that also enforces standardization of the build steps, with app specific requirements pushed to the build scripts. The bash build scripts are however, mostly duplicated across projects since it's not easy to modularize them, explains Fernández. Switching to another scripting language would introduce avoidable technical dependencies. Container builds also adopted the sharing model for non-application containers like ones running Terraform and ServerSpec, whereas application containers remained separate.

The test suites consisted of multiple linters for Typescript, CSS, Terraform and Docker files, and headless Chrome tests. ServerSpec, a test framework for infrastructure, can verify that an infrastructure resource is running as expected. In this case, since ConcourseCI runs its builds in containers, these tests ran a container inside a container using Docker-in-Docker.

Collective ownership was one of the goals of this effort. However, this is not black and white, says Fernández:

There will be always people who know more about a system, in this case the CI. We try to avoid forming knowledge silos while giving people the opportunity to work on what they like best. We do a lot of pairing, and to have everybody touch every area of the product every now and then, which helps spreading the knowledge. The people with more knowledge about a certain area can serve as "anchors", making sure that what is built is still of high quality in the meantime. Sharing stuff as openly and often as possible also helps in fighting this.

Team members leaving can pose a risk to collective knowledge, and it’s important to focus on both delivery and coaching. Stakeholders need to understand this trade-off. Focusing only on delivery, says Fernández, risks "having an unbalanced team that cannot survive if a team member leaves. The spreading of knowledge and upskilling of the team has to happen organically as part of the normal development process."

The production infrastructure for this project runs on AWS, and utilizes AWS IAM for access control. ConcourseCI runs with a role that has the permissions to create AWS infrastructure, which are used by Terraform scripts. Concourse itself runs on EC2 instances, and comes with a dashboard to visualize the pipeline flow. The migration threw up a few challenges like adapting to differences between Jenkins and Concourse, and complex setup requirements and scanty documentation for the new CI.

Rate this Article