At DockerCon EU 2015, Laura Frank presented “Stop Being Lazy, and Test Your Software”. Frank proposed that testing software is necessary, no matter the size or status of your company, and introducing Docker to the development workflow can assist with writing and running testing frameworks more efficiently, and ultimately enable the organisation to consistently deliver high quality software products to customers.
Frank, a senior software engineer at Codeship, began the talk by stating that testing software has been a practice since the very first machines were used for computing. The women who programmed the ENIAC, the first electronic general purpose computer, to calculate artillery firing trajectories for the United States Army’s Ballistic Research Laboratory would periodically check the computed results against a known correct, hand-computed table. Frank proposed that testing is a vital part of delivering ‘working software in production’, alongside other techniques, such as pair programming, version control, code reviews, and high-availability architecture.
Motivations for testing include the ability to update an application without introducing regressions, validate functionality of updates, and confirm that refactoring did not break existing functionality. Frustrations often encountered when writing and running tests include the test suite taking too long to run locally, distractions from normal development workflow, and the difficulty of correctly configuring a testing environment. Frank proposed that introducing the use of Linux containers and the Docker toolbox tool suite to the test creation and execution process can alleviate some of the frustrations.
Docker Compose allows easy creation of consistent, reproducible containerised environments. Frank stated that many software developers use Docker Compose to create local development environments, but stop at this. The same benefits for using Docker within development, such as creating predictable and reproducible environments, can be highly beneficial to the quality assurance process.
Docker delivers a predictable, reproducible testing environment. Period
Frank cautioned that some modifications may be required to the Docker Compose workflow when this tool is used for testing, and potentially even a different Dockerfile should be used to specify different environment variables and run targets, which Docker Compose supports with the ‘dockerfile’ property. Care should also be taken to avoid initialisation and execution race conditions when utilising a complex arrangement of containers, applications and data stores.
The Docker Compose ‘docker-compose up’ command can be used to execute an automated test suite encoded in a Compose YAML configuration file, and one-off commands can also be run against a service by using the Compose ‘run’ command e.g.
$ docker-compose up $ docker-compose run -e “RAILS_ENV=test” app rake db:setup $ docker-compose run -e “RAILS_ENV=test” app test-command path/to/spec.rb
Frank stated that continuous integration and testing go hand-in-hand, and relying on CI/CD without adequate test coverage is often a problem - developers would much rather see a failed run of a continuous delivery pipeline, in comparison with finding defects in production.
Care should be taken to run tests in an environment as similar as possible as production, and Frank argued that if applications are being run within containers (and developed using containers) then testing within a build pipeline must also be conducted within containers. In order to achieve this, developers may be tempted to run ‘Docker-in-Docker’, and Frank suggested that anyone considering this should consult the work by Jérôme Petazzoni, such as the article ‘Using Docker-in-Docker for your CI or testing environment? Think twice.’
When using containers within a development process, Frank posited that it is easy to (incorrectly) think of a container as a ‘small VM’ that runs a specific workload. However, if containers are treated more correctly as simple OS processes, then additional benefits to the testing process can be realised. For example, multiple test runs on a containerised application can be executed in parallel. Building upon DockerCon EU conference theme of improving quality, Frank suggested that the quality of the software being created can be increased by adding an additional parallelised build pipeline to fail builds as quality decreases, such as the reduction of test code coverage below a certain level, the increase of linting errors, or the violation of ratchets.
Frank concluded the talk with a quote from Edsger Dijkstra, and stated that although testing is essential, it does not guarantee that the software being developed is defect-free, and expectations should be set appropriately.
Testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence.
The slides for Frank’s talk “Stop Being Lazy, and Test Your Software” can be found on SlideShare, and the video recording of the presentation will shortly be available on the Docker YouTube channel.