A Google search for “devops” yields tons of interesting postings, tools, and even a couple of manifestos. Refine the search further to “devops quality” and the results become less direct; “devops testing” simply doesn’t exist according to Google.
That said, things are starting to change.
In late April, Greg Retkowski released Virtual Machine Test Harness (VMTH), for unit-testing infrastructure automation. Application developers are used to seeing a “green bar” or other reports that indicate that the changes they are making have been automatically tested and will not break predefined scenarios. VMTH does this for your configuration management (Puppet/Chef/CFEngine) setup, automatically building out virtual machines based on a configuration management setup, and then running tests against those machines.
As referred to in the source code, tests are set up in YAML. They are composed of directives to run uploads and downloads to the test VM, and of course to set up and run the tests. The desired tests can be specified in a plain-text scenarios file or passed in from the command line.
For example, one could set up a scenario file to kickstart a new machine, deploy and configure a database server, load-balancer, and web server, then ask VMTH to verify that the configuration works:
VM Test Harness Results.... Elapsed time to complete testing: 06:08:14 ------------------+------------+------------+----------------+ Scenario | Apply | Test | Execution Time | ------------------+------------+------------+----------------+ 0nulltest | Passed | Passed | 00:22:37 | dbserver | Passed | Passed | 00:09:29 | loadbalancer | Passed | Passed | 00:04:04 | webserver | Passed | Passed | 00:04:03 | ...
The test shows that the target machine’s services are configured correctly and (in this case) indeed actually up and running. While the tests take a long time -- several hours for a full run -- no human intervention is required, and the outcome is rock-solid confidence in your configuration. It’s roughly the DevOps equivalent of GUI testing tools like Squish and Selenium.
VMTH currently only works in a QEMU+KVM environment, but Greg points out that the VM-manager part of the software is abstracted and he’s “hoping to hear from some folks using xen or vmware to see their experiences.” VMTH is Apache licensed.
Increased test automation is probably the most significant change in the software quality and testing arena over the past few years. While developing for DevOps is simply different than developing a typical application -- infrastructure is hard to mock out, and cycle times are much longer -- VMTH shows that many application development techniques and tools are transferrable to DevOps.
You can find more information on Operations and Testing, right here on InfoQ!