Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Chaos Engineering Kubernetes with the Litmus Framework

Chaos Engineering Kubernetes with the Litmus Framework

This item in japanese


Litmus is an open source chaos engineering framework for Kubernetes environments running stateful applications. Sponsored by MayaData, Litmus enables users to run test suites, capture logs, generate reports and perform chaos tests. Litmus can be added to CI/CD pipelines as part of an end-to-end testing approach.

Litmus is designed to catch hard-to-detect bugs in Kubernetes that might be missed by unit or integration tests. Litmus can run two types of tests: chaos and functional. Chaos tests focus on application resilience and the framework contains several pre-existing tests for undesirable behavior such as a container crash, disk failure, or network delay and packet loss. Functional tests ensure the requirements of a system are met and Litmus includes tests for requirements such as application scaling. Litmus can also be used to determine if a Kubernetes deployment is suitable for stateful workloads.

Litmus is comprised of four primary components. Deployments setup the Kubernetes clusters for testing. Facilitators then assist with the test execution by running test suites, capturing logs, and generating reports. Test modules contain the containerized tests that are triggered from within the Kubernetes cluster when a test is executed. The final component is the tests themselves.

To run a Litmus test, referred to as experiments in the Litmus documentation, choose from the existing chaos experiments in the Litmus experiments directory, update application and test values passed in via environment variables, and run the litmusbook with kubectl.

kubectl create -f experiments/chaos/run_litmus_test.yml

Tests run in containers and after evaluating the test and environment variables, Litmus will orchestrate the necessary infrastructure to conduct an experiment, taking advantage of open source tools such as Pumba. When a test completes, Litmus will evaluation the success of an experiment based on predefined exit criteria. Test results are stored in a Kubernetes Custom Resource, along with test metadata. CI/CD pipelines that run Litmus tests can determine job status from these custom resources.

Litmus Architecture

Litmus Architecture [From the Litmus Documentation]

Tests can be written in BDD style English, taking advantage of the Go Cucumber BDD framework Godog, or in Ansible Playbooks and are grouped in the Litmus Test Playbook component. The majority of Litmus tests are Ansible Playbooks executed from within a Kubernetes pod managed by the Kubernetes job controller. Tests can take advantage of environment variables at runtime and pull in test libraries for common or specialized test utility tasks. Litmus also provides a Chaos-Exporter that exposes chaos metrics to a Prometheus endpoint.

Created by MayaData, the primary sponsor of OpenEBS, a Kubernetes native, open source container attached storage solution, the original purpose of Litmus was to provide end-to-end validation on stateful Kubernetes applications. Litmus was driven by the need to ensure that a stateful application could be run in Kubernetes with confidence that the data would be safe and accessible. Designed to be extendable and work with different persistent storage solutions, Litmus also supports stateful applications such as Percona MySQL, MongoDB, Apache Cassandra, and Minio, and covers common application lifecycle scenarios such as deployment, load, reschedule, scale-up, upgrade and de-provision.

To get started with Litmus, visit the Getting Started section of the Github repository or the official documentation.

Rate this Article


Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p