Implementing a test strategy helped an organization to move away from push and pray deployment toward continuous and confident deployment to production. The organization mapped their test strategy in a framework with different enablers, which has helped them align on quality metrics for the whole product together with a strong safety net of tests before moving to production.
Varuna Srivastava, a lead quality analyst at ThoughtWorks, presented at ConTEST NYC 2021 a case study of a product having severe deployment problems, and where the organization decided to increase their deployment maturity using test strategy as a driver.
A test strategy has its life cycle tied to your project. There could be multiple elements of strategy based on the project/product you are in, as Srivastava explained:
Our product was a mobile app and the elements of the strategy were what devices and versions were supported, the entire end-to-end quality process, different testing types, a definition of the path to production, and a risk mitigation plan.
As our product was on trunk base development, we also invested in feature and release toggles for frequent production deployments, and our strategy was to test both with a feature on and off for product before moving to clients. This helped to roll back any feature if defects or blockers were identified in the production environment.
To develop the skills required for the new test strategy, the organization identified the capability needs for the projects and based on those needs, invested in providing training for team members, Srivastava said. The organization as a whole also came up with ways of working that were customized based on the team’s needs; all team principles were similar with different flavors. The teams started working on autonomous and collaborative principles for the product, where the frequent team member rotation between different teams helped in spreading the knowledge across the organization.
According to Srivastava, some of the benefits that the test strategy has brought the organization include:
Quality metrics - The test strategy consists of measurable quality metrics. The scores give the health of any project and also reflect on the areas of improvement.
Engineering practice maturity - The test strategy has enabled matured engineering practices to optimize rapid reliable production releases. It helped in embedding all frameworks such as spotify or SAFe in a team. The test strategy also helped in project and tech governance to align the business and tech needs of an organization. It also enabled teams to invest in continuous backlog grooming and defining a roadmap with an interdependent team.
Infrastructure maturity - The test strategy enabled a delivery infrastructure pipeline for self-service deployments, rollback, and monitoring. It has helped in receiving the alerts for any downtime.
InfoQ interviewed Varuna Srivastava about how the organization implemented their test strategy.
InfoQ: What were the problems that the organization faced due to their "push and pray" deployment approach?
Varuna Srivastava: Push and Pray deployment was unreliable; sometimes features may work in production environments, but many times integration issues were creeping up. The stakeholders were losing trust in features going live to customers.
Most of the time the organization would see thousands of support tickets, and support calls were coming in all day. It was affecting the brand value and also the morale of individuals who were part of making these features go-live.
Time to market was at the longer end, which results in mistrust starting to creep in among team members and within teams, affecting team dynamics and work culture within an organization.
InfoQ: What does a test strategy look like and what benefits can it bring?
Srivastava: The test strategy is a framework similar to your programming language framework. It’s what you practice or intend to practice. Continuous refactoring and feedback evolves it throughout the project life cycle. A test strategy helps in bringing the common plan of action for both tech and business. In our project it has helped to define our needs, and then come up with agreed upon deliverables.
InfoQ: What feedback loops were set up to update the test strategy?
Srivastava: The test strategy feedback loop is tied back to the defined SMART (Specific, Measurable, Attainable, Relevant, Time framework) KPIs and RAIDs of the project. We had set different checkpoints on sprint iteration to keep getting the feedback and seeding those back in our KPIs. The project team started with a need and ended with an action derived from the KPI. These were some aspects that we decided to measure as part of KPIs:
- Defect escape rate
- Calculation of MTTR, MTBF and fail rate
- Test automation coverage & failure rate
- Test automation feedback time
- Defect matrix
The defect matrix gives a picture of the health of the product. We had created a matrix based on priority and severity of the defects, and this helped us to take action on how we could have prevented those defects and use that as input for new features.
RAIDs (Risk, Assumption, Issues, and Dependency) identify and analyze things that a project team needs to understand to effectively manage uncertainty, problems and things that are partially or wholly outside their direct control.
Here are few examples of RAIDs:
- Potential delays in going live if this set of testing is not performed
- Validating assumptions of interdependent systems available in test environments
- Acknowledging any issues
- Any dependency on production data, to execute the tests on lower environments
RAIDs being identified and agreed upon within a team help in deriving the go-live for a customer, and moves away from the push and pray stage of deployment.
InfoQ: One of the things implemented was to go for "strong tech champions". What was the idea behind this, and what steps were taken to realize this?
Srivastava: The company implemented the pathways which enable individuals to drive their own learning paths. It is built into a framework that reflects "Me Now", "Me Next," and "My plan". A consultant can create their own goal based on their interest, and attach a timeline against it to mark as "Accomplished" once they are done. An individual can self-nominate themself for a set of training (for example, implementing modernized architecture, and testing in delivery cycle) that they need from a pool of training available in the learning hub pool.
The idea behind cultivating strong tech champions was to enable passionate individuals in an organization, individuals who can drive their own learning path. It also creates visibility and an organizational alignment with individuals’ journeys.