Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles How to Achieve Collaboration as a Key Driver for Continuous Testing

How to Achieve Collaboration as a Key Driver for Continuous Testing

Leia em Português

Key Takeaways

  • Agile Teams are responsible for Test Artefacts creation and versioning
  • Contract-Tests prevents teams from deviating from interface definitions 
  • Achieve a “think beyond team borders” by involving Agile Teams in later test stages
  • Test Artefacts are Deliverables and shall be reused horizontally and vertically across Test Stages 
  • Various Testing Tools need to interact and integrate seamlessly 

A new rise in protectionism

At this year's World Economic Forum in Davos, Switzerland, the central theme was a new rise in protectionism and its negative consequences for the world economy. Angela Merkel, Federal Chancellor of Germany, issued a warning; a clear specification of the most important prerequisite for prosperity and growth in a globally connected and ever faster-moving world: collaboration.

Collaboration and the world of DevOps, Agile and Continuous Testing

It’s not just in political and economic climates that collaboration is rising as the most important aspect of business. Collaboration is also the key success factor for us leaders in the technical space - to deliver software and value faster to our customers with an Agile approach. 

With Agile, we have started transforming the old model of phase-gated software delivery into a team and value-centric approach, reducing delivery/batch sizes and automating manual interactions to lower transaction costs, simultaneously increasing the quality of deliverables and the speed of delivery (see figure 1). Typically, the costly manual interactions comprise two areas – installing, deploying and configuring applications on the one side and verifying application functionalities one the other side – from development, throughout the test stages, to production (Continuous Testing). We deliver more often, we deliver smaller packages – and we continuously test what we deliver.

(Click on the image to enlarge it)

Figure 1: 
Left Side: The traditional hierarchical team setup for Development, Test and Operations- Which typically leads to siloization, long processing times due to handovers, delayed deliveries and huge batch sizes. 
Right Side: Agile Team setup -  Minimizes hand over, allows sharing of information and improves collaboration across former boundaries. 
Bottom: Example of a business process (=value stream) -Component based teams focus typically on one component only, where more mature agile teams are setup feature based, responsible for a compound of components along the value stream.

Collaboration within the team, across teams and across stages

Three types of collaboration have proven to be the highway to success and reflect the typical maturity steps enterprises are undergoing in their Continuous Testing strategy: collaboration within the team, collaboration across teams and collaboration across stages. 

I: Collaboration within the team

Acceptance Test-Driven, Behavior Driven, and Test-Driven Development have all proven to be magnificent drivers for quality and efficiency. All possible initial situations, conditions, and expected results are exposed and by thinking through the process to the very end, the entire team really understands what’s expected for a certain user story. 

Ideally, the test setup in those Agile teams (see figure 2) follows the analogy of pair programming. Like pair programmers perform code reviews side-by-side, in a collaborative testing setup developers and testers work in parallel on the respective user story, support each other, and thus provide Test Artifacts from the first day on, in a “test as a self-service model”. As soon as a developer presses the compile button, a test driver is already available. This approach sets a high-quality standard, according to the checks-and-balances principle, and it additionally frees up developer resources 

Figure 2: Agile Teams are cross-functional, Testers are part of the team. Agile teams are self-organizing, self-managing, and deliver working, fully tested increments each sprint. Agile teams operate with vision, architecture and UX guidance. The team setup minizimes hand overs, enables sharing of information and improves collaboration across former boundaries. 

Setup your test suite and environment like you do pair programming

As Agile teams mature, they achieve a high-efficiencygain from model-based and object-oriented test approaches. Like the standard developmentparadigm has evolved to object-oriented principles, model-based testing is the evolutionary step that additionally us allows to abstract business layers from the technical layers. This way we can present to each team member – technical or non-technical -  the best fitting view possible, enabling those with different skills to participate in the overall success of testing the project.

Use Test Artifacts as a tool in collaboration

Test Artifacts (see figure 3) represent and store the information necessary to steer an application’s testing – for example,a UI form and its fields, an interface definition, or a file format - in a redundancy-free and reusable way, decreasing the effort in both creation and maintenance dramatically, allowing these models to be an integral part of the full business process. 

These Test Artifacts are not limited to UI or API test drivers, but additionally consist of: 

  • Test data management routines, which generate synthetic test data, searchesfor existent test data or snapshots and masks data
  • A test design repository, which contains: 
    • Required test variantsand for each test variant pre-requisite its input data, conditions, and expected result to be verified against
    • Non-functional test drivers and parameters, such as response time, expected maximum load capabilities or various network conditions
    • For each variant, its assigned business risk – to understand the implication of each test case and assign it to several cascaded execution lists like smoke, sanity or regression tests. 

(Click on the image to enlarge it)

Figure 3: Test Artifacts consists of a holistic test case repository, prioritized by the risk attached to each test case. The repository defines input parameters, conditions and expected results – in combination with test data management – which provides records for test execution, creates synthetic test data or snapshots and masks application databases. The repository is made up of model-based and object-oriented test artifacts for UI-tests, API tests or other non-UI test drivers like a database of file test triggers, non-functional test drivers like load or network simulation conditions. Service Virtualization simulates dependent applications and utilizes an identical test data repository. 

Transform Progression Tests into Regression Tests

Too often we see Agile teams creating progression tests for a new functionality with one certain test tool, and on the other hand,implementation teams creating regression tests – redundant work, which causes additional effort and delays. With a model-basedapproach, every progression test case, which initially verifies new functionality, can be reused and transformed into a full regression test portfolio (see figure 4).

By transforming Progression Tests into Regression Tests, we can reach a new level of accuracy in Testing that black-box testing never could provide

Another priceless advantage of this approach is that these Agile teams are the centerof knowledge and are best equipped to create, maintain and version these Test Artifacts - almost in real time, with the least effort and the best knowledge possible. This approach allows a new level of accuracy in testing, which black-box-testing never could provide.

Figure 4: For an efficient and accurate Test strategy, Progression Test artifacts (=Verify new functionality during the sprint) need to be reused as Regression Tests in the next sprint after incremental delivery (=Verify full existent system functionality)

Is a dedicated tester within each team really required?

Are you still hesitating to set upa dedicated tester role in your teams? In the criminal jurisdiction, the separation of powers and the system of checks-and-balances was introduced almost 200 years ago. Since prosecutors, defenders and judges are no longer united in the one person, we haven’t seen any spectacular witch trials anymore. A dedicated tester removes developer blindness – think back to how pair programming eliminates errors.

II: Collaboration across teams 

“Two parallels intersect in infinity!” – said my descriptive geometry teacher, shattering my confidence in science at the age of seventeen, while I was busy just trying to grasp how to draw central perspective ball shadows. Funnily though, for me it seems that in software science it is the exact opposite wayaround: two parallel teams, service provider and service consumer, converge while working from an identical interface description, after 1 or 2 sprints are already completed. In a way, that you never would believe both speak the same (definition) language.

The solution: Contract Tests prove interface compatibility, before deviation occurs

Components interact via interfaces, and we need to understand that those interfaces are the key to(team) collaboration. Instead of waiting for Integration Tests to prove any deviation at a later point, the use of Contract Testing allows each team to verify their components completely and immediately for correctness from the very beginning, without needing to have the dependent component already available (see figure 5).

Contract Tests decouple each component from its dependencies, and interpret the interface definition between serviceclient and service consumer as a “contract”. Through this contract,we create API test drivers against the service provider, as well as mirrored service virtualization tests for the service consumer. With this approach, these Contract Tests guarantee matching interface interactions at any given time.

Figure 5: (1) “Contract“ is the synonym for the interface definition between Service Client and Service Provider. With an early interface definition, (2) the API test driver and Service Virtualization artifacts can be created and provided for the Agile Team’s test (Test First Approach). Contract Tests are the foundation for efficient (simulated) System Integration Testing.

Service Virtualization allows fully sandboxed, business process tests

Service Virtualization is a simulation technology that lets you execute tests without the Application Under Test (AUT)’s dependent system components being available. Service virtualization ensures that your tests will encounter the appropriate dependency behaviorand data each and every time that they execute (see figure 6).

Once the AUT’s schema and request data are verified, data can be stored, reused, or modified for later service virtualization scenario steps. You can embed string, mathematical or external functions to make test scenarios more realistic. Moreover, you can tap into dynamic pattern matching systems to provide different response scenarios, structures, or failure simulations. Service Virtualization allows fully sandboxed, businessprocess tests – enabling a full and early quality feedback in the software lifecycle. [see also Test-Driven Service Virtualization]

Figure 6: Service Virtualization (“Sandbox testing“) allows the Purchase-Order-Service team to retrieve full integration test results on a daily basis, including full business scenarios. The Application under test (Purchase-Order-Service) is encapsulated in a test-driven approach using test drivers and service virtualization artifacts simulating dependent applications, both of which use the same test data repository.

Contract Tests (Reverse API) are the key tocollaboration 

On the service provider side, the APIs are the key to solve the Gordian Knot for quality at speed. By reversing Service Virtualization scenarios to API test drivers, it can act like two sides of the same coin: Service Virtualization for the service consumer and an API test driver against the service provider. 

Here we also break through the Agile team barriers – enabling the service provider and service client teams to execute and cover simulated integration tests in the early development phase, which typically only would have been possible to cover in later integration test environments.  

Service Virtualization routines are Test Artifacts

Contract Tests count as Test Artifacts; created, maintained and versioned in the same way as UI tests, which allows onsite, offshore or even external vendors to execute (simulated) integration tests.

API first!

API first - the directive to focus on API tests first not only comes about as a product of this more efficient setup and desire to lower maintenance costs,but predominantly from us enabling progressive Contract Tests across teams. 

These steps are essentially the same as those used for many years in Formula 1 simulators. Configuration and interaction of hundreds of components are simulated (did you know that F1-teams are limited to using under 25 Teraflops for simulation?) and once the simulation identifies an appropriate setup, the racing driver verifies and fine-tunes this setup in a fast two-minute lap at over 200 miles per hour with the real car on a real racing track. Or how about in pilot training programs – do you think they’d let a newbie crash a real A380 when they take their first flight? Might be just a little too expensive.

III: Collaboration across the stages

With the first company teams transformed to Agile, there is repeated pattern which shows again a rise of protectionism

“With more than 50% of organizations implementing DevOps”(8), we can see a similar and repeated pattern - as soon as more than a handfulof teams or even whole business lines are transformed, a familiar topic is back in the foreground: protectionism.

How do we prevent a flare of protectionism over team boundaries and thus repeated formation of walls? This time it is not between Dev, Test, Ops and Business, but between the Agile teams and the line of business that has often become independent through the Digital Transformation. How must collaboration be baked into the process to successfully implement a Continuous Testing strategy for the entire company, so that CEOs and board members do not still face massive delays for large-scale product initiatives?  

Once your number of Agile teams, grows you will form a compound of Agile Teams: Tribes, Agile Release Train or Line of Business – and you might have applied the Spotify model or follow the Scaled Agile Framework (see figure 7), yet still the mechanism of staged testing is present. This is assuming you have already matured in your test processes and the first and comprehensive quality verification happens within the team andis then covered via Contract Testing. 

(Click on the image to enlarge it)

Figure 7: Left - The Scaled Agile Framework(2). Right - The Spotify Model(3).
Enterprise Agile Frameworks describe the hierarchy and structure required to successfully scale Agile to the Enterprise, how to group Agile teams, how to deliver software across teams and how to build and manage a compound of teams (“Agile Release Train”, “ Tribe”).

Still, efficient tests are required to be executed in test stages – to preclude unknown or unidentified behaviors- especially for substantial use cases with a high business risk attached. In stage one, typically the interaction of components is verified, in stage two, full end-to-end use cases and final business acceptance tests are covered. 

Now imagine the additional resources, time and alignment required - and the delay consequently - on top of your Agile team member’s efforts, if you need to re-create Test Artifacts for those test stages, typically by a dedicated implementation team.  In order not to re-build another wall, there is one simple, but game-changingrule: 

Test Artifacts are deliverables

Test Artifacts are deliverables. The later that changes are made, typically the more brittle test automation is, but the reusability of Test Artifacts guarantees knowledge transfer and collaboration.  

Continuous Testing requires a balance between Agile Teams and Implementation (System) Teams

After a realized transformation from hierarchical to Agile team structures, in some enterprises,Agile teams might be responsible and accountable for their application from development all the way through to production, but the majority will have separate implementation (system) teams in place to deploy, maintain and test the compound of all applications.

The Scaled Agile Framework states that it “requires a sense of balance between Agile Teams and Implementation (System) Teams”, which requires these teams to tightly collaborate, sharing knowledge and Test Artifacts. 

Figure 8: Optimum velocity (and effort) is a shared (test) responsibility between Agile Teams and Implementation(System) Teams. 

How does a Continuous Delivery pipeline work?

The trio of an artifactory, version control system and release automation orchestrates a Continuous Delivery Pipeline, in combination with a Continuous Testing platform. It installs, configures and deploys the application to various test stages and to production, observes decision rules, quality thresholds, application dependencies and – if necessary - executes rollback scenarios. It triggers test execution for each test stage, based on the application and the version deployed. 

Figure 9: Test stages from left to right: (1) Dev: typically, on a local machine and Test executed in a kind of pair programming. (2) QA – build and test triggered via CI, focusing on the very team’s component only (sandbox test). (3) Component Integration Test for the compound of Agile Teams’ components (Tribe, Line of Business, Agile Release Train). 
(4) E2E -Test overarching the full solution (Tribes of Tribes/compound of Line of Businesses/Solution Train).  
Application Release Automation Tools are the foundation for Continuous Delivery – installing, deploying and configuring applications into each test stage and production, based on an underlying component versioning, an artifactory to prevent code from being recompiled, triggering test execution for each stage individually, moving on to next stage based on pre-defined decision criteria and performing rollback scenarios if necessary.

As these stages are highly dynamically built, initially in teams (created, versioned and maintained), Test Artifacts are best equipped to fulfillthe speed and the quality required.

Elevate Test Automation into true verification processes rather than a downstream change activity

System Tests typically have a different orientation, still,Test Artifacts like UI-test-drivers, interface tests, test data routines and a test repository can still be applied to end-to-end testing, whilst following the focus on business processes.

Model-based, once-in-place end-to-end test cases combine the pieces from the various teams.  Modifications in each underlying component and its test drivers, performed by the Agile teams a few sprints later, are inherited and merged into the end-to-end test process chains. Model-based tests enable vertical test execution for full value streams, utilizing the right version for each test stage, and elevating automated test execution into true verification processes, rather than downstream change activities.

(Click on the image to enlarge it)

Figure 10: Typically, a dedicated System Team verifies application functionality at the enterprise level. The best velocity, effort and quality can be achieved when Test Artifacts are handed over from teams to line of business to system teams.

Do methods such as feature toggling or testing-in-production change the way you perform critical end-to-end business case testing? Testing-in-production is typically used for lower-risk use cases, provided to a small customer base, using green-blue deployment and, if necessary, automatically rolled back in the event of errors. And feature toggles? Critical processes such as changes to the shopping cart payment process will still be tested sufficientlybefore customers might get a view of other customers' credit card data.

Do you stillthink that Agile teams should be fully self-organizing (and only focused on their owned component/s)? Never forget W. Edwards Deming

“A system must be managed. It will not manage itself. Left to themselves, components become selfish, competitive, independent profit centers, and thus destroy the system. The secret is cooperation between components toward the aim of the organization.”

Conclusion – The past, the present and the future

In the last two decades, central test centers, focusing dominantly on verifying full end-to-end business processes, have had a hard life. There exists a vast field of tension, between often more than 50 different suppliers, externally, near-shore or internally, mostly confronted with late deliveries, inferior quality inputs, where Developer Acceptance Test protocols are not followed, quality checks are carried out far too late in the process, in poorly-maintained test environments and in ever shorter periods of time. It’s a dysfunctional tangle.

We now see the pendulum swinging to the extreme left in the market, where it’s now expected that developers in their teams will now solve all quality issues.

Think beyond your team boundary

As is often the case, the truth is in the middle - and requires balance and collaboration between Agile teams and implementation teams - culturally, in-process, as well as in handover of Test Artifacts.

According to the three stages mentioned above, teams focus on only their components at first, and overall thinking takes place later, once the enterprise has matured in their Agile transformation. 

For the perfect approach for scaling Agile to the Enterprise, this requires more than just for every Agile team to be self-organizing – it requires the team to see cross-border collaboration as a crucial part of the Agile paradigm. A “thinking beyond team borders” attitude can be best achieved by involving teams in traditionally separate, but now cohesive testing activities that are conducted during the various Continuous Testing Stages, not just in development. With a free choice of tools in each team, testing can only work if the selected tools not only serve the purpose of the team but also allows seamless integration and further use in further test stages, by a variety of different eyes and roles.


  1. Header: Source: 683014141/; World Economic Forum
  2. Figure 7: © The Scaled Agile Framework
  3. Figure 7: © Spotify
  4. Figure 8: Shared Responsibility
  5. W. Edwards Deming:  Scaled Agile Framework – principle #2 Apply system thinking
  6. Model Based Test Automation
  7. Figure 9: Application Release Automation
  8. Forrester: 2018 –  The Year of Enterprise DevOps

About the Author

Alexander Mohr is working for Tricentis developing and providing Strategy, Best Practices and Thought Leadership around Enterprise Continuous Testing journeys, setup road-maps for Agile Transformation, CI/CT/CD and supervises enterprises throughout their transformation journey. Alexander has more than 19 years of IT experience, started early 2000 as a developer for a credit card company creating a system for issuing and acquiring. In the following 15 years, Alexander covered several roles as developer, product owner, business analyst, test manager, architect and project manager, learning a lot of do’s and don’ts in waterfall and agile projects in midsize and enterprise size companies.

Rate this Article