BT

Architecturally Aligned Testing

| Posted by Stefan Friese Follow 1 Followers , reviewed by Shane Hastie Follow 27 Followers on Apr 08, 2018. Estimated reading time: 9 minutes |

Key Takeaways

  • Microservices are typically owned by cross-functional and autonomous teams, responsible for implementing, testing, and releasing.
  • When implementing a microservices architecture there should be no separate testing phase.
  • Everybody is responsible for the quality of the product.
  • The complexity of integrating microservices requires carefully deciding the  appropriate integration test approach.
  • Continuous Testing means a shift left in testing to ensure that teams stay autonomous and a shift right in testing towards exploration and experimentation.

Spring Unit Testing Best Practices

The Spring framework comes with nice support for integration testing, but a lot of manual coding is required to set up test cases properly. This paper describes how Jtest’s Unit Test Assistant can improve the workflow and outcomes for Spring application testing. Download Now.

 

Sponsored Content

Architecture impacts heavily testing. The way you test a monolith is different than how you should test a loosely coupled system. Let’s look at microservices as a prominent exponent of loosely coupled systems. As we will see, they challenge our previous definition of testing.

Conway’s Law

Melvin Conway had observed that how organizations are structured has a strong impact on the products they create: "Any organization that designs a system […] will inevitably produce a design whose structure is a copy of the organization's communication structure".

Conway’s law says that a company’s software architecture usually reflects its organizational structure. On the other hand, how we organize the teams has a powerful effect on the architecture and testing approaches.

The interdependencies of architecture and organizational structure is also reflected in the BAPO model which assumes that in the context of software engineering, four concerns need to be addressed: Business, Architecture, Process and Organization. Basically, the idea is to define the architecture based on the business needs. However, as Jan Bosch (2017) states, most companies are not BAPO but instead they are OPAB which means that the organizational structure is the basis to define the architecture.

If an organization wants to change the architecture of a software product, its organizational structure (including how testing is organized) can be an enabler or impediment. Let’s look at different organizational structures:

  1. Functional-oriented organizations optimize for costs. Those organizations have typically a hierarchical structure with teams of specialists, responsible for their functional area.
  2. Market-oriented organizations optimize for speed. Those organizations are typically rather flat with cross-functional and autonomous teams, responsible for implementing and delivering their products even though this may lead to redundancies across the organization.
  3. Matrix-oriented organizations attempt to combine the functional and market orientation.

For more information, see: Gene Kim (2016).

Testing Microservices

How shall a loosely coupled architecture, i.e. microservices, be tested?

A loosely coupled system follows a service autonomy principle as its architecture is based on the decomposition in autonomous parts.

Microservices are increasingly being adopted by organizations to improve the autonomy of their teams and increase the speed of change. Microservice applications are composed of those small, independently versioned, and scalable customer-focused services that communicate with each other over standard protocols with well-defined interfaces.

Microservices

  • Are autonomous and truly loosely coupled since each micro service is physically separated from others.
  • Are responsible for one functionality (functional decomposition).
  • Can be deployed and released independently.
  • Shall embed dependencies to ensure that the number of integration points is manageable.
  • Shall be resilient and isolate failures by acting autonomously.

What does this mean for testing?

A domain-oriented and vertical slicing per business capabilities means that there can be still a layered architecture but the layering is the secondary organization mechanism. Autonomous product development teams are responsible for implementing, testing, and delivering the business capability:

Microservices are often accompanied by DevOps: In agile and DevOps, there is no separate design phase with an architect responsible to define the architecture prior to the development phase. Instead, the architecture is defined more federated, addressed across the project, and owned by the whole team.

And how is the test approach changing for those systems? Also testing is typically not done anymore in a separate test phase by an independent test team. Instead, everybody is responsible for the quality.

There is no separate testing phase:

DevOps and Continuous Integration & Continuous Delivery (CI & CD) are addressing the need to deliver to the customer fast and with high quality. This means that tests must provide fast and meaningful feedback. To be able to release with both speed and confidence requires having immediate feedback from Continuous Testing:

  • Almost all testing should be automated. Enabling Continuous Testing is crucial for CI & CD as it provides timely feedback about the quality. Also, it enables the team to learn and adapt fast.
  • However, this does not mean that Continuous Testing is only about automating tests. As we will see, it is a holistic approach to get timely feedback about the quality:

Source: Dan Ashby (2016)

Testing is not an isolated activity but rather integrated in a comprehensive set of practices to promote quality: “Still, many companies struggle with changing their process from 'testing or inspecting quality in' to achieving quality from the start – through culture, design, craftsmanship, and leadership.” (Ben Linders 2017).

Developers play an important role to ensure high quality, e.g. as it is also encouraged by software craftsmanship principles. Software craftsmanship is about professionalism in software development: “Well-crafted software means that, regardless of how old the application is, developers can understand it easily. The side effects are well known and controlled. It has high and reliable test coverage, clear and simple design, and business language well expressed in the code” (Sandro Mancuso, 2015).

Continuous Testing means (often) shift left and right:

What does shift left and right mean?

Tests are typically executed in a specific order, starting (on the left) with unit tests as they provide fast feedback, followed by tests that take more time to execute but on the other hand increase the confidence in the release candidate:

There is a shift left towards automated unit and component level tests, executed in Continuous Integration & Continuous Delivery (CI & CD) pipelines that are owned by product development teams. However, this has its costs: Given that the integration scenarios are complex and dynamic, integration testing is getting more challenging.

If we talk about test in isolation versus collaboration based tests, it is important to differentiate solitary and sociable test approaches:

  • Solitary testing is an activity by which the software unit under test is tested in isolation by replacing collaborators (upstream dependencies) with test doubles such as mocks or stubs.
  • Sociable testing is an activity by which the unit software under test may be tested together with collaborators.

Following the idea of the test pyramid, the focus of microservices testing is typically on having many unit tests and a comprehensive set of component tests. Microservices are tested in isolation from each other followed by a smaller number of integration and eventually End-to-End tests. Many End-to-End tests is typically seen as a problem as they take more time to execute than unit and component tests and are also often fragile. But interestingly, there is criticism about the value of the test pyramid for microservice testing and if focusing on unit and component tests in lieu of integration tests is still reasonable (see for example: Cindy Sridharan 2017 and André Schaffer 2018). That is a legitimized question because if the integration of microservices is most challenging, shall we focus not more on integration testing and rely more on sociable tests? In my opinion, the answer is yes and no: It may make sense to test a microservice together with its dependencies if those embedded dependencies do not provide independent business capabilities. Such a sociable test of a microservice should be done before testing the integration with other microservices, encapsulating other business capabilities. If there are not enough unit and component level tests, analyzing failed integration tests would get even more complicated. Also, besides traditional integration tests, API as well as contract integration tests are important as they help to keep the pipelines of collaborating microservices independent from each other. Therefore, it is important to test on the appropriate test level.

Another aspect is also important: Given that for microservices the integration scenarios are complex and dynamic, how many integration tests are sufficient? There is a trade-off between putting more efforts into testing and detecting issues faster in production. Besides the shift left, there is also a trend that testing, deploying and monitoring on production gets closer together. Controlled experiments in production help to identify problems fast and to get fast feedback from actual customers.

What kind of production traffic tests can be differentiated?

This means that there is also a shift right in testing.

As outlined earlier, market-oriented organizations optimize for speed and the shift right in testing helps to accomplish these goals. As part of continual experimentation, it is possible to explore and learn. And with controlled experiments, you get feedback from customers, using the application in real world conditions.

Controlled experiments can also support a resilient architecture. As we discussed earlier, microservices shall be resilient and isolate failures by acting autonomously. But what means resilient? "We usually think of robust systems as the opposite of fragile ones because they don't care too much about (i.e., neither like nor dislike) stress. In fact, robust systems are merely less fragile. The true opposite of a fragile system would be one that actually benefits from stress. Nassim Taleb calls such systems antifragile" (Dave Zwieback 2014, p. 3). A traditional test approach would be to test the system prior to releases to ensure the stability. However, those tests support the robustness but not the “antifragility” of a system. A different test approach is to run proactive controlled chaos experiments by introducing failures on production. This orchestrated chaos guides the product development teams to think about failure tolerance and possibilities to isolate failures.

Summary

For loosely coupled systems, i.e. microservices, tests shall not be done in a separate test phase, by a dedicated test team, but instead collaboratively by cross-functional product development teams. There is a shift left in testing to ensure that teams stay autonomous and a shift right in testing towards exploration and experimentation.

Continuous Testing, as automated testing within CI & CD as well as a culture of experimentation and exploration, are enablers to release loosely coupled services fast and reliable.

References

About the Author

Stefan Friese is a Test and R&D Manager at HERE, working in Berlin. In his current role, he leads a team that builds Continuous Integration & Delivery test capabilities for real-time location web services, hosted on the HERE cloud platforms. Stefan has 14 years of experience as test manager, performance tester, software engineer, and quality engineer. You can find Stefan on twitter as @y3key.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Is Shift Left / Right of Value? by Ken McCormack

Stefan, a great article!

Also check out Caitie MaCaffrey's talks / paper on distributed systems verification queue.acm.org/detail.cfm?id=2889274

One point I often make about Cohn's test pyramid is that he said you have to test at *all* levels. I think the "move test to the left" mantra can obscure this for many teams, who only unit test and fail to do sufficient control / observation designs early at sub/system levels* (this seems to affect all developers, young and old!)

In a test-driven cycle, I usually start with a happy case (probably a vanilla connected call), then next start to emulate types of external system failure... to drive this I'm then introducing new runtime composition / containerisation logic to allow me to abstract my data sources, as well as capture "canonical" inputs/outputs in source control, so other developers can understand the main interactions of the system. When done, the whole system can be run through its main states, without the need for gigabytes of production-like data or service virtualisation tools.

A great irony is, that if we did more "system test driven development", this would cause us to select and adopt far better architectures, and deliver faster. This point is nothing new (check out any work from Rebecca Parsons or Neal Ford on Evolutionary Architecture).

One further note on Cohn, I look at the ideal test pyramid as a *reflection* of a testable architecture i.e. if most system decisions can be made and constrained at unit level, that is probably a good design. In other words, the test pyramid implies you must *change the architecture of your system* in such a way to maximise its correctness (not just make your tests look like a pyramid!)

This idea has its roots in an ultraquality architecture tactic called progressive redesign, where we incrementally change a system to eliminate classes of failure. A great source on this is Eberhardt Rechtin's work - all based in Lean engineering and Deming, engineering excellence - NASA... from 1960s onwards
blog.mccormack.net.au/rechtin-on-ultraquality-i...

Cheers!

Re: Is Shift Left / Right of Value? by Stefan Friese

Hi Ken,

Thanks for adding the link to Caitie MaCaffrey's paper.

I particularly like your statement that the "ideal" test pyramid shall be a reflection of a testable architecture. :)

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

2 Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT