Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles Who is in Charge of Quality in Software Development

Who is in Charge of Quality in Software Development

Leia em Português

Key Takeaways

  • As software development continues to move left, quality is the responsibility of every team member.
  • Quality is no longer defined by just uptime and reliability — there are different aspects to quality that drive user choices.
  • David A. Garvin’s “Five Approaches to Quality" from 1984 defines quality as transcendent, value-based, user-based, product-based and manufacturing.
  • Besides manufacturing quality, most types of quality are immeasurable but still essential for teams to consider, and often involve working directly with your users.
  • Behavior-driven design or BDD is a way to map out the user journey and test cases before you even write code.

Agile software development and DevOps — and their emphasis on user experience — have us focusing on the people behind the products. But does the process matter or do the ends just justify the means? London’s P3X or People, Product, and Process Exchange focused a lot on the intersection of its three Ps, with perhaps the last one being the most interesting, as it brought together more acronyms — test-driven development (TDD), behavior-driven development (BDD), continuous delivery (CD), domain-driven development (DDD) and more — to help teams examine how to systematically build better systems.

Cofounder of the Agile Testing Fellowship Janet Gregory ended her keynote on the process of pursuing quality in software by asking the packed house to raise their hands if they felt the magic in their agile team, and if they felt they were spreading quality. Only a few people in a room assumably full of agile practitioners raised their hands.

How are we 17 years since the signing of the Agile Manifesto and still so few people are seeing past the shadows on the wall? Maybe we still aren’t having the right conversations. Maybe we aren’t having them with the right people. Or maybe conversations aren’t part of our processes at all.

While the manifesto puts individuals and interactions over processes and tools, there’s something inherently human in processes. Perhaps by examining our processes, we can better respond to change, increase our collaboration, and decrease bugs, all toward satisfying the customer early and often. Gregory took a generation-old approach to quality and applied it to modern agile software teams in a hope that everyone takes ownership for what is released.

Just what is ‘quality’ anyway

Gregory was pointing out that the often subjective definition of quality must be defined ahead. She offered David A. Garvin’s “Five Approaches to Quality" from 1984 as a way to start defining quality:

  • Transcendent - atmosphere, innate excellence, universally recognizable, achievements
  • Value-based - price and cost
  • User-based - value to some person, which person - this is the one most people think about when they think about quality
  • Product-based - What are your users looking for? (like kind of milk offered)
  • Manufacturing - practices, processes, standards, requirements, specifications — Did we build it right?

Gregory visualized the essentialness of each category as follows, radiating out from the most necessary center, and applied it to modern agile environments.

Manufacturing-Based Quality

First, things need to work, so manufacturing-based quality has to be there.

Gregory says this involves test-driven design because “by creating clean code, the rework is significantly reduced. Let’s do it right the first time so we won’t have other bugs. We can release with confidence.”

TDD, the practice of designing automated tests before the software it’s testing, which in turn leads to decoupling of said software, is an important part of manufacturing quality. Gregory quoted research that says teams that do TDD have have between 60 to 90 percent less defects than those who don’t, but TDD takes on average 15 to 30 percent longer.

Many teams are facing this quality-speed trade-off.

“Maybe it’s the PO [Product Owner] saying I’d rather you put the new feature in than quality. Who is making those decisions?”

Besides TDD, Gregory said manufacturing-based processes include:

  • Coding for maintainability
  • Monitoring error logs
  • Continuous integration
  • Exploratory testing on stories
  • Testing if the product meets specifications
  • Automated test creation for rapid feedback
  • Platform static analysis
  • A clear definition of Done

Finally, she said, “Practices like DevOps are trying to reduce the risk to the customer when releasing to the customer.”

Product-based quality

Simply put, if manufacturing-based quality is about making sure something works, product-based is about making sure it works as expected. For example, we expect to pay for higher quality, but we are more forgiving if something breaks if it cost little to nothing. The exception may be with applications which we usually expect to be free, but to really work well.

Gregory points out that what is defined as product-based quality varies by target audience. Someone in accounting would want that keyboard tray excluded from most laptop computers nowadays.

It’s all about asking:

  • Are we building the right thing?
  • Are we putting in the features that we want?

This includes:

  • Acceptance Test-Driven Development (ATDD) or sometimes called Story Test-Driven Development - bringing key customers into the TDD phase
  • Security testing
  • Bug bashes - like a team hackathon to find as many bugs as possible
  • Continuous delivery
  • Exploratory testing on features
  • Beta releases
  • Performance testing
  • Load testing

User-based quality

This is where perspectives most vary. As Gregory said, “Different people choose different things. They have different wants, different needs. If we’re trying to let the customer choose, make the customers happy.”

But don’t forget to keep in mind, she continued, “We are also making a big assumption that the consumers have enough information that they can make a qualified decision.”

She spoke of an app she once used that she found super unfriendly. It turned out the users loved it because it followed exactly how they worked. She didn’t work in that field. It’s all about meeting the specific users’ specific use cases.

Value-based quality

This is a simply what people are willing to pay for. Value is difficult to judge, basically impossible to judge without talking to your prospective customers.

Value-based quality is assessed with some combination of:

  • Effectiveness
  • Efficiency
  • Comfort
  • Trust
  • Complexity
  • Context-fit

Transcendent quality

Finally the most immeasurable quality — transcendence. Gregory said that’s because it’s hardest to measure emotion, making transcendent quality a blend of artistry, engagement, and customer loyalty.

How do we measure the quality of software?

Overall, if you accept Garvin’s quality scale, it’s difficult to measure most parts of software quality. She quoted Isabel Evans in measuring quality in software. There are plenty of examples of manufacturing-based quality:

  • Number of bugs in production
  • Severity of bugs in production
  • Days since last push to production
  • Number of new support tickets in X days since last production push
  • Build radiator remains green
  • No flakey automated tests (random fails)
  • Static code analysis of codebase is healthy
  • Rework rates are low
  • Bugs don’t get reopened

There can also be a measure of user-based quality in the form of user satisfaction surveys.

However, you can’t really measure product-based, value-based or transcendent quality. You can however discuss and evaluate all five layers of quality. Testing is an important measure of quality, but Gregory reminded that product teams can’t negate the value of discussing quality among each other, with the users and against competitors.

Of course, teams need to discover a balance between the desire to avoid mistakes versus those who want speed.

The whole team is responsible for quality

What is clear is that quality assurance — or an effort toward that misnamed impossibility — is not just the job of one department, testing, when the code is thrown over a wall.

The whole team owns quality

“If your organization, if your company starts with quality in mind, you’re likely to succeed because everything else just falls in place. Everything just kind of works. But if you start with the idea that speed is the most important thing, not paying attention to quality, the chances are, you are going to have in the long run a lot of rework, a lot of unmaintainable code and your quality will go down further and further,” Gregory said.

But she didn’t offer a secret recipe for the perfect pursuit of quality.

“Whether you use qualitative or qualitative, I don’t care, but you need to ask what you are looking at — does it lead to the ability to release with confidence?” Gregory said. “Does measuring the quality of the process measure the quality of the product?”

She quoted co-author of BDD Book 1: Discovery Seb Rose: “When a measure becomes a target, it ceases to be a good measure.”

“No matter how you measure it, it should lead to a conversation, a discussion about what you need,” Gregory said.

She continued that “The team owns quality but you have to think bigger. Quality in the processes and quality in the practices. Competence in your team, competence in how you deliver software.”

She ended saying that every conversation in this direction is best if it starts with people trying to solve a problem.

“Let’s build quality management into our process and learn to talk about what we do,” Gregory said.

About the Authors

Currently based in London, Jennifer Riggins is a tech storyteller and writer, where digital transformation meets culture, hopefully changing the world for a better place. Follow her on Twitter @jkriggins

Cofounder of the Agile Testing Fellowship, Janet Gregory has spent over 14 years helping teams transition to an agile software development environment, specializing in helping testers and business understand their role as part of the “whole team approach.”

Rate this Article