Categorizing Tests

Posted by Amr Elssamadisy on Aug 17, 2009

Sections

Process & Practices,

Architecture & Design,

Development

Topics

Software Testing ,

Agile Techniques ,

Agile

Tags

Testing

In a recent discussion on the test driven development Yahoo! group Carlos Ble shared his understanding of test categories after doing his research:

Eventually the picture I've got in my mind is:
Developer tests:
Unit tests : Isolated, atomic, innocuous: exercised with xUnit
Integration tests:
Isolated tests that might change the state of the system, i.e: saving into database, writing file... An integration test does not represent a functional requirement as is. Can be written for xUnit. They check the integration of our code with a third party tool or with the different layers of our own code, i.e: the business logic layer requires the data access layer
Functional tests: (also known as System tests)
A test that excercises part of the system as a whole, some functional requirement. It might change the status of the system.
Product Owner tests:
Acceptance tests: Functional tests which input and output can be validated by a non-technical person, the product owner.

 

John Donaldson shared a multi-dimensional model of tests that look at test roles and test types:

I like the test view you give. But I think it's an instance of a bigger model, where you have (at least) actor-roles and test-types.

Actor-roles: developer, tester, QA, user, sponsor, etc.

Test-types: unit, integration, functional, system, acceptance, soak, smoke, etc.

In any given situation you probably which role gets to do which test - but in the next project it may be different.

Dale Emery suggested that not knowing what kind of test you are writing is a code smell and indicates lack of clarity. At the same time, a test may be fall under several categories and what is important is your current point of view:

The challenge, I think, is that any given test could be reasonably classified in numerous ways, depending on what dimensions you're focusing on. And there are a lot of dimensions that people use to classify tests. I've identified some here: http://cwd.dhemery.com/2004/04/dimensions/

So I'm less interested in knowing "what kind" of test it is, and more interested in knowing where a given test falls along the dimensions that are most important to me, depending on my purposes at any given moment. The ones that I think about most often are:

• What "unit" does this test define and test? (What system, subsystem, object, collaboration...) - What feature does the test define and test?
• Who are this test's primary customers? Who most cares about the results of executing this test?
• What decisions will be made based on the results of executing this test?

Charlie Poole gives a detailed analysis of Carlos' categorization and suggests:

In my opinion, the most important distinction is between tests of developer intent and tests of customer intent.

This discussion highlights the fact that test categorization can be very confusing - especially to the newcomer.  The majority opinion is that a dimensional approach to test categorization is the way to go and that the type of categorization that is relevant is dependent on the user's current intentions and context.

1. Back to top

   The full test life cycle of software

   by Guo Du

   I summarised some time ago give more details about those tests:
   duguo.org/en/The_full_test_life_cycle_of_software

   Reply

2. Back to top

   Re: The full test life cycle of software

   by Amr Elssamadisy

   Thanks for your input. I think what you'll find is that there is no consensus on even the definitions, so your description of functional tests:
   

   code-facing, test features by sample data inside component

   doesn't match that of Charlie Poole's (for example) where he sees functional tests are synonyms to story tests.

   Reply

3. Back to top

   Categorizing Tests

   by rey bumalay

   Great article.
   
   I was laughing coz I had a talk with few people who thinks integration testing is unit testing. Hope this article helps them understand.

   Reply

4. Back to top

   Perhaps I'm an oxymoron, but...

   by Stuart Stern

   I recently authored the Infoq article, "FlexMonkey brings unit testing to Flex User Interface developers" (www.infoq.com/articles/flexmonkey-ui-unit-testing) and several people I know took issue with my characterization of any sort of UI testing as "unit" testing. In the article, however, I lay out the case (convincingly, of course!) for how it can be sufficient to divide testing into two broad categories: "Developer Tests" vs "QA Tests".
   
   Stu Stern
   Gorilla Logic CEO and Inventor of FlexMonkey

   Reply

• Older >