BT

InfoQ Homepage News Categorizing Tests

Categorizing Tests

Bookmarks

In a recent discussion on the test driven development Yahoo! group Carlos Ble shared his understanding of test categories after doing his research:

Eventually the picture I've got in my mind is:
Developer tests:
Unit tests : Isolated, atomic, innocuous: exercised with xUnit
Integration tests:
Isolated tests that might change the state of the system, i.e: saving into database, writing file... An integration test does not represent a functional requirement as is. Can be written for xUnit. They check the integration of our code with a third party tool or with the different layers of our own code, i.e: the business logic layer requires the data access layer
Functional tests: (also known as System tests)
A test that excercises part of the system as a whole, some functional requirement. It might change the status of the system.
Product Owner tests:
Acceptance tests: Functional tests which input and output can be validated by a non-technical person, the product owner.

 

John Donaldson shared a multi-dimensional model of tests that look at test roles and test types:

I like the test view you give. But I think it's an instance of a bigger model, where you have (at least) actor-roles and test-types.

Actor-roles: developer, tester, QA, user, sponsor, etc.

Test-types: unit, integration, functional, system, acceptance, soak, smoke, etc.

In any given situation you probably which role gets to do which test - but in the next project it may be different.

Dale Emery suggested that not knowing what kind of test you are writing is a code smell and indicates lack of clarity. At the same time, a test may be fall under several categories and what is important is your current point of view:

The challenge, I think, is that any given test could be reasonably classified in numerous ways, depending on what dimensions you're focusing on. And there are a lot of dimensions that people use to classify tests. I've identified some here: http://cwd.dhemery.com/2004/04/dimensions/

So I'm less interested in knowing "what kind" of test it is, and more interested in knowing where a given test falls along the dimensions that are most important to me, depending on my purposes at any given moment. The ones that I think about most often are:

  • What "unit" does this test define and test? (What system, subsystem, object, collaboration...) - What feature does the test define and test?
  • Who are this test's primary customers? Who most cares about the results of executing this test?
  • What decisions will be made based on the results of executing this test?

Charlie Poole gives a detailed analysis of Carlos' categorization and suggests:

In my opinion, the most important distinction is between tests of developer intent and tests of customer intent.

This discussion highlights the fact that test categorization can be very confusing - especially to the newcomer.  The majority opinion is that a dimensional approach to test categorization is the way to go and that the type of categorization that is relevant is dependent on the user's current intentions and context.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • The full test life cycle of software

    by Guo Du /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I summarised some time ago give more details about those tests:
    duguo.org/en/The_full_test_life_cycle_of_software

  • Re: The full test life cycle of software

    by Amr Elssamadisy /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Thanks for your input. I think what you'll find is that there is no consensus on even the definitions, so your description of functional tests:

    code-facing, test features by sample data inside component
    doesn't match that of Charlie Poole's (for example) where he sees functional tests are synonyms to story tests.

  • Categorizing Tests

    by rey bumalay /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Great article.

    I was laughing coz I had a talk with few people who thinks integration testing is unit testing. Hope this article helps them understand.

  • Perhaps I'm an oxymoron, but...

    by Stuart Stern /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I recently authored the Infoq article, "FlexMonkey brings unit testing to Flex User Interface developers" (www.infoq.com/articles/flexmonkey-ui-unit-testing) and several people I know took issue with my characterization of any sort of UI testing as "unit" testing. In the article, however, I lay out the case (convincingly, of course!) for how it can be sufficient to divide testing into two broad categories: "Developer Tests" vs "QA Tests".

    Stu Stern
    Gorilla Logic CEO and Inventor of FlexMonkey

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
Country/Zone:
State/Province/Region:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.