Testing: What Developers Are Expected To Do Versus What They Actually Do

At the PDC there was a panel discussion on "The Future of Unit Testing". Most of the conversation focused on mocking. It was the general consensus that mock frameworks were being over used.

The theory goes like this. Often, implementing all the necessary interfaces is too tedious or time consuming. So to make things easier, people turn to mocking frameworks. This is just hides the root problem, which is the API is too complex.

A popular theme was the difference between "developer testing" and the testing everyone else does. Consistently through the discussion was this idea that developers only write unit tests. Testing against requirements, acceptance tests, integration tests, and all the other forms of testing are someone else's problem.

This underscores a fundamental misunderstanding in the unit testing community. Specifically, the assumption that all developers have a QA team to handle all the other types of testing. Unfortunately even multi-million dollar companies often have no QA resources at all, leaving all testing for the developer and end user.

The primary excuse for not supporting more types of tests was that of speed. Unit tests are already too slow and therefore there is no room for slower tests that include network communication. What was unfortunate is that no other options were considered.

For example, unit testing frameworks could be smarter. They could use the code coverage results to only rerun tests for code that actually changes. Changing one class shouldn't require you to rerun the entire test suite. The very term "unit test" implies you can test just a small subset.

Another possibility not being discussed is leveraging distributed programming. The code and tests can be quickly uploaded to several servers and executed across them. We already have all the technology we need in from the work done in continuous integration.

Early in the session it was acknowledged that databases were being neglected. Most database developers have little or no concept of unit testing and there are few if any tools to support them. More troubling, there are not even being asked to come to the table. Unfortunately that was all that was said about them. At no point did the panel offer any real options for correcting the social problems.

On a positive note, there was some discussion about using modeling tools to make unit testing easier. There are a lot of options here such as starting at the contract level. The contracts would then be used by code generators to write the actually tests. Obviously this won't be a 100% solution, but it will reduce the pain for common scenarios.

Another promising idea was that of delta state management. Most people assume the test account starts with $100 and after the transaction he has $80. The alternative is to first read the current balance, then check to see if it was reduced by $20. This way a full reset of the test environment is not needed for each run.