Agile techniques recommend having adequate unit and acceptance tests to build a robust test harness around the application. However, in the real world, not all applications are fortunate enough to have a test harness. In an interesting discussion on the Agile Testing group, members suggested ways to test applications which do not have any automated tests.
Asad Safari started the discussion when he mentioned that his application did not have any tests, the developers on the team were not familiar with unit testing and the team was running against a 3 week deadline to test the application. He was seeking suggestions to test under the given constraints.
Phlip responded that he had been in this situation several times and recommended the following,
- Add an optional module that drives your application with random or canned input.
- Add logging to your program which spits out the error messages and assertions in a log file
- Write one big fat unit test that calls all of your program, feeds it that input, and scrapes the log.
- Write one big fat assertion that says the log shall have no errors.
- Add an exception to that assertion for each error the log does have.
- Now start burning-down the errors. Each time one goes away, remove its exception.
Phlip recommended that under the umbrella of this huge test, the team could start writing smaller focused tests, as and when the time permits. He also suggested that, though the team could just sit over the spills for the next 3 weeks however, the time to start writing and executing small unit tests is now.
Adam Sroka agreed to the suggestion and added,
Yep, most teams will respond to poor quality by slowing down and producing less value, which doesn't actually do anything for quality. We need a more pragmatic solution. … The fallacy is that testing now isn't valuable because we can't do it as completely as if we had done it from the start. We can't, but it is still valuable.
Unconvinced with the thoughts, Brian Spears added that Agile is not magic and it might not be possible to come up with a solution in a matter of 3 weeks. He said,
Agile it not magic. The solution to this kind of emergency situation, when there is one, is a whole lot of long hours, which is clearly not an Agile solution.
Adam countered this argument by suggesting that most teams adopt Agile once they get into a situation like this. This is the best chance for teams to be pragmatic and take the first steps towards making the software better.
Annette suggested that the current situation is ripe for doing hours and hours of manual testing as automated testing at this stage would be time consuming. The recommendation was to start with the high profile and revenue dependent features of the application. Annette also recommended the book, Agile Testing by Lisa Crispin and Janet Gregory.
Charles Bradley made a similar suggestion with a conditional promise in advance. He suggested,
Your time is limited, so maximize the ROI in whatever way seems best from a business perspective. Manually test the hell out of it, and try to get the decision makers above you to agree that they will NEVER EVER DO THIS TO YOUR TEAM again... and instead, they will PLAN IN time(and money) to automate tests... like AS SOON AS work on the next release begins or maybe even as post-release bug fixing begins.
Thus, the current situation might not be the best for writing an entire test harness and the team might be better off with manual testing, This however does not undermine the importance of having a proper test harness at the first given opportunity. As Jonathan Rasmusson put it,
All you can do is fix the bugs, and then manually test as best you can before you go live. That's about all you can do at this point. The bigger and much more important question is what you do the day AFTER you hit your three week deadline.