There will be many times in our careers when we support legacy code. Sometimes, in a new job, modifying legacy code is one's first assignment, or perhaps the company reorganizes and a legacy product ends up among an Agile team's assets. In fact, to stretch the definition a bit, we could consider any code written before today to be legacy code. Both situations are challenging and potentially risky if a team doesn't have a strategy in place for handling legacy code.
Incremental Test Driven Development is fine for brand new code... but how does one get started with a code base that doesn't have a regression suite? The traditional approach is to start making changes while doing your best to avoid unintentional collateral damage. Unfortunately, because the code is unfamiliar, you aren't sure what's really going to happen when you change a data structure or update a variable. This leads to interesting kluges when developers re-invent the wheel rather than refactoring code not supported by unit tests, bloating the code and making it still harder to maintain.
How does one avoid these pitfalls of working with existing code? Jared Richardson, author of ShipIt!, says in this exclusive InfoQ arcticle:
"Rather than wandering into this minefield blindly, let's create a plan of attack. Don't just start making changes and hope that everything still works. Instead, take aim and hit it out of the park with a 'BAT': Build, Automate, and Test. ...In addition to this article on testing legacy code, Jared has made a sample chapter of Ship It! available for InfoQ readers.
The BAT approach will ensure that your code continues to work the way you want it to work. It quickly catches unintended side effects and helps you to eliminate them."
Community comments
BA is great, T may be awful
by Alex Popescu,
Re: BA is great, T may be awful
by Deborah (Hartmann) Preuss,
Re: BA is great, T may be awful
by Alex Popescu,
Re: BA is great, T may be awful
by jared richardson,
Re: BA is great, T may be awful
by Les Walker,
Re: BA is great, T may be awful
by Lukasz Szyrmer,
Writing the tests is the hard part.
by Michael Burke,
Re: Writing the tests is the hard part.
by jared richardson,
Re: Writing the tests is the hard part.
by Michael Burke,
Re: Writing the tests is the hard part.
by Alex Popescu,
Re: Writing the tests is the hard part.
by jared richardson,
Re: Writing the tests is the hard part.
by Alex Popescu,
Re: Writing the tests is the hard part.
by jared richardson,
My case study for a legacy web application
by Dan Bunea,
Re: My case study for a legacy web application
by Demetrius Nunes,
Re: My case study for a legacy web application
by jared richardson,
Re: My case study for a legacy web application
by Alex Popescu,
Re: My case study for a legacy web application
by Demetrius Nunes,
Don't forget about legacy databases too
by Scott Ambler,
BA is great, T may be awful
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
All these are very "sane" steps to be done while working with legacy code. Unfortunatelly, in a lot of situations the T(esting) part will be very hard to be done even when using Smoke testing. Add to this some ASAP management requests for bug fixes or new requirements and you may be dead pretty soon :-).
./alex
--
.w( the_mindstorm )p.
Re: BA is great, T may be awful
by Deborah (Hartmann) Preuss,
Your message is awaiting moderation. Thank you for participating in the discussion.
Alex,
what are you envisioning, when you say testing will be very hard?
- Regression suite run times?
- Ability to automate?
It's not clear what problem you are thinking of ...
thanks
deb
Re: BA is great, T may be awful
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
I agree that it's often difficult to get management to sign off on a formal test automation strategy and commit time to it. In fact, it's often impossible. That's why I suggest you try Defect Driven Testing. From the article...
A great testing strategy is Defect Driven Testing. Every time you find a bug in the system, add a new test that covers that defect. While you are adding the specific test, look for other tests you can add that are close but not quite the same. Over time, this strategy provides good coverage in the areas of the product that need coverage the most.
Just like the XP practice, you add a test every time you encounter a bug. It's a lot easier to carve out enough time to add a test as you go.
This has the added advantadge of covering the areas of your code base where the highest profile bugs were discovered. They may not be the worst bugs in the system, but they are the ones that were reported.
It does take a lot longer to get testing coverage, but I've found the resulting tests are very focused and effective.
Jared
jaredrichardson.net
Writing the tests is the hard part.
by Michael Burke,
Your message is awaiting moderation. Thank you for participating in the discussion.
Apart from the 'defect-driven-testing' bit, which is a nice pithy name for a useful idea, this article says nothing that hasn't been said hundreds of times before - I don't think folks on InfoQ need to have 'continuous integration' explained to them yet again!
While the basic concept is sound enough, for any non-trivial system, the complexity comes in getting traction with your tests. From experience, you'll be facing a codebase that is poorly designed, not amenable to mocking, full of circular dependencies, dependent on a database with lots of pre-existing data, etc. This is where things get real complicated, real fast.
I'd be much more interested in people's insight on how to tackle these tarballs and bring them under control. Given that unit tests are used as a 'safety net' for refactoring design changes, how do we get the chicken before the egg in an incremental and safe fashion?
Any good books, articles, etc people would recommend?
Re: Writing the tests is the hard part.
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
It wasn't my intention to suggest mocking or unit tests. An automated test doesn't have to be a unit test.
Mock Client testing is usually implementing by coding an automated test against a product's API. Using a toolkit like JUnit does make it much easier, but it's certainly not a unit test. Move up to the highest level you can for a product (which is ideally just under the GUI or at the API of a subsystem) and put an automated test in place. This has the advantage of covering a lot of code... it's a very coarse-grained test. To balance out that large code coverage swath you need a very fine-grained build, which you get from the Continuous Integration.
I prefer to run my test suites against a completely live system, not a system of mock objects as well. If the database is too large to run in a reasonable amount of time you may need to create a smaller, yet representative, database. (DBUnit is a great tool for this.)
Jared
jaredrichardson.net
Re: Writing the tests is the hard part.
by Michael Burke,
Your message is awaiting moderation. Thank you for participating in the discussion.
Quite right and I apologise for the assumption - simply saw the article filed under 'Agile' and joined the dots (albeit incorrectly!)
It's good to hear people talking and writing about this less-sexy and politically-correct style of testing that does indeed happen on larger projects. Case studies, patterns, recommended tools etc are all of interest to me (haven't looked at DBUnit in a few years and am pleased to see its capabilities have grown considerably)
cheers
mike
My case study for a legacy web application
by Dan Bunea,
Your message is awaiting moderation. Thank you for participating in the discussion.
I recently was in the situation where I needed to extend a legacy web application, written in java for a client 2 years ago, where the domain was very strange (physics and medical measurements) having to keep the existing functionality in place while adding the extras.
I wrote about the way I could do it, using Selenium (www.openqa.org/selenium) at: danbunea.blogspot.com/2006/05/refactoring-legac...
My way was to record some web tests with Selenium IDE, that would need to run after I did my changes. In order to do that I needed to have a known state of the database, so the tests run exactly the same before and after the changes. My method proved very succesful for this particular case.
Re: Writing the tests is the hard part.
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
Unfortunately, I see a lot of places where this cannot be done. A lot more places than places where you can finally get an unit test done (in fact for other than simple applications, these tests will be a lot more difficult to be created if even possible). I don't think that anybody would like to run some tests on real, real-time air traffic data.
Speaking about Defect-Driven-Testing and others, I wasn't commenting about the strategy itself: which is, as I said, a very sane recommendation. I was rather pointing to the fact that whatever the strategy is, putting in a test (whatever type) will be very difficult on a "poor" legacy code. And I strongly believe, that in most cases the easiest way to do it is to go deeper than to go higher level.
./alex
--
.w( the_mindstorm )p.
Re: BA is great, T may be awful
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
I am speaking about writting/creating the tests. All other things are much easier than this step. There are lots of legacy applications for which it will be really complex to sneak in your unit tests at least. Not to talk about high-level testing which will require setting up a testing environment (if even possible) that may take lots and lots of time and effort.
./alex
--
.w( the_mindstorm )p.
Re: Writing the tests is the hard part.
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
I should have emphasized this bit "... Move up to the highest level you can..." In a really bad code base, that might be unit tests. I've found that you can usually move up a bit higher though if you keep an open mind. I've done this (or helped get this rolling) on a variety of applications.
So look at how high you can go. If it's a unit test, fine. Add a unit test to cover this bug. However, when the next bug comes along, can you go higher? Be agile not rigid and adjust as you go.
So I think we agree on this point.
But there are no silver bullets. There is no single answer to every single situation. I try to take ideas like this use them where appropriate.
I'm surprised you'd say this unless you work exclusively on exotic applications (like real-time air traffic control). I'd be curious to hear about places that you think no tests other than unit tests can be written. Or can this code even be unit tested? Do you drop back to completely manual testing?
It's also quite possible that I've just not explained the idea very well. :) This is an idea that comes out of experience, not theory. It has worked very well for me in a variety of environments on a variety of projects.
Really? I'd think you'd love to be able to test against a data source like that before your system goes live. Perhaps you can't do this for every project, but it doesn't mean we don't try to hit that mark.
There will always be an exoctic situation that can be used as a counter-example to any idea or practice, but that doesn't mean that the technique can't be used by 90 percent of the software projects in the field.
So can you share some of your strategies for creating automated tests? Is that something you try to do when you pick up a legacy project?
Jared
jaredrichardson.net
Re: Writing the tests is the hard part.
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
Good to see we agree on "move as high as you can" part, because this was one of the things that I think it must be clear from the beginning.
I will give you a few example where creating "next level" automated tests would have been a lot more complex:
1. an AOP framework (e.g. AspectJ, AspectWerkz): the next level testing would represent creating different usage scenarios and get assured that the framework makes this code behave according to the needs (even formulating the requirements for this next level already start sounding fuzzy). Instead, through unit code, we can get sure that the bytecode is correct. Upon receiving bugs, we add the code generating the bug as a test (and this indeed matches the "next level" idea), but than after investigating we try to have more unit tests for covering scenarios we identified by the original code.
2. a highly distributed application for moving data from local centers to central place: the next level testing would have mean to recreate an environment to test how the special protocol is behaving and what problems may happen with the real application topology. We finally did this too, because otherwise it would have been quite impossible to deliver and guarantee for the application, but it took us a lot of time to do it (we had no access to data sources, we had no access to that custom network to understand its weak points, etc.).
3. a site builder like app: the next level testing would have mean to register/replay different usage scenarios and make sure these were run correctly. At those times, there was no selenium, or tools like this, and it would have meant to develop in-house such a tool. The budget for this was never approved (and I still believe they were right not doing it). Instead we used lots of unit tests, a small QA team and another small team of testers that were producing some output that was run through some validation tool.
4. fixing/porting mainframe Cobol-based apps, fixing/porting special CRM solutions: there was impossible to create any unit test at all when working on the original application (due to lack of environment and due to policies regarding access to the application machines).
And I think I have more, but I don't want to bore everybody :-). Probably, the examples I gave may be considered exotic. Unfortunately, I don't think that every application on this world is a web application or some simple desktop one :-). I would like to hear from you what different types of apps you have succesfully driven this process on a "next level".
Bad formulation on my side: I would love to be able to do it, but it is sometimes completely impossible.
Unfortunately, till now I haven't found the silver bullet, so I don't have a receipe for it :-). I am approaching each and every project as open mind as possible and see there what can I do. That's why, I said in the first place that these recommendations are very "sane", but unfortunately not everybody will benefit or may be able to use them.
BR,
./alex
--
.w( the_mindstorm )p.
Don't forget about legacy databases too
by Scott Ambler,
Your message is awaiting moderation. Thank you for participating in the discussion.
Legacy code is only one half of the equation: we also need to deal with legacy databases too. At www.agiledata.org/essays/legacyDatabases.html I summarize some of the issues which you're likely to run into, and it ain't pretty.
Sadly, most organizations have given up on legacy databases, assuming that they're too hard to deal with and have therefore adopted a "let's try not to make it any worse" strategy. This of course is a strategy doomed to failure, but hopefully the strategy won't fail while the current data folks are still in power. Sigh. As I point out at www.agiledata.org/essays/databaseRefactoring.html it is possible to safely refactor your database schema over time, and at www.agiledata.org/essays/databaseTesting.html to test relational databases.
- Scott
Re: Writing the tests is the hard part.
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
I think we're both running dangerously close to that line, but I'll try once more but probably let the thread go after this one. ;)
1) If I inherited an AOP framework, I'd test the altered code behaved properly. I wouldn't drop down to bytecode verification until I had a specific problem I couldn't expose from the functional/integration/mock client test.
2) A highly distributed application with disparate data centers... I'd duplicate the environment in house with a few routers, firewalls, and representative (but smaller) computers. I'd try to get representative databases on each and run my tests against the "live" system. Against the actual code. And it sounds like that's what you did...
3) For the site builder, replay isn't the only to test such an app. Several years ago I had a chance to be involved with a product that had a client. We built the product with good Model-View-Controller separation (google on "Tracer Bullets") and then coded our tests to target the APIs called by the client application. Worked like a charm. It keeps you under that fragile GUI layer. Of course, with legacy code, you can't do that.
4) Fixing/porting Cobol apps. I fail to see your point here. Unit tests weren't appropriate... wouldn't writing Mock Client Tests that ran completely externally been the best way to be sure you didn't break the functionality?
No, but there's a lot of them out there...
An enterprise calendaring application. An N-tier bio-tech application. An enterprise platform encapsulation layer.
Any program in use is consumed at some level. If you're lucky it's a clean API. If not, there is at least a UI you can test, even if it's a console.
My strategy is to get a basic MCT suite in place to cover the basic, generic usage scenarios. Then, as additional bugs are found, try to add more. If you can't more at the highest level, drop down. Can you test the entire subsystem? Package? Unit test? You gotta trust the person doing the work to find the right place.
And in that topic... blogs.pragprog.com/cgi-bin/pragdave.cgi/Practic...
Jared
jaredrichardson.net
Re: My case study for a legacy web application
by Demetrius Nunes,
Your message is awaiting moderation. Thank you for participating in the discussion.
What about GUI apps?
I´ve successfully implemented CI thru a legacy Swing/Java application with many benefits, the only missing point being the automated tests, since we did not find a suitable tool to add GUI testing to it (which is the only viable way to go at this point).
Re: My case study for a legacy web application
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
You mentioned that you implemented the application. Ideally, you'd identify the routines called by your GUI components, and create your tests to use those underlying methods, not the GUI.
For instance, your login dialog probably calls a routine called Login(String username, String password). Code a Junit test that invokes the Login method with a good user name and password. Then, while you're in there, create some Test Jazz.... create variations on a theme. Check that a bad user name, a bad password, and both, all fail appropriately.
By writing your tests to target the code under the GUI you accomplish two things. First, you avoid coding around the fragile GUI layer for your tests. The GUI tends to change very frequently so any automated test driving your GUI will also change frequently. So where possble I drop under the GUI. It's not always possible with legacy code, but you mentioned you wrote this app.
The second thing you've done is forced (or identified) a separation in your GUI code and your logic code. You want your GUI code to harvest the information from the dialog and then feed it to the Login routine. This helps encourage good MVC habits (en.wikipedia.org/wiki/Model-view-controller). You don't want the same code that creates the dialog making calls to the server, validating users, etc. This ends up being a proverbial big ball of mud.
If the GUI is a thin shell around a very well-tested lower layer, GUI testing becomes trivial and the remainder of your system can be tested automatically.
The first project I worked on where we wrote Mock Client tests on purpose was exactly this type of project (a Swing application). After an a few rounds of adding tests, the coders on the GUI work added the separation between the GUI and the logic because it made their test creation easier, not because it was a good MVC pattern. :) But we had clean code and a well-tested system at the end of the day.
Jared
jaredrichardson.net
ps: if you just feel you have to automate the GUI, there are a few toolkits available, like Abbot swik.net/Abbot
Re: My case study for a legacy web application
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
Unfortunately, as Demetrius points out, the toolkits for testing GUI are pretty basic. Till now I haven't found one that would really do things simpler when trying to automate GUI tests.
I agree that all other things will help eliminating some of the problems, but GUI is about human interaction and this brings to the scene a lot more things to be tested than we can do on the code level.
BR,
./alex
--
.w( the_mindstorm )p.
Re: My case study for a legacy web application
by Demetrius Nunes,
Your message is awaiting moderation. Thank you for participating in the discussion.
Jared and Alexandru,
After posting the message here, I took another look around for GUI automation tests toolkits and found one the I could put to work in a matter of minutes.
It's called Jemmy (jemmy.netbeans.org/). The greatest advantage for me is that its tests can be written as regular Java code (not some weird XML-markup like in Abbot) and also be run as JUnit tests, which is then very easy to integrate with CruiseControl.
Also, it allows GUI testing without an actual display (meaning you can run the tests as a background service), because it manipulates the AWT event queue directly (instead of using awt.Robot as most toolkits do).
Anyway, also about Jared suggestions on MVC separation and testing the code under the GUI. That makes sense if you start doing it that way from the start (I did not wrote the app from the start, I´ve inherited it!), but after 250.000 lines of code being written, it is pretty damn hard to do.
So, that´s why I am focusing on automating GUI testing to get the best code coverage one could expect at this point without having to do major refactorings.
Re: BA is great, T may be awful
by Les Walker,
Your message is awaiting moderation. Thank you for participating in the discussion.
Writing a test per defect requires a significant investment in time even after you've put a testing framework in place. If you can't demonstrate a payback for that investment in short order, any automated testing initiative will be dead long before you can show any results.
Defect-driven testing isn't focused enough. Even if you do throw a couple of other tests in there while you're testing for one defect, the testing is too spread out and the coverage to thin. The results will not be as significant as if you tested a small set of features or module at a high coverage level. At that point you start to see the payback from eliminating manual testing because people are more confident in the automated tests.
If you force people to test only defect fixes I fear that you'll most likely turn them off to automated testing.
High level testing is good if your development cycle is months long. Those automated tests can take you from monthly to daily. However, on a large system you're going to have problems getting a developer to run those tests on their machine as part of their own dev cycle. To get inside that cycle you're going to have to have code-based tests that don't require a full build and "deployment" -- whatever deployment means in your particular context. That's going to require refactoring which I agree is difficult. But at some point, the high-level testing is going to cease to bring the returns that it did in the beginning.
Re: BA is great, T may be awful
by Lukasz Szyrmer,
Your message is awaiting moderation. Thank you for participating in the discussion.
I disagree, Les. There is a concept of bug clusters, which is discussed in detail on www.youtube.com/watch?v=zXRxsRgLRZ4
If you find one bug, you are more than likely to find a number of bugs in the same area. I discuss this a bit, along with implications on my blog post on the 80/20 rule in software: softwaretrading.co.uk/2013/02/23/do-less-of-wha...