Legacy Code is Inevitable
There will be many times in our careers when we support legacy code. Sometimes you accept a new job and the legacy code is your first assignment, or maybe your company reorganizes and this product ends up in your lap. For whatever reason, it happens. You wanted to code something new and shiny but instead you are now the owner of a new-to-you and completely unfamiliar block of code. The code appears to be rather intricate, is unfamiliar and you now have to wade in.
In fact, if I can stretch the definition a bit, you can consider any code written before today to be legacy code. Have you ever tried to revisit code that you wrote six months ago? It's not always as easy as you'd hope to support your own code, much less someone else's. Both situations are challenging if you don't follow some basic guidelines.
The traditional approach is to start making changes while doing your best to avoid unintentional collateral damage. Unfortunately, because the code is unfamiliar, you aren't sure what's really going to happen when you change a data structure or update a variable.
Rather than wandering into this minefield blindly, let's create a plan of attack. Don't just start making changes and hope that everything still works. Instead, take aim and hit it out of the park with a "BAT".
Here's a three-pronged attack you can use to attack the problem. Build, automate, and test. Use this BAT for your legacy code and create a safety net for yourself. The BAT approach will ensure that your code continues to work the way you want it to work. It quickly catches unintended side effects and helps you to eliminate them.
I'd like to challenge you to look at how you handle your legacy code in light of the BAT approach. See how your day-to-day work compares and see if you need to approach your work differently.
Build
The first problem to address is the build. It's difficult to ship and test a product unless you can reliably build it. Figure out how to cleanly build the product on your desktop and then script the process.
Occasionally this is a non-issue, but builds usually aren't nearly as clean as they should be. Often builds are limited to a single machine or a special environment. When teams pass code from owner to owner, it tends to accumulate odd build requirements. Each owner adds his or her own special case into the mix. There have been too many cooks in the kitchen by the time you inherit the mess.
A complicated build can trigger cascading problems to the entire product.
When a thing is difficult, people do it less often. When a build is difficult, people build less often. It's just human nature. The ability to run a clean build often becomes a dark art mastered by only a few people in your shop. No one wants the task because it's so difficult and painful.
Since you can't test what you haven't built, testing becomes less frequent. When people finally run their tests, they find more bugs... infrequent testing gives bugs more time to accumulate. If you are running tests daily, you'll only have one day's worth of bugs to report. However, if you wait six months to test, you'll have a lot more issues to pin down.
So your testing becomes burdensome. The testers get tired of all the work in a testing cycle so they try to avoid testing. Entering dozens or hundreds of bugs is boring work that no one enjoys.
Developers start to dread the testing cycle because they feel bombarded and attacked by the all the bug reports. So the developers start resenting and harassing the testers, which makes the testing cycle every more painful. It's a very destructive feedback loop.
The complicated build causes problems for the entire product lifecycle, so be sure your build is clean.
When anyone can build, anyone can test. Testing is run more frequently, leading to smaller groups of bug reports. Having less work to do at a time is less of a chore. Anyone will move a bucket of paint and not think twice, but ask someone to move five hundred buckets and see what they say.
Your goal is to create a clean build that runs on any development machine and is easy to maintain. Use a build scripting tool or language, like Rake, Ant, Maven or Nant. These high-level build languages let you focus on building your application, instead of build, language, or platform details.
When you can build your product with a single command (like "ant all"), you can move on to the next step. Be sure to test this on more than one machine.
Automate It
Now that you can build the product on an arbitrary machine automatically, let's automate.
The goal is to automate the entire build and test cycle on a clean machine with an absolute minimum of human intervention. It's not always possible to have everything completely automated but we want to reach a place where we script everything that can be reasonably automated.
Sometimes it's easier to install and configure a supporting piece of the software than to write a script to do it automatically. Applications that you install only once are prime candidates. Things like compilers, runtime libraries, and pre-existing data sets fall into this category. Don't try to reproduce the data set that takes you two hours to recreate. However, if you can rebuild a representative data set in thirty seconds, you should build from scratch. The benefits of starting with a clean, known data set are immense, but not always practical. Don't stretch a fifteen-minute test run into an hour by rebuilding all of your data.
Be sure to document any manual steps thoroughly and keep all the instructions around for anyone else that might want to duplicate the environment.
On the other hand, why should you sit around and watch complete builds all day long? Your time is valuable. Your IDE probably handles incremental builds and small unit test runs for you anyway. In most cases, this partial coverage is good enough. Having developers run a set of Smoke Tests, targeting active code areas, will cover most situations.
Smoke Tests
A Smoke Test is a short collection of tests that target the areas of the code that are actively being changed. Your Smoke Tests don't even try to exercise the entire product. A good set of Smoke Tests will be rotated... they aren't permanent. When you starting working on a different product area, move out the old Smoke Tests and cycle in others. You can select tests from your complete testing suite tests to run in your Smoke Test suite. I usually just add them to an Ant target (called smoke-test) that runs selected tests.
We still need a clean build and complete test run periodically. This is how we verify that no one forgot to commit a changed file or broke the product in an unexpected way. Smoke Testing often misses these types of breaks, so to keep the product in the best shape possible, we need to run the entire suite fairly frequently.
Instead of asking every developer to build the entire system from scratch and run every test available five times a day, tasks that can take quite a while, we're going to ask another computer to do that for you. Since we're tasking a computer to perform the automated build and test run, there's no reason we it can't run more than once a week. In fact, there's no reason this cycle can't run after every code change.
What's the best way to set up this type of automation? The quickest and easiest way is to use a Continuous Integration product. A CI product watches your code, builds after each change, runs your tests, and then notifies everyone involved.
Continuous Integration graphic
A CI system creates a fast feedback loop. When you change your code, you'll find out if anything was broken before you forget why you made the changes. (For more information on CI systems, visit my CI page at http://www.jaredrichardson.net/ci.html).
It's all about the pace of your development and keeping you moving. Having to revisit code edits from last week or last month is a poor way to keep things rolling. Catch and fix your problems within the hour and keep your team moving forward.
Here's how the system works. You edit the code on your desktop until you're sure you've got the feature completed or the bug fixed, so you put your changes into your source code management (SCM) system. Your Continuous Integration system is monitoring your SCM and sees that code has changed. It then checks out the product, builds it and runs all your tests. You'll immediately get an email telling you whether or not the compilation and test run passed or failed.
The concept is fairly simple but it's very powerful in action.
Test
The final section of our BAT is test automation. Now that our CI system is building the product and running our automated tests, we need to be sure that our tests cover the product properly. After all, a test that doesn't test the functionality the customer uses is pretty useless from the customer's point of view.
First, try to understand how your product is used... this can often be a challenge all by itself with a legacy product. You may even want to create a set of scenarios for customer usage. For example, you could have a scenario for creating daily reports, doing daily data imports, or adding new customers.
You may also wish to have categories of users, called "user personas". You could have "Joe the Power User", "Mary the System Administrator", or "Fred the New User". Each persona uses your product differently just like a power user uses the product differently than a rank novice.
Next create Mock Client Tests to duplicate the most common customer usage scenarios.
Mock Client Tests
A Mock Client Test isn't a special testing framework. It's a test created to ensure your basic, expected functionality doesn't break. Quite often with legacy code, you'll change something in one part of the system and not know that it affects other areas of the product. I once worked on a product where changing the communication protocol affected the GUI component layout. Your Mock Client Tests, inside your CI system, are your insurance policy against accidental change. Test your product the way that you expect it to be used and you'll have a solid product baseline. Add tests to cover the more interesting cases later, as you encounter them.
A great testing strategy is Defect Driven Testing. Every time you find a bug in the system, add a new test that covers that defect. While you are adding the specific test, look for other tests you can add that are close but not quite the same. Over time, this strategy provides good coverage in the areas of the product that need coverage the most.
Regardless of how you choose to add tests, make test addition a priority. Having a basic test suite in place is essential if you plan to make any changes to the product.
Your final step is getting the tests into your Continuous Integrations system.
You get feedback very quickly on any problems when your automated tests run in your Continuous Integration environment. Every time you change the code you get a "from scratch" build and a complete test run. Most developers get addicted to this additional coverage very quickly and soon depend on this "extra team member."
- Write Scenarios
- Create Mock Client Tests
- Use Continuous Integration
Summary
Build, automate and test (BAT) is good advice for anyone writing code, but it's an especially good formula for anyone inheriting legacy code. The ability to refactor with confidence is essential. I find it very difficult to be productive if I'm constantly looking over my shoulder to see what I'm breaking. A good test suite looks over my shoulder for me and let's me focus on the performance improvements I'm trying to make.
Remember, never change legacy code until you can test it, and never test purley by hand unless you have no other option.
Don't fear legacy code but handle it properly. Hit it with this BAT and you'll win every time.
About the author
Jared Richardson, co-author of Ship It! A Practical Guide to Successful Software Projects, is a speaker and independent consultant who specializes in using off-the-shelf technologies to solve tough problems. With more than 10 years of experience, Jared has been a consultant, developer, tester, and manager, including Director of Development at several companies. Until recently he managed a team of developers and testers at SAS Institute, Inc., and deployed a Continuous Integration System for nearly 300 projects, 5 million lines of code, and over 1,800 developers. He also led a company-wide effort to increase the use of test automation. Jared can be found online at http://www.JaredRichardson.net.
Click here to read an exclusive sample chapter of Ship It! A Practical Guide to Successful Software Projects, available only on InfoQ!
Community comments
BA is great, T may be awful
by Alex Popescu,
Re: BA is great, T may be awful
by Deborah (Hartmann) Preuss,
Re: BA is great, T may be awful
by Alex Popescu,
Re: BA is great, T may be awful
by jared richardson,
Re: BA is great, T may be awful
by Les Walker,
Re: BA is great, T may be awful
by Lukasz Szyrmer,
Writing the tests is the hard part.
by Michael Burke,
Re: Writing the tests is the hard part.
by jared richardson,
Re: Writing the tests is the hard part.
by Michael Burke,
Re: Writing the tests is the hard part.
by Alex Popescu,
Re: Writing the tests is the hard part.
by jared richardson,
Re: Writing the tests is the hard part.
by Alex Popescu,
Re: Writing the tests is the hard part.
by jared richardson,
My case study for a legacy web application
by Dan Bunea,
Re: My case study for a legacy web application
by Demetrius Nunes,
Re: My case study for a legacy web application
by jared richardson,
Re: My case study for a legacy web application
by Alex Popescu,
Re: My case study for a legacy web application
by Demetrius Nunes,
Don't forget about legacy databases too
by Scott Ambler,
BA is great, T may be awful
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
All these are very "sane" steps to be done while working with legacy code. Unfortunatelly, in a lot of situations the T(esting) part will be very hard to be done even when using Smoke testing. Add to this some ASAP management requests for bug fixes or new requirements and you may be dead pretty soon :-).
./alex
--
.w( the_mindstorm )p.
Re: BA is great, T may be awful
by Deborah (Hartmann) Preuss,
Your message is awaiting moderation. Thank you for participating in the discussion.
Alex,
what are you envisioning, when you say testing will be very hard?
- Regression suite run times?
- Ability to automate?
It's not clear what problem you are thinking of ...
thanks
deb
Re: BA is great, T may be awful
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
I agree that it's often difficult to get management to sign off on a formal test automation strategy and commit time to it. In fact, it's often impossible. That's why I suggest you try Defect Driven Testing. From the article...
A great testing strategy is Defect Driven Testing. Every time you find a bug in the system, add a new test that covers that defect. While you are adding the specific test, look for other tests you can add that are close but not quite the same. Over time, this strategy provides good coverage in the areas of the product that need coverage the most.
Just like the XP practice, you add a test every time you encounter a bug. It's a lot easier to carve out enough time to add a test as you go.
This has the added advantadge of covering the areas of your code base where the highest profile bugs were discovered. They may not be the worst bugs in the system, but they are the ones that were reported.
It does take a lot longer to get testing coverage, but I've found the resulting tests are very focused and effective.
Jared
jaredrichardson.net
Writing the tests is the hard part.
by Michael Burke,
Your message is awaiting moderation. Thank you for participating in the discussion.
Apart from the 'defect-driven-testing' bit, which is a nice pithy name for a useful idea, this article says nothing that hasn't been said hundreds of times before - I don't think folks on InfoQ need to have 'continuous integration' explained to them yet again!
While the basic concept is sound enough, for any non-trivial system, the complexity comes in getting traction with your tests. From experience, you'll be facing a codebase that is poorly designed, not amenable to mocking, full of circular dependencies, dependent on a database with lots of pre-existing data, etc. This is where things get real complicated, real fast.
I'd be much more interested in people's insight on how to tackle these tarballs and bring them under control. Given that unit tests are used as a 'safety net' for refactoring design changes, how do we get the chicken before the egg in an incremental and safe fashion?
Any good books, articles, etc people would recommend?
Re: Writing the tests is the hard part.
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
It wasn't my intention to suggest mocking or unit tests. An automated test doesn't have to be a unit test.
Mock Client testing is usually implementing by coding an automated test against a product's API. Using a toolkit like JUnit does make it much easier, but it's certainly not a unit test. Move up to the highest level you can for a product (which is ideally just under the GUI or at the API of a subsystem) and put an automated test in place. This has the advantage of covering a lot of code... it's a very coarse-grained test. To balance out that large code coverage swath you need a very fine-grained build, which you get from the Continuous Integration.
I prefer to run my test suites against a completely live system, not a system of mock objects as well. If the database is too large to run in a reasonable amount of time you may need to create a smaller, yet representative, database. (DBUnit is a great tool for this.)
Jared
jaredrichardson.net
Re: Writing the tests is the hard part.
by Michael Burke,
Your message is awaiting moderation. Thank you for participating in the discussion.
Quite right and I apologise for the assumption - simply saw the article filed under 'Agile' and joined the dots (albeit incorrectly!)
It's good to hear people talking and writing about this less-sexy and politically-correct style of testing that does indeed happen on larger projects. Case studies, patterns, recommended tools etc are all of interest to me (haven't looked at DBUnit in a few years and am pleased to see its capabilities have grown considerably)
cheers
mike
My case study for a legacy web application
by Dan Bunea,
Your message is awaiting moderation. Thank you for participating in the discussion.
I recently was in the situation where I needed to extend a legacy web application, written in java for a client 2 years ago, where the domain was very strange (physics and medical measurements) having to keep the existing functionality in place while adding the extras.
I wrote about the way I could do it, using Selenium (www.openqa.org/selenium) at: danbunea.blogspot.com/2006/05/refactoring-legac...
My way was to record some web tests with Selenium IDE, that would need to run after I did my changes. In order to do that I needed to have a known state of the database, so the tests run exactly the same before and after the changes. My method proved very succesful for this particular case.
Re: Writing the tests is the hard part.
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
Unfortunately, I see a lot of places where this cannot be done. A lot more places than places where you can finally get an unit test done (in fact for other than simple applications, these tests will be a lot more difficult to be created if even possible). I don't think that anybody would like to run some tests on real, real-time air traffic data.
Speaking about Defect-Driven-Testing and others, I wasn't commenting about the strategy itself: which is, as I said, a very sane recommendation. I was rather pointing to the fact that whatever the strategy is, putting in a test (whatever type) will be very difficult on a "poor" legacy code. And I strongly believe, that in most cases the easiest way to do it is to go deeper than to go higher level.
./alex
--
.w( the_mindstorm )p.
Re: BA is great, T may be awful
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
I am speaking about writting/creating the tests. All other things are much easier than this step. There are lots of legacy applications for which it will be really complex to sneak in your unit tests at least. Not to talk about high-level testing which will require setting up a testing environment (if even possible) that may take lots and lots of time and effort.
./alex
--
.w( the_mindstorm )p.
Re: Writing the tests is the hard part.
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
I should have emphasized this bit "... Move up to the highest level you can..." In a really bad code base, that might be unit tests. I've found that you can usually move up a bit higher though if you keep an open mind. I've done this (or helped get this rolling) on a variety of applications.
So look at how high you can go. If it's a unit test, fine. Add a unit test to cover this bug. However, when the next bug comes along, can you go higher? Be agile not rigid and adjust as you go.
So I think we agree on this point.
But there are no silver bullets. There is no single answer to every single situation. I try to take ideas like this use them where appropriate.
I'm surprised you'd say this unless you work exclusively on exotic applications (like real-time air traffic control). I'd be curious to hear about places that you think no tests other than unit tests can be written. Or can this code even be unit tested? Do you drop back to completely manual testing?
It's also quite possible that I've just not explained the idea very well. :) This is an idea that comes out of experience, not theory. It has worked very well for me in a variety of environments on a variety of projects.
Really? I'd think you'd love to be able to test against a data source like that before your system goes live. Perhaps you can't do this for every project, but it doesn't mean we don't try to hit that mark.
There will always be an exoctic situation that can be used as a counter-example to any idea or practice, but that doesn't mean that the technique can't be used by 90 percent of the software projects in the field.
So can you share some of your strategies for creating automated tests? Is that something you try to do when you pick up a legacy project?
Jared
jaredrichardson.net
Re: Writing the tests is the hard part.
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
Good to see we agree on "move as high as you can" part, because this was one of the things that I think it must be clear from the beginning.
I will give you a few example where creating "next level" automated tests would have been a lot more complex:
1. an AOP framework (e.g. AspectJ, AspectWerkz): the next level testing would represent creating different usage scenarios and get assured that the framework makes this code behave according to the needs (even formulating the requirements for this next level already start sounding fuzzy). Instead, through unit code, we can get sure that the bytecode is correct. Upon receiving bugs, we add the code generating the bug as a test (and this indeed matches the "next level" idea), but than after investigating we try to have more unit tests for covering scenarios we identified by the original code.
2. a highly distributed application for moving data from local centers to central place: the next level testing would have mean to recreate an environment to test how the special protocol is behaving and what problems may happen with the real application topology. We finally did this too, because otherwise it would have been quite impossible to deliver and guarantee for the application, but it took us a lot of time to do it (we had no access to data sources, we had no access to that custom network to understand its weak points, etc.).
3. a site builder like app: the next level testing would have mean to register/replay different usage scenarios and make sure these were run correctly. At those times, there was no selenium, or tools like this, and it would have meant to develop in-house such a tool. The budget for this was never approved (and I still believe they were right not doing it). Instead we used lots of unit tests, a small QA team and another small team of testers that were producing some output that was run through some validation tool.
4. fixing/porting mainframe Cobol-based apps, fixing/porting special CRM solutions: there was impossible to create any unit test at all when working on the original application (due to lack of environment and due to policies regarding access to the application machines).
And I think I have more, but I don't want to bore everybody :-). Probably, the examples I gave may be considered exotic. Unfortunately, I don't think that every application on this world is a web application or some simple desktop one :-). I would like to hear from you what different types of apps you have succesfully driven this process on a "next level".
Bad formulation on my side: I would love to be able to do it, but it is sometimes completely impossible.
Unfortunately, till now I haven't found the silver bullet, so I don't have a receipe for it :-). I am approaching each and every project as open mind as possible and see there what can I do. That's why, I said in the first place that these recommendations are very "sane", but unfortunately not everybody will benefit or may be able to use them.
BR,
./alex
--
.w( the_mindstorm )p.
Don't forget about legacy databases too
by Scott Ambler,
Your message is awaiting moderation. Thank you for participating in the discussion.
Legacy code is only one half of the equation: we also need to deal with legacy databases too. At www.agiledata.org/essays/legacyDatabases.html I summarize some of the issues which you're likely to run into, and it ain't pretty.
Sadly, most organizations have given up on legacy databases, assuming that they're too hard to deal with and have therefore adopted a "let's try not to make it any worse" strategy. This of course is a strategy doomed to failure, but hopefully the strategy won't fail while the current data folks are still in power. Sigh. As I point out at www.agiledata.org/essays/databaseRefactoring.html it is possible to safely refactor your database schema over time, and at www.agiledata.org/essays/databaseTesting.html to test relational databases.
- Scott
Re: Writing the tests is the hard part.
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
I think we're both running dangerously close to that line, but I'll try once more but probably let the thread go after this one. ;)
1) If I inherited an AOP framework, I'd test the altered code behaved properly. I wouldn't drop down to bytecode verification until I had a specific problem I couldn't expose from the functional/integration/mock client test.
2) A highly distributed application with disparate data centers... I'd duplicate the environment in house with a few routers, firewalls, and representative (but smaller) computers. I'd try to get representative databases on each and run my tests against the "live" system. Against the actual code. And it sounds like that's what you did...
3) For the site builder, replay isn't the only to test such an app. Several years ago I had a chance to be involved with a product that had a client. We built the product with good Model-View-Controller separation (google on "Tracer Bullets") and then coded our tests to target the APIs called by the client application. Worked like a charm. It keeps you under that fragile GUI layer. Of course, with legacy code, you can't do that.
4) Fixing/porting Cobol apps. I fail to see your point here. Unit tests weren't appropriate... wouldn't writing Mock Client Tests that ran completely externally been the best way to be sure you didn't break the functionality?
No, but there's a lot of them out there...
An enterprise calendaring application. An N-tier bio-tech application. An enterprise platform encapsulation layer.
Any program in use is consumed at some level. If you're lucky it's a clean API. If not, there is at least a UI you can test, even if it's a console.
My strategy is to get a basic MCT suite in place to cover the basic, generic usage scenarios. Then, as additional bugs are found, try to add more. If you can't more at the highest level, drop down. Can you test the entire subsystem? Package? Unit test? You gotta trust the person doing the work to find the right place.
And in that topic... blogs.pragprog.com/cgi-bin/pragdave.cgi/Practic...
Jared
jaredrichardson.net
Re: My case study for a legacy web application
by Demetrius Nunes,
Your message is awaiting moderation. Thank you for participating in the discussion.
What about GUI apps?
I´ve successfully implemented CI thru a legacy Swing/Java application with many benefits, the only missing point being the automated tests, since we did not find a suitable tool to add GUI testing to it (which is the only viable way to go at this point).
Re: My case study for a legacy web application
by jared richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
You mentioned that you implemented the application. Ideally, you'd identify the routines called by your GUI components, and create your tests to use those underlying methods, not the GUI.
For instance, your login dialog probably calls a routine called Login(String username, String password). Code a Junit test that invokes the Login method with a good user name and password. Then, while you're in there, create some Test Jazz.... create variations on a theme. Check that a bad user name, a bad password, and both, all fail appropriately.
By writing your tests to target the code under the GUI you accomplish two things. First, you avoid coding around the fragile GUI layer for your tests. The GUI tends to change very frequently so any automated test driving your GUI will also change frequently. So where possble I drop under the GUI. It's not always possible with legacy code, but you mentioned you wrote this app.
The second thing you've done is forced (or identified) a separation in your GUI code and your logic code. You want your GUI code to harvest the information from the dialog and then feed it to the Login routine. This helps encourage good MVC habits (en.wikipedia.org/wiki/Model-view-controller). You don't want the same code that creates the dialog making calls to the server, validating users, etc. This ends up being a proverbial big ball of mud.
If the GUI is a thin shell around a very well-tested lower layer, GUI testing becomes trivial and the remainder of your system can be tested automatically.
The first project I worked on where we wrote Mock Client tests on purpose was exactly this type of project (a Swing application). After an a few rounds of adding tests, the coders on the GUI work added the separation between the GUI and the logic because it made their test creation easier, not because it was a good MVC pattern. :) But we had clean code and a well-tested system at the end of the day.
Jared
jaredrichardson.net
ps: if you just feel you have to automate the GUI, there are a few toolkits available, like Abbot swik.net/Abbot
Re: My case study for a legacy web application
by Alex Popescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
Unfortunately, as Demetrius points out, the toolkits for testing GUI are pretty basic. Till now I haven't found one that would really do things simpler when trying to automate GUI tests.
I agree that all other things will help eliminating some of the problems, but GUI is about human interaction and this brings to the scene a lot more things to be tested than we can do on the code level.
BR,
./alex
--
.w( the_mindstorm )p.
Re: My case study for a legacy web application
by Demetrius Nunes,
Your message is awaiting moderation. Thank you for participating in the discussion.
Jared and Alexandru,
After posting the message here, I took another look around for GUI automation tests toolkits and found one the I could put to work in a matter of minutes.
It's called Jemmy (jemmy.netbeans.org/). The greatest advantage for me is that its tests can be written as regular Java code (not some weird XML-markup like in Abbot) and also be run as JUnit tests, which is then very easy to integrate with CruiseControl.
Also, it allows GUI testing without an actual display (meaning you can run the tests as a background service), because it manipulates the AWT event queue directly (instead of using awt.Robot as most toolkits do).
Anyway, also about Jared suggestions on MVC separation and testing the code under the GUI. That makes sense if you start doing it that way from the start (I did not wrote the app from the start, I´ve inherited it!), but after 250.000 lines of code being written, it is pretty damn hard to do.
So, that´s why I am focusing on automating GUI testing to get the best code coverage one could expect at this point without having to do major refactorings.
Re: BA is great, T may be awful
by Les Walker,
Your message is awaiting moderation. Thank you for participating in the discussion.
Writing a test per defect requires a significant investment in time even after you've put a testing framework in place. If you can't demonstrate a payback for that investment in short order, any automated testing initiative will be dead long before you can show any results.
Defect-driven testing isn't focused enough. Even if you do throw a couple of other tests in there while you're testing for one defect, the testing is too spread out and the coverage to thin. The results will not be as significant as if you tested a small set of features or module at a high coverage level. At that point you start to see the payback from eliminating manual testing because people are more confident in the automated tests.
If you force people to test only defect fixes I fear that you'll most likely turn them off to automated testing.
High level testing is good if your development cycle is months long. Those automated tests can take you from monthly to daily. However, on a large system you're going to have problems getting a developer to run those tests on their machine as part of their own dev cycle. To get inside that cycle you're going to have to have code-based tests that don't require a full build and "deployment" -- whatever deployment means in your particular context. That's going to require refactoring which I agree is difficult. But at some point, the high-level testing is going to cease to bring the returns that it did in the beginning.
Re: BA is great, T may be awful
by Lukasz Szyrmer,
Your message is awaiting moderation. Thank you for participating in the discussion.
I disagree, Les. There is a concept of bug clusters, which is discussed in detail on www.youtube.com/watch?v=zXRxsRgLRZ4
If you find one bug, you are more than likely to find a number of bugs in the same area. I discuss this a bit, along with implications on my blog post on the 80/20 rule in software: softwaretrading.co.uk/2013/02/23/do-less-of-wha...