Managing Technical Debt
Technical Debt is widely regarded as a bad thing; that should be avoided or should be paid back as soon as possible.
Should you? We don't think so. First, we compare Technical Debt with financial debt, explain its similarities with Strategic Design and which stakeholders it surprisingly has. Then we list the various possibilities to identify the Technical Debt in your code, which you perhaps have to take care of.
Finally, we describe different ways that a project could pay back Technical Debt and which considerations must be made in order to decide if you should better repay, convert debt or just pay the interest.
What is Technical Debt
Developers have the choice to implement new features in two different ways: one is to do it quickly and messily, which will make future changes very hard. The other is a clean and smart solution, which takes longer to implement, but makes changes easier in the future (see also Martin Fowler). But why should the sponsors of a project accept higher costs for a clean implementation of a feature if the same feature, implemented with a messy solution, delivers the same functionality and costs less? Why should they spend money on automated test coverage? Tests are not features and therefore don’t deliver business value!
Although messy code or code without tests works perfectly for customers if it delivers the desired business value, this will lead to an uncontrollable code base, extremely specialised developers and eventually to an inflexible software product. A sufficient amount of messy code may bring a whole engineering department to a stand-still.
The metaphor ‘Technical Debt’ - Similarities and differences to ‘financial debt’
Ward Cunningham used the metaphor ‘Technical Debt’ for the first time in 1992 to communicate this problem with non-technical stakeholders. Code with low quality and no automatic test coverage can be compared with financial debt. This code is like a financial burden, which imposes on all stakeholders - not only on the developers - a debt incurring interest for the future. The principal amount is the cost of refactoring the codebase to a clean design, which allows easy future changes. The interest is the extra costs, which have to be paid in the future if the team has to work with a messy codebase instead of a good one.
Unlike financial debt, you don’t have to pay back Technical Debt. Sometimes paying it back would even be pointless: some pieces of code are rarely or never read or changed. Therefore Technical Debt also needs to take into account a probability of occurrence - how probable and how often will the messy code be touched in the future? Another difference to financial debt is that the originator of Technical Debt won’t necessarily pay back the debt himself, but rather the developers who maintain the codebase later.
Like financial debt, Technical Debt is not necessarily a bad thing. Going into debt to buy a house is responsible if you know how to pay it back. Buying lots of luxury items on your credit card knowing very well you’re not able to cover the bill usually ends up in a disaster. Concerning software Technical Debt might give an advantage of an early release and profit the organization more than it costs to pay back the Technical Debt. In finance, debt is a good thing, if the principal amount plus the interest is lower than the yield of the investment. The same is true for software. If you sacrifice internal quality for being the first on the market, it only pays off if the yield you make with this decision is higher than being later to the market with better internal quality. There is, however, a risk, because it is hard to estimate these benefits upfront since there is a degree of uncertainty.
Technical Debt and Strategic Design
The concept of strategic design by Eric Evans gives us an idea how to deal with Technical Debt. Strategic Design says that a system can’t have the same high level of quality throughout the system. Therefore a team can choose to either leave to chance which parts of the system have a good or a bad quality or to actively control the quality. Messy code, which is rarely read or touched and doesn’t implement important requirements, does not have to be absolutely perfect and therefore we don’t need to spend a lot of effort on refactoring it into great code. So the question is which parts of the code should have high quality? It’s possible that a piece of the implementation has a bad design without having a bad quality - if no good design is required for that piece of the implementation. This argument is of course questionable: while each particular piece of code doesn’t need to have good quality, in summary you might be creating an unmaintainable system.
Understanding Strategic Design and Technical Debt leads to better communication within a project and therefore to better decisions from the stakeholders of a software project. Developers might realize that not every requirement has to be implemented elegantly, if this requires too much effort. Also the customer understands that quick & dirty solutions lead to debt the project eventually has to pay back. Technical Debt consists of hidden quality problems, which like an iceberg might lead to a failure of the project once they surface e.g. through too many bugs it is most likely too late to fix the quality problems cost-effective.
Stakeholders of Technical Debt
Technical Debt concerns many stakeholders of a software project:
- Customers are annoyed by bugs or missing features due to low productivity.
- This leads to additional costs for the helpdesk, which annoys the people there, too.
- Increased development time and quality issues are also a problem for marketing.
- Bugs lead to frequent patches, which annoys the operations team.
- Many annoyed parties definitely don’t make the management happy - especially if there is bad publicity due to bugs, delays or security problems.
- Last but not least also the developers are suffering. No one wants to deliver bad work. Furthermore, no one wants to take over the bad work of others.
Technical Debt is unavoidable
But why don’t many projects do it right the first time and avoid Technical Debt? Assuming that developers aren’t lazy and are experienced with the relevant technology and patterns, than Technical Debt is almost always enforced by time pressure. The developers hope to gain a higher short-term productivity with a quick & dirty implementation and lower costs for the current release - well aware that they cause lower productivity and higher costs in future releases. The team, however, can never be really sure that it actually achieves a higher productivity or lower costs within the current release. Possibly, the shortcuts boomerang earlier than expected and the team has to pay the interest earlier than expected.
But we as developers usually don’t really want shortcuts or compromises in quality. Sometimes the external conditions are just that things within a release have to be done quickly, because otherwise there won’t be a next release. Or the team assumes that the code won’t be read or changed often. The previously discussed “Strategic Design” definitely allows compromises in quality. In that case good code might rather be “gold-plating”, an overkill.
Thus we see, that the Technical Debt metaphor helps all stakeholders to communicate the problem of poor quality that might otherwise go unnoticed until it is too late. The subsequent damage due to bad code quality and the increasing degeneration of the quality are very well represented by the metaphor of interest. But the metaphor also falls a bit short: not all Technical Debt must always be repaid. The team also doesn’t know exactly how much the debt is and when it needs to pay it back. The person who obligates the debt is not necessarily the one to repay it, but usually someone else. Fancy that in real life!
Identifying Technical Debt
A major problem of Technical Debt is that it’s not obvious. Anyone can see the amount of debt on an account statement. But how can a team actually recognize Technical Debt? What are the indicators?
- General “smells” that manifest themselves in statements from the team members: “the only one who can ever change this code is Carl”, “let’s just copy & paste this code”, “if I touch that code everything will break” or too many TODOs or FIXMEs in the code (nicely described by Nico Zazworka).
- Scrum-teams have a velocity. The software system has probably too much Technical Debt if the velocity decreases even though the team wasn’t changed and the external circumstances did not change,
- Software ages. One indicator for Technical Debt is when a system uses very old libraries, that are no longer being maintained or in a new version are much more productive (e.g. EJB 2 vs. EJB 3). In the worst case, the libraries are already so old, that the developers of the libraries no longer even support them.
- Automatically measuring Technical Debt is partially possible. Correctly configured, tools like Sonar, SonarJ or Structure101 find important violations of best coding practices. As this blog post shows, this is not the ultimate solution, but a very good one. With Sonar you ensure that developers follow important code metrics like appropriate class and method size or low cyclomatic complexity. Tools like Structure101 find structural problems. These include cyclic dependencies. If two elements depend on each other, any change in one of the elements potentially affects the other element. These two elements can only be changed together - although they actually should be separated and should be developed independently. Furthermore Structure101 finds complex code: it is all about the dependencies among the elements. A system is really hard to understand and therefore to maintain if its classes or packages have too many dependencies to other classes or packages. If a developer wants to make a change he has to understand many classes and packages and has to consider many dependencies. These problems point to key challenges in the architecture.
- Code coverage tools can detect how much code is really covered by automatical tests. This metric should be used with care, since there are no general guidelines. Generally a coverage of more than 90% is a good indicator that there are enough test cases available. Conversely, a coverage below 75% may indicate a serious problem.
- But there are also a lot of cases in which systems have Technical Debt, but it cannot be measured directly in the code, e.g. clumsy solutions, wrong choice of technology, a well-executed, but wrong design for a solution or simply evil hacks that cannot be detected easily by tools. In these cases Technical Debt must be measured differently: the bugs per release rise rapidly, the velocity decreases permanently or the team is under extreme stress at the end of a release.
- A very bad indicator are frequent problems in production. This means that the problems in the system are so extensive that reliable operation is no longer possible.
All of these indicators can be measured, but not always directly in the code. The sooner these problems are solved, the cheaper. If developers identify Technical Debt, communicate it, and nothing happens, sooner or later business people will feel these problems too. This statement clearly shows: Technical Debt is not a gimmick of developers who want beautiful code. It is a tangible cost factor and may be a risk for a project. Therefore Technical Debt must be made visible and manageable. As in real life, debt is not necessarily a bad thing, but it must be used consciously and properly. This means for software projects that paying back Technical Debt is a pure business decision. It must not simply a developer's decision.
How to manage Technical Debt?
We have seen that Technical Debt cannot be ignored. Even the non-technical management on the client side must have an interest in managing Technical Debt as smartly as possible to achieve the best balance between short-, mid- and long-term success. What can a team do to avoid wasting its time with unimportant beautification but still make meaningful business decisions for improving their code quality?
Coarse-grained statements such as “do nothing” or “if the code is unmaintainable anymore, develop anew” do not help.
We consider two promising approaches in this article, which have already found useful in several projects:
- Technical Backlog
- Include cost for Technical Debt in requirements estimation
- Prior to that, we want to discuss two other critical processes that probably make sense in certain project contexts:
- Buffer-tasks for refactoring
One question, which must also always be addressed in the discussion of Technical Debt: must Technical Debt ever be repaid? Frank Buschmann describes 3 strategies, which we address later on:
- debt repayment
- debt conversion
- just pay the interest
The team creates one buffer-task per release with e.g. 10% of the available time. Team members can record the time on that task for not yet scheduled refactorings. So it is used for yet unknown problems that might appear in the future. Such a buffer is very easy to schedule and use. However, it also poses the risk that time is wasted on unimportant work. A buffer doesn’t force anyone to consider whether the time is spent for useful refactorings or not. The developers just record their time on the buffer task. Most likely the time of the buffer may not be optimally utilized - and especially deciding which refactorings will be done, although that should really be a business decision. Using buffer-tasks unfortunately means that what should be done is not really defined.
Some teams do a purely technical release to improve the codebase from time to time. This approach is only useful if a list with the really necessary refactorings already exists. Otherwise the team risks wasting time on unimportant refactorings. This approach must also be supported by the business side because it might delay new features. Of course this requires that business people understand Technical Debt. You should think about a pure technical release to clean up the codebase and to rework the architecture if some major effort is needed. For example the same parts of the code might always cause problems during development or operations, the current architecture might no longer fit the current requirements. Such problems cannot be solved within small refactorings. A cleanup release allows extensive changes.
It makes no sense to do a cleanup-release after a very hectic and time-critical release that has created a lot of Technical Debt. There is only little experience with the new code base, so no one can say which part of the code needs to be improved. The danger in changing code which does not really need improvement.
The technical backlog is an established best practice to define purely technical work packages. Tasks for this purpose are created in a task tracker or requirements management tool. Each task has a brief description of the technical change to be made, why this technical change is important for the project and in which part of the code the technical change has to be performed. As with any other task, we need an estimate of how long it takes to develop a good enough solution. In addition we need to estimate the interest which is inherent in this code. A precise estimation is difficult, but often a rough estimate such as 'small', 'medium' or 'high' is sufficient to guide a decision. Ultimately we also need a probability: how likely will this code be read or changed in the not distant future?
This approach has several advantages:
- Technical Debt is made visible and clear for everybody. A unanimous decision can be made if and when a refactoring task should be done based on the estimated effort and the impact on future code changes.
- The cost per task is easily trackable
- There is no mixture between technical tasks and feature tasks.
- However, this approach also has some disadvantages:
- The customer must decide about the backlog and prioritize tasks. The customer can do this only to some degree concerning technical issues since he cannot determine the value of a technical task without detailed knowledge of the software.
- In addition, the customer may not really understand the business benefit of a purely technical task. In many cases, there will be some mistrust why a technical task without a feature relevant to the customer is really needed.
In our opinion only in special situations is the customer able to decide that e.g. with software updates. It is obvious to most customers that outdated software causes problems. Whether an update should be done has to be decided case by case based on cost, risk and necessity of the upgrade. Updates of components such as the Java Runtime Environment, or an O/R mapper do usually not cause a lot of effort and have a rather small risk if they are done regularly. Very expensive changes e.g. the replacement of a web framework or a change from a relational database to a NoSQL solution must always have a business reason, e.g. the performance or the user experience of an application must be improved. For this purpose the technical backlog is also well suited. But for such tasks you may also initiate a separate project, depending on how much effort is expected.
The technical backlog is not so great if a new requirement should be implemented in a codebase and that codebase is not well designed for this new requirement. In such a case a refactoring might be needed before implementing the requirement. To do this, you have to add the extra effort for the refactoring to the estimation of the requirement.
Requirements estimation with effort for Technical Debt
Strategic Design has taught us that not every part of the codebase must be very good but only those parts whose changeability brings business value or need to be changed frequently for other reasons. So concerning Technical Debt code quality of rare or never changed locations can be ignored. Also important: Code should never be refactored without a good reason. So if a new requirement provides concrete business value then we should never implement this requirement based on badly written code but refactor beforehand. That way the really important requirements can be implemented to a “good enough” standard. When the team has been assigned to implement such an important feature in an upcoming release, the refactoring should be budgeted and then implemented. Of course there are exceptions, which we will discuss in the section “To pay or not pay”.
This procedure ensures that no unnecessary beautification work is done and that refactorings are always directly associated with a requirement. This allows the customer to discuss and prioritize the refactoring with the team. The customer can also decide whether or not a requirement should be implemented based on debt and interest. There might be other factors to take into account such as time-to-market. The customer might also already know about future changes to the code which are based on these requirements. He might therefore decide to refactor to a clean code base in order to save costs in the future.
Problems arise if the team runs into a problem with the code base during the implementation. Then the team must decide what to do: Maybe meeting the delivery date with all promised functionality is very important. Then the resulting Technical Debt and the interest payments have to be accepted. Or all development based on badly written code is considered to be waste and there is no real deadline pressure. Then the team should schedule the refactoring which causes a later release date. This can only be decided depending on the context and in particular the needs of customers.
To pay or not to pay
We have seen that only important parts of the code must be really clean. So code should only be refactored if necessary. However, it is hard to decide what solution might be best for a specific scenario. Frank Buschmann describes 3 strategies:
- Debt repayment: refactor or replace the code, framework or platform that is considered a Technical Debt.
- Debt conversion: Replace the current solution with a ‘good, but not perfect’ solution. The new solution has a lower interest rate. That might be a good option if a perfect solution is prohibitively expensive to build.
- Just pay the interest: live with the code, because refactoring is more expensive than work with the not-quite-right code
Developers and customers have to decide that on the basis of cost, risk and urgency. In summary, such a substantial refactoring should always be a business decision. Another important factor for paying interest: the "end-of-life" of the software - is a refactoring valuable at all, if the system will be redeveloped or retired soon anyway?
Technical Debt can be seen as a shortcut, which saves teams’ time, effort and/or money today, but might lead to increased costs in the future. Technical Debt cannot really be avoided in software projects. But it is not necessarily a bad thing, if handled properly.
Doing so is difficult: The effect bad code quality has for future requirements is difficult or even impossible to predict. However, completely ignoring Technical Debt can have disastrous impacts on the software. If the team and the customer handle the risk of Technical Debt properly, it should remain manageable. We have presented different ways to respond to Technical Debt. Each of these options should only be used in a specific context. It is important to accept:
- that there is always Technical Debt
- that Technical Debt is not always bad
- and that Technical Debt does not have to be (fully) repaid in every case.
Any repayment should be a business decision. Even if the repayment is so small that a consultation with the customer does not make sense, the developer must always ask himself whether an improvement of the code really justifies the invested time and effort.
About the Authors
Sven Johann is working as a Software Developer for Trifork Amsterdam. Sven is an avid user of XP practices like pair programming, TDD and small releases. Currently he’s developing online assessment software for schools in Switzerland and The Netherlands based on Trifork’s QTI engine.
Eberhard Wolff is working as Architecture and Technology Manager for adesso AG in Germany. His focus is on Java, Cloud and Software Architecture. He is a regular contributor at international conferences and author of several books and articles.
Excellent article on Technical Debt
Thinking beyond technical debt
Isnt the initial premise incorrect?
initially it appears as if "clean and smart" is referring to the "refactor-as-you-go" approach, but the rest of the discussion about the cost of the "clean and smart" approach seems to more closely match the case of "design up front" rather than "refactor is you go".
I think the developer has three choices, not two:
1. pay now "up front" (design or "factor" up-front)
2. pay later "with compounded interest"
3. pay as-you-go (just-in-time + just enough).
Much of the discussion about whether or not to remove the "debt" seems to require a degree of up-front certainty about whether or not the messy code will actually need to be revisited that much (which also doesnt seem to account for the likelihood that how messy the code is can play a significant factor in whether or not the developer will choose to revisit it versus change something else as a workaround).
I thought the whole point of the agile "pay as you go" approach was an application of lean's "right amount, right place, right time" approach to defer irreversible decisions to the last responsible moment (or somehow make them be reversible). The other point is that doing the clean and simple stuff simply becomes a force of habit, like breathing in and out, for every development task. And it would be only the bigger and more concerted refactorings that you leave for later (applying the "rule of three") -- hence the advice given here would be more applicable not so much for deciding whether or not to refactor at all, but rather when to stop refactoring (removing smells) beyond just the initial clean+simple stuff.
The approach suggested in this article seems to require the kind of pre-cognitive certainty to accurately predict (guess) the risk+likelihood of needing to revisit the code again. That sounds more like taking the YAGNI approach for up-front-design rather than the pay-as-you-go approach.
I would think a "real options"-based approach applied to managing technical debt w.r.t. cost-of-delay would make a lot more sense to use here (especially for the "pay-as-you-go" approach).
Re: Isnt the initial premise incorrect?
I believe "Refactoring as you go" is generally a good idea. However:
- if test coverage is low you might not be refactoring is risky and developers might limit it to the bare minimum.
- it only works for smaller refactorings Larger changes might be needed if you learn more about the requirements or if you add a new technology or new version of a technology that allows you to do things in a much easier manner.
Also I agree that refactoring should be a habit. But there are exceptions. If you have a large pile of hard to change code you probably should not try to refactor it - or maybe you should if you keep changing that code over and over again. In such scenarios the usual approach to aim at the highest quality possible must be replaced by a business decision. That decision must be based on an estimate whether it is better to improve quality and gain higher productivity or just deal with the bad quality. In particular for such scenarios the approaches described here are useful. This is a guess - but we do guesses all the time when we talk about the time and effort a feature need. These kinds of discussions are IMHO what the metaphor of technical debt was invented for - to talk to business people about such scenarios.
Re: Isnt the initial premise incorrect?
I'm sorry - are you talking about the actual practice of refactoring as described by Martin Fowler and Extreme Programming whereby "refactorings" are not separate tasks unto themselves but are very small changes made as part of a task.
Or are you referring to "restructuring" (where the changes being made are significant enough that they basically are a separate task (or even larger) on their own. Because those are very different things.
Because all the examples you give as exceptions don't look like examples of refactoring to me, but of restructuring (even the case of low/little test-coverage).
So your initial premise still seems incorrect to me. All of the cases where you say we shouldnt do it automatically and need to consider it more carefully first seem to be restructurings that require more effort than the refactorings Fowler and others describe.
If it fits within the scope and effort of the task being worked on, I don't think there should be a question of whether or not to refactor. The real question is if what youre about to do is really refactoring, or if it is something bigger (a.k.a. restructuring).
If it's bigger, THEN we have the "technical debt" discussion you mention. And look at risk and strategy and increased test-coverage and all that other good stuff).
I think maybe the real issue here is the common misuse of the team "refactoring" to refer to "after-the-fact restructuring" instead of the small refactoring-type changes Fowler and Beck initially described.
Rather second-guess all applications of refactoring, I think much better results would come from clarifying and correcting the intended meaning and usage of what refactoring is and is not, and that for true refactoring, most of the time it's probably more wasted effort to second guess them then to just do them as habit.
Then the primary focus can be more on honing and improving one's judgement (heuristics and "tests") for how to tell when what they're trying to do is more than just refactoring and hence warrants a discussion about the costs and risks of the "debt" being delayed or deferred.
Otherwise I think the wrong message is being conveyed as a result of the initial premise (and intended versus "popular" use of the term refactoring) being incorrect and misunderstood. I'd like to see the misunderstanding correct first (rather than leaving it stand), and then it actually makes the rest of the discussion a lot more consistent with when refactoring is and isnt already being recommended.
business decision on technical debt?
Re: Isnt the initial premise incorrect?
But what if you have the case, which Martin Fowler describes as "Now we know how we should have done it"? Then you're probably in the need of a larger refactoring (or restructuring) once you recognize your problem and you have to think about it as we described it.
You say: "If it's bigger, THEN we have the "technical debt" discussion you mention". That's what we mean. It's most of the time the bigger issues, where we have to think about how and when and if to pay back.
Regarding "I think much better results would come from clarifying and correcting the intended meaning and usage of what refactoring is and is not [...]": to be honest, I never thought about it, it was always refactoring for me. Large or medium or small. But while writing this comment and also discussing the article with colleagues, I agree with you, that the part is not really clear between larger and smaller refactorings. If you want to improve a nasty little part of the code you run into over and over again, which fix only costs you one day, there's no point in doing the things we described in the article. If you do continuous design, then you rarely run into these issues.
Re: Isnt the initial premise incorrect?
Then I suggest you familiarize yourself with martinfowler.com/bliki/RefactoringMalapropism.html
(and you may want to seriously reconsider your use of the term).
It probably would also help a lot for your article to refer to martinfowler.com/bliki/TechnicalDebtQuadrant.html when talking about the kinds of things to do/avoid and the risks involved.
Either way, you will probably want to revise your article to define what you mean by refactoring early on -- since there are obviously (at least) two very different meanings of "refactoring" floating around out there (i.e., the one that seems to be "popular" versus the one that was originally intended by the likes of Fowler and Beck)
I guess if you insist on using the term "refactoring" in the 'popular' sense, then you are right that there can be confusion as to when to do it or not. That confusion largely goes away if you use the intended meaning from the folks that actually popularized the term.
I would think it better to reduce the confusion (rather than promoting or sustaining it) by clarifying the intended meaning from the popular meaning and the importance of the difference between them.
Re: Isnt the initial premise incorrect?
Re: Isnt the initial premise incorrect?
thanks a lot for the thoughtful comments!
The Tech Debt quadrants are also quite interesting - www.youtube.com/watch?v=8kotnF6hfd8 is a video in which Martin talks about this among other things. I think the quandrant approach shows that it is important to discuss Technical Debt as something you can do deliberate - and it might in same cases even be a smart decision. That was one of the motivations to write the article.
Concerning the RefactoringMalapropism link: It emphasizes that Refactoring must not break the tests for a longer period of time. I completely agree with this. However, that still allows complex changes and eliminating Technical Debt. Actually even the original Refactoring book has a complete chapter (12) on big Refactorings.
So I guess we are talking about two different things here: The decision to pay back some technical debt - which might influence larger parts of the code base and might be a considerable effort. That is then broken down in Refactorings that each by itself is guarded by tests and might be automated.
I actually don't believe in the XP idea of "Continuous and Merciless Refactoring". It is very hard to reach the ideal quality and most of the time it doesn't make sense economically. Also developers often just don't invest the time to improve quality and therefore other solutions are needed - like setting aside some time. This is one of the points the article tries to make. It is based on our experience - as well as the experience of some others cited in the article.
Technical Debt: A Long standing issue
Re: Technical Debt: A Long standing issue