The Costs and Risks of Technical Debt
Forward
I first heard of Technical Debt while listening to an interview with Charles Hoskinson, co-creator of the Ethereum blockchain and founder of IOHK/Cardano (ADA). Hoskinson created IOHK (Input/Output Hong Kong) to develop what he calls the “Third Generation Blockchain.” One of the primary focuses of the worldwide development team on the Cardano project is the reduction or elimination of Technical Debt.
A methodology typically not used with blockchain development is the exhaustive analysis and Whitepaper development that undergoes rigorous peer review by experts of each component and feature of the project. This methodology has been put into place at Cardano/IOHK as a critical part of the third generation blockchains core development.
I feel strongly that many software development projects could benefit by putting into place strong guards and standards to protect the project from sustaining increasing costs from Technical Debt. I have gathered together some information from the internet that describes the concept, types, causes, and more. So that it might be understood and possibly heavily considered as projects are planned, defined, and ramped up.
Technical Debt
Definition - What does Technical Debt mean?
Technical debt is a concept in programming that reflects the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution.
Technical debt is commonly associated with extreme programming, especially in the context of refactoring. That is, it implies that restructuring existing code (refactoring) is required as part of the development process. Under this line of thinking, refactoring is not only a result of a poorly written law but is also done based on an evolving understanding of a problem and the best way to solve that problem.
Technical debt may also be known as design debt.
The term was coined by Ward Cunningham, a programmer who is also known for developing the first wiki in the early 1990s. Technical debt is a metaphor that equates software development to financial liability. Imagine that you have a project that has two potential options. One is quick and easy but will require modification in the future. The other has a better design but will take more time to implement. In development, releasing code as a quick and easy approach is like incurring debt - it comes with the obligation of interest, which, for technical debt, comes in the form of extra work in the future. Taking the time to refactor is equivalent to paying down principal. While this takes time in the short run, it also decreases future interest payments.
Cunningham’s summary explanation is as follows:
“If we failed to make our program align with what we then understood to be the proper way to think about our financial objects, then we were gonna continually stumble over that disagreement, and that would slow us down which was like paying interest on a loan.”
Ward Cunningham describes this concept in 1992 as follows:
"Shipping first-time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite. Objects make the cost of this transaction tolerable. The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise."
The concept does not mean that debt should never be incurred. Just as leverage can help a company when used correctly, a quick solution can expect a faster time to market in software development. In addition, technical debt is not just poor code. Bad code is bad code, and technical debt can result from the work of good programmers under unrealistic project constraints.
Causes
The concept does not mean that debt should never be incurred. Just as leverage can help a company when used correctly, a quick solution can expect a faster time to market in software development. In addition, technical debt is not just poor code. Bad code is bad code, and technical debt can result from the work of good programmers under unrealistic project constraints.
Common causes of technical debt include (a combination of):
- Insufficient up-front definition, where requirements are still being defined during development, development starts before any design takes place. This is done to save time but often has to be reworked later.
- Business pressures, where the business considers getting something released sooner before all of the necessary changes are complete, builds up technical debt comprising those uncompleted changes.
- Lack of process or understanding, where businesses are blind to the concept of technical debt, and make decisions without considering the implications.
- Tightly-coupled components, where functions are not modular, the software is not flexible enough to adapt to changes in business needs.
- Lack of a test suite, which encourages quick and risky band-aids to fix bugs.
- Lack of documentation, where code is created without necessary supporting documentation. The work to create any supporting documentation represents a debt that must be paid.
- Lack of collaboration, where knowledge isn't shared around the organization and business efficiency suffers, or junior developers are not properly mentored.
- Parallel development on two or more branches accrues technical debt because of the work required to merge the changes into a single source base. The more changes that are done in isolation, the more debt is piled up.
- Delayed refactoring – As the requirements for a project evolve, it may become clear that parts of the code have become inefficient or difficult to edit and must be refactored in order to support future requirements. The longer that refactoring is delayed, and the more code is added, the bigger the debt.
- Lack of alignment to standards, where industry standard features, frameworks, technologies are ignored. Eventually, integration with standards will come, doing sooner will cost less (similar to 'delayed refactoring').
- Lack of knowledge, when the developer simply doesn't know how to write elegant code.
- Lack of ownership, when outsourced software efforts result in in-house engineering being required to refactor or rewrite outsourced code.
- Poor technological leadership where poorly thought out commands handed down the chain of command increases the technical debt rather than reduce it.
- Last minute specification changes; these have potential to percolate throughout a project but no time or budget to see them through with documentation and checks.
Technical Debt Quadrant
It is useful to differentiate between types of technical debt. In a discussion blog "Technical Debt Quadrant", Fowler distinguishes four debt types based on two dichotomous categories: the first category is reckless vs prudent, the second, deliberate vs inadvertent.
Consequences
"The cost of never paying down this technical debt is clear; eventually, the cost to deliver functionality will become so slow that it is easy for a well-designed competitive software product to overtake the badly-designed software in terms of features. In my experience, poorly designed software can also lead to a more stressed engineering workforce, in turn leading higher staff churn (which in turn affects costs and productivity when delivering features). Additionally, due to the complexity in a given codebase, the ability to accurately estimate work will also disappear. In cases where development agencies charge on a feature-to-feature basis, the profit margin for delivering code will eventually deteriorate."
Junade Ali, Mastering PHP Design Patterns
"Interest payments" are caused by both the necessary local maintenance and the absence of maintenance by other users of the project. Ongoing development in the upstream project can increase the cost of "paying off the debt" in the future. One pays off the debt by merely completing the uncompleted work.
The buildup of technical debt is a significant cause for projects to miss deadlines. It is difficult to estimate precisely how much work is necessary to pay off the debt. For each change that is initiated, an uncertain amount of uncompleted work is committed to the project. The deadline is missed when the project realizes that there is more uncompleted work (debt) than there is time to complete it in. A development team should have predictable release schedules to limit the amount of work in progress to keep the amount of uncompleted work (or debt) small at all times.
If enough work is completed on a project to not present a barrier to submission, then a project will be released which still carries a substantial amount of technical debt. If this software reaches production, then the risks of implementing any future refactoring might address the technical debt increase dramatically. Modifying production code carries the risk of outages, actual financial losses, and possible legal repercussions if contracts involve service-level agreements (SLA). For this reason, we can view the carrying of technical debt to production almost as if it were an increase in interest rate and the only time this decrease is when deployments are turned down and retired.
"As an evolving program is continually changed, its complexity, reflecting the deteriorating structure, increases unless work is done to maintain or reduce it.”
— Meir Manny Lehman, 1980
While Manny Lehman's Law already indicated that evolving programs continually add to their complexity and deteriorating structure unless work is done to maintain them, Ward Cunningham first drew the comparison between technical complexity and debt in a 1992 experience report:
"Shipping first-time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise."
— Ward Cunningham, 1992
Causes
Because the definition of Technical Debt is clearly defined and stable, there is no need to rewrite a paper discussing the topic. Technical debt can take on many forms, but this summary is primarily regarding software/code development. Thus, the concepts and in many cases the actual text for this summary is from the following sources:
- Technical Debt Wiki page
- An Explanation from Ward Cunningham
- Techopedias description