A Guide to Technical debt!Software Craftsmanship Best Practices SDLC
As software evolves, so does its complexity. Updating existing code versus implementing new features is often a tradeoff and there are many battles to be fought when it comes to spending time on the former. This blog looks deeper into why it is so.
What is Technical Debt?
Technical debt in programming reflects the extra development effort needed to maintain a codebase. It is debt in the true sense of the word – something that will have to be paid off eventually or the bearer faces consequences.
It typically arises when short cuts are taken instead of implementing the best overall solution. As technical debt accrues in a project, it tends to increase estimates for new work items and bug fixes. This is the debt manifesting in terms of time to ship new features, extra effort involved, opportunity cost of the time spent and reduced developer morale.
Why is it worth understanding?
Technical debt affects both, the developers and the business.
High technical debt makes simple changes difficult to implement, taking a toll on developers. To keep the codebase in control, developers need to take time to pay down that debt, creating delays in feature delivery. It is also, often, the source of glitches and other problems that negatively affect user experience. All of this eventually trickles down to dissatisfied customers.
Many businesses are oblivious to the existence of tech debt, and they continue to endure the immense costs associated with having it – simply because that's how they have always done things!
That said, having some technical debt isn't always a bad thing either. If developers spend all their time perfecting code, nothing will ever get released! Not all technical debt is the same and development teams must always weigh the benefits of solving it vs. the cost of keeping it. "Software Design X-Rays" by Adam Tornhill is an excellent book on this subject.
Cost of Technical Debt
Development Team's Velocity is Hampered
Technical debt can cause software development teams to become overwhelmed. It has the potential to quickly turn an efficient team into a sluggish train wreck. In the longer run, it diverts the team's attention to either addressing the tech debt or slowly navigating the complexity to continue delivering features.
Consider this scenario: Your codebase has a perplexing module structure. You now need to add new features. It would take you four days to add the functionality if the module layout was simple, but it takes you six days with the current structure. The additional two days is the interest you pay on the debt, and you haven't even paid the debt down.
Working on new features becomes a struggle
Technical debt has an exponential effect on increasing the effort needed to build new features.
Teams might generate more clutter too in trying to keep up with the pace of delivery. And this end up increasing technical debt even further. To put it another way, we might assume that a team's "Performance efficiency" drops.
"Even making a minor security update with a modern framework in mind, where we wanted to integrate the authentication system to the current structure, proved difficult due to the application's conventional routing mechanism. Before we could even update the authentication, we had to first understand the existing framework, which was quite a tedious job", mentioned a fellow developer.
Negative Impact! Yes, you heard it right.
"End users are unaware of behind the scenes, it's the developers who care about the code and manage it." - Fellow developer. The development, testing, implementation, and documentation of new features' costs time and money. But developers are busy fixing the debt and it limits their efficiency. This creates a negative impact since this effort could be better used for long-term creativity or delivering new features.
Technical Debt, if not fixed on time, will just steer more broken windows, eventually leading to device downtimes, a poor user interface that affects customer/user loyalty, poor system efficiency, and so on, thus affecting customer experience.
Lack of Motivation for Developers
The entire process of fixing technical debt is extremely time-consuming. There are spikes, understanding the collective concept, generating a report, getting it approved, then incorporating the changes and several other obstacles along the way. If no test cases are available, developers may encounter more issues and waste too much time.
After a certain amount of time, it becomes frustrating and to be honest no one enjoys performing repeated unproductive tasks daily. If the same thing keeps happening over and over again, the frustration keeps piling up.
Employees can become demotivated as a result of any of the above.
Technical Debt Quadrant
Martin Fowler suggests a simple approach to navigate the complex matter of technical debt:
- "Did we incur this technical debt deliberately or inadvertently?"
- "Was that decision prudent or reckless?"
These two questions give you the four quadrants of technical debt:
Deliberate and Reckless
In this quadrant, you know you are making a design choice that is bound to create technical debt. Often the reasons revolve around working under immense pressure to meet thin deadlines and getting ahead in a competitive market.
Deliberate and Prudent
This quadrant is about making an informed decision to take on technical debt. Decision makers weigh the risks involved and carefully plan to mitigate them over time.
Start-ups can embrace this to make their product ship faster, get customer feedback early and adapt course accordingly.
Inadvertent and Reckless
This is a quadrant of things you should have known about, but didn't. Constant learning is part of the job. Knowing the underlying principles helps developers define the solution better and avoid patterns that are known to cause problems. This, along with following the right development practices help reduce issues in this quadrant.
Inadvertent and Prudent
Having followed best practices and design principles, we can still face challenges. There still will be times when you realize mis-steps after the job is done. Doing POC's helps you find these uh-oh moments before it's embedded with your actual code.
How to identify Technical Debt?
There are several telltale signs of technical debt's existence.
When technologies imbricate over each other over time, they become quite rigid to allow what could have been a simple feature addition. Any architecture that is not flexible enough to welcome new changes without complicating the modules should be acted upon at the earliest possible.
- Code Smells
Quoting Martin Fowler, a code smell is a surface indication that usually corresponds to a deeper problem in the system. They are subtler than logical errors and usually indicate problems that when not attended to at time may lead to even system crashes.
Tracking the overspilled bugs in each iteration and comparing the number of open bugs vs closed bugs can be helpful to get an idea on the direction the team is moving towards in addressing, handling and creating new bugs. An increasing graph is a good indicator of high technical debt in the system.
- Violation of non-functional requirements
With accumulation of tech debt, the system may start exhibiting symptoms such as slow loading speed, compatibility loss, security hacks, increasing difficulty and so on.
- Standard conventions
Code that doesn't follow standard conventions (to be decided by the team) be it as small as keeping the codebase modularized, to adhering to standard naming conventions and styling, will lead to unreadability and unmaintainability of the codebase.
- Developer Happiness
A simple and an abstract warning sign is when developers are frustrated working on the project (implying more than expected work for simple additions or fixes) can be an indication towards unattended tech debt in the system.
- TDR Score
TDR stands for technical debt ratio. It is calculated as the ratio of the cost to fix a software system to the cost of developing a new system.
Technical Debt Ratio = (Remediation Cost / Development Cost) x 100
Ideally, a remediation cost of zero is preferred, though practically, it may not be possible. It is usually favorable by teams to keep it under 5%. High TDR score reflects deplorable state of quality of the system. There are tools available that help compute similar metrics and are covered in a later part of the article.
How to avoid Technical Debt?
While different metrics are available to quantify tech debt, it is not inherently a metric. There are no sage advices to avoid, but tracking tech debt regularly, before it becomes too unwieldy, is definitely a step in the right direction. Analyzing the discussion with our developers, here are some suggestions that might help you.
- List all your tech debts. Make a note of all the instances where there is a definitive cruft smelled by the developers.
- Categorize each of them and sort them based on their impact on the system.
- Make space for your tech debts while keeping all the stakeholders in loop. This allows the process to be transparent and balanced with delivering new features and mitigating issues.
- Allocate time to dedicate towards tech debt on a regular basis.
- Apart from this, make sure not to add or ignore any code smell while adding new changes to the system.
- While new features might be prioritized based on business needs leading to tech debt, it is important to add it to the backlog to be able to act upon sooner rather than later.
- Most importantly, make sure you communicate with the business and make them aware of the existence and impact of technical debt in the system
Read more here: Technical debt - a problem with your code?
There are several online tools that help you analyse and prioritize your debt. It's not always obvious to spot buggy code. The modules with the most development activities are often filled with hotspots. Among several tools being widely used, one we enjoy using is CodeScene . It helps you visualize the most frequently changed modules. If you then overlap the graph with modules that have low code health, that's where you start seeing areas where you may have technical debt issues.
Refactoring should not be one-off events in short increments. It is easier to refactor a small piece of code than to refactor an entire module. X-Ray is one of the features from Codescene that helps you explore code smells at the function level.
You finally have something that can fit in a sprint! Now you can plan how much debt can be reduced per sprint and track progress.
Pay off your debt
There are a few options that can be considered to pay off your tech debt.
- Sacrificing the requirement completely : The organization doesn't deem the requirement valid any longer. Of course, this case is observed to be very rare, so we usually consider to go for the other two options.
- Refactoring : While understanding and clearly communicating the additional effort involved, refactor the system (change the internal structure of codebase) to allow for the feature to easily fit in. To avoid unintended behaviour, ensure there is a strong test suite first.
- Replace the module/application : As a developer, the thought to write the module or the application from scratch always comes to mind when tech debt is too high. While it may sound appealing, exercise a lot of caution while going this route. Not all requirements are 100% documented and the chances of breaking down the line dependencies, change in functional behaviour, and other such unintended consequences are high.
All software developers and businesses strive to facilitate releasing competitive products faster and generate commercial value. But, if the debt is not cleared in time, you'll go bankrupt (technically).
Nobody would have the patience to fix technical debt, if it is left unresolved for too long. That would just result in more havoc. So, don't give up on Clean Code! Practice conventions that will help reduce technical debt and prevent introduction of more technical debt.
Make a deliberate decision that will encourage you to step forward while still encouraging future refactoring. Create that "YAY" moment and leave the code in a condition that makes it easy for someone else to scan and rework on.
As a developer, it's up to you to begin improving things!