How to Get Tech-Debt on the Roadmap

Over the years, Honeycomb has experienced growth, steadily attracting more customers and facing the challenges of scaling. As the company navigated through these growing pains, it managed each new hurdle, learning and adapting. This growth journey brought about various side effects, each pushing the company's systems to their limits in unique and unpredictable ways. Despite these challenges, the organization’s infrastructure has remained robust, albeit showing signs of strain as it encounters new boundaries.

During my presentation at QCon San Francisco 2023, I recounted the story of how we navigated the challenge of doubling our business.

Initially, it was imperative to determine how to develop a persuasive business case to ensure technical projects were prioritized in the schedule.

Understanding Priorities

There's always a surplus of work compared to the capacity to execute it. Engineers focusing on product roadmap items - new features or enhancements - often face the challenge of how to prioritize necessary work to maintain existing operations.

Scheduled work from product managers might not align with the immediate needs to sustain infrastructure. From a product manager's viewpoint, prioritizing technical needs over roadmap tasks may seem to question the engineering team's focus. For example, in a startup environment, cost-saving measures like reducing hosting expenses by 20% might be overlooked in favor of projects perceived as providing more value to end users or potential customers.

The Prioritization Process

Sophisticated tools assist product managers in consolidating numerous ideas and inputs into proposals for engineering evaluation. These inputs, ranging from customer feedback to continuous ideas and specific business appreciations, must be synthesized into representations of the business's services, solutions, or problems addressed. By reframing each piece of feedback or idea as a problem the business solves, a comprehensive dataset is formed. This in turn highlights the various challenges the business faces, their impact on customers, and opportunities for growth or improvement, all curated by the product organization.

After understanding these ideas, you can categorize them into three groups: the clear benefits, the not-so-beneficial, and the ambiguous middle. During a workshop led by Jeff Patton, I was introduced to a graph called Constable's Truth Curve, which was presented at QCon San Francisco 2013.

There's a necessity to leverage tools for the majority of ideas that fall into the blue middle area, aiming to clarify and direct them toward more concrete decisions. There are effective frameworks such as the opportunity canvas and learning canvas that help in structuring the problem understanding, identifying who benefits or doesn't, assessing costs, and facilitating discussions around these aspects.

The primary aim here is to clarify the return on investment for any project by evaluating its impact on users, how significantly they value the change, and whether it enhances features they frequently use. This process is central to product management, focusing on thoroughly answering these crucial questions.

Software Costs More

Addressing technical debt is costly, emphasizing the need to view engineering efforts beyond the individual contributor level. Companies often use the revenue per employee metric as a gauge of the value each employee contributes towards the company's success. Quick calculations suggest that engineers, who typically constitute about 30-35% of a company's workforce, are expected to generate approximately one million dollars in revenue each through their efforts. For top-performing companies, this figure can soar to between two and five million dollars per engineer.

It's important to acknowledge the significant effort the product organization invests in curating features and deciding which to prioritize. Considering technical debt, such as a database upgrade, requires assessing its business value and potential return using the same criteria applied to other features.

A particular approach involves the work and prioritization being anchored in constructing a solid business case. This involves striking a balance between the benefits derived from undertaking the work and the costs involved in its execution. Such decisions must get triggered via a framework of the business's current priorities and overarching goals, ensuring that our efforts are aligned with what is most crucial for the business at any given time.

What is Tech Debt?

Identifying technical debt is complex. It encompasses customer-facing features, such as new functionalities and bug fixes, as well as behind-the-scenes work like toolchains, testing, and compliance, which become apparent only when issues arise. Additionally, operational aspects like CI/CD processes, training, and incident response are crucial, non-code components of system management.

Emily Nakashima, the VP of Engineering at Honeycomb put forth the below graph in Anything But Tech Debt.

Technical debt is often thought of narrowly as code refactoring or dependency upgrades, but in reality, it includes a broad range of tasks necessary for adapting the product to new business requirements. The term "technical debt" itself, given its varied interpretations, can sometimes prevent clear communication. A more effective approach might involve discussing work in terms of its business impact, thereby facilitating its integration into the product roadmap.

Changing the Language

In the process of determining how to prioritize and schedule engineering tasks, it's crucial to shift our focus from the specifics of what needs to be done, such as database upgrades, to understanding why these tasks are essential and linking them back to business objectives. For instance, an engineer might argue for a database upgrade because it's reaching its end of life. However, the urgency becomes clear when considering business implications, like a hosting provider's refusal to support an outdated database, threatening the continuity of the product.

Highlighting the direct impact on business operations, such as potential downtime, makes a compelling case for prioritizing such upgrades.

As engineering teams, we possess the unique ability to generate and harness data from our systems, or even modify these systems to gather new insights. This capability allows us to substantiate our perspectives in discussions, moving from speculative arguments to data-driven conclusions.

Service Level Objectives (SLOs) stand out as a preferred tool for connecting technical metrics with business value, primarily because they encapsulate user experience metrics, offering a concrete way to demonstrate the impact of technical decisions on business outcomes. There is also a talk by Liz Fong-Jones about Pitfalls in Measuring SLOs on InfoQ. The past incidents also provide useful metrics.

Engineers also have access to metrics that reveal the capacities of our services, allowing us to conduct chaos experiments. By intentionally limiting a service, we can simulate stress conditions to uncover potential issues before they escalate into real incidents.

Additionally, gathering qualitative feedback through surveys on the engineering experience, especially regarding technical debt, provides valuable insights. For instance, areas of the codebase perceived as "haunted graveyards" due to their complexity can hinder progress. Addressing these areas not only cleans up technical debt but also facilitates business advancement.

We'll align our previous points with an experience from a year and a half ago. As the organisation is a Software as a Service(Saas) company, Annual Recurring Revenue (ARR) serves as a central business metric. Understanding ARR's significance—involving customer acquisitions, upgrades, and churn—helped us prioritize actions that didn’t directly correlate with technical tasks like database upgrades. By engaging with the sales and product teams, we gained detailed insights into customer behavior and requirements. This information, translated into tangible technical metrics, enabled us to connect engineering efforts directly to business value, guiding our infrastructure planning and optimization.

We tasked each engineering team with evaluating their services, starting with a comprehensive overview and identifying potential bottlenecks and scalability. They assessed how their services related to our sales targets, considering relevant factors like ingestion rates. This inclusive process, supported by our SRE team, led to standardized reports detailing service functionalities, dependencies, and scalability challenges.

For instance, our primary API service, crucial for customer telemetry ingestion, underwent analysis revealing its scaling limits and dependencies. By aligning these findings with our sales projections, we determined the urgency of addressing specific scalability concerns, such as database load issues, to meet our growth targets. This approach not only identified immediate technical priorities but also informed our strategy for future scalability testing and adjustments.

Associating with the Business Case

The justification for the project instantly became apparent. We could convincingly inform our sales team about the necessity to address the database load issue to achieve the sales targets. This led to a straightforward approval for scheduling the required work.

This systematic approach was applied across all services, resulting in an organized Asana board categorizing tasks based on urgency and relevance. This exercise also brought to light certain engineering challenges, notably those dubbed 'explody,' highlighting critical scalability issues. Such insights not only informed immediate action plans but also facilitated discussions on broader engineering practices and scalability considerations, emphasizing a balanced approach to system design and prioritization.

Close the loop

In this way, we've effectively leveraged business priorities to address critical technical issues, scheduling necessary fixes while deferring less urgent tasks. As we near completing this scheduling phase, there's an additional step that goes beyond mere planning: celebrating our achievements. Highlighting the successful resolution of these projects underscores the engineering team's pivotal role in solving business challenges and fostering growth. This recognition not only validates our efforts but also strengthens the case for future technical projects.

Reflecting on a past incident revealed the significant improvements in our service capacity, showcasing the tangible benefits of our engineering advancements. This serves as a reminder of our progress and the value of bringing visibility to often overlooked technical work, enhancing our organizational agility and product quality.

Summary

In summary, to effectively integrate technical debt resolutions into your roadmap, it's crucial to align them with your business's core needs and the potential impact of addressing such debt. It is essential to champion the success of these projects as integral to the business's growth, and to articulate their value beyond mere technical fixes.

Use data to substantiate the necessity and outcome of these projects. Once completed, it's important to communicate the positive effects that were realized, and to highlight how overcoming these challenges has propelled the business beyond its previous constraints.