Last year, the Wikispeed team took the world by storm. They took agile practices and applied them to the stodgiest industry possible: the car industry. The developed a green car in 3 months, proving that cars don't need to be developed in 10-25 year product cycles.
Moreover, their new car was engineered to high quality standards, thanks to test driven development of individual components. The car performed to a much higher safety standard. They developed a beautiful car body in 3 days. The chassis was the lightest chassis in the world to achieve a 5 star equivalency safety rating. They iterated in changes quickly. Much of this was possible, because they were very consciously modular in their design. They could change it from a convertible to a pickup truck by just changing the chassis.
According to team founder Joe Justice, the following factors were the most important elements of their success. Team WikiSpeed:
- Used less stuff
- Actively minimized the cost of making changes
- Collaborated on distributed teams with high morale
- Used pairs to reduced time spent on training and documentation
- Visualized workflow to find any time not spent creatively solving problems
While Wikispeed's initial success isn't news to the Agile community any more, it has significant implications for Agile. In particular, applying agile outside of software development demonstrated how ineffective product management styles are outside of software development. It exposed mind-numbing inefficiency.
By reducing the product life-cycle from 10 years down to 3 months, the Wikispeed team performed at least 40x faster than product development teams in the automotive industry. While the product is the same, the tools are generally available and accessible to high school students on a budget. The people participating were not professional car designers but dedicated and starry-eyed amateurs.
The results they achieved point to the importance of process and agile culture. In particular, they constantly removed any obstacle to creative problem solving. The results they achieved point to the impact of process debt in the automotive industry, and possibly many others.
For Product Developers, Process Is A Useful But Necessary Evil
Let's say you want a particular problem to be solved. To you, it’s irrelevant whether the solution is delivered as:
- software
- following a process
- a physical good which does something.
You know you have a headache, a migraine. You want to get rid of it. You couldn't care less how.
You want a solution. As Ted Levitt quipped, "People don’t want to buy a quarter-inch drill, they want a quarter-inch hole". They only want a specific type of hole. Everything else is irrelevant, including how it's delivered. If that’s the case, at its essence, process is the "business system" created by an entrepreneur, to deliver that solution.
Product development itself is a process. Often it's done as a team. Product development may ultimately result in a process, software, or a physical good which addresses the identified need. If successful, this product development process accrues financial benefits to the entrepreneur and any other future business owners.
In short, process is a necessary evil for many product developers. Process is a delivery mechanism for satisfying a need. It organizes our work. Over time, a good process generates not only value, but wealth in a for-profit company. The company generates profits and is worth more. It’s everyone's interest to keep the product development process lightweight. We want a process that is easy to manage, particularly if we own the company. We want a process that is easy to live with, if we work in it. Team Wikispeed proved that a lightweight process can make a massive productivity difference.
Process debt
In this context, process debt is what Team Wikispeed actively tried to remove, as they minimized the cost of making changes. Process debt is an accumulation of shortcuts, workarounds, and deadlocks in an existing process. Over time, everything gets tangled up. Like spaghetti, you pull one strand, and half of what’s on your plate moves. Process debt is the business equivalent of technical debt. It's useless complexity. Most importantly, process debt results in a breakdown of useful conversations. Instead of discussing and resolving problems, people hide behind processes, "best practices", and "preferred vendors". Processes institutionalize "No". Would-be team members feel alienated and disenfranchised.
If you have accumulated odd workarounds over the years, never really bothering to make things work efficiently, you are paying “interest” in terms of time, because of shortcuts taken in the past. Such shortcuts are a transfer of time. A problem could have been resolved earlier once. It could have been prevented from ever happening again by being completely "done". Instead that time is shifted into the future. It comes up again and again. Worse, it might affect other departments, who have to work around the problem. This, in turn, can cause further problems. Ultimately, this impacts customers, whether they know it or not.
In this case, the debt is counted in terms of time, especially employee time. A shortcut taken once ends up causing significant rework later. This is typically considered standard “technical debt”, which is a well-documented problem. Technical debt itself is a subset of the larger business problem of "time shift". It instigates a loss of ownership.
If your technical shortcut goes into production, it forces operations staff to perform time consuming workarounds. They also need extra training and documentation to deal with the additional complexity. The development team takes on technical debt, which subsequently impacts the whole company. So it’s not technical debt any more, it’s a more general problem.
As Steve McConnell's previously pointed out with technical debt, its significance is contextual. In this sense, process debt's similar to financial debt. Sometimes, taking a loan is highly beneficial. Debt helps you achieve a strategic goal. For example, you take a mortgage to buy a house. You let out the house. The people and the renting the place pay off the interest and the mortgage. In this case, the mortgage helps you generate wealth you wouldn't be able to generate otherwise. In contrast, certain types of debt rapidly destroy wealth. While you can get a new credit card with a 0% offer for 15 months on incoming balance transfers, it's not necessarily a good idea.
As many developers know, often a process can be automated. Like process, software can also be used as a delivery mechanism. Scripts can effectively replace and automate a service. How can you be “agile”, if it takes an epic effort of many man-moons to go into production? This assumption is embedded in the recently popular DevOps movement, particularly in the demanding agile practice of Continuous Delivery.
If you can script your release process, you have a completely different definition of agility. While continuous delivery seems to be about code, it's really about having a very simple, streamlined, and push-button process. It also frees up the product development team to talk about more pressing issues.
As a customer, excessive process debt keeps you from getting what you want much faster. So how is that migraine again?
Process Debt is Different Than Technical Debt
At its core, process debt is excessive complexity in business processes, just like technical debt is excessive complexity in code. It depends on how well the abstractions used to talk about a problem map to the actual problem domain. If you can accomplish exactly the same thing with less complexity, you are feeling the effects of process debt slowly. It's tightening its grip around your throat. If you've ever tried to script an existing process, only to discover byzantine, redundant, and inconsistent requirements from different factions within a company, you'll see it's not exactly the same thing. No code exists. How can that be technical debt?
Both types of debt involve unresolved logical conflicts in workflows.
- In the case of technical debt, this includes mis-scoped variables and member functions, unclear separation of concerns, unintended side effects.
- In the case of process debt, this means unclear goals, states, and functions in organizations, a muddled business identity, and unintended side effects caused by a lack of communication among departments.
In 1968, Melvin Conway said: "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations". According to Conway's Law, complexity tends to trickle down into the code; however, it's fundamentally caused broken communication among people and a lack of trust. Process debt creates technical debt.
Most software companies endure some kind of process debt. Some of it may be taken on intentionally-strategically even-just like the mortgage example above. Nonetheless, most process debt isn’t strategically managed, simply because companies don't see it as a problem. For example, even if you have no technical debt in your software, if your release process to production isn’t fully scripted, you have process debt.
How To Deal With Process Debt
Most of the advice below falls in the category of "I know about it, but I'll deal with it later." Deal with it now, especially if it slows you down. It's keeping you from moving faster. You will delight your customers, particularly if they are unhappy about your inflexibility or lack of speed.
Recursively clarify strategy until it's painfully obvious
According to Fortune Magazine, over 70% of companies fail to execute strategy. In The Strategy Focussed Organization, Kaplan and Norton found that most companies spend about an hour a month discussing strategy, with most of their time being sucked up by busywork. If companies do talk about strategy, it's limited to a few senior decision-makers behind closed doors. This approach sabotages implementing the strategy. It ensures a lack of buy in from the front line employees, who actually need to change how they work.
In contrast, in Motivate Like A CEO, Suzanne Bates argues that the CEO's job is to constantly communicate the strategy, tying it into day-to-day activities. Ideally, this strategy is captured as a pithy one-liner. Strategy's discovered together, involving everyone from the company. It captures the single-most important thing that needs to be accomplished. Ideally, the CEO ideally helps employees apply the strategy on a day-to-day basis. This implementation gap is hard enough when you know what the strategy is; therefore, the strategy needs to be blindingly obvious to everyone involved if you want them to act accordingly.
When in charge of a local non-profit club, I realized complexity of this task. By talking with everyone individually, I helped the group decide on the most important goals for the organization. Then once this was clear, my primary objective became communicating this priority in fresh and relevant ways, so that the volunteers felt their contributions were useful. Unlike in a for-profit setting, the volunteers helping me weren't paid for their time and contributions. They primarily wanted a sense of satisfaction, that they've achieved something, that they've helped others. By collaboratively identifying the top priorities and systematically communicating them, I helped double the size of the organization in one year.
Muddled strategic thinking leads to muddled execution. Clarifying the strategic "pinch points" allows you to take advantage of them. Big wins are easy, if your strategy is right. Clear thinking with substantiated reasoning enables you to accomplish a lot with very little.
It doesn't make sense to address process debt, until you are clear on strategy. Your strategy will define which processes are strategic, and which can be eliminated. It's pointless to improve efficiency, if you don't have a clear sense of effectiveness. In other words, going faster doesn't help, if you are driving in the wrong direction.
Push-button workflows with software
Once you clarify your strategy, you can streamline or replace processes which are not useful. Useful ones can be automated. For software companies, automating work produces the biggest gains.
As a client facing developer, I once had a client who had his accountants spend over a week producing reports for his investors-every single month. This started with a manual extract from a database, followed by massaging and formatting the data exquisitely. As his number of investors grew, he had to provide more and more customized reporting.
Enter the geek (me). After writing a few VBA based scripts, driven by a few dynamically populated drop-downs combined with SQL queries, they could produce exactly the same reports in a few seconds. From their investors' point of view, the software generated reports were equivalent. They provided exactly the same value.
Instead of costing them weeks of accountant time, effectively a laborious, manual, and error-prone reporting process, they had a developer reduce it to a few seconds. From an aggressively low estimate of 1 week down to a conservatively high estimate of 5 seconds: a 12,096,000% productivity improvement. Moreover, once the reporting logic was accepted by the end client and his investors, i.e. there were no logical errors, this process became effectively error-free. The accountants were freed up to do something else, to address issues that required more thought, discussion, and analysis. They were thrilled.
Eliminate dependencies
In a process, we often introduce dependencies, when there are multiple uses for a particular resource. Competition for the resource turns it into a bottleneck. Overall progress slows down significantly.
Multithreaded software development serves as a useful analogy. Just as two threads conflict when they access a resource, you use a mutex to ensure the threads don't overwrite data. Depending on the context, this can create a deadlock. Removing such a deadlock can often speed up code which hangs indefinitely down to near-instant execution.
Let’s say you are developing a multi-threaded application, which is experiencing from performance problems due to a recent feature addition. You look for easy wins. In practice, this typically means you look for threads which lock, and prevent other parts of the application to work as you are waiting for a long running process to complete. If threads do lock, you will typically have messages piling up in a queue feeding a thread, as you wait for the thread to be freed.
Don Reinersten, the author of The Flow of Product Development, considers half-finished, unreleased features to be the bane of most software development initiatives. “Product development inventory is observable through its effects: increased cycle time, delayed feedback, constantly shifting priorities, and status reporting.” When a pile up of unfinished features in queues is clearly a serious form of waste, the underlying culprit is often project deadlock. Dependencies among projects slowly kills them all.
Removing dependencies is the fastest way to improve speed of delivery in a software organization. Ideally, this means less projects. You can focus on products and customer priorities.
It becomes easier to make customers happy. Ultimately, features are released faster. They don't waste time sitting in queues.
Reduce number of variables, by removing unnecessary detail
If you have excessive process debt, you will achieve big productivity gains if you reduce the number of "symbols", variables, or data points that you deal with on a daily basis. This increases the signal-to-noise ratio in your communications. It lets you focus on interesting problems, as opposed to constantly dealing with a "complexity hangover".
With source code, version control repositories which versioned each individual file were popular in the late nineties. Nowadays, most source control systems keep track of the version of the system as a whole. For a large system with many files and moving parts, you dramatically reduce the complexity of most processes by moving to one system wide version number. It's much easier to talk about specific problems. You need much less documentation. This is actually a form of technical debt that impacts processes. Once version control is streamlined, testing, release, and operations are simplified.
Of course, there are a number of difficulties with this particular approach, such as versioning database packages or inconsistent release schedules of various components. Facing this complexity forces you to reduce any "friction" caused by these unnecessarily complex processes.
Where possible, eliminate commitment by default
Lack of trust causes a number of problems, often causing managers to introduce techniques which force commitment early; this destroys value. Many tools, like MS Project, attempt to create detailed schedules based on a number of dependencies and deadlines. While this level of detail is meant to increase trust levels in a plan, unnecessary commitment to intermediate deadlines often causes you to lose sight of underlying priorities. At best, you optimize locally, with a hefty global price tag. One project is delivered on time, causing slippage on other projects. This reduces trust in the detailed plans, which then means even more detailed planning.
In MS Project, "Finish-To-Start" (FTS) task dependencies implicitly capture much of the process debt inherent in waterfall. It's also the default type of task dependency in MS Project. With FTS, you have to finish a certain task by a certain date, so you are specifying that the start date of any dependent tasks must start early enough to finish the other task on time.
While this sounds like a good idea at a high level, i.e. "start with the end in mind", at the level of tasks it creates a lot of unnecessary inflexibility. With FTS, you commit to certain deadlines. These deadlines impact work preceding it. If there are a lot of tasks, it becomes very difficult to identify the most important tasks or features. Dates commited overshadow priorities. Over time, you forget about priorities. Everyone remembers promises of due dates.
Amongst software professionals, a #NoEstimates movement has started, which is pushing back against any forecasting of effort. #NoEstimates proponents claim that forcing developers to give estimates causes the business to fall back into this FTS thinking. It ends up interfering with the developers' work, as expectations often become a source of unnecessary stress. Excessive stress prevents creative problem solving. Ultimately, it causes unnecessary conflict between the business and developers.
Unnecessary commitment has far reaching implications for processes beyond just development. Beyond Budgeting by Hope and Fraser argues a strong case against the traditional budgeting process. While budgeting was highly useful way to impose fiscal discipline the big car companies in the 1950s, budgeting can also cause a significant amount of waste. Months of executive time are spent debating about how the future might look, and what financial resources they need, committing despite not really knowing what the future holds. Instead, they could be getting a very clear picture of what's working for them now.
In contrast, there are approaches like activity-based costing (ABC). ABC essentially calculates profit and loss per activity, per customer, and per product on a regular cadence; ideally daily. This gives everyone the tools to make financially sound decisions, as they rapidly get feedback about the financial impact of their decisions. Information's not kept behind closed doors, but out in the open. Everyone, including front line employees, benefits. Immediate feedback impacts employee decisions, which impact company profitability: virtuous circle.
Enable problem solvers to listen intently
Quite often in a big bureaucratic process, artificial walls and boundaries are set up, in order to help "control" complexity. Unfortunately, this typically means that the most important information never really flows. No one acts, because they don't have a full picture. By giving everyone a voice, you permit them to express their biggest frustrations. Often the ground floor employees have a good sense of what's wrong with a company, they're just being ignored. When they do talk, they don't spread their message widely enough. They fear negative consequences. No one wants to be considered a killjoy.
If you think giving a presentation in public is hard, try making one up on the spot, with a few co-workers. While this might seem wacky, it's actually a highly refined set of skills practiced in improv comedy. While the don't know what any particular scene will be about before they start, each improv actor has to be completely plugged into everyone else on stage.
As a result, they find ideal ripostes to every offer. They discover conflict. It builds naturally. It culminates. Usually, not always, there is a completion in the scene. Systematic creativity comes from intense listening.
It's an ideal example of collaborative creative flow. The improv principle of "Yes, And" drives the creative flow on the team, according to Tom Salinsky in The Improv Handbook. When developing a scene, two improv actors enter the stage. They carry absolutely nothing, except their imagination. Once one of them starts, the other immediately responds to the "offer" of the first. That offer becomes "reality", for the purposes of the skit. The fact that they both agree to the offer "creates" it in the audience's mind. Each actor doesn't know what his compatriots will offer. The magic of the scene comes from this unpredictability. It only works if each actor intently listens to every other actor on the stage. It requires an uncommon level of intensity and trust.
When coming up with dialogue, improv comedians always accept what the previous person said. Saying "No" or "Yes, But" to an offer stillbirths the creative dynamic of a scene. The dynamic decelerates. Rapidly. In contrast, "Yes, and" pushes collaboration forwards. If the environment allows everyone to build on everyone else's creative ideas, it accelerates the rate at which these ideas come about.
In improv, the focus is on the quality of the interaction itself. At each step forward, you advance the team's thinking, with "Yes, And". You are moving forwards rapidly. You discover what you are making as you make it (think Dan North's deliberate discovery). It requires full attention. It requires being plugged into what your teammates are doing, as you want to run with any offer they make.
This same type of productivity happens when using "Yes, And" when designing a new feature or software architecture. The ideal structure is unknown at first, as it depends on the problem being solved. The team discusses the problem's aspects. First they diverge and explore a number of possible forms of the software. This is a series of "yes, and" offers. Finally, they converge on various solutions. The best solution emerges. There are conflicts in the ideas, but the people involved are constantly holding on to a "yes, and" attitude.
Process....gets in the way. It often institutionalizes "No" or "Yes, but", killing creative problem solving. While some process is essential, process which destroys creative problem solving decelerates your entire business.
@theagilebastard: "People and interactions over processes and tools" Nelson Mandela, from his prison cell 1985. #Philosophical #Agile #Kanban
The Good News
By reducing process debt, you:
- Speed up your ability to deliver your product
- Reduce the amount of errors
- Re-energize the entire company and raise morale
Ideally, your development and production processes can be embedded into your company as software, like flexibly written scripts within an automated build that composes your product for customers based on their unique needs. You can drive your release process with Continuous Integration running a suite of acceptance and unit tests, followed by sponsor sign off. This could even go directly into production, depending on the type of software you make, e.g. web software. Some teams, like for example at arts and crafts wholesaler Etsy.com, release into production 30-50 times a day.
The best process is no process, just a well written script. It captures the essence of the problem. It delivers a solution at the push of a button. It frees people to solve problems creatively. Team members can do their best work, not forced to repeat the same tedious tasks.
Despite the focus on process in agile particularly by newcomers to the agile world, it's easy to forget why agile processes exist. Agile processes are intentionally lightweight. For once in your career, you can focus on individuals and interactions. Because you remove deadweight from processes with agile, fantastic people drive agile performance. Ask Team Wikispeed.
If you want to reduce process debt in your business, go to Overcoming Process Debt, to download a free workbook.
About the Author
Lukasz Szyrmer has over 12 years of experience as a software developer, product manager, public speaker, a mentor, and community activist. He enjoys the challenge of distilling complex technical and organizational ideas down to their essence, so that others can benefit. He has previously inspired many people to systematic self-improvement. He currently develops multithreaded, real-time financial software in C++ and C#. Luke learned to code as a child, fascinated by software, well enough to become an embedded C developer after earning an Economics degree. In addition to holding degrees in Economics and Finance, he is also a Chartered Alternative Investment Analyst. He blogs here .