Q&A with Jez Humble, Joanne Molesky and Barry O’Reilly on Lean Enterprise
Being lean is the modus operandi for successful startups (at least those on a tight and shrinking budget). Growing with minimal resources and ever expanding tasks, experiments and projects demands radical prioritization of work and waste avoidance.
Enterprises have more time and resources so they should be able to move faster and go further than startups. But the all too common focus on resource (over)utilization and budget fulfillment leads to missed opportunities and work being prioritized according to politics more than by value delivered.
"Lean Enterprise" by Jez Humble, Joanne Molesky & Barry O'Reilly addresses the fallacies of traditional management practices that often promote "being busy" instead of generating (financial) value quickly and sustainably (by managing multiple horizons according to product development lifecycle).
Realizing the fundamental distinction (in terms of strategy, structure, processes and more) between new ideas exploration (the Lean Startup way) and existing (successful) product exploitation is a crucial first step for enterprises aspiring to become lean as a whole, not just within specific departments or teams.
The principles and practices described in the book aim at creating alignment and enabling autonomy not only in engineering departments but also across financial management, program management, governance and process improvement areas.
InfoQ talked to the book authors to understand the challenges and benefits of the lean enterprise transformation.
InfoQ: What motivated you to write this book?
As technology consultants who get exposed to many different organizations, we are frustrated with the way many of them approach the use of technology. The capabilities and capacity for technology has increased dramatically over the past decade, and capital costs have fallen enormously. Unfortunately, many enterprise organizations have not kept pace. Although some have managed to deliver faster through the use of principles and concepts of Agile and Continuous Delivery, we find that there is still a gap in how they view and use technology. Companies toil at building products, services and businesses that simply do not deliver the expected value to customers.
Done right, technology can drastically reduce time-to-market for new businesses and products. It can also transform the relationships between businesses and their customers. The techniques and practices we describe in the book are not new, and they are known to work, but many organizations don’t use them. Our book is designed to teach leaders and managers how to best leverage the unique advantages of software at every stage of business and product lifecycles. The goal is to test many ideas and discard the bad ones quickly, while iterating rapidly on the good ones, developing and harnessing the unique capabilities of your employees.
InfoQ: Your book covers a wide array of topics, from project management to HR, product development, engineering best practices, psychology, process improvement. How would you like to see the book filed?
This is a business book about how to do software-based product development in medium and large organisations. Technology can no longer be viewed as a specialised domain that is the purview of an IT department that takes orders, or that can be farmed out to cheap labour that has no knowledge or relationship with your business or customers.
We’re all in the technology business today. Even if our main business is tied to physical assets and people, it is technology the enables the delivery of our services to customers. Our economy is being driven by the use of technology. Business leaders must have a good understanding of the capabilities of technology and create an environment that fosters learning and innovation to stay in the game. This requires fundamental changes in the way we manage and direct knowledge workers throughout our organization.
InfoQ: In your experience as consultants, do you see positive signs of lean adoption at organizational level (beyond adoption in individual dev or ops teams)?
Absolutely. It has been happening a for a long time: you only have to look at how the manufacturing industry has adopted the Toyota Production System. Healthcare embracing lean to improving quality, patient safety, and employee engagement. In some respects technology is late to the game. Barry was at the first ever KataCon this year in Miami, a conference organised by Mike Rother that included Jeff Liker and other leading lean thinkers. He spoke on the technology track which was actually the smallest track with roughly 10 people attending the session, compared with 100+ in manufacturing, healthcare, service/office.
Focus on value for customers has always been a hallmark of many long lived, successful organizations and those that get the changing role of technology in the success of their organization are now paying attention to technology. We used to see Directors and Vice Presidents in IT departments coming to us to deliver new web-based products that are reengineered from older legacy systems. Since we have released the book we have been approached by senior executives from numerous Fortune 500 companies asking how they can get started and where we can help them to use technology to deliver better value to customers.
InfoQ: "Learning anxiety is the most serious barrier to creating a learning organization and organizational change". What other major barriers are to be expected?
The biggest barrier is lack of executive understanding, sponsorship and support. Paying lip service to the concepts isn’t enough. Many executives need to change their own behaviors and organization policies and processes to create a culture that supports learning and experimentation, as John Shook highlighted from his experience at Toyota. You don’t talk your way to a new way culture, you have to act a new way to change it.
Secondly, this takes time. Teams will actually go slower as they learn new methods and techniques before they start to see improvements. This requires a long-term commitment and investment to stick with it during difficult times. This is where true leadership qualities are needed to help people keep faith and drive forward.
We also see many pathologic cultures where information is used as power source and is not shared. They will also use performance measures and rewards that drive behaviors to achieve short term shareholder earnings, but negatively affecting the long term ability to continue to deliver value to customers.
We consistently see failure to take an end to end view of business processes and how they affect all parties along the value stream. Rigid processes and structures designed to optimize functional silos such as Finance, Risk and Compliance inhibit the ability of teams to experiment and learn, or even worse prevent them from taking responsibility and accountability for the outcome of their work.
InfoQ: One of the ideas underpinning this book is the separation between explore/discovery and exploit/execution stages of a product. Would you say this is at odds with traditional program and project management practices prevalent in enterprises today?
Definitely there are some areas that are at odds with traditional project management. We’d actually like to see many traditional project management practices go the way of the dodo bird. We talk about the shift from project thinking to product thinking in the book.
Program management is largely about managing the dependencies between teams working together to deliver a service. This shouldn’t be affected by separating explore from exploit initiatives. A good program manager recognizes what stage each product is at, and knows when the transition from one stage to another occurs and new dependencies are created.
To manage explore, you need good portfolio management, something that many organizations fail to achieve. They spend most of their time and money on sustaining outdated products when they should be planning to replace and retire them, simplifying their portfolio in the process. Consequently, they actually discourage innovation because the majority of their traditional controls and processes are set up for managing execution not learning.
InfoQ: How can an enterprise bootstrap the mindshift required from being fully execution-driven to alternating between exploring and exploiting products? Does it need to start at the top, at the bottom, or both?
To get maximum value, the shift must occur both from the top and bottom. Execs are responsible for driving the culture and creating the environment where both explore and exploit succeed. They also need to pay attention to the balance of the portfolio as mentioned above.
Teams at the bottom need to embrace the technology and methods that will allow them to learn at a faster pace and make better decisions.
InfoQ: Acknowledging the idea of multiple product horizons to manage can have a massive effect in organization structure. In case of organizations with cross-functional product teams in place, how can they be sure they're not transitioning between stages too early or even too late?
We know we can never be absolutely certain, but we can reduce the risks and impact of jumping too soon or too late by looking at real data, making smaller bets and not being afraid to delay, or stop work completely.
Outlining a guiding set of principles for each horizon in terms of how teams are set up, funded and qualify to transition to next stage is a good first step. We think the Government Digital Services in the UK have a great model that they share with everyone in terms of their digital transformation. Discover, Alpha, Beta, Live outlines who and what should be involved at each stage and even goes so far as defining principles of mission for each.
Ultimately it is up to each business to create their own structures, outcomes to be achieved and measures for success, and to evolve them over time. Performing this exercise as a cross functional group is a great way to foster collaboration and break down silos in the organization and align business strategy and objectives.
InfoQ: Successful startups might face similar kinds of problems transitioning from an innovation-driven modus operandi to an execution-driven one. Plus they're more constrained in terms of people and skills. Do you have any advice to handle these growing pains and avoid becoming a traditional, command & control organization?
“Don’t F up the culture.” (quote from Peter Thiel, AirBnB)
As you grow, what works for you today will not necessarily work tomorrow.
The role of leadership is always to maintain a sense of mission and purpose for the organization, especially as it starts to grow. Then they need to create alignment at all levels of the organization, while also working out how to delegate authority and increase information flow. This is very hard. Crucially, the actions and behaviours of the leaders will dictate the culture that is propagated throughout the business.
One of the traps that growing organizations fall into is believing that adding more structure, policies, rules and processes will increase efficiencies and create more predictable outcomes. In reality, they actually decrease efficiency in communication, and the outcomes, although predictable, may not be a good thing. Policies will be needed, but they should focus on defining boundaries and providing guidance on how to make good decisions - not list rules that must be followed by all in every circumstance. This will allow teams to continue to experiment and learn in all aspects of work as the company grows. As Reed Hastings, CEO of NetFlix, says, “The best managers figure out how to get great outcomes by setting the appropriate context, rather than by trying to control their people.”
You need to accept that people that led the initial change may not be the people to lead the next stage. You will have to bring new people into the organization, others will leave because “it’s not how it used to be”. Put simply, what got you here won’t get you there. This is why discipline at the leadership level in terms of applying the principles of mission command - setting the appropriate context, defining measurable outcomes at various time horizons, and then working with teams so they can define, test and implement their own route to the desired outcomes - is so essential.
InfoQ: Independent R&D departments are common in large enterprises. Do you consider them a blocking anti-pattern for a lean enterprise and, if not, how would you go about ensuring an adequate transition process to the delivery teams?
They are not a blocker when their work is tied to business objectives and the mission of the organization. There are three big traps they can fall into. They can be so far removed from the wider organization that their work is totally disconnected from the organisation business objectives and others see them as ‘playing’ or just having fun. The second is that they come up with great ideas, but hand them off to other teams who have no ownership over the product to be developed. Finally, when R&D departments are managed and measured against the same criteria for mature product offerings they get crushed and no one is able to do R&D.
A good way to handle the transition between independent R&D and product teams is to move the R&D team (or part of it) to become the product team responsible for exploiting the ideas.
Finally, R&D should be part of regular work for all product teams to foster the learning and experimentation culture. This is why 3M invented the concept of 20% time, also adopted by Google, to encourage product teams to create and test new ideas. This not only creates new business ideas but also enriches the work experience for everyone.
InfoQ: In the book you make several propositions sure to shock C-level executives and middle management in most enterprises today. One of the most challenging is to empower product teams to prioritize work themselves (using economic measures linked to business value) instead of just pulling from a backlog of pre-prioritized work items. Can you explain the rationale and feasibility of this approach? Especially in command & control organizations where development teams might not even be allowed to use their technical environment/tools of choice?
Just to be clear, pulling from a global backlog of pre-prioritized work items is actually a good place to start for many organizations. If your engineering teams are constantly churning work due to conflicting demands from multiple customers, getting the stakeholders to meet regularly and agree on the priority of all the work that needs to be done is an enormous step forward. This is one of the most critical insights of Kanban, Scrum, and the various Agile-at-Scale frameworks.
However anyone who has actually tried the approach of taking work, breaking it up into little bits, handing it out to teams, assembling the resulting output and expecting it to achieve the desired customer and organizational outcomes knows that -- at scale -- it’s pretty hit-and-miss affair that takes an awfully long time.
You can achieve an order-of-magnitude improvement in productivity by defining very clearly and concisely (on a single sheet of paper) in measurable terms the desired program-level outcomes for customers and the organization over the course of the next month, and then letting teams work out how to achieve these outcomes through a process of experimentation. You can take this approach for product development, process improvement, and large-scale architectural change.
To make this work, you first need to lose the command and control approach and instead have a clearly communicated mission and direction, along with principles for decision making, and the ability to see what’s going on.
Second, the product teams should be cross functional and include appropriate representation from all areas of the business, not solely technology delivery experts. They aren’t making the decisions in isolation, their decision making process includes business representatives and is focused on customer value. By being transparent about how their decisions are made, anyone can easily see what is being done and why.
In the end, the teams are responsible for working out how best to deliver the desired outcomes. They often have the best ideas on how to do what is required, and when they get to try out their own ideas, they are much more likely to put in the effort to make them work. The role of executives and middle management is to support and coach the teams.
This approach requires extreme discipline at all levels of the organization. It is the opposite of chaos. Management needs to clearly communicate the measurable customer and organizational outcomes that must be achieved, and then help teams to work together to achieve them. The boundaries for decision making still need to be defined, especially the criteria to apply when a team feels the rest of the organization is working against them.
InfoQ: "Minimize output, maximize outcomes" is another recurring theme. Why are (managers at) enterprises still obsessed with getting work done rather than with value produced for customers?
Put simply, it's because the majority of them are measured against outputs (How much work did I do?), as opposed to outcomes (Was that work valuable?). Agile teams that report velocity are measured on velocity. Agile teams that report on business and customer outcomes are measured on business and customer outcomes achieved.
The challenge is that most teams are so disconnected from customers and the outcomes of their work it is hard for them to measure anything but output. It is also related to the project context. Project teams are created to deliver features. Teams understand they will only exist as long as they keep delivering new features, otherwise the project will no longer exist. These teams essentially fall into the trap of becoming feature factories, with little regard to the actual outcome of their work.
Forming customer facing product teams allows us to create feedback loops with the people we are trying to help. It enables the product team to empathize with customers, understand the outcomes of their work, and solve real customer problems.
InfoQ: What is "the local optimization folly" as you call it and what are the signs that an organization is falling prey to it?
This usually happens when an organization is structured around functional teams, which are measured by different and often conflicting metrics. The tendency is for teams to optimize their work to achieve the best results for them without considering the teams who are impacted upstream or downstream in the end to end process - not necessarily because they are stupid or evil, but because they never get to see the outcomes of their decisions.
A really good example is technical change management in IT. Project teams are measured on how fast they can crank out features, so are working hard at that, and often accept a higher risk level than the production support and operations team, who are measured on how stable the production systems are. To manage the tension between these opposing goals, there is a change management process. This is viewed as onerous and a bottleneck by product teams, but operational teams use it as a way to limit the amount of errors that go into production. It is extra work for everyone, much of it unnecessary. If both teams were measured and accountable for the same outcome - more features delivered with no downtime or rollbacks, the change management process would look a lot different for everyone. This is one of the key insights of the DevOps movement.
A few signs of local optimization include increasing wait periods to handover work between teams, high levels of re-work, low collaboration and trust between teams, constant churn, and the inability for anyone to get time to do improvement work - they’re told to just get on and deliver the features. Value-stream mapping is an incredibly powerful tool to expose these problems, and to experiment with solutions -- provided you can actually get the stakeholders from all the functions in the value stream in a room together.
InfoQ: You are very critical of current financial practices, in particular how annual, project-based budgeting kills innovation. Can you explain why?
There are several flawed assumptions made when organizations use a traditional approach to annual budgeting. The biggest one is we assume we can predict relatively far into the future what the markets and local environments are going to look like. The process requires us to crystal ball the numbers 1 - 3 years in advance and estimate accurately based on unrealistic measures like: “How much will our share price increase as a result of this work?” The decisions to allocate funding is then based on these submissions with little validated learning to support it. This doesn’t work with today's rate of change in technology, customer needs, and the impact technology can have on business. Simply put, it is getting harder and harder to do long term predictions with any accuracy. Asking your entire organization to plan this way is a poor use of their time because they will be wrong, and then you’ll yoke the entire organization’s output to pursuing this wrong path .
The other problem with annual, project based budgeting is it assumes that unless there is positive return on investment, the work is useless. Funding is awarded to those who will claim the highest ROI. There is no room for experimenting and unless you appear to be certain, you aren’t given funding. Feedback cycles and decisions on whether to continue or not are made until the end of the funding period, after most of the money is spent. Often there is no working software or validated learning on which to base future funding decisions. It is considered risky to try something new when using the annual project based funding model. It is hard to convince someone you are certain on ROI when no one has done it before.
Innovation stems from experimenting in a disciplined fashion with restrictions that encourage creativity, frequent feedback loops and adjusting based on what you learn. One of the hardest problems IT departments face is how to be creative and innovative when most of the operational time and costs are consumed in improving and supporting a bloated portfolio created by projects. Until we stop the project based approach to improving IT, there will very little time and investment for technical innovation to happen.
InfoQ: Getting financial departments to change their ways seems a total different ball game from implementing lean practices in product development. Where to start?
Large scale change of financial departments is complicated and usually easier said than done. It requires complete buy in and support from the entire executive team and the Board of Directors. It’s not going to work in an organization when short term shareholder return is the number one priority and executives rewards are based on short term objectives such as quarterly results and share price.
However, if you start to talk about the financial risks associated with the large project based approach to development, the conversation on how things might change becomes easier. Talk to your Finance department about smaller bets with more frequent funding decisions, even on ideas that have been previously approved.
The capital vs. operational conundrum often crops up in these conversations. It’s really not that hard. We can still figure out how much we want to spend on capital expenses in a year and use that as a target. Instead of divvying it out all at once, we just break it out to small chunks allocated more frequently. The allocation of a product team to capital and operational can be made using estimations of how much work is done in the different work categories as defined in SOP 98-1 (Statement of Position 98-1 Allocation of Costs of Computer Software Developed or Obtained for Internal Use, Financial Accounting Standards Board, 1998 ) (note this was written in 1998, seriously out of date with the way technology is developed and used today). A good financial analyst should be able to sit down and find a way to allocate the costs in the right bucket with a little help from the product managers, becoming part of a high performing cross functional team. The trick is to get the feedback and decision making on continued funding down to shorter and shorter time frames - which may require (you guessed it!) a bit of experimentation.
InfoQ: You also call out the myth of "technical talent shortage", pointing out flawed recruitment practices which focus on candidates' existing skillset instead of focusing on their ability to rapidly acquire new skills and knowledge. Can you exemplify how would you evaluate a candidate's growth potential? In other words, what sort of recruitment practices can effectively assess candidates' learning skills?
We’re not HR experts, so the first thing is go ask them for help. They can help with any psychological and personality assessments that may provide insights into the candidate's’ ability to learn. Although we give some suggestions that may help you with a decision on if a person is a lifelong learner, these tests can reveal significant traits that you may not pick up on.
A couple of ideas:
Don’t restrict your requirements to specific education and experience profiles. You won’t find people with 5 years experience in using the latest technology. People who apply for a job outside of their own experience and education are more likely to be those willing to learn. Look at their experience for examples of learning in different situations and growth over time. When screening written applications to short list, ask for removal of any personal information that may cause you to apply your own unconscious biases to select a person most like yourself.
One of the key observations we make is that moving away from the shopping list of existing skills is also going to enable you to increase diversity. There is good evidence that diversity at all levels of the organization provides a competitive advantage in terms of fostering a creative learning environment. The fact that so many engineering departments (at least in developed economies) are overwhelmingly white and male indicates we’re missing out on an enormous pool of potential talent. We need to put in the effort to seek out and develop it.
Pay close attention to the ability of candidates to listen and communicate during interviews, as it is an important skill for learning in the work place. Do they take the time to understand the questions? Do they answer the questions thoughtfully? Do they actually admit to not knowing the answer, but give some idea of how they would go about finding the answer? Do they acknowledge everyone in the room during the interview? Do they refer to past successes as of their individual efforts, or do they recognise the contributions of others?
Don’t forget to get a diverse set of people to perform interviews and observe their interactions. As mentioned earlier, diversity plays an important role in developing creativity and innovation in a team. If you have a group of people who all come from similar backgrounds and experience who can’t relate to people outside of their own comfort zone, you can’t expect them to empathize well with customers. That’s going to severely constrain your ability to innovate.
InfoQ: You reveal at the end of the book your epiphany that effective process improvements require the scientific experimentation method as well. Is it possible to combine such approach with frameworks like CMMI or ITIL that prescribe maturity levels where a very concrete set of practices is expected to be in place simultaneously?
We don’t see any conflict with using framework and standards to help you figure out how to get better at what you are doing. However, we do worry when people dogmatically try to follow practices and rules from these frameworks. No two organisations are the same. What worked in one will not always work in another. What is more important is teach people to solve problems in their own organisation and improve the value delivered to customers. This is at the very heart of Lean thinking. Set the direction you aim to go. Understand where you are and define a target condition to get to. Then create an experiment to move towards that target condition. Then use what you learn to update your vision and inform your next step.
Frameworks and standards like ITIL, CobiT, TOGAF, SAFe, ISO 27000 series, etc. are really good for helping organizations determine what good might look like and setting targets for improvement. One of the five main sections of ITIL is called Continuous Improvement. Maturity models, including CMMI, are just a tool that helps you define where you are in a journey to get the best use of technology. Level 5 is where the process is fully automated and continuous improvement is applied.
All of these are good at defining what better could look like. The traps organizations fall into is either thinking they need to get to Level 5 everywhere, or believing that applying the framework will solve all of their problems. But there are no silver bullets, or shortcuts to thinking for yourself and experimenting. If you treat frameworks and standards as activities that must be performed by everyone in the same way to get defined outputs, you will never be successful. Instead: Plan, Do, Check, Act - taking ideas from many different sources and seeing what gives you better outcomes; then get ready to do it again, and again, and again.
If you are in an organization that must be certified to a certain standard or compliant with a regulation, the situation changes. Most certification will be based on audits that assess if you have the required documentation for processes and then can produce evidence that you follow your own process. Most do not measure if you are good and deliver value. The number of payment card data breaches that have happened in PCI compliant organizations over the past 2 years is only one example of how compliance and certification doesn’t necessarily mean you are going to achieve the outcome the standard is designed for. (In the case of PCI, it is largely the case of bad guys being smarter in the use of technology - they constantly perform experiments to locate and hack into susceptible systems that have been designed to meet the standard)
About the Book Authors
Jez Humble is a vice president at Chef, a lecturer at UC Berkeley, and co-author of the Jolt Award winning Continuous Delivery, published in Martin Fowler’s Signature Series (Addison Wesley, 2010), and Lean Enterprise, in Eric Ries’ Lean series (O’Reilly, 2014). He has worked as a software developer, product manager, executive, consultant and trainer across a wide variety of domains and technologies. His focus is on helping organisations deliver valuable, high-quality software frequently and reliably through implementing effective engineering practices.
Joanne Molesky is a Principal Consultant with ThoughtWorks, where she works on internal IT Risk and Compliance, and provides consulting services to clients in the area of continuous delivery and process improvement, particularly as it applies to controls, risk, and compliance. She holds CISA and CRISC certifications from ISACA.
Barry O'Reilly works with ThoughtWorks, consulting with leading global organizations on continuous improvement using lean and agile practices and principles. He has been an entrepreneur, employee, and consultant. His passion is business model innovation, product development, organizational design and cultural transformation.