In the book Leading the Transformation: Applying Agile and DevOps Principles at Scale executives Gary Gruver and Tommy Mouser share their experiences with applying lean and agile development methodologies in enterprise development teams.
InfoQ readers can download a sample of the book Leading the Transformation.
InfoQ interviewed Gruver and Mouser about problems that large organizations experience with agile, how enterprise wide continuous improvement helps to increase agility, managing continuous improvement, experiences from the HP LaserJet firmware development lab, the benefits of combining DevOps and continuous delivery, what executives should do to enable a transition to continuous delivery and advice for executives if they want to adopt agile and DevOps in their organization.
InfoQ: What made you decide to write this book? For whom is it intended?
Gruver: The decision to write this book was based on wanting to help other executives that were trying to improve the development processes in their organizations. There are lots of places that engineers can go to learn the latest approaches to software development but if the executives don’t foster the right environment for implementation the spark of enthusiasm from the engineers will be smothered by resistance to change in the organization. Therefore, the executives need to understand the changes that are required and their role in leading/supporting the improvements. Our goal in writing this book was to share everything we wished we had known before we started leading large scale transformations so others won’t have the learn the same lessons the hard way.
Mouser: We decided to write this book because over the past several years there has been a growing trend of applying Lean and Agile development methodologies to large Enterprise development teams. While the methodologies certainly apply to both small and large teams, we have found that many of the tools and process need to be modified before they can provide their maximum benefit within a large organization. For example, the tools to manage Continuous Integration (CI) activities for a large, geographically dispersed team are much different than those needed to manage CI for a team of 6 engineers all sitting in the same room. When we made our first attempted at applying Agile methodologies to Enterprise class software development we discovered that there was very little information available on how to do it and we ended up doing the trial and error method. Gary’s first book was used to document what we did and what we learned. This book is a continuation of those learnings and is targeted at the Enterprise executives who are sponsoring and supporting an effort to move their organization to Agile methodologies. Our intent was to provide them with a roadmap to help guide them through the transition. This roadmap is based on our experiences and we hope to provide them with some our learning and to help them enter into the transition with the proper expectations.
InfoQ: Can you give examples of the problems that large organizations experience when they start with agile?
Mouser: I think one of the more common problems that large organizations face is the fact that they attempt to take individual development and qualification teams within the organization and make them Agile, as opposed to making their entire organization and it’s processes Agile. For example, many large development organizations have a Program Management Office (PMO) and historically the PMO and Marketing organizations would participate in a Waterfall type process where they would plan out and get funding for next year’s software projects. As part of this planning, the development teams are asked to size the projects and the funding is then based on those estimate. The result of this process was to lock in funding and high level requirements for the next year. These projects are then given to the development teams and they are expected to “be Agile” and stay within the scope, schedules and budgets that were likely set months before the work actually began. All of this results in parts of the organization using Waterfall type processes and other parts attempting to use Agile. As you might expect this inevitably results in conflicts that the Executives typically have to resolve.
One of the objectives we had was to highlight situations like this and make Executives aware of the need to look holistically at their organization and then begin this journey fully aware of these types of pitfalls.
Gruver: The biggest challenges of any large transformation are organizational change management and getting people to change how they work on a day-to-day basis. Additionally, in large organizations typically the biggest inefficiencies are in coordinating the work across teams to deliver value to the customer. Most organizations though start with applying the Agile processes at the team level instead of applying the basic Agile principles of continuous improvement, prioritized backlogs, and releasing code to the customers on a frequent basis for feedback at the system level. They convince the high-level managers to transform to Agile and then bring in Agile coaches to start working with a few prototype teams. Once these teams have demonstrated the advantages of Agile and they have defined how everyone in the organization will work and it is just a matter of training them how to do it right. Then overtime they start to realize they still have big problems coordinating the work across teams and start to figure out how to address the enterprise level issues.
This approach tends to overlook that the coordination across teams is typically the biggest source of inefficiencies in larger organizations so it is where they should start. It also overlooks the fact that people are much more likely to change if they have some input and ownership for the new approach. As much as possible you want to let people get their fingerprints on the plans so they will ensure their ideas are successful. This works well for the first few prototype teams that get to define how agile will work for your organization but then as these approaches are rolled out that principle is frequently lost with the additional teams. It also ignores how important it is to get the management teams fingerprints on the plan if they are going to help make it successful. Most transformations start with getting the top-level executives signed up and then have Agile coaches start working with the teams. This approach completely skips the management chain in the middle other than telling them we are going to do this and your job is just to stay out of the way and be supportive. This might seam expedient but overtime you will start to realize that this results in significant resistance to change. Instead we would recommend that you include the entire management team from the beginning coordinating the all important improvements across teams with the executive led continuous improvement process described in the book. You should also give the additional teams during the rollout as much flexibility as possible with their processes so they take ownership for its success. Force commonality only where it is required for coordinating the work across teams and provide as much flexibility as possible everywhere else.
InfoQ: Can you elaborate how having an enterprise wide continuous improvement process helps to increase agility?
Gruver: From my perspective learning and adjusting with a continuous improvement process is the most important principle of Agile. Each transformation, development process, and organization is going to have unique challenges and barriers to efficiency and effectiveness. Therefore you have to be constantly examining your development processes to identify the biggest opportunity for improvement. Once you have prioritized and fixed that issue you need to discover and address the next biggest issue. It is this on-going process that enables you to achieve the business results that are possible with a large-scale transformation. Having your teams implement scrum processes with an Agile coach is just not going to deliver the results that are possible by focusing on the specific challenges of your organization with a continuous improvement process.
Mouser: Having your entire organization working within a continuous integration(CI) process allows you the react to customer feedback or competitive pressure much more quickly because every part of the process, from idea generation to delivery to the customer, is streamlined. The CI process also allows you to quickly find and deliver the minimum viable solution and get it to your customer without taking the time to work on other capabilities that can be delivered at a later time.
InfoQ: My experience is that continuous improvement can be hard to do and difficult to manage. Can you give some examples how organizations can deal with this?
Mouser: This is especially true in large enterprise organizations. There are often many geographically dispersed teams that do not have the same goals and objectives. This is why it is imperative to start and then manage any continuous improvement program based on a set of well understood and document business objectives. For example, you might want to increase your feature throughput by 50% so that you can get new features and capabilities to your customer faster. With an objective like this each team can work to measure their current throughput and put new processes and tools in place to increase their throughput. In addition, as a senior executive in an organization you now have a set of metrics around throughput that you can measure. This will help you understand where the bottlenecks are and you can focus your attention on these areas.
One important note about these metrics that Gary has emphasized in both of his books is that these metrics should never be used as a stick to “beat” on the teams that are a bottleneck. Instead, they are simply indicators that point you to the places where you need to go have a conversation. You’ll often find that the team that is the bottle neck is dealing with issue far outside their direct control and they will need you as a senior leader to engage with the broader organization to resolve the issues.
Gruver: It is interesting that your experience has led you to believe this is hard. In my experience as an executive leading large organizations this is pretty straightforward and easy. You just need to write down your top priorities for the month, share it with the organization, track progress during the month, and hold a checkpoint at the end of each month to review what got done, what didn’t get done, and what you learned during the month to set the goals for the next month. This part is pretty straight forward if you are leading an organization and has been fairly natural for me.
The harder part I am finding while I am working with more and more organizations is getting a cross organization team (Dev, QA, Ops, Sec, Business) of executives to engage in leading the process with a common set of priorities. This is where I am hoping the book will provide them a common language and approach for coming together and lead the transformation. If that is not enough then I also do workshops to align the executives with the organization and get the executive lead continuous improvement process started but hopefully the book will be enough for most executives. If you have this level of aligned and engaged executives, I truly believe the continuous improvement is fairly straightforward and natural. Getting them aligned, engaged, and started is the hard part.
InfoQ: Can you describe the approach that the HP LaserJet firmware development lab uses to do continuous improvement?
Gruver: We used the process described above where my staff that was responsible for leading hundreds of engineers around the World would agree on our priorities for our monthly iteration. These priorities would include both business deliverables and continuous improvement efforts. Once we agreed on these objectives our focus was entirely on executing to those priorities. The engineers understood that if they had something on the list or could help someone else that had something on the list it was their number one priority. If there was nothing on the list then they could work on their team level priorities. It was a nice combination of top down coordination of changes and bottoms up effort.
The management team then spent our time during the iteration playing the role of investigative reporters trying to understand the reasons why the things we felt were achievable and our top priority were not getting done. We would spend time out in the organization talking with everyone to better understand what was getting in the way. We would talk with the engineers that were assigned tasks that were not getting done to understand why. We didn’t do this by accusing them for why it didn’t get done. Instead we would start by saying that we both agreed it was important and achievable but it still was not getting done so why not? What happened? It is through this process that we started understanding the biggest obstacles to achieving our goals so we could prioritize removing the barriers during the next iteration.
To be honest the first few times as a leader over ~800 developers when I talked with an engineer that was not completing the objective it was a little intimidating. Then over time they realized I was helping to prioritize fixing issues that were getting in their way it started creating a level of trust. With this trust came transparency and with transparency came an even better ability to fix issues. A lot of people talk about the cultural changes associated with DevOps but there are very few concrete examples. Hopefully, this starts to show how the executives can start playing an active roll in helping to lead the cultural transformation by engaging in the continuous improvement process.
InfoQ: Do you have examples showing how managers have played an active role in an agile transition?
Mouser: When a large organization makes the decision to transition to agile development methodologies there is a tendency by most senior leaders to disengage from the day to day activities after the decision is made. They assume that the middle and first level managers now have the needed directions and they will then go make it happen. More often than not this leads to different parts of the organization going off to work on different things that they believe will make them more agile. They will pick different tools and different processes that may or may not be compatible.
Senior managers need to stay more closely engaged with this transition. During each iteration planning and review session they need to see the metrics around the original business objectives that the organization aligned on as the drivers for the transition. If progress is not being made on those objectives they need to ask why, what is the blocker? This often leads to the discovery of issues that on the surface don’t seem to have anything to do with an agile transition. For example, perhaps the teams need more VM’s because the build and test process are taking too and developers are unproductive while they wait on test results.
InfoQ: Which approach did HP use to find waste in their software development process?
Gruver: We started out by being very clear about the business objectives of our transformation. The Firmware organization had been the bottleneck of the LaserJet business for a couple of decades and we struggled to add the features the organization really needed to differentiate in the market. Therefore the business objective was to ensure that the organization could add new printers to our product line without having to worry if Firmware could support it and free up capacity for innovation. This really led to a focus on productivity. We looked at anything that drove our cycle-time or costs that was not key to our business objective and looked to either automate it or engineer it out of the system. This helped to focus our continuous improvement processes where they would add the most value to the business. It was this focus on business objectives and continuous improvement over time that really drove our dramatic improvements.
Mouser: At the end of each monthly iteration cycle the entire managerial and technical leadership team met to discuss how we did in the previous iteration. We looked at each objective we had set and how we did against that objective. For any objective that we did not fully complete we asked why, what was the blocker? This process quickly led us to discover the wasteful and ineffective activities that were built into our processes. Each month we identified the top blockers and then prioritized for the next iteration the work required to improve or eliminate the blockers. We repeated this monthly process for several years and we always had a backlog of process improvements that could make us better. This process worked because the senior most manager in the technology group was directly involved in these discussions and could make real-time decision regarding how to prioritize the work to fix the issues.
InfoQ: Can you elaborate on the benefits of combining DevOps and continuous delivery and explain why you should do both?
Gruver: This question gets a little more difficult to answer because it depends on how you define DevOps. In the beginning I think DevOps was just associated with the cultural changes of getting Development and Operations to work better together and continuous delivery was about creating an automated deployment pipeline. Overtime though as DevOps has started gaining more momentum in the industry I have started hearing it defined as including all the technical and cultural changes required to release code to the customer on a more frequent basis while maintaining or improving quality, security, and performance. I try not to get to wrapped up in the names and which is which and tend latch on to that definition of the business objectives with the understanding that it really can’t be achieved without the appropriate technical and cultural changes.
I guess if you have a small organization or even a large organization that has a loosely coupled architecture where small independent team can develop, qualify, and deploy code in isolation then you can probably release quality code on a more frequent basis by just putting an operations person on the team and having them work closer together. The reality though for most large organizations I work with is that they have large groups of people working with tightly coupled architectures that have to coordinate their work to develop, qualify, and deploy changes. This is just not possible without the structure and rigor that a well-defined continuous delivery pipeline provides. Additionally, a well-defined continuous delivery pipeline will not help that much if the organization does not change how they work on a day to day basis to take advantage of the new system. Culture is not enough in a large organization with a tightly coupled architecture and the technical changes will have limited benefits if the culture does not change to take advantage of the new system. Therefore you really need both or at least your definition of DevOps needs to at least cover both aspects.
InfoQ: What do you suggest that executives should do to enable a transition to continuous delivery?
Mouser: I think the most important thing that senior executives can do to enable a transition to CD is to ensure that their entire organization is aligned and working towards this transition. It is not enough to simply tell the development teams to have “always releasable” code on trunk. All the teams involved must streamline their processes. If your marketing team has to go do 6 months of research and customer surveys before they can tell you what is needed you will never achieve a true continuous delivery processes. Everyone must align on the idea of small, incremental changes brought in by a continuous integration process that are then presented to the customer so that you can get real-time feedback and adjust your strategy based on that feedback.
Gruver: They should probably start by reading the book to give them a broad overview of the entire landscape of a successful transformation. They should also have as many of their peers read the book as possible to provide a common language and perspective for the leadership team. The next step is clarifying the business objectives of the transformation. You should not do Agile or DevOps if your current development and delivery processes are meeting the objectives of your business. It is just too much change and turmoil if what you have is meeting your needs. The problem though for a lot of organizations like the LaserJet business is that software is becoming more and more important to the business and the current development processes are not sufficient. If this is the case for executives reading this review, I would suggest they start off by clarifying the business objectives of the transformation. This will be used to gain support across the organization, prioritize improvements, and show progress over time.
The next step depending on your role is either getting your staff or peers to join you in leading the continuous improvement process. Once you have the team in place have the executives bring together your management and technical leadership together to identify the objectives for the first iteration and start to process of continually getting better.
InfoQ: Can you share some learning from the test automation journey that he HP LaserJet firmware development lab took?
Gruver: We started by highlighting test automation as one of our biggest objectives across my staff. We all agreed it was important and critical to our success. We reviewed the status and progress on a regular basis. Looking forward we were always trying to do more and get better. That said to be honest every time I looked back at where we were struggling I realized that we had been under investing in terms of technical and management leadership. You can’t create a solid stable, maintainable, and triagable test automation framework without strong leadership on both the development and QA front. Probably the biggest challenges I have had over time is getting the top technical talent in development to engage in driving the test automation framework. It is just not seen as sexy and as fun but it is so critical to successfully changing how you do software development I don’t think you can avoid having your top people across development and QA working together on a solid approach. Looking forward I always felt like I was investing a lot in test automation. Looking back I always realized I was under investing in the horsepower we brought to the problem.
Mouser: I think one of the most important things I have learned about test automation is that for it work properly and add it’s true value, you must also automate all the steps in the process leading up to the automated test execution. Given the same set of code artifacts you must be able to exactly replicate the environment you are going to be testing. If any parts of your build, provision, or deployment processes are still manual you must first address this issue. For example, if your hardware provisioning process sometimes results in your software being deployed on a VM with 8 CPUs and other times it results in a VM with 4 CPUs, your automated test results will likely vary for reasons you can’t easily explain. A test that passed on the 8 CPU system may timeout on the 4 CPU system. Your teams will end up wasting an incredible amount of time triaging these failures that have nothing to do with the code that was just checked in.
For automated testing to work properly you MUST have a completely repeatable and reliable build, provision, and deployment process. Focus here first, then build up your automated test suite.
InfoQ: Do you have some final advice for executives if they want to adopt agile and DevOps in their organization?
Mouser: My advice to senior executives is to get involved and stay involved in this transition. You don’t need to be a software expert, but your business management skills are imperative to the success of this transition. Always remember that you are doing this transition to meet some specific business objectives. Your technology teams can lead and drive the details of what needs to be done, but you must keep them and their other business partners (Ops, PMO, Marketing, etc) focused on the business objective they are trying to reach and ensure that everyone in the organization is making the needed changes to ensure success.
Gruver: The biggest challenges by far are organizational change management. How are you going to get everyone onboard and heading in a common direction? Also how are you going to get support from the business side to support sharpening the saw activities instead of just spending all your resources cutting with a dull saw? It is important to get everyone’s support and fingerprints on the plan to help avoid resistance over time. This is also going to be a journey that is going to take some time so work to be as transparent and inclusive as possible during the journey. If everyone is part of the team and feels they have some ability to influence the direction it will help in terms of getting them onboard to help lead the changes.
About the Authors
Gary Gruver is an experienced executive with a proven track record of transforming software development and delivery processes in large organizations. He is currently working with Executives across the industry to help them transform their development and delivery processes. As Co-author of “Leading the Transformation” he shows executives of large traditional organizations how to lead their transformation by applying Agile and DevOps principles at scale. As co-author of “A Practical Approach to Large-Scale Agile Development” he documents how they revolutionized software development processes at HP while he was the Director of the LaserJet Firmware development lab. As VP of QE, Release and Operations at Macy’s.com he led their transition to continuous delivery. Blog: practicallargescaleagile.com Twitter: @GRUVERGary
Tommy Mouser has been directly involved in the design, development, qualification, and delivery of software systems for over 30 years. For the past 8 years, while working at HP and Macy’s.com, his primary focus has been on working with his teams and partners on a journey towards Agile methodologies. He has led, managed, and applied these principles with many teams, both large and small, and has experienced the significant quality and productivity gains that are possible. Tommy currently resides in Boise, Idaho with his wife Debbie. He enjoys a wide range of outdoor activities and has recently gotten into fly fishing with some of his longtime friends and co-workers.