Interview and Book Review: A practical approach to large-scale Agile development
In the book “A Practical Approach to Large-Scale Agile Development”, the authors Gary Gruver, Mike Young and Pat Fulghum tell their story about applying agile and lean principles in a large scale software development program for the HP laserjet futuresmart firmware.
The authors explain in the book why agile was chosen, and what their expectation were when they decided to implement agile practices. Focusing on the why first, the business needs, gave them a better understanding of the potential benefits of agile, and helped to keep focus while adopting agile and lean practices. The book also describes how they have implemented agile, which practices were chosen and which ones not, and how they fine tuned the way of working along the way.
This book is a description of a large scale agile implementation, with the nuts and bolts of actually doing agile. It shows that the agile journey that HP laserjet futuresmart firmware took wasn't easy, but that is was worthwhile doing it.
An excerpt of the book is available for download (Chapter 2).
Ben Linders from InfoQ did an interview with two of the authors of “A Practical Approach to Large-Scale Agile Development”, Gary Gruver and Mike Young:
InfoQ: In the book you describe your own top 6 agile/lean principles. Why did you choose these 6 principles, and why only 6?
Gary: We just felt these were the 6 that in our experience were most relevant. Kind of a combination of what we found was working from both Lean and agile.
Mike: We tried a lot of things, and like Gary said, some worked better than others, and these specific principles worked the best of all (if it doesn’t help the business, don’t do it; if it works, keep doing it). So we made them our underlying focus for the organization. We picked a small number to focus on in keeping with Lean principles. We actually only listed 5 to start with, but unless something actually works in the real world, it’s not worth pursuing – so we added #6 as really the most important (Practitioners should define agile/lean practices). A principle always eventually ends up as a practice, and when it does, it must be palatable and even motivating to engineers/programmers.
The 1st one (Reduce Overhead/Waste) was picked because of a personal pet peeve we had. We had so much to do and never enough time, so we focused on eliminating anything that would slow us down and wasn’t getting us towards our overall business objectives.
The 2nd principle (Don’t overfill your plate) was picked not only because it’s the essence of Lean, but also because it has been the hardest one to keep going and needs the focus. There’s always such a business demand for more and more, and since managers and engineers always want to please upper management and marketing, we have to keep this front and center or we get bogged down in doing too much again and ends up reducing our throughput and quality (counterintuitive but true).
The 3rd principle (Cater to the bottleneck) was picked because of where we’d come from. Before we started this transformation, everyone was measured on what their team contributed, not on how they helped the whole. We were big believers in the age-old classic book by Goldratt, “The Goal”, where he talks about Herbie, the slow hiker, and how the whole group of hikers ended up going faster when they catered to Herbie. It really works!
The 4th principle (Integrate early and often) was picked because of #6 (Practitioners should define agile/lean practices). We came up with several supposedly impossible visions, the first one being “10x developer productivity”. Slow integration times was the developers’ biggest complaint. So we decided to solve it. Plus, continuous integration is such a heartbeat of the whole agile experience.
The 5th principle (Planning rhythm) became a focus because we had to set the right expectations with the broader business. We were previously being asked to make a high-confidence commit to the full set of features for each product a year before product introduction. So we would stop nearly the whole development effort for two or more weeks, get everyone into planning mode, and come up with detailed designs and work breakdowns and resource plans for everything we thought we could fit in for the coming year. Even though we established a “feature cut line” from this exercise, the moment we discovered more work, had someone change jobs, or had marketing come running in with a last-minute high priority request, we were stuck and couldn’t respond. And we were missing our schedules because of what we’d committed to. So we turned planning upside-down and, thanks to some key believers in marketing, we reset expectations – that you won’t get more just because you plan more. Frequent, short planning cycles have been key to both keeping the organization focused on delivering to the current priorities, and allowing the requestors to be agile in giving new input. Using a 4-week planning rhythm now allows us to adjust our focus and priorities for every 4-week Sprint based on learnings from the last one (so the biggest challenges and business imperatives are our top priority).
InfoQ: At the start of your agile journey, you investigated the “capacity of the organization to absorb change”. How did you do that?
Gary: This was and always is challenging. As the leader of the organization, I constantly have to figure out how much a group can do. Take on too much and you never finish anything plus the team will never feel successful. This was really a learning process that was part of our Mini-Milestones objectives evolution (editor note: A Mini-Milestone is the term used at HP for a 4 week agile sprint). In the beginning we would sign up for more than we could do within every Mini-Milestone. At the end of each month we would review what we got done and set new objectives for the next month. Over time you get more realistic in terms of the organizational capacity and force the priorities very hard. The reason monthly felt like it worked well for us is that it takes awhile to drive really good realistic priorities and objectives. These objectives were much more than an aggregate of stories.
Mike: I ditto Gary on this one. Gary had a keen sense of when the organization was taking on too much. So we’d back off where we needed to. It was interesting, though. The more we made the focus of “change” be with the intent of improving things for the developers, the more change everyone was able and desirous to absorb. It’s the whole WIFM concept (What’s In It For Me?). Once a team sees where they’re going and understands how much better it will be when we get there, they almost always asked for and yearned for more change. Gary was our visionary, and was constantly following the favorite practice of HP’s founders (Bill & Dave) of MBWA (Managing by Wandering Around) – going into developers’ cubes and finding out what was going well and what wasn’t, and we continuously adjusted our rate of change accordingly.
InfoQ: The book mentions that your architect “would purposely make changes that would break the code until it was architecturally correct”. How did your software developer react to this?
Gary: The developers would come to me complaining . As the leader I had to back him up in that the hit to short term metrics was worth the benefits of a long term supportable platform. This was hard as I would have to take flack from the broader organization when the visible metrics took a hit but it was the right thing to do. Was it an effective way to manage your technical debt? It is not the only approach for managing technical debt but it sure did help. I am glad the architect drove this even though it was a pain at times.
Mike: Once everyone saw we were serious about this, they reacted by making architectural integrity a part of their daily focus.
InfoQ: You tried both top-down and bottom up approaches to implement agile. Which approach did you finally chose, and why?
Gary: It was a balance. The leadership team was driving overall objectives and doing Mini-Milestones reviews every month. At the same time we were reviewing metrics and spending a lot of time with the engineers learning what was and wasn’t working. Most of the good ideas for what to fix came from the bottom (i.e. the people doing the work). That said, the leaders needed to be driving, listening, and shifting focus based on what they heard.
Mike: Our previous top-down approach failed because it felt forced on us. Our previous bottom-up approach failed because it couldn’t get traction or funding. We could never have done this without an executive sponsor and a director who cared deeply about making it real (and funded it). The key to actual success was both top-down excitement and sponsorship combined with “in the trenches” ideas to make it real. Gary would come up with what we called “hallucinations” (like “what if we could build and integrate changes in an hour” [when it was taking a whole week]; or “what if we could run every test every night” [when it was taking 6 weeks]). He would share these ideas with some thought leaders in the organization, who would tell him he was crazy, then they would come back a few weeks later with details on how it could be done. All of these visions were focused on making life better for engineers/developers, not for management. That was the key to success. It’s all about vision plus motivation.
InfoQ: You introduced a new role, the system engineer. Why this role, and how does it help you to work agile?
Gary: It really enabled us to move planning from a major waterfall type event that happens by everyone at certain time slots, to a process that was constantly moving works-in-progress through the requirments process. A focused group of people on requirements that did not have to context-switch and when the developers were ready for the next batch of work, the system engineer would lead a feature kickoff meeting with a pretty clear definition of what the business was requesting.
Mike: I’m pretty passionate about this as I managed the system engineering team for a few years. Gary described it well. In agile, having just-in-time user story definition is key to efficient development flow. The other big impact is that the stakeholders in the business who request most of the user stories (marketing, customer support, etc.) have a strong relationship with a real person that they come to trust and who they know is on their side to do the best thing for the business and customer. And who can effectively bridge the gap between the business and the technical community.
InfoQ: How did you manage the agile collaboration between the teams in the US and India? What worked, and what didn’t work, why?
Gary: Collaboration was handled at the management level and the individual level. We got to where the team members in India were just an extension of the team in the US. Have them spend some time in the US getting to know their team. Then migrate that working relationship back to phone and e-mail. At the management level we did video conferences every other week with the expectation that anything being presented had been agreed upon by the first- level management teams on both sides. The biggest ways for the management teams to disgree and not align is if we were hearing different things from each of our teams. We worked hard to make sure the first- level managers knew that was not acceptable to go to “Mom and Dad” with different stories, so to speak.
Mike: It’s also critical to convince the U.S.-based teams that their jobs are not at stake. We had some situations early on where developers in the U.S. were very nervous they were “training their replacement.” Over the years, as everyone has realized that going global creates more flexibility but also allows for jobs and opportunities to remain intact, it creates a powerful team. And Gary’s right on that it’s critical to develop a trusting relationship, and it’s amazing what happens when engineers and managers actually get to know each other. At critical times, you can literally work around the clock on getting things done. One thing it really magnifies is how critical having clear priorities and real-time automated metrics is. If you have to rely on daily meetings it can ruin work-life balance for everyone.
InfoQ: The chapter “Real-world agile results” describes some of the benefits that were realised using agile. It mentions that R&D and Developer productivity has increased. Can you elaborate on that?
Gary: Our focus was really on driving developer productivity. You can measure this lots of different ways which we tried to present in the book. The other thing that is interesting to watch is what happens when people move out of that development environment into a more traditional SW development group. I have mentored several employees that left FutureSmart Firmware for other groups or companies and their reaction is universal. They don’t know how to work without large scale test automation and processes with continuous integration_(CI), like auto-revert. They feel exposed and struggle to be as productive as they were in this environment.
Mike: Productive developers are happy developers. And when they feel like concern for their productivity is driving ongoing decisions in the business, they feel supported and motivated.
InfoQ: If a company wants to become more agile, what can they do to take a first step?
Gary: This may sound funny coming from an agile author but the first thing I would recommend is that they never start by developing a plan for an agile transformation. It is a losing battle. There will always be someone in the organization that says they read being agile means you have to do XX. There are just too many options for XX such that you will never be done and thus can’t ever successfully do agile. That said there are a lot of good tools in the agile toolbox you can use to help transform your business.
The first step is making sure you clearly understand your business objectives and value proposition. I too frequently see people who pursue agile and then struggle to document the benefits to business leaders. You should start with a clear value proposition so there is no question about how to measure value.
The next step is the basic activity- based accounting exercise of how you are currently spending your resources. Then you know what you want to do and where you have waste in the system. At that point you can start looking at your agile toolbox to see where you will get the most bang for your buck. Personally, I would almost always start with automated testing and the Continuous Integration process. If you don’t have that I don’t understand how you meet the definition of “done”, which is critical to any agile effort.
Mike: The other big mistake businesses make (besides “doing agile for the sake of doing agile”) is trying to bite off too much. Coming up with realistic, prioritized objectives that can be achieved in a 4-week Sprint is key. Nothing in our agile world exists except completing the current 4-week Sprint and doing light-touch planning for the next one. If you focus on the right things in a Sprint (with actual working demos at the end), then you show the business you can actually deliver what it needs and you start to get an understanding of what you’re capable of delivering as an organization.
About the Book Authors
Gary Gruver is formerly the Director of Engineering for HP’s LaserJet Core Firmware Lab, and he worked at HP for 22 years. He is currently VP of Release, QA, and Operations at macys.com. Gary has been able to bring a “manage to metrics” approach that rallies everyone to common measurable objectives without requiring lots of meeting and coordination overhead. Of course, his most critical role is buying lunch during particularly busy sprints for anyone working weekends to finish off key features. His favorite hobbies are cycling and skiing with family (he’s married with two daughters).
Mike Young is the senior program manager directing day-to-day efforts across many distributed teams. Mike has been involved in development of HP LaserJet Printers for 17 years and previously designed satellite control systems for Hughes Aircraft Company. He also is one of the strongest advocates of agile approaches and helped get the organization started down this path before anyone really knew we were doing agile. His hobbies are family (he’s married, with two daughters and two sons) and playing racquetball. In agile, we’ve found that a program manager should spend most of their time watching the metrics and quietly coordinating behind-the-scenes to cater to the bottleneck. In our sprint checkpoints, we tend to minimize slideware and maximize problem-solving and demos of new user stories.
A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware” by Gary Gruver, Mike Young and Pat Fulghum, is published by Pearson/Addison-Wesley Professional, Nov 2012, ISBN 0321821726, Copyright 2013 Pearson Education, Inc. For more info please visit the publisher website.
Deployment Automation 101IBM