Transcript
Hendrickson: My name is Elisabeth Hendrickson. Let's talk about speed, more specifically, speed of software project delivery. It's amazing, nobody ever says, "I want to figure out how to go slower and be more methodical, and cautious." Everyone I talk to typically wants to figure out how to go faster. This was true when I was a consultant. It's been true in most of the jobs that I've held. Everyone wants to go faster, and yet, our intuition is a terrible source of ideas for how to go faster. The most common ways that I hear is, got to hold somebody's feet to the fire. Fairly violent. Got to set deadlines, because that's the way to create a sense of urgency. In my experience, these techniques, they tend to backfire. They don't go quite the way people expect.
Pressuring Engineers to Deliver
In fact, let's start with a story. This story takes place a few years ago, in an open office plan, back when we had those, before COVID. I want you to imagine, it's me and one of my peers, the head of product management. The two of us are in a tense discussion. For our purposes, I'm going to call him Jay. That's not his real name, but he needs a name in this story. Jay has been pressuring me for a while, not just on this particular day, about how to get the engineers to go faster. He's frustrated with the pace of delivery. He has, after all, made commitments to our customers, and engineering is not holding up their end of the bargain. For the moment, I'm going to skip the fact that he made these commitments without actually consulting engineering, but that's neither here nor there.
Jay and I are having a tense but still very professional discussion. Neither one of us is raising our voices. Neither one of us is losing it. We are clearly on opposite sides in a very critical issue. At this point in the conversation, Jay snaps. Jay says, "Come on, Elisabeth, you know developers. You have to light a fire under them to get anything done." This does not go over well with me. In fact, you can probably imagine what's going through my head. At this point, what I'm thinking is, what? No, I don't know that. I don't know that you have to light a fire under developers. The developers that I know all care deeply about delivering value. Nobody is malingering or being lazy. You don't need to light a fire under them. In fact, what I do know is that we have too many top priorities in flight. We have too much work in progress. Nothing is getting all the way done, because we are attempting to do too much. Furthermore, your assumptions, Jay, about the duration that something should take are based largely on what you're hearing from your product managers. I'm hearing from those same product managers. One of them cornered me to inform me that the particular chunk of work that was taking a couple of weeks was, in fact, the work of only a couple of days if we had competent developers. That I should be firing anybody who couldn't deliver said work in a couple of days. That conversation incidentally, also did not go well.
I'm not saying this out loud to Jay. That would have been counterproductive. Inside my head, what I'm thinking is, you're making assumptions about how long it should take to deliver based on estimates that somebody who is not the developer gave you, and the only people with valid estimates are the ones who are actually doing the work. Finally, I think that we're probably confusing output and outcomes. You're focused on output, and hurrying up the speed of delivery instead of being focused on the outcomes that we actually want. We're at risk of confusing speed and progress. You're trying to manage by deadlines. The irony is that deadlines can ultimately make projects go slower. You see, this particular group was working on software that had been around for a while. There had been, over the course of the years, many people like Jay, who had tried to manage by deadline. As a result, there had been numerous circumstances where the developers had to make a trade-off, because something had to give. Sometimes the trade-off they made was against their own personal lives. They worked a lot of hours. That thing is, about working a lot of hours, exhausted developers don't produce good work. The code was incredibly complex, difficult to reason about.
Then the other thing that happens is when something has to give, inevitably, it often is in the code quality or in the tests. There were numerous places in the code where logic had been shoved in wherever it was expedient, instead of where it actually belonged. Of course, there had never been time to go clean things up and refactor to make the code easier to work with. Furthermore, writing good automated tests does take time. We had a set of tests, but they weren't necessarily the right tests, and they didn't run consistently. They tended to have flakes, tests that would fail for no good reason, causing our CI to go red. We couldn't trust our CI. This was a natural outcome of having managed to deadlines, and now Jay was turning up the heat even harder on those deadlines. The engineers were in the impossible situation of having to figure out what trade-offs they could make. Frankly, we were pretty much out of trade-offs that we could make.
Risk Is Invisible
One of the most insidious things about this is that when a developer has to make a trade-off because of a deadline, what they're really trading off is future risk against current velocity. That risk, it's nearly invisible. People like Jay, who had put pressure on the development organization previously, probably didn't stick around long enough to discover the implications of the decisions that the developers had made on their behalf. We found ourselves in a legacy code situation, and it was pretty difficult to dig out of it.
What Does Work?
I had to figure out how to frame a response to Jay. In my head, what I was thinking about was everything that I know, that did work. That's what I really wanted to talk about today, what does work? I want to tell you a story. This is a good story. It's a happy story. It's a story of a project that was tremendously successful that delivered business value, where the engineering team was incredibly effective. We worked incredibly collaboratively with our product manager who was also the company founder. Once upon a time, back in 2007 or so, Drew McManus and Melissa Dyrdahl had formed a company called Bring Light. They engaged Pivotal Labs to build the website. Bring Light was a charitable giving app. This was long before GoFundMe was a thing. You can see from the UI, it's a very 2007-ish website. This is a screenshot of the original Bring Light website. This website, it was a charitable giving website, which meant that it was financially critical. It had to take payments. It needed to stay up so that we could take payments.
It had a wealth of capabilities. A 503(C) could register themselves, and after going through a vetting process could start setting up projects. Projects were the unit of funding. For example, in this particular screenshot, we see that the Humane Society of Silicon Valley had a project around raising funds that would save animals. There were all sorts of projects across all types of charities. Donors could not just donate to given projects and search for projects that matched their values and charitable giving goals, but donors could also organize themselves into giving groups. Then giving groups could challenge each other or challenge the giving group to raise a certain amount of money cumulatively.
As you see, there were a fair number of features. The team that delivered this was four engineers, and one product manager, Drew. There were just five of us, and we delivered it in four months. Furthermore, after the go-live date, there really wasn't a whole lot more that needed to happen to realize the vision of what Bring Light was supposed to be. Ultimately, Drew ended up hiring a part-time developer for a few hours a month to do routine maintenance on the site. We went live with virtually no bugs, certainly none that were serious bugs. The maintenance cost was comparatively low. Eventually, Drew and Melissa sold Bring Light. Overall, I would say that this project was a phenomenal success, and we delivered quite quickly, as well as delivering with quality. At no time did Drew ever set deadlines. At no time did we feel pressured to cut corners.
Typical Sources of Friction
How did this work? Why was this such a success and so unusual for most software projects? Before I answer that, let's take a look at some typical sources of friction on any given software project. From the time that someone has an idea to the time that it's realized, delivered out in the real world, we'd like to believe that's a straight shot, a straight path forward. It never is. These paths typically have twists and turns and speed bumps, which is what this illustration shows. What are typical sources of speed bumps? One is consistently slow feedback. Another is when you have to wait on someone else. When maybe there's another engineering group that is responsible for maintaining the API that you have to rely on, and that group needs to add the endpoint for that API before you are able to do your work and deliver the new capability in the website. Or maybe that other group is an ops group, and you have to file a ticket and wait for your VM to be provisioned before you can stand up your staging site. Whatever it is, oftentimes, software development projects involve waiting.
Then once the waiting is over, there's a whole bunch of manual work to do. Whether it's manual verification, or manual deployments, or mostly automated deployments except for those few little tiny things, or mostly automated everything except you have to go SSH around nodes in order to collect logs when something goes wrong. Whatever it is, manual work is often a source of friction. Then of course, after all of this, surprise, there's bugs. Because of course there are. You end up having to go back around and go through the entire cycle, run through all those speed bumps again, in order to deliver the fixes for the bugs. Excruciating.
Dedicated, Engaged, Self-Contained Team
We didn't have any of that on Bring Light. None of it. We were a self-contained team with all the skills that we needed and full access to our staging environment and our production environment, full control of it. We did not have to file a ticket to get somebody else to go do a thing before we could proceed. We were all dedicated, specifically 100% to that project, except for Drew. Because remember, he was a founder of the company. He did have other things to do besides work the backlog with the engineers. The engineers were all 100% dedicated, no multitasking. We were all deeply engaged in the work. Drew, like I said, he was a special case. He split his time between being in the office with us, and making sure that we were all aligned on what we were building. Then going out into the wilds of Silicon Valley and trying to raise funds for his startup.
Single Flow of Work
We were working with a single flow of work. As is common with many agile projects, Drew as the product manager had the steering wheel. He got to say what we built and in what order. As engineers, we got to say how we were going to build it and how long it took. It was a genuine partnership. There was none of this tension between the groups and negotiation. As soon as you start negotiating estimates, you've got a serious problem. Instead, we would give an estimate. Then, if Drew didn't like the estimate, because it was more than he had mentally budgeted in terms of time for a given capability, we would have a conversation about, if this is going to cost this much, what could we do for half the time. It never felt like a tense negotiation. It always felt like partnering around. We know that we only have so many hours total, let's figure out how to make the best possible use of them with the highest leverage from a business perspective. Drew was the one who could tell us about the business perspective.
Drew worked really hard to keep us all aligned, in being so available to the engineers, and so incredibly open to answering questions. He helped us calibrate what was and was not important to him. Consequently, we got really good at figuring out what kinds of bugs we should be looking for as we were developing and ensure that they didn't happen, versus what kinds of things he would eventually say he just didn't care about. We would frequently go to him and say, "Drew, we noticed that when we do bool in this site, we get this result. Is that something you care about?" Drew would say something like, "Yes, please just fix that," or he would say, "No, I don't care about that. Just move on." Over time, we got much better at guessing what his response would be to a given thing.
Work In Small Shippable Units
Along the way, we were always working in small shippable units. This was a Rails app, and that did give us an advantage. It was relatively straightforward for us to deliver a capability that was all the way ready for production with nothing left to do. We had a shared definition of done. We worked to done, and we worked on small capabilities. A typical thing would take us somewhere between half a day and two or three days to deliver. We had fast feedback, because as we were working on those small shippable units, as developers, we were doing test-driven development. That meant that we were writing a little tiny executable unit test before we wrote the code that would make that pass. Then of course, we were doing the full cycle: red, green, clean. Make the test, watch it fail, write the code that makes it pass. Then clean up the code base to make it more malleable for the next little cycle of TDD.
Furthermore, Drew was amazing at giving us fast feedback. Drew, even when he was out of the office, if he had an hour between meetings, as he was trying to get VC funding, he would go sit in a Starbucks and hover over the project tracking software we were using, just waiting for us to deliver a feature, a new thing, so that he could try it out and accept or reject the work. We knew as engineers that he was hovering and waiting, and that there was a high probability if we delivered something he liked that he would be demoing it when he had a demo, the half an hour from that point.
That is where I learned about momentum. Drew really is the one who by example, taught me how important momentum is. We knew that minutes mattered to Drew. We never felt rushed. We never compromised our integrity with respect to engineering practices. We knew that we were in a partnership, and that we needed to deliver solid, high quality code that would not have risk of not being able to accept payments, for example. At the same time, we also knew that Drew was waiting, and eagerly, never punitively, always eagerly waiting for the next thing. The way this played out was there were several times where we were just inches away from being able to deliver something. If we hadn't had that sense of momentum, and that sense of Drew cares, and wants to see this so that he could maybe demo it. We might have said, as lunchtime approached, "We'll get back to this after lunch." Instead, what we said was, "If we take 5 more minutes now, I think we could deliver this. Do you want to just take the 5 minutes and deliver this?" As a result, Drew got things sooner, and we got feedback sooner. It became this virtuous cycle. We were eager to get the feedback, so we were eager to deliver. I can't emphasize enough the extent to which this was a sense of momentum and not a sense of pressure. Consequently, we never cut corners. We did make sure that we used every hour of every day as effectively as we possibly could have.
What about Efficiency?
That is how this project, Bring Light, worked. That all sounds great. Don't you want to live in that world? Sounds fabulous. It's counterintuitive to so many people who would say, but what about the efficiencies? It is a common thing to say, if we want to go faster, we should make the most efficient use possible of every single developer. That means that we want to ensure that the utilization of every developer is as close to 100% as possible. That they are leaning into their specialties where they're going to have a much higher point of leverage. We're going to create a system that's really focused around efficiency of use of developer time. It's a trap, incidentally. Eliyahu Goldratt talks about that trap in "The Goal" beautifully. It's something that our brains tell us sometimes, seems intuitive that if we want it to be efficient we should make the most efficient use of somebody's time. By contrast, the way that we worked on Bring Light, we were focused on throughput through the entire system. We weren't always operating in the most efficient way possible. The end result was quite effective.
The Importance of Shaving the Right Yak
I also want to take a quick pause to talk about yak shaving, and the importance of learning how to shave the right yaks. Because the other thing about efficiency that some people sometimes think is, it will be more efficient, if while I am in this piece of code, I just go ahead and do all this other work that I know needs to happen anyway. Or, it will be more efficient, if we hold off on delivering so that we can just batch everything all together. The end result of that will be less overhead. That will be more efficient. Part of what I learned on that project was the importance of shaving the right yak. That phrase that probably seems a little bit weird to some of you anyway, it isn't a common phrase used in the software engineering field and elsewhere. It came out of the MIT lab. Seth Godin has a lovely blog post, describing his story of the yak story. I have my own. I'm going to tell that very briefly.
It's a lovely Saturday afternoon, and you decide that it's a perfect time to go trim your hedges. You go into the garage to get the hedge trimmers, and that's when you realize, they're not there. I lent them to the neighbor. I better go get them. You're about to go over to your neighbor's house to get your hedge trimmers, and that's when you remember, the reason I haven't asked for them back yet is that I still have their lawn furniture from the time that we did a socially distanced, small family gathering but needed enough seats outside for everybody. You head back into the garage to get the lawn furniture so that you can return it. That's when you discover that the dog got to the cushions on the lawn furniture and they have all been ripped to shreds.
You're about to go into the house to get the sewing kit so that you can re-stuff the cushions and sew them back up so that you can return the lawn furniture, so that you can get your hedge trimmers, so that you can trim your hedges because it's the perfect day for it. That's when you remember, your neighbor was bragging about how they had special ordered this from a catalog that specialized in fair trade, yak stuffed furniture. You realize that you can't just re-stuff it with whatever poly-fill you happen to have on hand. That's how you find yourself with trimmers and a bag packed in a suitcase heading to the airport hoping that despite COVID, you'll be able to catch a flight to the Himalayas where you can shave a yak. Stuff the bag. Bring it back. Stuff the furniture. Sew it back up. Return it to the neighbor. Get your hedge clippers so that some number of weeks later, you would be able to trim your hedges.
Seth Godin, in describing the yak shaving problem, says, don't shave the yak. Go to Lowe's and get a new pair of hedge trimmers. I have a different take. My take is the difference between a more senior engineer and a more junior engineer is the senior engineer has learned which yaks to shave. It's not that we never did the, while we're here, let's do this thing, while we were on Bring Light. There were times that we did. It was always in service of the work that we were doing at the time. We were hyper-focused on delivering the capability that we were actually working on. When we chose to shave yaks, they were always the yaks that would do our near-term future selves a favor. We weren't trying to future-proof for two years from that point. We knew that if we were going to have to go back and do this in the next couple of days anyway, and it would help things go faster now, it was the right digression. It was the right yak to shave.
Eliminate Friction
Ultimately, everything that we did on that project was all about eliminating friction, those speed bumps that get in the way. This wasn't the only friction that I learned about at Labs. Eliminating friction involves identifying every source of an interrupt between idea and implementation. The things that I just talked about on Bring Light were really at a project level. This happens at a micro level every single day. I will admit that there are times when I get accustomed to certain speed bumps or sources of friction, and just start ignoring them. I think it's a common thing.
In one particular case, I was working on a different project, not Bring Light, much long before that in the early 2000s, probably 2004 or so. I was pairing with Alex Chafee. Alex taught me something so critical in that moment. I was relatively new to pairing. We were pairing on my personal laptop. He was helping me get something set up. My personal laptop was quirky. The mouse was a touchpad mouse, and the mouse had a tendency to move wherever it felt like on the screen. You'd be working along and things were going fine. Then all of a sudden, the mouse cursor would teleport to someplace else on the screen. You'd have to patiently get the mouse back where it was supposed to go.
As it was, Alex was exhibiting a fair amount of patience with me and my mousing because he's the person who memorizes keyboard shortcuts in order to eliminate that friction between the time he has an idea of how to express something in code and the actual writing of the code. Mousing does get in the way a bit on that. However, he was willing to put up with that. The first time he noticed that my mouse decided to teleport to some other part of the screen, he just looked at me and he said, "Can I get you an external mouse? We have external mice. An external mouse would be good right now." I being relatively inured to the pain of the mouse moving around, said, "No, I'm fine." I was relatively new to pairing and was not picking up on the cues that my pair was giving me, but we're going to gloss over that for the purposes of the story as well.
At this point, Alex's knee started doing this. You can picture it. Somebody whose knee is bouncing. Then the mouse thing happened again, and Alex's knee sped up a little bit. Alex turned to me and he said, "We have external mice. Would you like an external mouse? External mice are good." I said, "No, it's fine. This won't take very long. It'll take you longer to go get the mouse. Let's just keep going." We kept going. Alex's knee is doing this. Alex suddenly shouted, "Three." I looked around like, what? He wasn't explaining. I just kept doing what I was doing. Alex's knee. Then he says, "Four." That's when I realized he was counting the number of times that my quirky mouse had interrupted our flow. At that point, I said, "Ok, fine, go get an external mouse."
Alex was right. In that moment, he taught me the value of seconds, and the cost of ignoring the sources of friction. It took less time to go get an external mouse and plug it in, and get it working. Even though that felt like a big interruption, it took less time to do that than it would have taken to recover each time the mouse got in the way. It's an example of shaving the right yak. Alex taught me something very important about workflow as a developer. Ultimately, it's about increasing momentum. At every step on the way, as we eliminate friction, we increase the possibility of momentum. Developing software goes from feeling like you're driving a truck through sand to feeling like you're skating across wonderful ice.
Epilogue
You may recall, we started this story with me standing with Jay in an open plan office having intense discussion. What happened? He had just said, "Come on, Elisabeth, you know developers. You have to light a fire under them to get anything done." I needed to frame a response. I took a very deep breath. I calmly said, "Jay, the pressure you're putting on the team is going to backfire. We will end up going slower. Let's work on establishing momentum, instead of increasing the pressure." I really wish that I could tell you that in that moment, Jay had an epiphany and realized that I was right. That we should work on momentum and supportive culture that would help engineers do their best work. That is not exactly what happened. Jay and I really struggled to get along. He continued to put pressure on me. Honestly, I don't remember what happened with the particular deadline that he was concerned about. In the grand scheme of things, that one particular deadline didn't really matter.
Eventually, Jay decided that he wanted to go work on other things in the organization, and not have to deal with me and this particular engineering team. He moved on, and presumably put the same pressure on whatever group he moved on to. What happened to our group? You may recall, we were shipping enterprise software. It was fairly complex. It was enterprise software that had been around for a while, and had a number of circumstances where shortcuts had been made. We were slogging through the increased risk that had resulted from all of those shortcuts over time. This was ultimately why Jay was so frustrated with the pace of development, because there are no quick fixes and it does take time to address the underlying issues. The underlying sources of friction.
Over the course of the next two years, we managed to organize ourselves so that we were enabled to enable teams to focus. One of the differences between a fairly small scope Rails web app, and a honkin' big piece of enterprise software, is that it's not the thing you can develop with four people in four months. We did have multiple teams, and the teams had to coordinate, and so we didn't end up with a fully self-contained team. We were able to structure ourselves to enable teams to focus. We changed our branching strategy. Historically, teams had worked in massive batch sizes. Instead of the ideal of small, incremental delivery teams, historically had had incredibly long running branches that were more than even feature branches. There had been circumstances where we had lost huge amounts of work. Huge amounts of work had ended up going to waste because for whatever reason, the branch was never going to be able to be merged back together with other branches that were all being developed in long running projects. We reduced the amount of work on any given branch and changed our branching strategy. We paid down a lot of technical debt. We invested heavily in test automation, in automating manual tasks, and invested in our CI.
Ultimately, I'm so proud of the team. We went from struggling to ship on an annual basis to being able to ship incredibly predictably, monthly. At which point we could have trade-off discussions that weren't about, how are we going to make this deadline? Were more about, there's going to be a bus leaving every month, what do we want to have, be on it? Ultimately, these are the stories and the lessons that I have learned around how to speed up a team. It's really hard to do this. It's incredibly valuable. Honestly, I think attempting to manage by the pressure of deadlines is an easy way out that honestly doesn't work all that well. Doing these things, this is how you speed up a team and get a sense of momentum.
See more presentations with transcripts