Bio Gabrielle Benefield is the founder of Evolve Beyond and has led many international Product Organisations, both large and small, with outstanding results. Using Agile and Lean since 1999, she has taken San Francisco start-up to Initial Public Offering, and at the other end of the spectrum, led the Agile transformation at Yahoo!, scaling to over 250 teams globally.
Software is Changing the World. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.
Shane: Last time we spoke was few months back at the Agile 2013 Conference and at that point you were doing some pretty interesting stuff with the Flexible Contracts and I know something has happened since that with that.
Yes, that was kind of an interesting, Susan and I ran a pilot class and we thought that was just an experiment, 2 days we are taking through, actually 1 day, about Flexible Contracts stuff and then we started trying to figure out who with invite. It was kind of interesting because when I was talking to some of the big suppliers, the managers were saying: “You know is really challenging, we work with central governments, with big companies and we are in there. We know that are building the wrong thing and actually interestingly enough from the customer side, the people working with us directly, they know is the wrong thing. So everybody is aware of it, we are tied into a contract and so everyone knows we should be doing something else and as a supplier, we can see the shortest path, the quickest way to reach it, yet we are completely contractually bound. We need a different way to work”.
Then when I speak to people like in UK Governments, some of these big customers, they blame suppliers saying: “Oh it’s interesting that these guys, you know we’re bound the contracts, they are building the wrong thing” and we say: “Why don’t you use something else” and they say: “They’d never change it because it’s what the suppliers want”. And we said “you know what, this is not making sense, we need to get you into a room”, so we got some of the really big (like) Accenture MD’s, we got UK Government people, we got people who’ve been working with different European Governments, we got a woman flying out from the US who worked with the department of Defense, we had people from lawyers. You know, a really interesting mix of people coming in and we had a Forrester researcher for Agile. When we got them all together it was a fascinating day; it was less about the class and more about the conversations. There was a recognition that everything that we are doing in contracts is not working.
So whenever I bring up with people why don’t we try something else they’re like: “Oh that will never work, it will be too hard” and then I put it to them and I say: “What you got now, what is working now” and there is silence, so what was seeing is there is a big cultural for shift and there are some very interesting things happening with outcome-based contracts. I’ve noticed in the US in Social Impacts, so groups that are trying to do things like everything from bringing down prisoners re-offense, rates trying to help abused children, trying to do healthcare; you know social outcomes. That was what they are trying to bring in and so there is a lot of work that we are not using in the technology space that we should, so, you know, is not that we invented any of this stuff, it’s out there. So we starting to see people now opening up to the ideas that there is a different way of doing things and I think trying to slot together the pieces, because when you talk about outcomes people start getting them very confused with outputs, so when we are talking about outcomes, we are saying what are the desired business results, and what we found is that the customers, the reason that they outsource work in the first place is because they need the suppliers because they understand the technical domain, they understand how to build the things, that is why they want them in the first place.
So what you’ve got is customers who are fairly clueless about how to put that together, being asked to write down the requirements specifications, product backlog items, playing be it the customer, be it the product owner role in Agile, we haven’t solved any of these things just by being Agile, we are just kind of delivering wrong thing faster. And so we’ve got to reeducate the roles a little bit, so when we start saying: “What are the outcomes you want?” they say: “Well ok, what we want to do is to automate things, we want to get to this data easily, we want to be able to see these different reports”. They don’t want to tell you how, they do know where they want to go; so if we take the outcomes, drive the contract by that, now the suppliers understand what we are trying to achieve, their job is to figure out the easiest way, the least effort to get there fastest, which the suppliers are what why we hiring them for, so it’s interesting how the roles swap around, how the thinking changes and you actually get a sign relieve from the customer side because that scared, that scared of being told that you have to come up with everything upfront and in a domain they don’t understand very well and then it causes an instant conflict because that constantly fighting in this sort of CYA behavior that we see coming out.
Shane: One of the things that I often here I’m talking to organizations about making this change and thinking about outcomes and so forth this but we want a fix price.
Ok, for a fix price what they’re trying to do is limit risk; and if I want to get a bathroom renovated I would want to know what are the constraints that we can’t go over. So it’s ok to say to people what is your budget, “we’ve got 800.000 we can’t go beyond it”; ok fine. However don’t just throw that money out there because one of the touted Agile Contracts is fixed price and yet there’s nothing around what are you getting for that money and again to me that is massive risk. A contract is meant to mitigate risk rather than create it, so when we start talking about a fixed price that is fine, that’s constrains that we can set but what we want to do is beak it down so for that 800000 you want to say, “ok let’s do a smaller portion of that and start validating, can we get some of your outcomes met for a small investment, maybe you put in 50.000 at a time”; it’s very much flow based budgeting. Let’s tie that back so we can now gauge are we heading towards the outcomes together, do we have some of the leading indicators, we don’t want to measure at the very end when we’ve spent all the money, did it work? Oh, it didn’t, bad!
If you look at the contract failures out there, the ones that end up in court, you see millions, billions sometimes being lost and I saw an interesting quote from Senator Gains from the US, I think it was one of the big consulting homes who been brought in and I think the project it’s going to be whatever was ten million, it ended up being 5 times more expensive, taking a lot more time and at the end it was a bit of a disaster and the Senator said “we need to move towards outcomes, is the only way we could start understanding how we really meeting the goals”. So I think being cognesent of the constrains, which we do set, we say: “What your constrained by ?”; budget, timing all those things matter, now let’s get indicators progressively to see are we on track or not, because if we are not, don’t waste your money, let’s pull the plug early rather than have that big failure. There is no point, you know, hitting it on budget and on time if you don’t get the success out of it.
Well this is where when you think about outcomes they can exist at multiple levels. So let’s say one of our clients said we have an outcome which is to improve the amount of sales we make in an ecommerce platform. When we start braking that down because what we are looking for is smaller chunks to validate this, we said: “Why are people not buying the service, what are the constraints what is preventing it?” When we break that down it turned out there are a myriad of them, some were usability issues, so now we found that certain pages, certain layouts, we thought might be problematic so we could start running experiments; so very quickly and again Agile’s ideal (and Lean) for getting that rapid feedback, we could start saying an outcome that we are going to measure and one of the more specific chunks is going to be if we take this problem which has major abandonment rates, let’s say 50% of our people live at one page, why don’t we run multiple prototypes and we will test it out and will see if we fix maybe this usability, does it increase the amount of conversions we get, people actually buying our service. Ok that is one level. Another level we discovered was around the service levels and let’s gets back into your operational backend even. We found the pages that took too long to load, over 4 seconds in this case, had 85 % abandonment - so that is a very easy one, this is where we are looking for quick wins too because I want to deliver he value rapidly, that is the promise of Agile. So what we did was break it down and say: “why is that page so slow?” and it was everything from with the data center was based which was a long way away on the other side of the world; to where a lot of the customers paying for the stuff were located; network problems. Some of them we could solve very quickly, a quick experiment actually was to just to cache some of the non-realtime page elements and we got massive upside, and again we link them because what I’m looking for is not just that we optimize the page load time, did we actually get the conversions because if you don’t do that, what happens is that we get that sub optimization, so people are chasing the wrong metrics without understanding the business layers above it.
So this is where again for me coming back to the contracts, a good contract is about taking the alignment between what the business wants to create and what the supplier wants to deliver. So you have to link those very closely otherwise you go up down the wrong path and then it’s very haphazard, it’s with the agilistas “building the wrong thing faster”, with the Lean Startup it’s “pivoting without purpose”. So we need to link that purpose and this is why getting a real clarity what are the problems we are trying to solve, what is good look like, now let’s rapidly test that. Those leading indicators are those short experimental leads that we can start saying did this impact something else. Some of them take longer, I do understand that so we sometimes have to make some leaps faith, often we driven by the data itself, so sometimes we can look at the data, look for problems and go “ah – that’s it” and we can push that out very quickly, we don’t always have to go through longer cycles so that is always the feedback loop that we are looking for and then we pay for it as we go rather than just paying time and materials, which is the common one used in Agile, again I look at it and I’ve said: “You spend a lot of money but are you really getting the value and how do you know?”
I think some of those things, like Cycle Times and Story Points, etc, to me those throughput metrics are somewhere useful when you are looking at the system to see where you’ve got bottlenecks in flows,; so I’m ok with them, some of my teams use them; it’s just that what I find is every time in the worst Agile contracts are the ones that puts things like Story Points and how many Story Points you are going to deliver, because since you put an incentive like that people try to optimize that metrics which means we are all about more throughput, but througput is not the goal because it doesn’t tell anything about have we met outcomes, it just means we are going faster and teams will game it especially contractual situation, it’s pretty easy to game Story Points; this is why a lot of traditionalists really have a problem with it. So those measures, we found every single time we put anything about Story Points, people optimize flows even if we have outcome metrics in there, so personally I’m like if they’re for the team, the team chooses to use them, do it but I have to go in and reverse engineer this out of organizations, one organization not that long ago, had velocity measures for all the teams and to be “hyper-productive” they have to make them get better or at least not go down. To me is an innovator, that is a sign that your teams are never going to improve because you have to get worse sometimes before you get better, so unless we are willing to use that just as something for the team to see maybe where there are flow gaps, I don’t even bother honestly, that is not the game for me, I find personally I tend to be unless teams want it, don’t do it, it’s actually a distraction.
Other metrics, and this is we are talking about Agile at the moment, but most metrics I’m finding, we are talking about outputs so features delivered by this date, we are talking about Function Points, remember Cosmic Function Points, whatever they means, but again these are all about pushing more stuff through the system whereas what we want to be doing stepping back and saying: “Here is the outcome, what is the least amount we can do to hit that”. That reduces time to market, it reduces the amount of code we write which reduces the amount of defects we might create and when you are in critical systems like Life Science, every line of code matches up to defects which matches up to people dying, so our responsibility as software craftsmen is to be able to improve that world, so when you think next time you get on a plane, you start thinking about “did I really want these guys to put more features in this airplane creating more lines of code which means there is more levels of failure”, you don’t want that, right? I want a simple elegant system so the way that we drive towards it should be around outcomes with the least effort.
Shane: But if I’m an outsource supplier I want features, I want time, I want your money.
You want your money! Do you want features or time? Of course you don’t, that means more work for you. If you could go in, I mean this is Utopia, we usually do blender charging because this is so Utopian that is a little hard to achieve, that I went into a company and I said to them: “Hey, you’ve got this problem, right? What if we came in and every percentage improvement we make is worth two million annually for you. What if we improve this and we took 10 percents of that fee, we are going to come and do that for you”. What if I could go and do that within three days of being in the organization, what if that change with 15 million. As a supplier I could walk away with 1.5 million within three days; the irony of this story is, it happen to me. I went into a company and did this, run a workshop, found a day and went: “Oh hang on, something is happening here”. One of the guys, the product owner looked at it, made a switch, started measuring the impact, we did this on the third day being of there, they were going to save that amount annually. And I’m sitting there going “Oh, why didn’t I use the outcome based contract?”
When I get that, most suppliers would respond like “that would be fantastic, we’d love it” and honestly that would be where we should end up. The caveat there is it’s uncomfortable to go into those models, so what we found the reality is, and I’ve noticed this in the non-software space too, with outcome based contract is you blend it, so you tend to do things depending on your level of confidence - if you are in an area that you know well you might take more of the upside and less of a day rate. So you’ll find it’s we tend to do blended rated where they could do time materials but they’ll get a bonus; they could do a lower day rate like cost plus 20% with more of the outcome or if you really believe you know that area, you might take more because risk award is extremely high and also for the customer we found that you have to talk about what this might means because they even if it’s the coolest thing in the universe, they make 100 million, if they have to shell out 10 million that might just hurt them so much, so sometimes we set caps and threshholds to do that.
It’s interesting, I get very mixed thinking on this. When I sit down with them there is an instant “why would we change what we do”, but then when you talk to them, the people that I think are not going to be into this actually are because they have had so much pain, for example one of the big contract mediation dispute resolution people, they have to get in the middle of these things when they go wrong and they said honestly: “We have to get these people in a room, we don’t want to go to the court, the objective is not to do that” and it’s all about aligning what is in the people’s heads, what they expect to what they wanted, so ultimately often times this comes too late when we are trying to figure out what would the outcomes you really wanted, now how can we help you get there. We sort of moving that to the front and when you say look, if you take your contract models, time materials very easy actually, you can add outcomes to it not difficult, fix price again, we can start adding outcomes, so I think often we say: “We are not reinventing the world, you can do this in a light way” also the risk is very low if you are able to exit the contract as you go because that means that you don’t have a risk, who cares?
If you go for a month, not seeing the results and you can exit, you don’t have to go keep renewing, that is fine, and with a lot of the big contracts because I thought I would get more push back on that fact with a lot of suppliers saying:”We get these three year contracts”. However what I find is most of the contract models that people are using, have a one month fix the cause anyway right, so the suppliers are still in a risky situation, in fact they are looking for a way to validate that what they are doing is worth continuing; and longer term you might do a short term contract, get the money and then I’m not going to rehire you again, and for these big suppliers that is an ongoing issue, so procurement; it’ getting people comfortable with here is what it means to you, here is how to put together. There are other people sort of saying “we can never do that” but I think it’s more education that they don’t know how to do it, not that they’re not willing. Again some experiment and we will see how it goes.
This is one of our biggest constrains, I was so excited, I’m telling people this is the way and then it was a lot of trying to get people to feel comfortable with using these models. So we actually created something (we being Ryan Shriver and myself) called Mobius. Now do you know what is Mobius is, the Mobius loop? It’s kind of, I describe it as if you could get a piece of paper and it’s a strip of paper that it’s attached, if you sort of cut it and twisted one side and reattached it, it creates this almost giegeresque sort of the constant flow and so the idea is; I’m big on don’t just do planning once, the whole idea is about continuous, the continuous improvement is we go through, so Mobius for us was a little visualization, a simple vision model, like anything people need an anchor point. So the way that Mobius works it’s an infinity loop and the way it started is if you think about something like the Iteration, Scrum, etc, we’ve got that nice little “let’s constantly iterate.” What that looks like to me is deliver and adapt, inspect and adapt, do something look at it, do something else, and I think we tend to be all about delivery. What is now happening and it’s almost a 180 degree shift from where we were, I think the traditional model was all about analysis, we got stuck in analysis paralysis; we’d do so much upfront, then throw that over the wall then we’d deliver a Big Bang, looked back and went: “Oh, that is not good”. The Agilest said: “Ok let’s get better at delivering and inspecting, delivering and inspecting” and somewhere along the line we lost the discovery, we lost where we are heading, which means is the Shakespeare’s Monkeys in the room, right? Keep typing hoping that eventually you’ll get the complete work of Shakespeare. That is wasteful and takes a long time and it’s one of the big pain points I think in the Agile Community, is we are often building rapidly but we are not always building the outcomes we want.
so Mobius is a way to say: “Ok, on the left hand side we start with the problem, so we figure out what is that you really trying to achieve, why are we doing this”. We do a deep dive and this is where we bring in a lot of the, we use both data so we do deep dive into what are the data sources that we’ve already got, what do we need to gather; we look at the customers, what are they trying to achieve, what are their goals, we get data from whatever we can, so we deeply, deeply understand the problems because most companies come to me with problems like “we need a new shopping cart” to use the earlier example. When we break it down we are like we can put a new shopping card but why do we do that? And as I said it turns out it’s usability, it’s all these things that just putting a new system in doesn’t fix that. So once we understand the problems, we have some data to compare against, which again back to our indicators is very useful, we create outcomes. Outcomes are what we are going to deliver and measure against. Once you have the outcomes and you say this is where want to head to, we want to have options, how do we get there, and often you get multiple. I’m a big fan of concurrent set-based design where we try multiple experiments, rapidly get feedback because I don’t know; when people talk about requirements or backlogs, to me that are a guess list, there are random guesses that we have absolutely no proof that they work; so let’s be a lot more disciplined in that product management styles. Let’s come up with options that we say option A could be high risk, high expense but it could be high value, option C could be very quick and easy to do so let’s implement it, like that page caching. We can just put that out to market, low risk, low cost. It may actually fix our problems, brilliant, maybe setting up a whole new data center locally very expensive but again, run an experiment, so what we have in Mobius is the deliver, measure and adapt; so deliver and adapt very Agile, measure it’s what is important to get the feedback and so by doing that we can now say, ok we’ve run this experiment, we have implemented something, did we see that the outcomes have moved or not?
So getting that blending and what I find is when I’m talking to senior management I very much talk about left side but talk about how they need to interact with the right side, so we’re actually bringing in the delivery development side with the business and strategy, and it’s that intersection that we get in Mobius that keeps us going around the loops that we don’t just do a big handoff and as soon as you describe it visually something clicks; I’ve had various senior managers turning around recently who the Agile people could not get them involved, they could not get the business to be interested. As soon as you start talking about outcomes, value, investment and risk and how they need to be setting those, it’s a no brainer; they are very much involved and they understand now how this all comes together, so these visual models as a way to anchor thinking, turn out to be incredibly important, and Mobius looks incredibly simple it’s weird how long it took to come up with that simple model.
It is, it’s all of those things; I’m very highly influenced by all of the Lean thinking, also the OODA Loops stuff to comes out of John Boyd, so a lot of the military strategy. That actually is interesting a friend who does Joint Services training out the US and he is actually within the military here, he is big on the Commander’s Intent, military commander’s idea. Were the military leaders set where they want to get to but don’t tell the troops how to do it and he said that they were struggling to get this idea across and again he starting to apply Mobius in that world because they needed that visual model to link all pieces together, so it’s fascinating how this way of thinking and you know the John Boyd stuff and all these major military thinkers comes into it, so I like to see patterns from different industry and verticals, not just software that is when I think it has more strength.
Shane: Thank you so much for taking the time to talk to us today, it’s really been interesting and good to see you!
My pleasure Shane, thank you!