Jeff Sutherland welcome to InfoQ. Tell us what's your claim to fame?
In 1993 I was trying to figure out a better way to build software. I've been CTO and VPO Engineer at nine software companies; this was my fifth company in 1993, and we figured out a way to use a practice which is now called Scrum that worked really, really well. In 1995 I got together with Ken Schwaber who was CEO for a project management and software company, I brought him in and showed him the team who I'd been working with and delivering software for many iterations, and he looked at it and he said "this is really the right thing". So Ken has helped us for many years to roll this out across the world. And in recent years I have been able to spring free at least some of the time from my company, because I'm still CTO of a software company, but I spent a lot of my time travelling around the world talking about Scrum.
Last time I visited the Scrum alliance I noticed that there were 10.000 people trained as certified Scrum masters. Tell me, who's doing Scrum?
Scrum is really taking off, there are a lot of people doing Scrum, we have a certification course that is doubling the number of Scrum masters every year, there are now about 12,000 Certified Scrum Masters, and we know for every Certified Scrum Master there are at least 10 Scrum teams out there that haven't been to any kind of training. I expect that probably this morning there are 120,000 Scrum teams around the world having their daily meeting. There are a lot of people doing Scrum.
Is it all really Scrum? Are there many flavors of Scrum that you are seeing?
This is really interesting, because as with anything that explodes in adoption, there are people at various phases of doing it. In my recent meetings with people who are in training sessions, I have been asking "how many of you are doing Scrum?" In fact here at this conference, at Qcon, on Monday we had 30 people at a Certified Scrum Master training course, and I said "how many people are doing Scrum?" and 15 people raised their hands. So then I said: Nokia has maybe more Scrum masters than almost any other company, in fact we know Finland is the highest per capita Scrum master in the entire world, mainly because of Nokia. And they have come up with some simple ideas that allow them to determine whether a team is really doing Scrum."
How does Nokia tell if a team is doing Scrum?
They first address the issue of iterative development. In object technology we've been doing iterative, incremental development for many years and that's kind of fundamental to all the Agile processes. First they want to ask the teams: "are you doing iterative development?" because you can't even do Agile process unless you are. The first thing that they say is "Do you have fixed iterations? Do you have an iteration that starts at a specified time and ends at a fixed time?" And that iteration has got to be less than 6 weeks. This is not Scrum but this is just their policy. It's not iterative unless it's a time box that's less than 6 weeks. If you are not doing that you are not doing iterative development. Then if people answer "yes" to that they go to the next step and they say "Well, at the end of the iteration, do you have working software?" and this tends to rule out a lot of people, because if you don't have working software you are not doing iterative development. And then they say "Ok, you want to have this working software at the end, so at the beginning you want to be able to start it in an Agile way. Do you have to have a specification in complete detail before you can start the iteration?" If you need to have that, that is not incremental development. And finally they say "It's important for working software at the end to have testing as part of the increment: Do you have testing in the middle of the development?" That tends to rule out about half of the Scrum teams before they even need to go on to Scrum issues.
Why does it matter if teams are "doing iterative development?"
It matters because the whole point of Agile development, in fact there are several interlocking points: you want to get everybody in the process working together, you want to get the people responsible for formulating the product: the people that are building the product, the people that are testing the product, and the users that are going to use the product. You want them all working together and if you separate things into "these people over here build the spec and they hand over a document to the people that are going to build it, and they hand over software to the testers, and then the testers hand it over to the customer," the customer says it's not what they wanted. And so you cycle back to the beginning and then you go through it again. And if you go through this 3 times they cancel the project. That's why there are so many projects killed all over the world. You have to be doing everything at once.
You sound like you agree with the criteria that Nokia is applying. Is it important that all those things be happening at once?
Yes, this is really a subset of a bunch of points that need to be met to run Scrum well. And before I just enumerate it, these are just 4 of Nokia's 8 points that they use. They use those 8 points to determine if you are even doing Scrum. Then there's a whole bunch of questions about "are you doing Scrum really well? Is your velocity of delivery really high? Are your customers ecstatic about your product?" All of these things are about doing Scrum well. But that check list is to determine if you are even doing Scrum, then we can start talking about improving the process.
Nokia has rules about general iterative development, but can you tell us what their additional "Scrum rules" are?
They have 4 additional rules that apply to Scrum. The first question they ask the team is "Do you have a product owner? Do you have someone that represents the customer who is working with you?" That's one person that you can interact with to determine what the product is that you should build. And there are many teams, when you ask them that, they say "no, we don't know who the product owner is." The second thing that Nokia asks is "If you have a product owner, do they have a product backlog, a list of features to be developed? Is that backlog prioritized by business value, and is it estimated how long it is going to take to build those pieces?". This is what allows the product owner to build a road map for a release. And if the answer to the question is "yes" then they go on to "When the team is doing the development, do you have a burndown chart that shows how much work is remaining in the iteration as you are moving forward, so you can track your progress? And from that, can you tell the velocity of the team, how fast is the team moving?" This is basic to: first, for the product owner to be able to build a release plan; but also for the team themselves to actually improve their progress. They want to know what their velocity is; it helps them estimate better it also helps them improve the velocity with which they move forward. And then finally the Scrum teams depend on self organization in the sense that the team is responsible for choosing the work, assigning themselves the work and the response for figuring out the fastest path to deliver the work. Nokia has a rule that you can't have a project manager interfering with the team in the middle of an iteration because that stops the self organization and it guarantees you are going to have a suboptimal path to a solution.
Why do these rules matter?
Agile processes in general and Scrum in particular are a way to get a team working together with the customer, with the people that are going to use the product, and to actually improve the software and be able to deliver it faster, and be able to improve the quality of the product. In order to do that you need all the pieces working together. Both Scrum and XP and other Agile processes were designed at a time when we were figuring out how to do object oriented development best. They are built out of a set of interlocking pieces, and one of the things that companies are trying to do is say "well, let's pick this Agile process because it's hard to do the other ones, and we'll get some improvement". But the improvement is often not what they expect, is as if you look at object oriented technology and say "well our developers can do everything with objects except they have a tough time with inheritance and besides we have this rule in the company coding standards and it doesn't fit into our company standard. So we are going to do everything but inheritance". Now the product comes out, and it's brittle, it's not adaptable, it's not flexible, all the qualities that you expect to get are missing. And then management says "well, we are doing object oriented development, we invested a lot in this, and we are not getting the benefits".
How does this selective implementation show up in Scrum?
Scrum was designed basically to put a team into a state where they can deliver 5 to 10 times as much as normal and with really good Scrum teams that's what you'll see. With the average Scrum implementation a company should be able to double its throughput with software, and more than double the quality of the implementation. That's one of the reasons why Scrum is extending because where people really implement it, it does work and it gives you that. If a company is not getting that a fact the first thing I do is go down the list of Nokia questions, and I almost always find out they don't meet the basics of doing Scrum, and they can't expect to get the benefit unless they are doing the basics.
Is it better then, not to just pick a couple of Agile practices that look useful and bring them into your traditional team?
There are many companies that will start with some pieces and actually doing things like having a daily meeting are probably going to make things better. But that by itself will only make it slightly better you won't get the big benefit. There is an interesting example at Google with the AdWords project:, it is fascinating because AdWords is software that puts the ads on Gmail or on your blog, if you have Google ads, and it is the thing that makes most of the money for Google. It has to integrate with every other product at Google, plus anybody that's external to Google that's putting ads on their website. They have a bigger release problem than most Google applications, and they are trying to get better with their delivery dates and with their integration, everything has to work. And so the product owner Mark Striebeck presented the paper at Agile 2006 conference last year and what he did to improve the process with ad words, and to do it in a way he calls "a googly way" because at Google, they don't have much process and they don't want any process overlaid on them.
His strategy was he knew what he wanted to get, he knew all the pieces of Scrum, but he asked the development team "What is our biggest problem? Let's list our biggest problems and let's put them in priority order". And they said "Our biggest problem is we are not meeting our dates and we have a whole bunch of people counting on that". And he said "Why don't we use one of the practices from Agile called the burndown chart (and actually he just used a release burn down chart), and let's see what happens?"
Now Mark knew all the pieces of Scrum and he knew that wouldn't fix the problem but that it will surface more problems. And as it began to surface more problems he began to asking questions: "Ok, let's assume the release burn down is going to get us to the date, what's your problem now?" And they started to say "Well, we have a lot of problems, people are checking out code, AdWords has 5 distributed teams in 5 different places in the world, so we have people checking our code, working on the same code, doing duplicate effort, the code is clashing, it is causing bugs, this is a big problem." And Mark said "Why don't we try another Agile practice, it is called the daily meeting, you get together and you talk about what people have been doing yesterday and what they are doing today, what problems are getting in the way, I think that this might fix this problem". So they started their daily meetings. And it did fix a lot of it but it also began to surface more problems because in the daily meeting you begin to build a larger impediment list, and prioritizing that.
But Mark systematically put the entire process of Scrum in place, and in the end when it was all done he said "Guess what guys? This is Scrum." He did it without even telling you he was doing it and he did it in a way that really worked. So I recommend when people are trying to do pieces of Agile, that they look at that paper at Google - http://doi.ieeecomputersociety.org/10.1109/AGILE.2006.48 - and they use the same intelligence that Mark used, in the way he systematically put the process in place.
What kind of experience would you need to shepherd an organization through that kind of incremental change (as at Google AdWords)?
You need to have Agile experience, you really need to have done work with Agile teams, work with good teams that would meet the Nokia test, that have been delivering software well using Scrum or any other Agile process. You need to understand what makes that work. Then you can go to another company or another team and you can guide them through the process of adoption as they did at Google.
In this environment everybody is working hard to improve the software process to increase the bottom line, to produce a better product. So, what should people be calling their processes when they are using only parts of Scrum?
People are looking at Agile processes, in particular Scrum, and they are noticing that other people are having some success and they know they need to change their process, deliver software earlier and deliver higher quality. And also improve the lives of the developers, so they can hire better developers and they can keep the ones they have and have a good work experience. It's normal for them to start introducing incrementally Agile practices. What they need to understand is that they won't get the full benefits unless they have a full package of processes. They need to have a plan where they consciously implement step by step, and they need to understand, just as they did at Google, when they do one thing it doesn't solve the problem it actually makes other problems more apparent. They need to address them and again use the Scrum strategy: prioritize your impediment list and start working on the top of the list and then introduce the next practice that removes that impediment, and if they systematically do that step by step, they would get there, and they will get minor improvements all along the way. That is a good thing but it needs to be done consciously, and they should get some Scrum training or bring in people who have done this well in other teams that actually know what they are doing in introducing piece-by-piece Agile development.
While they are on their way there, is it Scrum?
I like the Nokia test. The people at Nokia say that test saved them so much pain and confusion over so many years, they would not be without it.
Jeff I am fascinated by your six degrees of separation from the animated robots on Mars. Can you tell me about that, and what does it have to do with Scrum?
The insect-like robots with many legs are very interesting, and the company. Before Scrum started, I had leased a little bit of space to a startup, a robot startup that was founded by professor Rodney Brooks, from MIT. They were building these robots with multiple legs and I learned a lot about them because they kept on running into my office trying to hunt me down, they had infrared sensors. And we were trying to think, as we looked into models of software development, IBM had published a study showing that a "surgical team" was the highest productivity for building software that they had seen within IBM.
And the surgical team is a single person who is "the surgeon," who has the entire design, architecture in his or her head and does all the coding, and is surrounded by helpers. We were building a product at the first Scrum company, that needed to go out to information technology developers in Ford motor company. For example, we had a thousand of them that were customers of our old product, and we knew they didn't have any surgeons; in fact most companies don't have surgeons of that capability. So, how do you build a process that could get a team working together well enough that they could operate at the level of a surgeon?
Well, when I saw the robot come in and professor Brooks explained how it worked, he called it the subsumption architecture, he had a layered architecture. He said: you know, in artificial intelligence we've spent 30 years trying to build a smart system. We tried to build the biggest database in the world and bring together all the information that we could put into it, we tried to build the biggest computer that would essentially create a white board in the sky, that we could do an infinite number of calculations, and what did we get? We got a smart chess program, it has been a total and complete failure. So he said "I'm taking a radically different approach.
We are going to build a system that has no database, its data is all going to be the data coming in from the world, it is going to be sensor-based, and it will have no central processor. It will have a processor in each leg that knows how to flap away. It will have a processor on the spine that knows how to coordinate legs, and it will have a neural chip in his head that, when you plug it in, the chip is blank." And you turn on this robot and the legs will start flapping and then all of a sudden it will be wobbling upright and pretty soon it is scurrying around the room like a toddler learning to walk. It's quite amazing to see one of these things.
So, by creating this layered architecture of simple things that work together you get something that appears really intelligent. And that's what you need to drop a robot in on Mars, or they are using them in Iraq to survey urban areas. And they are making vacuum cleaners out of them (maybe you are a Roomba owner?), this company is known as I Robot and it's one of the leading robot companies in the world today. This idea of creating a set of simple rules that would help a team self organize such that they achieve an extraordinary level of performance is fundamental to Scrum and it comes from the robots that used to chase me down in the office.
5 comments
Watch Thread Reply