I am here with James Shore at Agile 2007. James, tell us a little bit about what you do and who you are?

I have been involved in the Agile community for quite a while. I presented my first experience report at the first XP Universe in 2000 I think, and I have really been interested in Agile methods and XP ever since that time. In 2005 I was one of the first recipients to Gordon Pask Award and for the last year and a half or so I have been working on a book called "The Art of Agile Development".

So you have been developing this book with online visibility and feedback from the community? Tell us about the book

Well, in a way I have been writing this book for the last seven years, because this is really about taking everything that I have learnt over that time. For the community aspect I had an online review list and I have a co-author too I should mention Shane Warden and we solicited feedback on every single section of the book as it was produced and we got over a thousand emails on that list.

And what is it about?

There are a lot of books on Agile development out there, but I felt that there was a gap in one particular area, and that is if you want to learn about pair programming you can read the pair programming book; if you want to learn about test driven development, you can learn the test driven development book; but if you want to see how everything fits together, it's been a while since a book showed you how to do that and that's what this book is about, it's about how everything comes together.

So do you start right from scratch? Can a team pick this book up if they are considering using Agile and start to use it?

Yes, the book is in three parts. In the first part of the book we take the reader through all the basics: what is Agile development, why would you care, we don't spent a lot of time on that because it has been done to death. And then we go into a lot of material on how would you get started using this. And most importantly when it's in our book appropriate to and when is it not. Because our book can't cover every possibility, they will just take up too much space.

It must be challenging when you write a book and you have to make choices about what to put in. So what did you put in and what did you leave out?

It is challenging and this is a fairly large book, it is about three hundred and fifty pages maybe four hundred when it's all set and done. And we just decided that we are going to focus on a specific audience for the book and the audience that we wanted to focus on was sort of the early majority audience, the people who have been hearing about Agile development for a while now, and now it's popular enough, they have heard enough stories from their friends, where they think they would like to give it a try, they would like to give it a serious look. And so this has a very pragmatic tone to it, it's very focused on "Let's see how this really works in practice". So to do that we chose just one approach to follow. And there are lots of different Agile methods, there are lots of ways to do Agile development. But we chose an approach that was largely based on Extreme Programming and that allowed us to cover everything from planning to developing, to less concrete things like how are you going to reflect on what you are doing and how are you going to work together effectively.

And all these things really come together into one concrete whole: now on the flip side of that, that means that there are some situations where our book isn't appropriate. And rather than try and cover all those situations, what we do is at the end of every practice section in the middle of the book, we say "Well here is where you shouldn't do this". We say "We recommend that you pair program", which can be controversial so we explain why we recommend it. And we say "And here is when you shouldn't do it". Some people really don't like the idea of pair programming so you shouldn't do it then. Some people don't want to sit in the same office, well it is going to be hard to pair program if you can't sit together. And then we say: "Well, if you are not going to do it, here are some things to consider instead. But be aware that we are recommending this for a reason, and if you choose not to do it your life might be more difficult".

So you are helping people understand the trade offs that they are making when they are choosing practices?

To the degree that we can, yes.

So tell me more about the exercises in the book.

All right. Our editor at one point said "Let's put something for people to do in the book". Which really led to something that I thought was the neatest part of the book and we called it etudes, and this is an idea I first heard from Ken Beck at one of the conferences several years ago. An etude is a term from the music industry and it's something that people do to practice their technical skills, like a piano skills are etudes although they are not very interesting etudes there are some other etudes out there. There are something you do over and over again to gain a skill. And we put these etudes in the book to begin each of our five major practice chapters, to help people practice those skills. So for example in the planning chapter, we have an etude that discusses different aspects of planning. But they tend to go a little deeper than just the practices, so for the planning chapter we say "Let's look at the sources of waste in your process so over time we are going to do this etude multiple times. Every time you do this, come up with a new thing that you have to do to get right software from idea to delivery". And then let's look at what can you do to reduce the amount of time it takes to do that. Is that a wasteful activity that you can actually eliminate entirely? And then these etudes actually tie into the third part of the book, which is about the philosophy underlining Agile development.

What is the philosophy underlining Agile development?

You know I don't know if I am qualified to answer that question. And Shane and I didn't just want to say "Well here is the philosophy of Agile development". So what we did is we went through all the Agile books we could find "Lean Software Development", "Agile Software Development" by Alistair Cockburn, "Agile Software Ecosystems" by Jim Highsmith, many, many books and we looked through all them for what were called values, or principles or some of the underlying ideas of Agile development. And we wrote all those down on index cards, and we ended up with about two hundred of them. And we spreaded them out on a big surface, a carpet in his living room, and we started doing an infinity mapping, which is where you take things that are alike and put them together and things that are not and separate them apart, and we just wanted to see is there something there that we could say is really universally what Agile development is? And we grouped them down in about eighteen principles which I think really do represent the heart of Agile development and could apply to any practice. And this was important to us because we spent a lot of time in this book talking about specific practices; we say "You should do these things if you want to have success". And the problem with that is that it's not going to be applicable to every situation. And Agile development isn't about following a list of rules but following a list of rules is a really good way to learn, so we had to resolve this conflict. And that's why we had this third part of the book which is all about the principles, because it allows us to go from following rules to breaking the rules.

Will you share your principles with us?

Sure, there's about seventeen and we broke it down into five categories of principles and those are Improve the Project, Rely on People, Eliminate Waste, Deliver Value and Seek Technical Excellence. And then the actual principles are in those chapters so under Rely on People we have build effective relationships, let the right people do the right things and build the process for the people. Under Improve the Project we have understand your project, tune it out and break the rules, which is one of my favorites. One of the neat things about the book that I think many of the readers won't even notice, is that there are five chapters in part three of the book which is about how to break the rules and master agility. And there are five chapters in part two of the book which is very specific "Well, you are going to do this and you are going to do this and this". And those five chapters map back and forth to each other exactly. The first chapter of practicing XP part two is thinking, how are you going to work as you are thinking about your process. And then we have collaborating, releasing, planning and developing. The releasing chapter, which says how you can release effectively ties into the eliminate waste chapter which talks about how you can remove sources of waste and that's release more often and more frequently. And we used the etudes to tie these two things together.

What are the principles in the book called Eliminate Waste?

That's chapter thirteen in the book and the principles are work in small reversible steps, fail fast, maximize work not-done and pursue throughput. And as I said that ties back to part two, or part two ties to part three, so in the releasing chapter which is the corollary, we got things like "Done-Done", which says that whenever you are going to finish a story or something to do make sure it's all the way done so you can actually release, because if you don't that leads to waste. Make sure you don't have any bugs or very few bugs in your software because that too is a form of waste. Use version control, have a ten minute build, use continuous integration. We even talked about documentation a little bit because despite people's prejudices documentation is part of an Agile process but we want to do it in a way that's going to eliminate waste, allow you to fill fast and work in small steps. So the next categories deliver value, so the corollary to eliminate waste, that is going to reduce our costs, but deliver value is what's necessary to really be successful. And one thing that I really care a lot about is success, what does it mean to be successful, and maybe we can talk about that a little bit later. In the deliver value section of the book we have the principles of exploit your agility, only release when code has value, deliver business results and deliver frequently. There's one more. That one ties back to the practices of planning, because you need to plan in order to deliver value, which again is something that people sometimes assume that Agile methods don't do. But of course we do planning all the time we are just willing to change our plans.

And the last core category is?

The last category is Seek Technical Excellence. And we didn't try to break that down into principles because there's a lot of books out there about technical excellence and how you achieve technical excellence and instead we have an essay on design principles.

At the beginning you suggested that this is not an Agile methodology book but rather a book about succeeding in creating software. So what does success look like?

That's a great question because my understanding of success has really evolved over time. At first I was happy if I just had a technical success, if the software worked it was great, I was happy, but of course that doesn't always work, in practice, you got to have people using the software. Then I thought if we have people using the software and the software is usable, then we have success. But even that's not true, I had that sort of success, I had software that was usable that still was a business failure. So, this is a question I have spent quite a bit of time thinking about because I think it's the heart of the software question, how can we deliver something that is a success? Well what does success mean? And for this book we realized there were three separate elements. You do need a technical success because without a technical success you are going to deliver something that might work now, but probably won't work in a couple of years and we have seen a lot teams building software that just fails after time and then their companies just have to throw them away and rewrite them, these are the type of projects people call legacy projects that people don't want to be on.

So technical success is good but not enough?

It's not enough, exactly. You also need an organizational success. You have to deliver something that the organization is going to value. And that's unfortunately not even enough, not even just usable software or software that the users love. It has to have something that you can sell or that somehow gives you a higher return on investment. And I think that's a real important idea that isn't talked about enough. We talked about it in the Agile community and I think that's one of the strengths of Agile development. And that's one of the reasons we spent a lot of time on that in this book.

You wanted to teach people about software techniques and about thinking about business value all at the same time.

Yes, and not just thinking about business value and software techniques, but there's a really important part of Agile development as well and that's how do you work together, how can you be successful in the way that you work together? Which brings us to the third category of success and all three of these really have to come together for you to have a great project. And I want people, I want everybody that I know to have great projects, projects that they can go away from and say this is the most fun I have ever had; this was the most successful project I have ever had. So the third category is personal success.

Tell me what did you and your co-author conclude about personal success in software projects?

Personal success I think is much harder to guarantee than a technical success or an organizational success. And of course we can't guarantee anything, we are writing a book, we are not developing software for you. So it's really going to be up to the people who are doing the work whether or not this is going to be successful, but we are doing as much as we can to help them get to that point. Personal success, that's even harder because personal success is different for everybody, it's personal. So what is success for one person may not be success for another. But in general people tend to really like having the other sorts of success, they like feeling they have accomplished something significant, they like having a life outside of work and they like getting along with their co-workers. So Agile development I think enables this type of success as well. And we actually spent a whole chapter in our book on collaborating and how the practices you can use to collaborate successfully like, and some of those are very important but not often talked about in books because it is hard to talk about them. So we talk about trust and how can you really engender trust among your team members. We talked about something that XP calls ubiquitous language, how can you communicate well between all the different people on the team, you have got programmers and testers and domain experts and interaction designers and people that XP calls customers. How can you have everybody work together smoothly? One of the interesting things that I discovered is the practice of sitting together really can do a lot for that level of trust and collaboration.

At the Agile 2007 conference there has been a lot of talk on how to succeed with distributed teams. So it's interesting to me that you talk about sitting together as an important factor in building trust and having personal success on a project.

Yes, and that is really interesting. I think that sitting together is one of the most important things that you can do on a team. And yet we have all these teams that aren't co-located. Now for those teams that are actually in the same building, I really recommend that they try to find a room, and that can be hard, you have to work with the facilities, there's not always space, but if you can find a room it's really amazing the kinds of results you can get. But what do you do if you are not in the same room? What do you do if you've got some people in Canada and some people in Sweden, or in India or in China, or just even in different parts of the US? And that's a hard problem and I think it comes back to what's the reason behind sitting together and this is something we tried to talk about in the book when we say if you can't do this, so we said we would really recommend that you sit together, and of course, here is the situations when you can't. And one of them is when you are not in the same office space, when you are not in the same time zone. So then you have to do something else, and when you are thinking of something else that you can do you have to think about what's really going underneath and what's really going on underneath sitting together I think, it's not about knowledge sharing it's about forming that bond and really having the strong feelings of working together as a team. That's something that you get when you sit together and eat together, share jokes and go out to the pub afterwards. So for distributive teams my recommendation is do whatever you can to perform and strengthen that bond. And I think the teams that are successful with distributive development in Agile method really spend a lot of time on that. Monica Yap had an experience report a few years ago where she talked about one XP team in three different time zones where they were actually doing 24-hours development around the world, and their coaches and their team members flew back and forth to other sites all the time and really spent a lot of time with each other, and I think that's what really made it work for them.

So you see an airplane as a team building tool?

Absolutely, yes. Airplanes and pubs are great team building tools.

And toys.

And toys. Anything you can do to make the work more than just "I have got this list of stuff to do and I got to do it". Because that turns into "I have got this list of stuff to do and I got to do it, so why are you interrupting me and preventing me from getting my work done?" But if it could be "We have this goal we are trying to accomplish and we are going to accomplish this goal, for a great purpose, we got something, maybe it's not saving the world, maybe it's not solving world hunger, but it's some purpose, some important thing that we are doing for the company, to get them to an organizational success. And we are going to do the best work we know how, and we are going to have a technical success". If you can work together as a team and have that call, that starts to be forming when somebody says "Hey can you help me out with this?" the answer is "Yes, I would love to help you out with that because that's going to make us all be more successful".

I think it is easy to get stuck on thinking about the technical practices, especially people passionate about coding and testing and doing the software. Do you think it's important that there is somebody designated to think about the process, to think about facilitation, and to be looking at a meta level over the team?

Yes I do, and I would like everybody on the team to do that at some point. And also I don't want to put down the technical practices, I think the technical practices are a very important tool and we spent a lot of time on those. But we also spend a lot of time on this team building and there are thinking chapters, we call thinking, is really about this meta level, how do you make sure everything is still going well. And in Scrum we got the Scrum Master and in XP we got the XP coach, and that person's job is really to be thinking at that meta level, I think that's very important, without somebody really thinking about what's working well and what's not, the team is not going to be able to continue and improve. And of course we want to do retrospectives and we talk about, we have a great recipe for doing a retrospective. But you still need somebody that's constantly thinking about that, I think. Now my preference is that everybody be constantly thinking about that. And the ideal of the Agile team is the self organizing team where everybody has their own strengths and everybody on the team is aware of those strengths and people are constantly shifting who is leading in what area. And ideally I would like everybody to have a strength in terms of thinking about processes and how can we improve and how can we be great and how can we have fun. And in reality there are usually some people that are more interested in that than others. And especially for beginning teams people are getting started and even in intermediate teams it takes several years to get to this level where everybody just fluidly taking responsibility. I think for these newer teams it's really important to have somebody who is specifically been put in that role for taking responsibility, because otherwise it doesn't happen and then it will not improve like it should.

Any ideas on how that person can be measured in that role?

Well, measurements are tricky subject, because once you start putting money or some sort of punishment or reward behind something, people change. They start really stressing out about how am I going to do on this measurement, and it's funny you don't even have to have a reward there you can just start tell somebody that you are measuring them and they will change their behavior. So on one hand measurements can be a great way to encourage the kinds of changes you want and we have a kind of practice that comes from XP called Informative Workspace, the big visible chart, and if the teams decides there is something they want to improve, for example let's say that they want their customer to be involved more often, they might say let's start measuring that, let's put a big chart on the board and watch the level of our customer involvement go up. And just having that chart will make people want to see the number go up. The problem is you really want those measurements to be short term, because once you start having somebody who judges that somebody that is saying it's their performance criteria and this is going to determine whether or not they get raises, for the year or the next two years or three tears, they will alter their behavior so much that it will be a little dysfunctional everywhere else, they will just focus on doing that one thing really well. And then you get to the point where you have to start putting other measurements in because you don't want them to neglect the other things and it gets very complicated. And I think it's very difficult to come up with a sequence of measurements that is going to force the behavior you want because behaviors are complex, it's not that simple, it's not mechanical.

And everything is changing so what you are measuring today may not be what you need to measure tomorrow.

Exactly. So how would I measure a coach whether or not they are doing a good job? I would be very careful about doing something like that, because I don't want to measure an individual of a team, I don't want to say "You are doing a good job and you are doing a bad job", because I want the team to come together and work to solve this problem all together. And measuring individuals and ranking them, let's say "You're the best and you're the worst", that's certainly not going to help people and feel like they are working together, is it? I wouldn't try to measure the coach, I would measure the team. And I would say "Is the team having a lot of success in the way that they are continuing to improve the process?" And actually in the book we have a whole chapter on assessing your Agility which can be a little controversial. And we put that in because people do need some guidance. Ideally the success of your team, and the success of your coach, and the success of everything, it depends on whether or not your project was successful. If your project was successful it doesn't matter what else you did, you did it right. And that said, you can't tell that until afterwards and you might want to know if you are going in the right direction. So we have a whole chapter called "Asses Your Agility", and we have a bunch of questions that we ask that you can plot out in the chart and that will not tell you if you are Agile or not, because it's not my place to tell you that, but it will tell you if you are facing common risk areas. And we have got a really neat algorithm that gives you a red, yellow or green risk assessment.

So the person who is in a mentoring or coach role, the word I used is they were thinking at the meta level. I think of that if they are thinking beyond the current iterations, they are looking at the trends. And you have talked about the improvements of the team. So perhaps that is a good indicator of the health of the team and of the coaching they are receiving, is the improvement that they are able to accomplish together.

I think that's part of it and we have a question that says: "Are you making at least some small improvements to your process every week in our assessment?" I think the coach is responsible for a lot of the intangibles, a Scrum master if you are in a Scrum team, so one is "Is the team improving regularly?" But another would be, particularly with the technical practices they require a lot of discipline, a lot of self discipline, this isn't imposed discipline, this is self discipline where people are remembering to do their tests, and they are remembering to think about the design, so I think the coach keeps an eye out for when people are letting loose their inner pig, as some people like to say. And pair programming is very helpful with that which is one of the reasons I think it's a good idea, pair programming helps people remain self disciplined, because there is a peer pressure there. So when I coach, one of the things I listen for is, if two people that pair together they are talking back and forth all the time. When people get into trouble and when they let loose their inner pig.

Is that pig like in the Scrum pig and chicken?

No, it's just your inner monster. We know we are supposed to do the right things but we are humans, and sometimes we are tired and we just don't. As a coach one of the things I would listen for when I see two people working together, they will be talking to each other, but sometimes they will stop talking and they will start whispering to each other and that's because they are ashamed of what they are doing. So when I see that happening as a coach I won't go over and say "Are you guys being disciplined?" As a coach I am a member of the team too, and the whole team needs to be friendly and cohesive and get along well, and argue well and solve problems together. So I will just ask "Hey guys when was the last time you had a passing test?" Because you are supposed to have a passing test every couple of minutes, and their answer is often "Oh, it's been about half an hour". And just that question is enough to remind people what they should do, because they just get into a ride. That kind of subtle help is what I think the coach does. And it's not just on the technical practices, it's on people working together, it's keeping an eye out for conflicts and tensions in the team and figuring out ways to help people get back together. I often see a lot of tension between programmers and customers, and one of the ways I can help programmers and customers work better together is just have them sit together again, eat together, see what each other does and have empathy for what each other does. There's a lot of things like this that can help the team run more smoothly and I think that's what the coach can do.

So what's coming up next for you James?

I am going to take a break. I worked full time on this book for just over a year and it's been longer than that, like I said in the beginning over a thousand comments on the review list, my co-author and I went through revision after revision, after revision, and I am ready to just stop and think about what I want to do next for a while. So I will take a break and spent some time with my family and do some work for my customers.

We had an insert in the package that we received at Agile 2007, which was an excerpt from the book. Can you show us that you've been referring to in this interview?

This is the excerpt; O'Reilly was very generous in putting together this excerpt for us, I am really happy to have it at this conference, at the Agile 2007 conference. They also provided a nice discount that can be found at O'Reilly.com/go/Agile.