We are here with Scott Ambler. Scott, can you tell us a little bit about yourself? What are you up to these days?

Yes, I am the Practice Lead for Agile Development at IBM Software group, and they basically go around the world helping organizations adopt Agile with skills, so large Agile, distributed Agile - it's sort of interesting.

What's going on in the Agile Community right now? What's the progress of Agile adoption?

In short story, it's that the majority of organizations are now doing Agile. At Dr. Dobb's Journal we ran the annual Agile Adoption Survey in February and we found that 69% of organizations had one or more Agile projects. 82% of them were beyond the power project phase. That tells me that Agile is - at least in the majority of organizations - now real and it appears to be here to stay. In a more recent survey that we did in August, we found that even though a lot of people were talking about Test-Driven development, not as many people were doing it, which is a real shame. I think that TDD is great, but unfortunately is also very hard to pick up, so I think we are seeing a little bit of that. It's also being inhibited by organizations, so to learn TDD properly you really do need to be able to pair with other people and if you are not doing pair programming, then it becomes really hard to pick up a skill like Test-Driven Development. A lot of organizations are really losing out on a real proactivity improvement issue just because they are not really investing in their staff, which I think is a real shame.

The other thing we found in a recent modeling and documentation survey was that agilists are doing a lot of modeling and that they are doing far more modeling than most people would think. The vast majority of Agile teams are doing upfront low, but if upfront modeling keeps going and they are modeling throughout. We also found that they are seem to be a little more likely to model in traditional teams, which was a bit of a surprise and they are just as likely to document as traditional teams. Some of the ..., you might have heard, are simpler and accurate.

What kind of modeling are the Agile teams doing? You don't really think of modeling in Agile typically in the same breath.

They are doing the full range. It's sort of interesting they are doing some upfront requirements modeling, some upfront architecture modeling. Fundamentally, regardless of your development paradigm, you have to answer questions like "What are you going to build? How long is it going to take? How much is it going to cost?" If you can't answer those things, you are not going to get funding. Reality is sort of trumping some of the Agile writers that we keep hearing. We also see teams that do sketching throughout the project, sometimes they capture it with drawing tools - just Visio, or with digital cameras. We also see teams using hardcore CASE tools - Erwin for data modeling, RSA for class modeling, object oriented modeling. We also found that although we might not be statistically true - but at least the numbers we have are pretty close - but the numbers that we got in this survey show that Agile teams were more likely to do the sophisticated levels of modeling in particular full round-trip engineering as compared to the traditional teams. For the most part, the traditional teams were using the fancy modeling tools simply for documentation, which is not what they are meant for.

What kind of proportions are we talking about here? I mean what percent of people were saying that they are doing modeling of this nature?

I don't know exactly on the traditional teams, but the numbers were high. On the Agile teams it was something like 87-88%, so it was a large number of Agile teams who were claiming that they were doing modeling. In that question, we also included things like anything from sketching on whiteboards all the way up to really hardcore complex modeling using fancy CASE tools - we included the full range. I think this is a challenge to the community - we need to get better at recognizing what we are actually doing and getting away from some of the theory, because the theory tells you that if you are not using these big fancy CASE tools you are not modeling. The reality is that if you are doing sketching, if you are doing paper based modeling, you are, in fact, modeling. I would also argue that you need to do those simpler forms of modeling first and get good at that, and then you are going to move into this CASE tools. I think the CASE tool vendors could learn a few things from the way they promote the concept of modeling.

Eventually you are focusing on introducing Agile in large scale development teams. What kind of issues are you seeing people run into?

When organizations are applying Agile in the larger Agile scale, I think there are various factors we need to consider. The two most popular ones - most common ones I should say - are large Agile teams - "How do you do Agile with teams of 50 or 200 or 500 or 5000 perhaps?" and "How do you do Agile in distributed teams?" - those are the common two things. The other factors that people generally don't think about, but are equally as important, are "How do you do Agile under a regulatory compliance scenario? How do you do Agile when not everybody is working as this big happy family that you keep reading about and that you have very serious political issues? What do you do when you are working with Agile at multiple organizations - let's say you have some contractors in place, you are partnering with another company and that dramatically increases your complexity? What happens if you want to get good at governance?

A lot of organizations do and a lot of organizations seem to struggle with governance and it's strange because, when you step back and think about it, Agile actually enables you to govern projects more effectively than traditional. I couldn't tell you how you could accurately and safely govern traditional IT projects - there are just too opaque, there are too few leavers for control from management, whereas in Agile it's very open, it's very visible and there are these phenomenal leavers for the stakeholders. If you allow them, you can let them control the scope, the budget and the schedule. It's a beautiful thing - it requires accountability but if you are given that accountability and your Agile teams have the discipline, then this is a very good thing.

How you noticed distributed teams affecting successful Agile applications?

Some of the rhetoric in the Agile community and in the traditional community too is that Agile is only good for small collocated teams. Working in small collocated teams is phenomenally intelligent - you might as well lower your risk and good stuff like that -, but the reality is that many Agile teams are distributed. Actually, in the Agile 2008 Survey that we did at Doctor Dobb's, we asked how are people doing Agile development and how distributed is it and how successful is it. We were actually surprised: we found that the majority of Agile teams were distributed, but there is also our definition of "distributed". If you have people working on different floors, you are distributed. If you have people working from home, you are distributed. If they work in different buildings, they are very clearly distributed. If they are working on different continents, they are obviously distributed.

What we did was we asked that for 3 different types of projects, so we asked about collocated Agile teams, where everybody is in a single room, and we found that on the average they had an 83% success rate. We also asked about near located Agile teams, so that's when you could get together on a daily basis but cars might need to be involved because you've got some people working from home or in a different building in the same campus, or on different floors. They are fairly close to each other but they are not in the same room. We found that those teams had a 72% success rate. Finally we asked about far located Agile teams where an airplane need to be involved to get people together and we found that those teams had a 60% success rate. This was success as defined in the eyes of the beholder.

In previous surveys we found that Agile teams had an average success rate of 72 -73% compared to the traditional teams, which had a 63% success rate. In various surveys over the years we've seen that sort of a delta in the success rates between traditional and Agile teams with Agile coming in roughly 10% higher success rate than the traditional, but this is always when success is defined by the actual stakeholder. This is an interesting point, because we've been told for several years now by the CHAOS Report by the Standish Group that we have a 34% success rate in this industry. I would argue that the CHAOS Report - even though I think the numbers are accurate - are not measuring the success rate.

What they are actually measuring is "Are we reasonably on time? Are we reasonably on budget? Are we reasonably to specification?" and then the assumption is that that's the way we define success. I think the numbers are accurate, but the assumption is wrong. The work that we've done at Dr. Dobb's is showing that it's actually a bit different. This inherently makes sense - I get to go to the organizations around the world, big and small, private industry and public and I've never seen a company or even a government agency that had that low of a success rate. It just doesn't happen. If I'm not seeing somebody - at least 1 or 2 organizations anywhere close to that, there is no way the average is that. We need to improve things in the industry and start getting better numbers.

In your opinion, what is a more appropriate definition of success on a software development project?

What's going to happen is that different teams are going to have different success criteria. It's a regulatory issue, so you've got a certain date you've got to get out, that's just the way it is. Other teams are going to want to make sure that they conform to a given specification, some are just going to want to spend the money wisely. Different teams have different success criteria, but some other thing we found in the Dobb's 2007 Projects Success Survey was that people, in particular stakeholders, valued spending the money wisely over coming in under budget.

They don't really trust in our ability estimates. That's interesting. They also would rather have software, which meets their needs as opposed to something that's built to specification. They also want something when it's ready to be shipped, as opposed that's on time and on budgets. They really don't want crap that's on time, they want good quality stuff and this is also interesting. We asked about 10 different job categories and every single category, without exception, said that quality was job 1.

Even the customers?

Even the customers, by far. That was really interesting and it tells you that - and what's going to happen is depending on your success criteria - you are going to change the way you work. I think it's an interesting observation that, when you look at the things people actually value, the Agile community seems to be far more in line with that set of values than the traditional community. Many times, the traditional community assumes the exact opposite and this is one of the reasons why we see a lower success rate in the traditional community. They've got a few challenges with the way they're organized, but fundamentally, they are just not geared towards higher levels of success, simply because their philosophical base is a bit out of whack.

Back to distributed Agile development: what are some of the solutions that you've seen work in practice to enable better success for distributed teams?

I get asked this a lot. I work with a lot of companies that are doing distributed Agile, so my advice is always the same - the best advice I can give is "Don't do distributed development! It's very risky because distributed development is a lot harder than collocated development." The Agile rhetoric is dead on. If you can, definitely prefer collocated teams and do anything possible to make sure you are collocated. I go to companies all the time and I see people grieve for having people working in cubes or working on different floors or stuff like that. There is definitely a competitive advantage to let people work from home. I'm a firm believer in that, but when I see people working in different cubes, particularly on different floors, or even on the same floor but at different ends, that's just crazy! I mean that's an interior decorating decision and when you consider the success rate for collocated teams is 83% and the success rate is 72% for near located teams - that's a little over 10% difference for interior decorating concerns.

What I tell clients is first of all "Don't distribute if you can avoid it. If you can't avoid it, then you've got 2 options and you've got to do both options. The first is you got to get better at your process, be more intelligent about the way you are going to organize the team. The fundamental advice there is to organize your team around the architecture. If you've got a distributed team - it's distributed for a good reason, that's fine! -, let the various sub-teams on various locations work on their different parts, because what happens there is that when you step back and observe what the communication collaboration is amongst people on a software development effort, the most of the collaboration is between the people that are working on the same sub-systems, the same components. If you've got the different locations working on their components, what you've done is you've reduced the communication you require between the locations, so you reduce your risk and your cost. You are not going to remove all of the communication, but you are going to get rid of a lot of it. The second strategy is better tooling.

Speaking of better tooling, the Rational Group that has been working on Team Concept has been talking a lot about Jazz and such. Tell us a little more about that and what it offers.

Our new product Rational Team Concept, which can be downloaded from jazz.net, free of charge - the demo -, is based on Eclipse, so for developers it's like an Eclipse IDE for distributed teams. The idea here is that if you've got a couple of people working from home, or on a different floor, or on a different side of the planet, you can see it when they are logged in, all the work item management, the product backlog and Scrum, all that is automated so you've got your databases stuff that you are going to do, then you are chatting. The two of us are working together on a project, I am working from home and you are working at the office, if I need your help, I can just bring up a chat box and we can start chatting away and you can get me help on whatever it is that I am working on and all that chat is put against the work item or the defect I am currently working on. You get this full traceability, you don't lose the information.

A lot of really nice little things are automatic like that. For the developer, it really makes life a little bit easier. It also automates a lot of the regulatory and traceability stuff that you just don't want to have to do by hand, so you get that stuff for free. Then, for the project managers, for management generally and the stakeholders, it has this project dashboard stuff. A lot of the important metrics, like your defect backlog and all your defect tracking and all your trend analysis, where your status is, it is all generated automatically based on the status of your work items. You can see where the team is, how much work they've got to do, how the defects are coming through - stuff like that. All that is generated ..., you have a data warehouse type of paradigm. This is really a beautiful thing for management because you get this complete visibility and this complete reporting automated for free from this tool. The result makes life a lot easier for everybody involved - the management has a clue what's going on and the developers are not being forced to write status reports and stuff like that, because a lot of the status information is coming out of that.

The point here is you are not going to get rid of all your risk that's associated with distributed, but, by being smarter on the process, having better tooling, you are going to end up at least mitigating a lot of that risk. I think there is a pretty good value proposition there, particularly for developers, - in a few years to Eclipse, then RTC is pretty much a no brainer, so it's really a beautiful tool, it's definitely worth looking at.

You talked about how RTC helps with the management side of metrics, but how is it actually helping with the team collaboration?

There are a few things there. First, I think the neat thing from a developer's point of view is that I know the people on my team and I can also, when I log into the tool, see who's currently online and what their status is, if they are available for help and stuff like that. A neat feature as well is because the tool implements the product backlog - the work item list, because at the beginning, when you are doing sprint planning, iteration planning you put your estimates against the work items, then you are part way through the iteration.

What happens in the beginning of iteration we are doing planning is we've got the things the team wants to do this iteration, we put numbers against it, then each of us will pick off the things that we are going to do this iteration, so my personal backlog says "I've got these things and as I am working down through them, once I complete them, the tool knows it's complete and stuff like that." What will happen is you can calculate where I am, if it's a 2 week sprint, if I'm 5 days in and I've still got 7 days with the work, it can accurately calculate that I'm in trouble and I can't take on more work, whereas if I was in a position where I was 5 days in and I've got 5 days left and I've only got 3 days for the work, then I've got some time on my hands and that's going to show them my status.

If you need some help, then you can give me a call and bring up chat and say "Hey, Scott, I misestimated, can you take over this work item?" or "I'm having problems with this work item. Could you help me out?" What will happen is, if there are 10 people on the team, then you can see who's currently online and what their current status is, so you've got a choice of asking for help from 2 or 4 people. If one of them is really busy, you know to leave him alone and if 1 or 2 seem to have some time on their hands, then those are the ones you are going to approach first. It's little things like that that make life easy. If you are part way through something and some other high priority thing comes in, you can actually stop your work, save it off as work in progress and then roll back to where you were before you started whatever it is that you were just in the middle of working on, then work on that defect that just came in or the new work that just came in that was in an ultimate priority and get over with it.

Do that work and then come back later to what you'd put off to the side. Or you can even push things over to the other people, or they can pull them from you and stuff like that. It just makes it easy to communicate and collaborate with each other and to share work and to share what your actual status is and where you are. It's a beautiful thing, like you should still be doing daily stand-ups and things like that, which becomes harder if your team is distributed. Also, what happens if I am in the middle of the day and I need some help and maybe he wasn't paying attention to what I was saying or the status has changed anyway - this stuff happens during the day? I've got a better clue of what's going on because I can see it automatically online, so that's a good thing. That sort of stuff is going on and it's nice.

What are the patterns for Agile development that you feel is an awfully good idea that RTC is designed to enable?

There is a collection of fairly important process patterns it should be interested in like, obviously, visibility and openness is a key one; definitely tester development focused on quality; parallel independent testing or investigative testing - we still haven't settled on a name for that, but the basic idea is to have an independent tester, independent test team working in parallel to the development team. You push your working build on a regular basis every weekend or at the end of your iteration and let them pound on it, find the bugs and then push them back with those defects and have them put on your work item list.

Another good pattern would be treating defects and requirements - it's the same thing -, just having them on your work item list and you are not really worrying "Is it coming from the business stakeholders?" or is it coming from your independent testers because they found something they think is a problem - that's just a requirement, you've got to deal with it - and that dramatically improves your overall change management process. Working close with the stakeholders is absolutely critical; having a good relationship with management is an important thing; continuous integration without a doubt. We've a build tool built right in and that's nice. You don't have to couple together all this using open source stuff, it's just all there, it's all working together. Testing often and early, not quite TDD, but minimally you should be doing a regression testing on a regular basis and the developers should be doing it, not just testers. I personally prefer test-driven development, but that is pretty hard, so it might not be there yet, but it's definitely something you should be aiming for.

Speaking about collaboration business, I was at a session recently with Martin Fowler. He was predicting the death of the IT department, saying that developers should soon be working close with business people, right there, collocated with business. What do you think about that idea, that prediction?

Not a chance! I've heard this a couple of times. This might happen in a few organizations and if you look at the history, different organizations will be a different part in this pendulum. Years ago, the pendulum - particularly in the middle-late 80s - swung over to highly distributed and user computing type of things - we saw products like dBASE, FoxPro - very popular -, Visual Basic became very popular and people were doing it. Actually it's one of the things why I would think that Visual Basic is embedded in all of the Microsoft Office products because there is this inherent need for end users to do their own programming and it happens on a regular basis. The IT department is not going to go away. Who the hack is going to run these systems?

Who is going to run the databases? Who is going to keep the network up and running? Who is going to support the hardware? There are a lot of business applications running and mainframe COBOL and they are not going to go away soon. I just read a headline - in the current issue of Dobb's - that was asking something like "Are you about the next COBOL programmer?", because they were running the numbers on the retirement rate and all the COBOL programmers were about to retire. Today's Java coder or today's C# coder is probably tomorrow's COBOL programmer. That stuff is not going to go away. For some organizations, particularly smaller ones, maybe it's possible, but I would argue that, yes, I think the business should be more aware of IT.

A lot of companies really get into trouble when the business makes relatively poor decisions - they think it's good business decisions, but it's a poor IT decision. The perfect example is doing fixed price projects or fixed estimate type projects - that's almost a kiss of death for an IT project. It pretty much guarantees that you are going to spend the money really poorly because, for the IT department to protect itself from that business decision, puts change prevention processes in place and it really starts working inefficiently, so it increases your financial risk when your real goal is to minimize your financial risk. The business needs to learn about IT a little more and that behooves us to educate the business. We need to have a better relationship with them, but the IT department is not going away any time soon.

Back to your distributed development: what are some other processes that you think are good ideas for enabling distributed teams?

Any Agile team in general has definitely to do some upfront modeling, but it's particularly true for Agile distributed teams or just distributed teams in general because, if you want to organize your team around the architecture, you need to know what the architecture is. You are going to do some architecture modeling upfront to figure out "Here are my subsystems, or my services or my components or however you are organizing your architecture" and "Here is what you need and then look your team and see if they really are distributed around the world, around the city or around the building." Then you say "OK, this sub-team is doing this, this sub-team is doing that" and so on. Your architecture and your team structure are going to evolve together and reflect each other and you want to get that right away.

You also want to identify at least the initial versions of the interfaces to those components or services right away. They will evolve, so you are going to need architecture owners on each of the sub-teams. This is something that the Agile community doesn't talk about all that much. Just like you have a product owner and Scrum who's responsible for the requirements, each of these sub-teams also has an architecture owner, who's inevitably responsible for the architecture. That doesn't mean that they are an ivory tower architect that we see in the traditional world to inflict the architecture on people, but it does mean that they try to mentor people in architecture, they lead the architecture effort. Then, when there isn't consensus amongst the team on how to proceed, they're the ones who are responsible for making decisions.

It's like in Star Trek: The Next Generation - if you look at the way that Jean Luc Picard would organize his team. The situation got interesting, he'd have a staff meeting in the ready room, they'd talk about options, Worf would say some crazy things that would blast him out of the skies, so he would be ignored, then everybody else would put in their ideas and Jean Luc Picard would say "OK, fine, here is what we are going to do: let's go forward!". Is that sort of thing - mostly leadership and ideally, if they could come to consensus, then that's what they would do, but if not, he would take the best ideas and say "Here is what I think we need to do to go forward" and would lead the team through that.

There is all the coordination effort between the sub-teams as well and the technical side they would be responsible for - that's actually an important role. I guess it's a combination of more discipline practices and a little bit of rethinking of some of the rhetoric around the roles. Some of the stuff we hear from the Agile community is great on smaller collocated teams and the vast majority says it's a great way to work. Once you get to more complicated situations to scale, then these other roles start to become important again. We need to learn to take the best of the traditional community because they are good and they do have some good ideas about scaling and how to build complex systems - take the baby, throw out the bathwater - that's what needs to happen.

How do you count for the role and the work that the architecture owner does in your iteration planning and estimation?

The architecture owner is just another team member - that might be technical team lead or the senior developer or whoever got motivated to take on that role at the time. They are just a person, but for whatever reason they've been tabbed to be on that sub-team, they really do own the architecture more because they are the most senior person and the most experienced. They are that person that the team feels can lead them technically, but that is just another role.

You mentioned the new role of the architecture owner. This person has to do a bit of upfront design because he is responsible for the long term stability of the architecture. How do you count for his time and work on iteration planning and on iteration estimations?

This one area where some of the original writings in Agile missed it. In the last couple of years we've been introducing this concept of iteration zero or sprint zero where you do this upfront work. Some of this work you got to do is put the team together to begin with, get your desk base, get your machine setup, do a little bit of the initial requirements modeling because that stack the user stories. It's got to come from somewhere, it just doesn't get beamed down from the starship Enterprise - somebody has actually got to create it, to get it going and basically to identify the scope. You're also going to use some architecture modeling because, as I was saying earlier, you are going to be asked questions about what are you going to build, how you are going to build it, how much is it going to cost, how long is it going to take, so you've got to handle an architecture, but that's something that's done by the entire team not just the architects. This is where it differs from the traditional world, at least - hopefully - it should work that way. You might spend the first week or two of the project doing that sort of work.

Since you mentioned Next Generation, I want to ask you what is the next generation of Agile? There is no clear definition, but what do you see as some of the practices and ideas that could form Agile: The Next Generation?

I actually just recently wrote an article, which I called The Astronomy Analogy for Software Development. What I've observed is that, for the most part, the traditional community has pretty much figured out how to develop low-quality silo systems. I let the legacy source code and the legacy data sources out there speak for themselves. The Agile community for the most part, has figured how to develop high-quality silo applications. I think the next generation is going to be the next paradigm shift is "How do you develop high-quality enterprise class applications?". I'm not sure I'm going to see this in my life time, but this is definitely the next paradigm shift.

Do you mean SOA?

SOA is good, but we saw that with components and objects, start procedures and staff like that. SOA is just another opportunity for bundling functionality.

What is the Astronomy analogy? 2000 years ago we figured out, through observation and good theory, that everything revolved around the Earth and this is an obvious thing. Then, 400-500 years ago some heretics came along and they said "Wait a minute! No, I'm doing better observations and my theory is that, with the exception of the Moon, which is this obviously the weird thing that is revolving around the Earth, everything is pretty much revolving around the Sun, not the Earth, so it's the center of the Universe." About 100 years later, Newton came along with these laws of motion and said "Wait a minute! Everything is revolving around everything else."

So this is what the analogy is because when I go to companies to help them adopt Agile, I'm often telling them "Here is the Agile theory. Here is how you scale it." It seems strange, working there closely, doing test-driven development, doing Agile modeling, but not doing the traditional documentation stuff - it's obviously breaking the laws of physics! I said "Let's say the laws of physics apply and we stop to do this stuff" and he said "Yes, the laws of physics apply, but we don't know what they are!"

I would argue that the traditional community is equivalent of the people believing that everything revolved around the Earth, but in their case, they thought everything revolved around engineering and around documentation. In the Agile community, there are the people who think that everything revolves around the Sun, or around the developer. We need one more paradigm where we figure out that everything has to reflect everything else and have to work together as a big team. This is very difficult - having an enterprise view of IT is a phenomenally difficult thing. This is something I wrote about a fair bit in the Enterprise Unified Process book a couple of years ago. When we wrote that book, I think we were pretty clearly saying "Maybe 5% of the organizations were mature enough to pull this off." It has a lot of advice for how to do Agile at the enterprise level, but I don't have a lot of hope for companies picking that stuff up in my lifetime.

I know that now 1 or 2 are trying it out and I wish them good luck and hope they do it. I'll probably see 10 organizations maybe that pull this off, but for the most part, I think we are going to spend the next 20 or 30 years figuring out how to develop high-quality silo applications. It's not ideal but it's a lot better than what we were doing 10-15 years ago. The next generation for Agile or the next generation for software development is going to be high-quality enterprise development. Some people in the traditional community have an enterprise view, but their underlying philosophy just doesn't allow them succeed - they are too command and control, too bureaucracy driven.

The Agile community has a better chance to succeeding at it, but they seem to struggle with this enterprise view and they are just too developer focused and I see a lot of people make some very big mistakes. We really haven't acted yet to get end-user interface development, which is strange because this is the most important aspect of a system to the end-users. Very few developers seem to have a handle on database stuff or security stuff or networking stuff, let alone we have the entire range, not just any one of those issues, but all of those issues and if you don't understand those things, you don't understand operations and support and basic production issues - you've got no hope with developing a decent system. For example, a few months ago I got brought in by a customer to review the work of an Agile team.

They have done a great job - it was a Ruby on Rails project and they've done some really good work, it was a great application, high-quality, test-driven development, all the Agile stuff. From the Agile point of view it was great - working software, good to go, stakeholders loved it, but it introduced yet another silo database, exactly what the organization didn't need. It didn't really fit in with everything else to support people; it didn't have any Ruby experience, it didn't have any Rails experience. Here is another technology being thrown in the mix there - exactly what they don't want. At the end of this review, the CEO just turned to me and said "Great! The Agile dumb nuts have done it again!" He just didn't appreciate it and the problem is that, even if they built yet another high-quality silo application, they just didn't have the experience or the background to actually do the job properly.

It was a real shame because they made just a few basic mistakes and coincidentally they were architecture mistakes that they should have nailed at the beginning of the project if they had done a little more modeling and calm down a bit on the Agile rhetoric and had worried a little more about scaling stuff. It's definitely not an easy thing to do, but I think that's where the next generation is going to be going.