So can you tell us a bit about yourself and what you're up to?

Sure, my name is Per Kroll and I am the product leader for a recently new open source project called "Eclipse Process framework" and also the head of development of IBM Rational commercials methods which include Rational Unified Process. I'm also a book author, I have written 2 books, and the latest is "Agility and discipline made easy".

Agility and discipline, sounds like an oxymoron. Can you tell us some more? (purposefully sarcastic)

It's not really an oxymoron, you want to have agility, agility means that you need to be able to respond to change, and you need to have a development approach that allows you to do appropriate changes rather than trying to upfront produce plans and stick to the plan. On the other side, you also have to live in reality in many organizations, you need to have some level of discipline, and that is not contradicting to being agile. But in some cases there are some trade-offs. So adding discipline you can often keep being agile but sometimes you can not be as agile.

How are agility and discipline sometimes contradictory but sometimes not?

Let me give 2 examples: one thing you often look for when talking about discipline is some level of assurance that things are going reasonably well. If you're looking at one way showing how things are going is to have an iteration burn down and a project burn down, or in Scrum you call that product burn down. So, those are things you can produce to give a certain element of rigour to a project, if you know where you are at, you can tell the stakeholders "this is where we are at, this is how well we are doing" and more important for the product team to understand how they are doing. You can do that without adding a lot of formality, it's pretty easy to do assuming you track some very few things.

So this is an area where you can add some discipline but in this case it actually makes you more agile because you understand where you are at, so you can take the appropriate actions. Since agility is about reacting to different factors and change, it gives you information you need so you can change effectively. Another example would be that you need to pass an audit. Now, to pass an audit you need to produce artifacts that the auditor wants to review. That may not help the team be more productive, it might actually take away time from productive activities but it might be a necessary evil so you need to add discipline which actually in this case reduces your agility. So, in some cases you can have more discipline that helps you be more agile, in other cases the discipline would actually turn you to be less agile.

What are some of the main advice from your book "Agility and discipline"?

So the biggest problem we are trying to address is that people don't know how to get started with being agile. And people don't know how to do the right thing. What we describe in the book is a number of practices, and for each practice we talk about when it applies, how you can adopt it, what additional resources there are - we reference a lot of literature on that practice - and we give you the "how to" adopt a practice. And for each practice we then give you 3 levels. Sometimes it's very hard for a team to be really good at a practice so if there are some baby steps you can take, maybe you want to go to level one, maybe you can directly jump to level 2, maybe level 3 is not appropriate for your type of project because it's dealing more with issues you need to deal with for a dispute in a product team. Maybe if you are collocated you wouldn't go to level 3. That is the overall strategy or intend with a book, to give you bit sizeed practices that you can adopt one at a time, but also we describe how one practice links to other practices to give you a sense of how you can adopt a series of practices to improve your capability.

So in your opinion what are the minimum practices a team needs to apply in order to be considered agile?

I don't think agile is a digital thing, like "this is one practice, once you learn that you turn from being not agile to becoming agile". If you define agility as the ability to change there would be a number of things you can do that allow you to react more rapidly to change. And this is a continuum, so people often are try to do..."you are agile or not agile". There are many different ways to guide you, for example "how agile are you". An excellent way would be to see how close you are to the agile manifesto. So for example some core practices: "it's hard to be agile if you don't do iterative development". So probably one crucial practice is that you need to divide your project into iterations and figure out what are the most important things to focus on within the next month for whatever length your iteration is. But then you can say "to be successful doing it in the development you need to have a number of other practices that you need to implement to various degree", and it will vary from organization to organization. What does it take for them to actually be able to do good iterative development.

So what are the most important practices to support iterative development?

There are a number of practices that are really important. The first one is that you need to prioritize what to focus on and break it down into small pieces so you can decide, implement and test each of these pieces within an iteration. So you have to make sure that the iteration delivers working software that has some value. So you both from a requirement stand point need to brake out a requirement so you can trace them to some final customer value. That's one of the practices: how can you make sure that you define a requirement in a way that will have value to the customers.

Another one is that you need to do continuous integration, that's a widely accepted practice, and many organizations are doing it, but it takes some effort for some organizations to move to continuous integration. Otherwise it would be hard to get feedback fast enough to succeed with an iteration. That will also require a level of sophistication in the testing, that you're testing continuously what you have produced, it's not only about integrating but making sure that you're testing it. So those are example practices. Then you have a number of other things that will help you to do a better job and figure out what you should do in this iteration versus next iteration - as an example, risk management. But that might be seen as one more practice to help you figure out what goes into an iteration but maybe not as critical as some of the core practices around having your testing practice work well, having continuous integration, and user-centric requirements.

You mentioned that your book also covers distributed teams. There's a lot of difficulty in applying agile in distributed teams. What would you say are some of the main practices distributed teams can use to be effective with Agile methodologies?

The biggest problem with distributed development is that you have problems communicating effectively. What we find essential is that you need to find effective ways of replacing or trying to find substitutes for some of the face to face communication when you are distributed. So in some cases what we try to do is make sure that the team meets and at least get some kind of social bonding. That is not always possible, but if you can do that it's great. Choose always the highest means of communication; email might be an important way of communicating but for example in my teams you can send more than 3 emails back and forth, you have to lift up the phone and communicate, because after you can email back and forth, but if you lift the phone you can resolve it in a few minutes.

Is that a policy?

Not a policy, we have it as a guideline and whether people follow it or not is more their judgment, but I found that sometimes people just crash back and forth with emails when a phone meeting with the appropriate people resolves the issues in no time. And often emails are escalating to not good collaborative discussions vs phone discussions become much more collaborative. The crucial thing is that in the end, no matter if you dispute or not, to have an integrated build that you are validating. Validating by testing or validating by trying out to using it and that will drive out a lot of issues. So those are some key practices.

But in general you also need to have the right infrastructure in place to support distributed environment. Things that you could work out when you were face to face need to now be supported by some level of infrastructure. So if I have a whiteboard to have a sticky notes for user stories, that becomes quite a trick when you're distributed, so now I still want to know 5 minutes after you change the user story what it is. But how do you deal with that in a disputed environment, and that's where tools can help you to have the right infrastructure placed, to enable distributed development.

What tools have you found to be most effective for distributed development?

We have a number of tools at Rational. In general you want to have some basic tools to manage your requirements, you want to have a crucial list, to have CM infrastructure that works for distributed development, you need to have test automation; all of those things need to be in place no matter what. It's just that as you are working in a distributed fashion you have a lot of additional requirements in terms of "can you work in an effective way with your CM system in a distributed environment or not?" Then you need to look at does the CM scale environment to handle the complexity of distributed environment. And it's the same with whatever tool you choose to use. It's not that you necessarily need a different type of tool, but the requirement on that tooling solution is slightly different because it needs to sometimes handle the more difficult situations with distributed development.

So you've written about RUP agile. Tell us more about that.

The first thing, I'm not writing quite on making RUP agile, I'm writing about agility and how to develop software using agility. Many people seem to incorrectly do a contradiction between RUP and agility; I see no contradiction at all. Agility is about reacting to change, it's about having iteration planning, it's about making sure that you as a team take ownership of issues and react to required changes. All of those are key values of rational unified process.

The biggest problem we have with RUP is that RUP is a rich knowledge base and what many teams try to do, which is something we recommend against, is because it is in RUP they try to adopt it all; when you try to adopt too many things, too many artifacts and so on, it slows you down. So when that slows you down you become less agile because it's too much weight to change everything to keep up with the necessary changes. If you try to adopt everything in RUP for a small collocated team, it's very hard to be agile, because it's too much weight to carry.

So what we keep on articulating is that you should only adopt the key aspects that are out there for your situation. If you are running a very large DUD project with hundred plus people which would last for 2 years, you probably have an interest in adopting very large portions of RUP especially when you need to comply this with certain standards that force you to call out a quality plan separate from a project plan. But when we're looking at how we use RUP internally, we don't produce separate different types of management plans, just because different government or military standards say so, we use only what makes sense to us and allows us to keep agile.

The biggest problem we have is to get the message out that just because it's in RUP you shouldn't use it. And to help do that have over the past few years we done a number of things in the product to help you sort out what to use and what not. As an example, in the last release we have a specific delivery process describing: "this is how you do maintenance projects, this is how you do a COTS package application development project, how you do small scale projects", and each one of those takes out a small subset of what is in RUP just to give you a starting point of what is an appropriate subset, but you need to get the team together and decide what aspects of these make sense for your team. We just give you a starting point, and if you are doing a maintenance project this small subset is probably the most relevant, that doesn't mean that you should use all of that or that you shouldn't use anything else, it just gives you a hint of what we thought could make sense for that type of project. But you need to make the right call on what makes sense for your project. But never use "it's in RUP" as an excuse to do things that doesn't make sense.

So who makes the call on what subsets of RUP should be used?

If you are looking at the failure in the 70s and 80s around roll out of processes is that you have management to PMO trying to say exactly how each team should develop software. And they come up with a very long list of things you need to do and things you need to develop. They don't do it because they are mean people trying to make life for developers as hard as possible; they do it because they think they are helping the development team. What is reasonable for the management is to put some requirements on "these are things that I need to feel that I know to some extent what this team is doing, I would like to see some kind of vision whatever form that takes". Maybe I need to make sure that there is a project plan at some level, to make sure that is some kind of notion "I will get these capabilities delivered at some time to a cost". Those are reasonable things to expect. But you may wonder why high level managers care about exactly how the team delivers on that.

So typically what you would like to have is a very few number of things, that are mandatory, that are enforced by the organization. And then the process should contain appropriate process patterns guidance that you may or may not use, to help you produce those mandatory things. The problem is that you put too many things as mandatory and you don't see many of the guidelines as just what they are: guidelines for "if this helps you, do it that way. If you know how to produce a vision document, just do it, you don't need to read up on it. If you need some guidance use the guidance". So it's very important to people to make the difference between this is mandatory, they might be a very good reason for why things are mandatory, might be regulatory reasons, some ways for management to have some insight, for when we need to find or when we need to restate objectives, when we need to rethink when to launch a product and so on. But it shouldn't get in the way of getting the job done.

How did you come up with your recommended subsets of RUP?

So we tried a number of different approaches to develop the content, depending on how mature it is, but typically we go through the following approach: first we look at projects reducing RUP or other approaches, see what seems to work, what characterizes the most successful projects.

Then we notice that over the past few years a number of customers have used RUP for doing COTS package application development, but have figured out on their own, we didn't give them "this is what has worked for other customers". So we look at that work with customers, we look at what other people in the industry are doing; in the area of COTS for example, we work for an SEI called AG Melon that has written a process for COTS called "Epic". So we worked with them and worked with a number of key customers to come up with notions like "this is how to use RUP for COTS package application development", then we tried out with some customers, and when we got past the feedback then we make it a part of the product.

And that's a typical cycle. In some other areas we need to do something more rapidly because we can't wait for 2 or 3 years to try something out.

It's always trick and trade: should we wait until we used it long enough to say that this really works, we know it for sure, or should we put something out that seems to work and seems to reflect what many people think is good practice and what some customers have used, but we haven't validated it. So we typically have the more mature content in the actual product, and the less mature content as a downloadable, you can then download, because then we can update it without any problem, the least mature content will be labeled beta release which means "use it on your own risk".

I saw you've been using RUP for some time and partially behind it, it seems that the industry has moved towards agile. So why is that? What did you get wrong?

Well I don't think we got anything wrong, I think that Agile movement is something that is a natural extension of what we've done. If you're looking at many of the topics at the conference (Agile2006) or just listen to a presentation on enterprise Scrum, one of the topics was that in Scrum over the past years have added a release plan. While we had the concept of iteration plan and a light weight release plan that just covered the high levels more than 10 years ago.

So what are the keys differentiators between Agile and RUP?

There is no differentiation or contradiction, is it RUP or agile. What we've done with rational unified process is to capture an agile process talking about continuous integration, testing, iteration planning, you need to have a light project plan or what in Scrum is called release plan. The big difference is that we are trying to capture and write down some of these best practices. And since we are working with a wide variety of project teams we're working on a lot of different practices to be used for different contexts. So RUP should be seen more as a knowledge base for software development.

There's nothing in RUP that is secret; you can go in to any book store you will find 100 books, and find anything in RUP there. What we've done is we've produced an online knowledge base and we made sure that we configure it to fit your specific needs. There are some downsides with that. The risk is that people read a text and always everything that has been built can be mis-used. So you can read the text and say "this is the way and only way of doing it", rather than read the text and say "there's some good experiences here that can't be applied to our situation, but I learnt something, let's do it slightly different".

So what we tried to do is to create an additional vehicle for communicating best practices or process patterns. What is best practice depends on the context what is a best practice for one project is a really bad practice for another project. And we think it's a strength to have it documented so people can learn, read, debate and criticize. We get for example requests for changing our content every year; people read the content and say "no, this doesn't work for us". So it gives us a natural feedback loop. This should not replace the need for coaching because in the end, especially when you are moving from waterfall to iterative it ends up being a mind shift you need to create, a cultural shift; you can't tell somebody to read something and then change how they operate it, change how they do their business.

You need to use a number of mechanisms and coaching is an example of a key way of transferring the way people operate into a different way of doing business. So we see no contradiction but we think that Agile movement has done a fantastic job focusing on the cultural issues, I think the Agile movement would benefit from having some of those practices documented, we think there are other ways of documenting these practices - similar approaches as RUP - and we need to make sure that when it's documented people don't just follow it because it's written down but really stop and think "does it apply to me?". Maybe not then you shouldn't follow it.