The Importance of Technical Practices in Agile

1. This is Ben Linders with InfoQ and I am at QCon London talking to Tim Ottinger. It is great to have you at QCon, Tim!

It is great to be here.

2. You gave a talk about taking back Agile where you talked about things that can and do go wrong with Agile transitions. Can you give some examples of that?

Certainly. The most common thing that happens is that people think they are going to Agile to make things go faster. So, as a result, they try to put 10-weeks’ worth of work into a 2-week sprint. That is the most common thing that happens. Agile does not actually go faster, it does not actually produce more code, it produces more value and it releases sooner. That is confusing for people. So people who adopted straight by the book just do not look into it that far to see the differences and how they plan and operate. The other one is that a lot of people adopt the formalisms, but they miss the principles and as a result, they have something that is pretty dry and uninteresting and unhelpful.

3. Looking back at your first answer – do you think that organizations sometimes have false expectations of what Agile can bring them?

I think that is definitely true and sometimes it seems like it is being adopted without a real purpose. Why are we doing this? Because someone told us to. And I think that is not necessarily a good way to give an executive directive: “Hi. I want you to do this thing. Don’t worry about why!” Managing is like software because there is a lot of decision-making, right? Every day you make a million decisions, small ones and large, and if you do not understand the goals and the purposes, then you are not going to make the decisions in an optimal way. I see Agile implemented like bad software is implemented.

4. Agile could actually learn about how we are developing software then in Agile transitions you think?

I think so. I think that is fair.

5. That is great to hear. Sometimes we can hear Agile teams complaining about having to attend many meetings and they may feel like Agile forces them to do backlog grooming, planning games, stand ups, demos, retrospectives – all of that stuff. Do you recognize this?

I do and I think it shows a difference in time. So, if we go back to the late 90’s when Agile methods were born, early XP, early Scrum, we had been suffering for decades under a really onerous process, with lots of paperwork and lots of meetings. It was really hard to find any time to do any software at all. And then we developed these Agile methods that have a few meetings, a planning meeting,a retrospective, a demo, a daily standup to attack the day. That was a lightweight process. It relieved us from all of the paperwork and meetings we had before, but people coming into the field today did not live through the Waterfall years, they did not go through the paper-heavy process. So, they are going from no process to suddenly having five designated meetings they must attend and also, frankly some of the meetings are not well performed. So, a 15-minute meeting can be 45 minutes with no results whatsoever and a one-hour retrospective can be an all day thing that produces nothing. So, they are right! They are absolutely right! The meetings are too long, there are too many of them for someone who has never lived through the prior process. What is a relief to us is a burden to them. We gotta to fix that.

6. Any suggestions on how we can fix that? What is the way to make sure that these meetings are productive and are bringing value to the team?

Well, I think there are two pieces to it. One is that we have to remember not to batch our work. We really should not wait until the retrospective and bring every complaint we have had. We probably should fix things on the fly. And then our retrospectives are rather short. It is how did work and what do you want to do next? On standup meetings, I have seen teams spend 40 minutes on “What have you done yesterday?” because everybody is multi-tasking on a thousand things. If we reduce the WIP, the work in progress, then the meeting can take 5 minutes or 10 minutes or it can go 45 minutes, but it can produce a lot of shortcuts and timesavings and learning. So, it is all how you conduct it. But again, it is about knowing the purpose – if we are having the meeting because it is 9 a.m., then it is not going to be a very useful meeting; if we are having the meeting so we know how to attack the day, that is a different meeting. It is productive and short.

7. Do you think the facilitation of those kinds of meetings, of stand-ups and also retrospectives, plays a role in there? Does that matter?

I think it matters rather a lot. Again, if you have an executive order from somebody, they should tell you the reason why you are doing the things you are doing. I am afraid I am going to say this and it is going to sound disparaging – but there is a category of people called 2-day wonders whose entire experience with their Agile method was a 2-day class and they walked out with a certificate and now they are running the meeting exactly as they were told to run the meeting because that is how they were told – exactly four questions, what have you. They do not really understand the purpose behind it all and if they did, they would really revise their process. So, people who do rote facilitation by the book are not going to get the same results as someone who understands their purpose and goals and can move things forward. It is just understanding.

8. What can we do to increase the understanding to the people, to get them more aware what the purpose is of the meeting and to know how to do it?

The first thing is that you should never run a team, unless you have been on a team. So I think that people who have no experience with Scrum, should not be the PO and the Scrum Master. Developers come into it, testers come into it and they learn, and they develop and they understand getting the work done. As a top-down directive, without direct Agile experience, it is not going to happen. Now, if you do not have experience and it is your job to be the Scrum master, there are a lot of forums, there are a lot of papers and one of the best things that you can do is to go back to the original forms, read the Scrum guide – by the way, I found Scrum masters who had been Scrum mastering for five years and have never read the Scrum guide. It is mind blowing! Go back to the original XP – XP installed, XP explained. These are places where you can learn an awful lot about the real intent of the methods and without that background, what you are doing is a dry process. Don’t be surprised if people hate it.

9. I recognize that and I am glad we had those solutions to this, solutions that people can do to learn a lot more about this. Last year you did an interview with InfoQ about taking back Agile in which you mentioned that Agile was suffering from its own commercial success. Can you elaborate on this?

How much time do you have? So, the problem is that, often, when you are selling a thing, you may have to make certain changes or compromises to make it more attractive. So, the Agile methods were the most anti-Taylorist methods ever, whereas Taylorism tended to dehumanize people, although, give it a break! Taylor was the first one who decided that workers should have rest breaks so that they can recover from their work. So, he was more human than a lot of the Taylorism we see in software. But even so, there is a dehumanizing effect – so, people are supposed to be like cogs in a machine – that is a brutal, mechanistic metaphor.

What a reduction of human beings! But there are organizations around on a very Taylorist, top-down, master and minion, control based strategies and to sell them Agile, you will have to sell them an Agile that is amenable to the kind of work they have been doing. So, when this amazing, productive, semi-humanistic methods meets Taylorism and then we start selling off pieces of Agile to make it more profitable and make it more comfortable, it goes down the tubes and as a result, we end up with companies that are doing everything that you would have done in Waterfall, plus everything you would do in Agile and the developers have no time to develop, the testers have no time to test, they are still carrying on work for months at a time, they still have hardening sprints because their code has never been integrated and they have the same kind of failures they always had. If we are going to take back Agile, if we want to, as developers, own it again, we have to go back to the principles and we have to remember that cogs don't make good decisions and all software developers do all day long is to make decisions. 11/12 of our work is entirely intellectual, the typing is nothing. And if we are in a system that does not respect the intellect and decision-making capabilities and supports those, we are never going to get good software, Agile or not.

Ben: But selling such a solution will be harder, I think.

I think it is and I think that reminds us of what Kent Beck said in the early days. We don’t know that this is for everybody. There are teams that can do this. There are organizations that can be Agile and there are some that are not willing to go there. I think that is OK. I think that there are probably undiscovered methods of developing software out there and somebody does not have to go Agile and they do not have to go Waterfall. They may come up with something much better and I think that if you have an inferior method, you are going to get inferior results and that is going to show up, eventually, in the market. Some companies are juggernaut with so much money that if they never did anything right again it would take 10 years for them to die. But I think that is OK, too. I think that it is a reasonable thing that if you cannot go Agile, don’t go Agile, and if you use an inferior method, then eventually the market will deselect you.

Ben: So it is a kind of evolution.

I think so.

10. What, in your opinion, is needed to do successful Agile transitions?

You could acquire a very Agile method if you did nothing but three things: first - deliver frequently; second – do everything you can to make it easy to deliver frequently and the third one is to give people enough trust to grow into it. I think with just those three things, any company could be Agile in a matter of time.

11. What is needed to create trust in the organization? Is there something that organizations can do?

There absolutely is. The amazing thing about trust in an organization, as opposed to trust in a family, is that it is transactional. In a family, you have to give a certain level of trust to your family members and sometimes they are going to fall flat and you just have to deal with that and move forward. You do not have to give your criminal uncle your bank account number, but there is a certain amount of faith you have to give each other and sometimes it is disappointing – you are ready for that.

We have trouble with that in business, but that is OK. In business it is entirely transactional. If you deliver results, then your manager can give you some trust – a little bit more leniency, a little more latitude. If you give full transparency, so your manager can absolutely see what is going on, then he can participate by giving you more latitude to make decisions or helping you make those decisions. Over time that performance and transparency is traded for enough trust that you can have quite a lot of reasonable status, a lot of autonomy, a lot of certainty in your work, all the SCARF threats start to disappear.

12. I have heard more people emphasizing the importance of technical practices in Agile. So, you are certainly not alone in this. Do you feel that things are changing? Are more people getting this?

I have a sampling problem. Usually, I am called into a company that is having some struggles. They are either very, very new in the process or they have been operating in a way that is not optimal. Sometimes, these people have realized after a year, a year and a half that the problem they have is that they do not have any technical practices. So, I tend to come into places where it is already missing. Now, I go to the forums or I go to the conferences and I say something about – TDD uptake is hard for people or pair programming is hard and I find dozens of people who say “Oh, no! We do this all the time!” So, as long as I am visiting - a doctor who sees a lot of patients does not see how many healthy people are around and I am afraid that I have the same situation.

The bigger problems we see are the inability to slice work thin and get it completed. Technical practices are a way of doing that. I think that without TDD you just don’t have the cycle shortness. The cycle is: there is a time when you inject a defect, there is a time when you detect the defect and then there is a time when you correct the defect. That cycle has to be incredibly short if you are releasing all the time and the technical practices – TDD, pairing, BDD, Continuous Integration, Continuous Deployment – those are ways of shorting that cycle and all of errors tend to be made with a locality. It is all close to home. If something happens in your team, it is like it never happened. It’s escaped defects that scare you. These different practices allow you to prevent the escape of defects and allows you to make more mistakes so that you can learn faster, in a safe way. Without the technical practices, I think it would be impossible to deliver every two weeks, continually, without creating crushing technical debt and just not realizing the low quality that you have created. Sutherland said that all the Scrum teams that have reached hyper-productivity have done so through the XP practices. I think that is still true.

13. I agree with you on the cycle of defects. If I look back, this is something that we have known for many years already. Barry Boehm has published research on this showing how shortening the cycle can dramatically reduce the cost of defects in there. So, what you are saying is that XP can even make it go better than all the practice in there?

I would say yes. In the cycle, a mistake I make before I type it, when I have already corrected it in my head before I type, that is infinitesimally cheap. There is no cost at all. So what we really want to do is to train our brains and our team’s brains so that we just don’t make those kinds of mistakes. But to do that, we have to see what all the mistakes look like. We have to have seen a lot of errors. The second best is as I am typing, I notice the mistake and I back up. Well, that is great for working by myself, but that requires a lot of attention. It turns out that pair programming, among other things, will catch the logical errors and the errors that I am typing, as I am typing them.

That is a very, very short period. Those have never existed and not cost me anything. Then I get to the unit test. Now, my partner and I have made a mistake that our code works, but it breaks something else. We know immediately, we understand the breakage. Now we have another thing in our head. We know how not to make this breakage again. That becomes infinitesimally cheap* forever. This is a training system. XP is way that we learn to make stuff better. That is unique and it reduces cost tremendously.[*correction applied in the transcript per author's note]

14. Do you have some recent examples of teams that have successfully improved their technical skills and practices? How did they approach that?

I have teams that did not believe they had permission to do so and so they have always worked in silos and independently and they were not aware of integration problems because they had never integrated. In a few weeks of coaching, they had gotten to a point where they were integrating constantly, they found errors way before going out the door. We had one team who had not been delivering for months because their build didn’t work, but they did not feel like they had permission to fix their build. With a little bit of coaching, on the management and team side, they fixed their build and now they are releasing to QA, I think, every day.

First getting things integrated is key and then, as far as getting to the TDD side of it, I have a number of teams now that are doing tests now first development. They found that there’s no mistakes getting out. My favorite is: James Grenning had a team whose release was being delayed. So, here is the reason: the company has a long history in software and they know that for every so many thousand lines of code, how many bugs to expect. James’s team was working on an embedded product and they handed it over to the QA group and it sat for months. They could not get it out the door. The problem was they could not find the rest of the bugs. So, they worked for months but they could not get the bug count high enough to satisfy the requirements that they have so many bugs per thousand lines of code. Yeah, those are recent examples. It makes a huge difference and it is such a large difference that the organizations are not prepared to deal with it. I would love to be so successful that my company does not know what to do with me.

15. Are there any new developments in the “Taking back Agile” initiative, ongoing or coming up?

We do not have much planned right now. It is more of a conversation than a movement at this point. We are re-opening the forums again and we have gotten a lot of attention now through Twitter that I think that we are ready to start moving again. Ruud and I have been discussing a video series and you will probably see something from that in the near future. But, as if right now, there is not a lot of development there, but a lot of enthusiasm. I think that the phrase has gotten the attention of many people. By the way – the phrase is not mine; Tim Gifford had spoken this, that maybe we should take Agile back after seeing some bad attempts and Curtis Cooley picked it up from him and brought it into our company, at Industrial Logic.

When four or five of us heard it, it lit our imagination and we have not been able to let it go now for two years and you see this talk here. We are looking for people to join up with us. We are looking for other people who would like to see Agile return to its original state where we are productive and we are lightweight. It is going to take a lot of people to start to talk about this and start sharing the materials. I think we can do it. I think that we can create a social campaign around it. I think that we can be teaching through conferences and through user groups, but we are going to have to overcome the fact that Agile has kind of been taken away from us by stale, dead, 2-day wonder type implementations. People need to know that is not all there is.

16. If people want to get involved in this, where can they go?

“Taking back Agile” is now a Google group. Also, they can contact me – I am Tim Ottinger at Industriallogic.com or Ruud Wijnands at – oh, no! Is Ruud Wijnands at Phillips.com now?

Ben: I think it is.

OK. So, both of us. Our conference talk will be up with our e-mail addresses, but the Google group is a good place to start. You don’t need us. That is the nicest part! You don’t need us at all. You can take back Agile wherever you go, just dig into the materials, find the original principles and get things done.

Ben: Thank you very much for this interview, Tim.

Thank you, Ben.