James Grenning is the founder of Wingman Software, author of TDD for Embedded C, signatory of the Agile Manifesto and inventor of Planning Poker (which he now advises people not to use). He delivers training and coaching in technical practices around the world. At the recent Agile Singapore conference he presented two technically focused talks - one on the importance of technical excellence and the other a hands-on workshop teaching the skills of test-driven development. He spoke to InfoQ about the importance of strong technical practices to enable true agility in software development.
InfoQ: James, thank you for taking the time to talk to InfoQ today. It's really great to see you. You've come to the Agile Singapore Conference and you're talking on Technical Excellence. What's the extent of technical excellence in software development today?
James: In my travels, I visit a lot of companies and I find people programming the way I learned in 1979, something that I call "Debug-later Programming". You write some code and you figure out what's wrong with it later. You usually find some of the problems but not all of them and it leads to big problems in delivery and later. A lot has changed since 1979; I discovered in 1999 Extreme Programming and it changed my world from a reactive life of debug-later programming to a proactive life of test-driven development where I can prevent defects. Here in Singapore I was invited to give this talk and I had some specific messages for the attendees: for managers I wanted them to understand that defect are the result of the way we work; it causes the problems they are experiencing: defects being found late and in the field. I wanted developers to understand why these practices are important. They're not important because they're written down on a book. They're important because they solve specific problems. As an Engineer I'm interested in solving those problems and helping other people solve them.
InfoQ: We've known about these practices since at least 1999 and before. Why are they not just the way we do work now?
James: I don't know the answer to that but I've got some suspicions. Work life is very demanding when you work for a big company and there's a lot of ideas out there fighting for your attention. It's hard to know what the good ones are. People know what they know and they don't necessarily know what they don't know. And so if you hear about something like Test-Driven Development you form a vision of what that is in your mind. It sounds strange, people form opinions with little information. They don't know what to do with it so you keep doing what you know. The pressure is on, so the change that you might want to do is hard to find time and support for. The organization makes it hard for you to change and you don't really understand what it is and why it would be important. Most people are doing the best that they know but they don't know what they don't know.
InfoQ: How come we've done such a poor job of getting this into the educational programs for programmers? The university system, education system, organizational training systems because these good technical practices can actually saving organizations lots of money.
James: Yes. That's my experience and belief as well, that the practices of extreme programming save people money. First off it appears that they cost more because people are concerned about this lines of code that need to be written. When I teach people how to start doing test driven development the first simple exercise results in writing a hundred lines of code and 150 lines of test code. We spent less time than it would take someone to write the production code and get it to work through manual testing but they don't think about that. They think about how many lines of code were written and it must mean that is going to take us way longer.
InfoQ: But surely we’re beyond lines of code as a metric. Surely the work is the creative output of the thinking process?
James: About lines of code, people still see it that way, until they experience it. One thing I can't do for someone is experience how much better they will be in several weeks or months if they learn this. I can't make them experience that because these approaches have to be voluntary. For someone to get the benefit from TDD, they need to see that is might solve a problem they want to solve, try it, they are likely to convince themselves through their own experiences.
Of the experienced people that started learning this years ago, most of them wouldn't stop and it's not because they want to be slower, less effective developers. It's because they want to be more effective. Though you've got to take the time to learn. That often means picking the right time to start.
InfoQ: Agile has been around since you and others created the Manifesto in 2001. One of the underlying principles is Technical Excellence. How effective has the Agile community been at getting that message through?
James: It's interesting. On the tenth anniversary of the Agile Manifesto Alistair Cockburn held another meeting and he invited the original people involved in manifesto as well as other influencers in the community. It was in Utah. So I am in. The reason I was in the Manifesto to start with is Bob Martin invited me, it was in Utah and ski season. This was an easy choice for me. Obviously, I'm going to go to Utah to go skiing. The second meeting was a retrospective on Agile and again it was in the right place and hanging out with cool people again.
The retrospective on the state of Agile resulted in four bullet points, surprise. Unlike the original meeting, four bullet points was an objective of the tenth anniversary meeting. The top two were very important to me. The first bullet point was that we need to demand technical excellence. The second bullet point was we have to help foster change. The other two I don't remember. The first two surfaced on many fronts in the workshop. It was pretty clear that the attendees think the industry has adopted the business practices of Agile, but not the more difficult technical practice of Agile.
The genesis of this talk came from an invitation to do a keynote at the London Scrum gathering a few years ago. When the organizers invited me I said "Well, I'm not really part of Scrum. Why do you want me to talk there?" "Because you're not part of Scrum." they answered. I asked who would be the audience. Scrum masters mostly.
It has been a concern of mine that Scrum seems to be practiced the by the book. The continuous improvement intended by the creators of Scrum was not really happening across the industry. I asked, "Can I talk about this things that the Scrum community is not doing?" They said "Yes, that would be great.", so began this talk. I probably should write something about this too. I've been talking about it now at a few conferences, it’s a fun story for me to tell with about a million sides.
InfoQ: So what's wrong with the Scrum and other Agile methods that we're just not getting? What's missing?
James: I think there's something going in which is people go to the two-day Scrum Master class. Now they're certified and they go and practice just what was taught. When they're challenged they say, "Well, the books says to do it like this," and people get used to saying, "Well, the book says to do it like this," and they miss the part which is about, "Now look at your work and decide what's broken and try to fix it."
Having been in the first-ever Scrum training class, given publicly by Ken Schwaber in 2002 at Object Mentor in the Chicago suburbs, the original materials had the Extreme Programming practices. I discovered later they were removed. I think Ken’s expectations were that people would go find XP or other technical solutions as part of the retrospectives of Scrum. "We didn't want to prescribe that as part of the Scrum," is what I remember Ken saying. (I could have this wrong). What I see is people aren't doing that part of Scrum. Scrum, at its code, is intended to be a continuous improvement cycle.
In the 80's there was TQM, Total Quality Management, where we learned the PDCA cycle, Plan Do, Check, Act. And what I feel is happening in Scrum in many implementations is not the whole cycle, just the "Act" part where they're doing the easy part of Scrum and programming the same old way, debug-later programming. (See a comparison of debug-later programming and TDD here)
InfoQ: Evaluating the process; because certainly if one looks at the Scrum framework the retrospective is an absolutely imperative part. I will say in my experience it's horrifying the number of teams for whom the retrospective is groundhog day, they say the same thing again and again but nothing changes.
James: Right. If there's numerous things that are wrong and they all seem out of the team’s control, it's hard to make any progress. Fortunately, there's solutions to many of the problems that the Scrum teams continue to seek. For instance, not being able to deliver software every iteration. (I don't call them sprints; I think it's the wrong metaphor.)
Failure to be able to deliver every iteration, it is failure of defect prevention. If you don't have automated tests, the cost of retest will be high, making releasable product increments very risky. cost of retest must be low to be able to do effective releases every two weeks. You must have The the cost of retest needs to be close to zero. Test-Driven Development and Acceptance Test-Driven Development are solutions to that problem. It turns out they're hard to learn, which is very nice to me because it's a fun business. But they're hard to learn and it requires discipline and mistakes until you figure it out and get addicted to this improved way to program.
InfoQ: Why is it so hard to learn? The people were talking about here are technical, they're programmers, they're natural problem solvers. They want to do this.
James: I'm no cognition expert or anything, but the way that people learn to program is a very unique thing. I know for me it was -- here's how you set a variable. Here's an "if statement". Here's a "for loop". Now go figure out how to write an operating system or how to simulate Newton’s methods for calculating the area of a curve. Go figure it out.
No one was really taught to program. Everyone's programming is a very personal thing, although most people generally do what I call Debug-later Programming. Write the code and figure out what's wrong with it later. With the entrance of Test-Driven Development, you're turning the whole programming upside down. You one behavior at a time, thinking about what you want the code to do. Express one small code behavior in a test, one test at a time. Then you evolve the code to do just that behavior. Then choose the next behavior, express that in a test. I can work in a tight feedback loop that catches my mistakes so that I can prevent many defects. I also produce a regression test suite as a side effect, virtually for free.
Bob Martin compares TDD to double entry accounting. There’s always one credit, and one debit. They cancel each other out when the books are right. This is a very specific technique for programming but very foreign because it's not what people know. People will generally look with suspicion at things that they don't know. "Oh. That's too weird. No, I don't do it that way."
What I try to do to get some buy-in though is before I even start with saying anything about TDD, I ask the attendees of my training to tell me what they like about development; tell me what they don't like about development; and to tell me why they're at the training class. I get some interesting feedback. I started to put these on my website.
The activities engineers put in the ‘Likes’ category are usually "problem-solving", "I like to code", "I like to clean code", "I like to design", "I like to solve a problem for a customer" I like to see my work be used to solve a problem in the real world." This is what engineering is all about. I can confirm that I have the right people in this room for my training.
Engineers tell me "I don't like testing." I say, "Now, wait a second. Remember the class is Test Driven Development. I'm going to go check to see what you answered about why you're here. I bet your boss made you come."
If you're interested to see some of the answers for yourself, you can find recent replies here: Likes and Dislikes . So I can I have a little fun with them and find out what motivates them as programmers.
I don't like testing either, traditional testing is boring and repetitive. Its unfortunate Test-Driven Development has the word test in it because it's not the focus. Other things they don't like are bad code, unrealistic expectations, many of the things Agile and TDD are supposed to help with.
I use this to get buy-in to try TDD during the training. I’ll ask "What if I by the end of this training class you could do more of what you like and less of what you don't like?" Let’s reduce boring manual tests and trade them for something fun - problem solving. The problem to solve is the challenge of writing the test so that I don't have to do boring manual tests afterwards. I've found this to help open minds to the possibility that there is a better way.
I find that more people try to learn TDD when they see the possibility that their lives could be better. I had this weird idea in the beginning in 1999 about people adopting the engineering practices of Extreme Programming. The people I worked with at Bob Martin’s company were convinced XP was loaded with good techniques and ideas. We thought these ideas are so good that we can just tell them and people will want to do them. Unfortunately it's not the case.
I mean they are better but telling someone about a better idea does not make them want to do it. For wider adoption people have to want to avoid the problems like Debug-Later Programming, and degraded designs. They have to accept there are problems, then see that technical practices of extreme programming may be part of the solution. Then they have to prove it to themselves.
About the Interviewee
James Grenning, founder of Wingman Software, trains, coaches and consults worldwide. With more than thirty years of software development experience, both technical and managerial, James brings a wealth of knowledge, skill, and creativity to software development teams and their management. As his professional roots are in embedded software, he is leading the way to introduce Agile development practices to that challenging world. See James' articles for applying Agile to embedded software development. Areas of interest are software process improvement, Object Oriented Design, programming, embedded systems, project management, Extreme Programming, Test Driven Development, test automation and Agile software development. James knows his way around Scrum, with Scrum Master and Product Owner certifications.