Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Interviews ThoughtWorks’ Jez Humble Delivers on Continuous Delivery

ThoughtWorks’ Jez Humble Delivers on Continuous Delivery


1. Jez, what brings you to Agile 2010?

I work for ThoughtWorks Studios which is the product wing of ThoughtWorks. We build products to help organizations be more productive. We have a project management tool, a build and release management platform which I am the product manager for called Go, and also an automated testing tool. I also have another hat, which is consulting around build and release management stuff and I just finished writing a book on that called Continuous Delivery. That being finished I now decided to go and talk a bunch about it and me and Martin Fowler just did a presentation, a tutorial yesterday on the stuff that’s in that book.


3. And it’s worth reading.

It’s an excellent volume.


4. Could you summarize it and give me a five minute recap of what we will see in that book?

Absolutely. I guess the key point to the book is it’s about going from dev complete to live, so a lot of the problems that we see in the organizations. This book comes from years of experience, not only of me but also of my colleagues at ThoughtWorks. We were going to organizations and finding that they were pretty good at developing software, but actually going from dev complete to live was very painful. It often took weeks or even months to get the stuff live. Stuff gets thrown over the wall to the testing organization and they find that the software is not fit for the purpose and then you try and deploy it and you find there are all kinds of problems actually getting the software deployed.

One of the very first gigs I worked on software was written on Windows boxes and then it was deployed into a Solaris cluster. The first time we tried to do that, deploy into a Solaris, it took about a week and we found that there were some serious problems with the architecture side. People were caching stuff by writing to disk, which works fine if you are on a laptop. If you are on a cluster you write stuff to disk on one node, you try to read it on another node and guess what’s not there? We had to make some changes to the way the application worked. Problems like this are very typical and they can cause serious delays to the process of getting stuff live.

What the book is about is trying to make sure that doesn’t happen and what we say is your software should always be production ready right from the beginning of the project, all the way through to the end. It should not be a phase where you do integration and then do testing and then try to deploy into production-like environment. You should be deploying to a production-like environment from day one and you should be running automated functional tests and automated capacity tests and all the other kinds of things you want to do right from day one. That’s one part of it and that involves automation and things like building automation of testing, automation of the deployment process.

The other part of it is around collaboration. One of the typical problems you see is that stuff gets thrown over the wall to testers and the testers go like "This is rubbish, this doesn’t work!" and they point the finger to the developers. Then you try to deploy to production and the operations people say "The developers have thrown some undeployable code over the wall to us. We can’t possibly deploy this. These guys know nothing! It’s going to destabilize our environment." Then the developers go like "The operations people are stopping us from getting our software live and they are so conservative!"

What we want to try and do is to change that and there is a movement that sprang out called the DevOps Movement which is about taking the Agile principles and applying them to operations and also helping to bring operations people and developers closer together. I think that’s big part of what we talk about in the book: trying to help everyone involved in delivering software - developers, testers, operations people get together right from the beginning of projects and work together all the way through. The operation people’s concerns around things like monitorability, logging, traceability, performance - all those requirements are considered right from the beginning of the project rather than being afterthoughts.

Automation and collaboration already are at the core of what we’re trying to do. In order to talk about software always being production-ready we have to talk about a bunch of other things as well, like how you use version control, avoiding branching which we think is evil, which I guess is my controversy sort of thing. Trying to make sure that people who check in are mainline and understanding how to develop in an incremental way, so you don’t have to create branches. Talking about things like base-to-base migration, how you do incremental base-to-base development and also releasing software, doing things like continuous deployments are covered, zero downside deployments and techniques for deploying to clouds and clustered environments.

All this stuff is covered in the book and continuous integration and testing strategy, so a pretty wide ranging book. The idea is that you have to take a holistic approach. If you focus just on one part of it you are not going to get the benefits that you want. You have to look at the delivery process as a whole and try and find where the bottlenecks are in that process and then attack them one by one. We kind of borrow from the Lean movement and the idea of value stream mapping - it is a way to get the organization from where it is now to gradually continuously improve and become more efficient delivering software. The value of it is getting software to the market fast, getting ideas out into the hands of the users fast so you can get feedback fast, so that you can respond to what your users want.


5. Correct me if I’m overstating things or misstating things, but I seem to recall listening to Kent Beck back in 2001 saying that this is exactly what XP does: it takes the whole thing, which means customer to production, including the testers and everybody else, and every two weeks delivering production ready software. Where did we go wrong that we aren’t doing that yet?

Absolutely. Martin Fowler wrote the foreword to the book and the very first thing he writes is "I remember visiting Kent Beck in 1990-something. When he was doing this, he was deploying every week (maybe even every day, I think) to production." People have been doing this for ages and, absolutely, XP talks about this and this stuff is in XP. I guess the reason we’re talking about this all over again is that that stuff is going back in the days when Agile was practiced on small teams. This book is written with medium and large organizations in mind, so when it was written lot of the XP stuff, the idea was "You don’t want a team bigger than 10 people."

I think what this book comes out of is that as ThoughtWorks we go into organizations which have hundreds of people working on projects and you just need a different set of techniques in order to actually get continuously deployable, production ready software when you are working on larger projects. There are more tools that you have to have in your toolkit in order to do that. So it really comes out of our experience in actually scaling stuff into larger and larger organizations. I think one thing in particular is Scrum has taken off in a really big way over the last 10 years. One of the problems with Scrum is that people think it’s a panacea.

Scrum focuses on the process of managing requirements and (I’m going to get into trouble for saying this) Scrum says you need engineering techniques. Scrum doesn’t really talk about those engineering techniques. A lot of organizations have said "We’ve adopted Scrum, we’ve gone Agile, but things aren’t working out and we still have problems with getting our software live and we are not getting the expected efficiency." You need the engineering techniques as well and Scrum says that you need the engineering techniques, but it’s not really part of the remit of Scrum, which is something we are trying to fix at the moment.

But XP has that stuff in it. I think what we’ve done with continuous delivery is said "Here are the core bits out of XP that we think you really need to do this stuff, which is automated testing, continuous integration, good configuration management and applied it in particular to the world of operations. XP doesn’t really talk about operations and again the DevOps things that we are trying to talk about the needs of operations people and infrastructure management, that’s the big part. It’s a very roundabout way of applying stuff to medium and large organizations.

Also there is a bunch of new techniques that come out of things like automated provisioning and production environment tools like Puppet and Chef and things like ongoing automated management of infrastructure, even huge clusters of servers, large numbers of machines, techniques for dealing with the automated provisioning and managing of those environments are the things that we talk about.


6. What’s the relationship between continuous delivery and the concept of flow that we’re seeing?

It’s very strong. We start off the book on the very first page with a quote from Mary and Tom Poppendieck where they say in their very first book (Lean Software Development I think it was called) "How long does it take you to deploy a change that involves a single line of code to production? Can you do this repeatedly and reliably?" That is a really great definition of cycle time and that is really the staring and the ending point of the book. How can you get the organization cycle time down as fast as possible? Make that delivery process as efficient as possible so you can get stuff out of the door as fast as possible. That’s a recurring theme throughout the book. All the techniques in the book are dedicated to making that cycle time as short as possible.

But I would say you have a value stream, when you go from concept to cache, what we talk about is the part of that value stream that goes from check into release. We completely ignore the part of the value stream that goes from having an idea, analysis, going to the stage of having stories written. We start from checking and go through to release and that’s the part of the flow. The process of flow doesn’t have a lot of literature on it, frankly.


7. One of the more interesting concepts that I see in both the flow notion and the "continuous" word in your book title is the notion that somehow or another we can get away from the very episodic way that software is developed. We have a project and this project is 18 months long and we may flow like crazy within that boundary, but we still stop and then we start a new project. Somehow it seems the whole organization should be in this kind of flow state, there shouldn’t be these cycles of projects.

I think that’s absolutely right. The way we talk about that in the book is the idea of "done," so when you’re done. You’re not done when you’re dev complete; you’re not done when it’s tested. You’re done when what you’re doing is delivered to the users and you should be delivering stuff to users all the time. That should be a continuous process of getting stuff into the hands of the users, getting feedback - "Is that stuff useful?" and then reacting to that feedback straight away. One of the interesting things that organizations like Flickr and Netflix and Facebook are doing is they don’t wait for weeks or months to deliver new features.

They start incrementally delivering features into production. They might not make them available to users straight away, but they are always delivering features to production and when they actually do turn on a feature, they monitor immediately what’s the usage pattern - are people actually using this stuff? Do they like it? Organizations like Netflix they want to make revenue, they want to make more money out of this. That should be a continuous process. It shouldn’t be "What features are we going to deliver in the next version in 9 months?" It should be continuously delivering features to users and getting feedback all the time. You’re right: it’s shouldn’t be episodic, it should be everyone involved into delivery working together to get stuff out there on continuous basis.


8. One of the impediments that seems to be is the size and the scale are a defined deliverable. Instead of delivering one story, our ultimate goal is to deliver X with 400 features or 400 stories. How do we change that mindset? Do we need to?

It’s a deep-rooted psychological thing, I think, for anyone who’s worked in the software for any period of time. That’s just how things are done. It’s a psychological thing, a mental thing. You have to change the way you think about stuff and sometimes it’s not the right thing to do. The obvious example is embedded, it’s just too expensive to keep delivering new versions of hardware to the users and it often doesn’t make sense. Economically it might make not sense to deliver more than once a year or even less frequently than that. But what you can still do is when you are developing software, you can still make sure that your software is always production-ready.

You can always be deploying software inside your embedded environment and running automated tests to see whether the change you made worked. I own a synthesizer, a Virus synthesizer which is a really great piece of hardware and they are always putting out new updates for that. They change the reverb engine so they have more powerful reverb. They do some really cool things around oscillators. Even after they brought the hardware out, they were pushing out new OS updates where you could get new features in your synthesizer. That’s one example. If you are using a continuous delivery technique, you can be constantly pushing stuff in and then testing stuff and playing around and doing new things.

I have no idea if they were using the techniques in the book, but it’s a great example of where you think it just doesn’t make sense to use continuous delivery. Actually, it makes a lot more sense than people think even when you think "We can’t possibly do this." The other common objection is around support and this is a more difficult one to get around. As an organization delivering product you might not want to support more than two or three versions at any one time, so you might not want to do full releases, but you can still do early access programs. Eclipse has been doing daily builds forever pretty much. You can still use the concept of continuous delivery even where there are other kinds of constraints around your delivery process.


9. You said you worked on the tools side at ThoughtWorks. During today’s keynote we had a rather disparaging comment made on tools in general and the waving of the old 3x5 card, which is the original philosophy of Agile. What role do tools play in Agile and what should they be?

Tools are my silver bullet and I’d be the first person to admit that an edict from on high saying that we are going to use this tool is in no way sufficient to change an organization to be more Agile. However, tools can be useful. Going to the tool I work on, within three months we’ve been self hosting on that tool, and we had other teams within ThoughtWorks using that tool. And all the stuff we’ve done on that tool, a lot of the information has come from feedback from those people. You have to constantly validate the work you are doing is useful, that’s really the key to it. There is some really great stuff that comes out of that. I can’t remember what Dave Thomas’s name for it was; he was talking about doing prototyping and showing people work-in-process type of stuff.

This whole product development methodology that’s been developing over the last few years where rather than building something for six months and then showing it, you build little prototypes and get users to try them out, play with them and experiment with real working software. I think that is how good tools develop. There is no argument that tools can’t be useful, they can be. The mistake people make is that they are putting a tool in place that has insufficient condition for organization transformation and that’s clearly wrong.


10. One of the other complaints about Agile is notably by those that are in the software craftsmanship movement. They claim that we’ve got away from our roots, which is the skilled professional craftsman in favor of Scrum management and to tools that allow senior managers, business managers to micro-manage developers and put them right back into the commodity market again. They are splitting out from Agile trying to do their own thing again. Do you have viewpoints on that?

I’m philosophically opposed to that. It’s been my own personal experience as well because I started off as a systems administrator and developer before I became product manager and consultant. My experience has been that you get the best quality and the most innovative thinking when you trust your team to deliver stuff they think it’s cool. One of the things that we do at ThoughtWorks Studios and other organizations do as well is give our developers slack time to work whatever they want and come up with new ideas. That stuff gets fed back into the product roadmap and I think it’s just basic psychology, research has been done. If you tell people what to do and treat them like fungible, billable units you do not get good quality stuff out the other end.

You get good quality stuff and you get people who work hard when you empower them to be creative about what they are doing. This research has been done in psychology and people ignore it to their peril, so it’s just ignorance in my opinion that that kind of technique will work.


11. You have recently spent some significant time offshore in India and China. Would you speak a little bit about cultural differences and the way the Agile is? I’m assuming that you’re doing Agile kinds of things because ThoughtWorks is all about Agile, but can you give us some insights into the cross-cultural differences in Agility?

I can talk about my own experiences. I went to India and worked on an offshore Agile product in India that we did for a UK client. It was certainly very instructive to be on the other end. One of the great things about my experiences in India and China was that I met just a bunch of really innovative and interesting people working out there and I had a really great time. It’s very tough - the main problem is just communication. That’s not a cultural problem, that’s just a problem around it. There are people at the client who have expectations and there are people delivering software and if they don’t talk face-to-face, isn’t very easy to lose trust.

That’s the first thing that goes when you offshore something and people aren’t meeting face-to-face is something goes wrong and you have an entirely reasonable excuse why you weren’t able to do such and such or it didn’t work, but if you are not able to communicate that because the person who is the customer is not in front of you. You can’t have a discussion and trust immediately disappears and people go "These stupid guys in India, they don’t know what they are doing." So one of the things that we did is we ended up flying people backwards and forwards and have the customers fly to India.

They spent time with us, we went out with them and have a good time and that made a huge difference to our ability to come to a common understanding of what we wanted to do. You are always having negotiations - "We can’t do this. This is going to be more expensive and we thought what we would do instead. We want this new feature and we’re going to cut…" That is the lifeblood of the Agile process. And when you are distributed, it’s very difficult to get the levels of trust you need to do that. That for me was the overriding problem rather than any kind of specific cultural dilemmas. There are definitely differences; the common thing is that Asian cultures tend to be more command and control.

You find less argument in China; people don’t really challenge you so much as you would be in the UK. In India it’s the other way around. One of the things I love about going to India is that we have all kinds of very spirited discussions between everyone on the team on what are we going to do and the people in India that I met were very creative and they are always coming up with new ideas and we’d have big fights over stuff. That was really great! There are definitely cultural differences like that, but they don’t prevent anyone being able to deliver great software. There are just different ways of working that you have to be sensitive to, I think.


12. Are there any burning issues or topics that I didn’t manage to drag out of you?

We could go on like this all day. We covered flow, we covered DevOps stuff, automated testing I guess is one thing that I think it’s going to be an interesting thing to watch out for. The last year we’ve seen lots of controversy over whether automated functional testing is a good thing or a bad thing. In the book we come out very strongly in favor of automated functional testing, end-to-end functional testing. There are people like J.B. Rainsberger and Jim Shaw who think this is a bad idea. J.B. Rainsberger famously did this article that automated tests are a scam. That’s the cost-benefit argument essentially.

We’ll be writing a book, me and Dave, that there are ways to reduce the cost of automated functional testing. That’s one of the things that I’m interested in pursuing and I think Martin and I are going to be doing some work on this. It’s like real life techniques and practices to make automated functional testing lower cost and more maintainable so that people can do it more effectively. Agile is still evolving very rapidly and new stuff is coming along all the time. People are learning things and taking them out there. The other thing that I should say is I recently got to do a keynote at Agile Japan. They’ve only been doing Agile conferences in Japan for two years. They can make Waterfall work pretty well for them, so Agile is new in Japan and all these discussions are going to have to happen again over there.

And we’re going to see new stuff coming out of there. I think it’s a really exciting time to be working in software right now and I look forward to lots more evolution over the next two years.

Dec 11, 2010