Bio Jez Humble is the product manager for the Go product at ThoughtWorks Studios and is co-author of “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation.” Martin Fowler an author, speaker, and pundit on the topic of software development. He is chief scientist at ThoughtWorks and has written six books on software development.
QCon is a conference that is organized by the community, for the community.The result is a high quality conference experience where a tremendous amount of attention and investment has gone into having the best content on the most important topics presented by the leaders in our community.QCon is designed with the technical depth and enterprise focus of interest to technical team leads, architects, and project managers.
Jez Humble: All right. My name is Jez Humble, I work for ThoughtWorks Studios where I am product manager for Go amongst other things. I also recently wrote a book called "Continuous Delivery" which was released this year.
Martin Fowler: And I am Martin Fowler, A loudmouth, pundit on software development topics, writer of too many books, and fortunately managed to nab Jez’s book into my series.
2. A couple of years ago the "ThoughtWorks Anthology" book came up and it mentioned a lot about the last mile. Can you please explain to us what was the idea behind the last mile and what is the deployment pipeline?
Martin Fowler: The way I look at it is that the Agile movement has done a great deal to encourage the practice of continuous integration of the idea that all of the people working on a software project should integrate their work frequently so that you don’t get long periods where different people are making different assumptions and therefore leads to integration problems. However, most of this integration work has been just within the development organization itself. There is still the journey of getting software from within the development group to actually deployed in production and running. And this can be more or less complicated often depending on the size of the organizations.
We deal a lot with big enterprises, and they often have very complicated procedures around testing and deployment to get stuff into production. But even small organizations can take a long time to do this. So the last mile is really that distance, the distance between when we have integrated software that works in the lab that we are pretty confident and does its job, and actually getting the thing into production and live.
Jez Humble: And the deployment pipeline effectively is an automated implementation of your build-deploy-test-release process, so I you take a part of the value stream of your delivery process from check-in to release, the deployment pipeline is the system that automatically takes each commit and then makes a build and then progresses that build through the various processes of testing and staging and then into production.
3. What is the size of a project in a team that starts mixing, automating every step of the deployment pipeline? Is it that small one week project which benefits the most from that procedure or those ideas? Or an entire two years product that’s going to be delivered and maintained forever? Which ones benefit the most and how do they benefit from it?
Jez Humble: I think that one of the interesting things about this pattern is that it is a very widely applicable pattern. So obviously our background is in enterprise organizations and it’s extremely beneficial there because the delivery process where you have large numbers of teams and distributed teams, normally that delivery process is very painful and still quite "Waterfall-y" especially going from dev complete to testing to prod. And there it makes a huge difference not only to the ability to release software rapidly and reliably with little downtime, but also to the development process. You get much faster feedback between the testers and the operations people and the developers with new builds going through much faster.
But, the interesting thing is that a bunch of people working on web start-ups in Silicon Valley and other areas have been doing the same thing for some time in much smaller organizations. One of the great things for me was when Timothy Fitz’s continuous deployment article came out and the Lean Startup people Eric Ries, those guys are talking about the same practices in small organizations, so that was great because it proves that this pattern can be used everywhere.
Martin Fowler: Yes, I mean the way I look at it size doesn’t matter when it comes to this thing. I mean the point is: do you want to go through a manual process where you’ve got to remember what to do and make sure you do everything in the right order or do you want to press one button or issue a single command line command and get the computer to do it all predictably and without errors because that’s what computers are good for? I have only started in this business as a sys admin 20-odd years ago and I automated the hell out of everything. I came from the Unix world where if you ever had to type two commands in a row you would make a shell script and that way you only had to type one command and it’s really the same basic principle.
4. But there are probably some pitfalls some common failures that people are falling through when trying to implement for the first time a deployment pipeline trying to deliver continuously and you have been talking about a few anti-patterns lately. Can you mention some of those? Some of what happens with companies?
Jez Humble: Sure I think actually the biggest problem is usually a cultural problem trying to get development organizations and testing organizations and operations people to work together, because in many ways that’s what the deployment pipeline is. It’s a system that connects all the different parts of the organization that need to work together to deliver software. So one of the common anti-patterns is either people saying from above: "We are going to implement this." The same way that happens with Agile, right? Some manager says, "We are going to implement Agile now, go and implement Agile." If you try and do that with continuous delivery you’ll see the same problems where people will either try and stop what they are doing and then try to automate everything which is a terrible idea.
So one good implementation pattern for this is that you should do it incrementally: find where the pain is, fix that problem, find out where the next bottleneck is, and fix that problem. It’s an incremental thing that you should do piece by piece, rather than trying to down tools and fix everything at once. The other thing is just about trying to get all those different groups to collaborate more effectively. So when developers come along and say "We are going to automate the delivery process and we are going to automate deployment to production" operations people are normally really terrified and go "Wait, what are you talking about? That’s our job."
So I think one of the things that it’s important to do in terms of implementation patterns is to get everyone together and collaborate in the small, find out what small improvements you can make - how operations can make life easier for developers and how developers can make life easier for operations by creating software that is deployable and involving the operations team in creating requirements for the software, and in creating the deployment systems.Because operations teams have requirements of software the same way that the business does: that it should be deployable, it should be manageable, it should have good logging, it should have ways to archive the data, all these are requirements of the software and they should be heard from the operations team right from the beginning.
5. A product which deals with multi-tendency for example and delivers different tagged versions in different servers and you have to deal with branches, tags and whatever many times during the same day or week: how can we deliver continuously using that without going to a feature branch hell - like branching it for every feature or trying to merge everything later and avoiding continuous integration? How can we still do continuous integration while dealing with those complex situations?
Martin Fowler: Well that’s something you will find most of my colleagues rant on about a great deal, trying to push away from this notion of feature branching and the idea that you should use a version control system to solve these kinds of problems. Because it is one of those things where if it worked it would actually be quite a good approach, but it doesn’t work. And you end up having very complicated merges and a great deal of stress involved in these merge operations. So we very much follow the principles of continuous integration, which is not something you can do at the same time as feature branching. Continuous integration means everybody works together on the main line, I am hearing phrases about trunk based development and things of that line for this. But it really comes down to the fact that everybody is committing at least every day onto the main line.
That will remove both the problems with complicated merges and also the tendency that comes with feature branching that people are afraid to refactor common code. That of course introduces other issues, and one issue is the one you mentioned which is "How do we deal with features that last longer than your release cycle so you don’t want that half-built feature released either because you don’t want to do that or simply because the business doesn’t want that particular feature switched on yet?" The basic way to do this is to use a technique that we call feature toggles, you have also heard terms like feature bits and flaps and flippers and flags and lots of other F’s. But we use toggles as our terminology and it’s basically something built in a configuration file which you then use within the code to interrogate the information from that configuration file to decide whether to make certain features visible or not.
You usually have certain entry points in the interface, the user interface of the application and you wrap those with checks to the feature toggles. If things work out really nicely you might be able to have just one little link or menu item that you can hide or show, if it gets more messy which it does from time to time you may have a lot of them. But the point is that these are things that are determined on or off within the product and within the tool. If you do use this technique a very important thing that has to go with it is you must make sure that you retire the toggles once you are done. And this is the base technical technique that you use as an alternative to feature branching and a great plus of it is that it is switchable at runtime so as a result you can have the system on making change at runtime. You can put one version over here and another version over there, do thing like canary releasing, A/B testing and stuff of that kind.
Jez Humble: In terms of multi-tendency software I think what we’d recommend is I mean it is a very common pattern for people to release one version of the software to one group and then they create a branch for that and then that version of the product for that company lives on that branch and then it creates another branch for another company. That is the wrong thing to do. That’s just another application of the fact that branching by company is as bad as branching by feature. Every organization should work off the same trunk-based development process and you should ship versions of software to different people based on that. And use feature-based to turn features on and off to different people. Make that part of the configuration; don’t make it part of the source code branching strategy. And we do that with Go and all our other products and we know other companies that do the same thing. It just requires careful management of your relationship with your clients and your development process.
Jez Humble: Or deployment time - that works too.
Martin Fowler: And the point is then all of the code is being integrated together and so you know it all works together. I mean the branching thing would be fine, if you never had to merge. But the fact that you have to merge means you have to make sure things are always working together. I should also point out as well I do advocate using feature toggles for features that take more than a release to implement, but it’s only the second best solution in that case. The best solution is to find a way to release the feature’s capabilities incrementally and usually you can do that and people need encouragement to go down that path. And only when you can’t and absolutely can’t should you use toggles.
Jez Humble: And that’s kind of an analysis problem as much as a development problem.
Jez Humble: Maybe there is a small part that we can do and then we can build on that. And that’s a good way to develop features anyway. Because if you develop a feature that takes two months to create and then at the end nobody wants it you have wasted two months worth of time. It’s like minimum viable products. If you can create a small part of that functionality and test it on real users, then you get feedback and then you can work out what you should do next. Because the more you work on a feature without showing it to someone the more likely it is that it will go off in some direction that actually people aren’t interested in.
Martin Fowler: And you are not getting value. I mean you only get any value in return from software by putting the thing into production. So you want to get as much out there in production as rapidly as you possibly can and that again encourages you to incrementally release these capabilities.
8. Delivery pipeline features are typically implemented to a build server configuration such as Cruise, Go, or any similar products. But still some of the open source products don’t provide support to that, you have to hack it somehow around and play with it in order to support it. So depending on server configuration means being vendor locked in so is there any way out of it? Can we do it just with basic simple scripts? As Ant, Maven or NAnt or whatever in order to support pipeline steps and phases and manual and automated movement and so on?
Jez Humble: Let me first declare that I am product manager for Go so you will get a somewhat subjective answer from me. The reason we created Go in the first place was because this pattern of deployment pipelines came from many projects that worked on at ThoughtWorks and so we have been hacking it using the open source tools, and it was just quite painful. So we created the product because we wanted a tool that would actually support the practices that we were using all the time on all our projects. One of the interesting things that happened is some other vendors are starting to copy us, so Bamboo is coming out in the next version with support for deployment pipelines. I have not seen any of the open source tools doing any of this stuff yet.
There is kind of stuff like this, there is a release plug-in for Hudson and Maven has a release plug-in. But you do have to do a lot of work and it’s more painful and that may be the right thing to do but one of the things that we always say is that "If you are paying people to develop software would you rather pay them to hack Hudson or would you rather buy a product that actually does the right thing?" And vendor lock-in is something that I am very sensitive to. I want people to buy Go because it is the best tool, not because they have been locked into it. So we have open support for pulling data in and out of Go and it stores artifacts just on the file system in plain format. So we try to make the system as open as possible so you don’t have to be locked into it, because that is how a product should work. It is possible to hack it. In our experience the open source stuff hasn’t caught up yet to make it really pleasant and straightforward and simple.
Some of the products you have to work quite hard to get them to do the right thing. Maven for example - you can make Maven behave in a good way but it certainly doesn’t do out of the box, out of the box it updates itself without telling you, people use snapshots and then you can easily repeat building a given snapshot. Then in the deployment pipeline you build once on commit time and then promote and it’s hard to do that with Maven. I hope that the Maven guys will find a way to do the right thing with Maven and that they’ll make it possible to use the deployment pipeline stuff in Maven. I would love the open source tools to support this paradigm because I think it’s very useful but they are not there yet.
Martin Fowler: It’s actually a very interesting source of internal tension in ThoughtWorks, because a large part of our business is on the professional services side and we tend to hire people who are very fiercely independent and they don’t want any sense of a corporate constraint on them, and some have had nasty experiences with other companies. I remember one of our guys talking about how he used to work with a Microsoft partner company and to recommend any competitive product or to even make any criticism of a Microsoft product was a firing offense. We don’t want that kind of world, so we have this tension with our tools group and our professional services work because a lot of people in professional services are almost extra-reluctant to recommend our own tools because they fear they wouldn’t be perceived as not independent and "In any case I want to make sure I am independent by extra asserting here."
I mean yes, you can take open source tools and hack stuff together, but the client shouldn’t be paying more to do that than they would be paying to get the tools that have been developed to do the job. So it is a very real source of tension for us internally and I know it’s a source of frustration on both sides, but in the end my feeling is still that it comes down as anything else: You look at how much cost it will actually cost you to do it yourself and that’s going to be people’s time to build and maintain and then you look at the tool and you compare the costs and you go with whichever is the better way to go.
Jez Humble: ThoughtWorks is actually our fiercest critic - our professional services organization. Our customers are not nearly as critical of us as other ThoughtWork-ers and that is a good thing. Other companies have a "Not invented here syndrome," ThoughtWorks has an "Invented here" syndrome, which is kind of entertaining, but it’s a good thing and a bad thing. But ultimately I would rather have the feedback. The other thing that I would like to emphasize is that continuous delivery is something that we want everyone to do. It is not something that we want to say ours is the only product that can do that. I would be really happy if the open source community would embrace this, and provide really great tools support for it, because it’s something that everyone should be able to do and I am committed to trying to help other people try to help, other people in ThoughtWorks as well, try to help the tools support for this grow.
Jez Humble: I think we kind of addressed that in a way talking about the cultural aspects because that is normally the biggest problem. And in a way the "Continuous Delivery" book and the talks that we do around it are actually great way of bringing different parts of the organization together, but the organizational challenges are definitely the biggest problem that we face and that is because those organizational problems go right up to the senior leadership in the organizations. You have these stovepipe functional areas where operations report right up to the VP or CXO level and the development side of the organization reports all the way up through a completely different chain to a different VP or a different CXO level person. And that really is the biggest obstacle I think.
Martin Fowler: It also ties in with the nature of software for a lot of organizations itself. I’ve argued and my colleague, Ross Pettit, has argued that there are two very different kinds of software that most organizations are dealing with: one which is utility software which doesn’t have any particular competitive advantage but is what you need in order to keep the enterprise running; and then strategic software which has to do with competitive advantage. There is much less strategic software than utility software; the strategic software is 10 percent less maybe of it but they have to be managed utterly differently and a lot of organizations struggle because they kind of take one kind of way of managing software and they are trying to apply it everywhere and it doesn’t quite apply everywhere.
And that’s the thing that we tend to run into a lot at ThoughtWorks because people bring us in to do strategic work not to do utility work as a result we particularly rub up against those kinds of organizations. And continuous delivery is valuable in both cases but the difference is it’s not that important for utility software because utility software isn’t that important anyway. But it really matters when it comes to strategic software where you need to be able to get cycle time down and get stuff out there and people using it.
10. In the end it’s only worth doing something if you can measure the impact of it in its costs or its costs and it impacts on your profit. What did you two think about it? Is this the next step on continuous integration, continuous delivery, and then some kind of continuous feedback, financial feedback or continuous feedback on information that we get from the product?
Guilherme's full question: Eric Ries at Business of Software 2010 doing his talk mentioned getting continuous feedback from every commit when developing a product. So seems like one step further from continuous delivery. Every release that gets into production will let us know what was the financial impact in your product from those commits. So I can use some measure function to see what was the impact, what is the next step or what is the next impact that will have on some changes and how that benefits or what is the problem with that in my company. It actually reminds me of some of the Lean values that says that in the end it’s only worth doing something if you can measure the impact of it in its costs or its costs and it impacts on your profit. What did you two think about it? Is this the next step on continuous integration, continuous delivery, and then some kind of continuous feedback, financial feedback or continuous feedback on information that we get from the product?
Jez Humble: I don’t think Eric Ries is talking about something different from continuous delivery, I actually think he is talking about exactly the same thing that continuous delivery is talking about. The value proposition of continuous delivery is three-fold really: one point about continuous delivery is exactly what Eric Ries is talking about that you get stuff out of the door as quickly as possible so that you can get feedback from your users. And that actually comes back to the first principle of the Agile Manifesto - I can’t remember exactly what it says.
Martin Fowler: You are looking at me as if I can remember what it says. It says something like "The highest priority is continuous delivery of valuable software".
Jez Humble: The reason why you want to deliver fast is so you can get feedback. And I think Eric Ries would agree with this because Eric Ries says the same kind of things that we are saying. One of the interesting things for me is that as I was finishing the book I came across Eric Ries’ work and he was saying the same kind of thing from a different perspective. So I violently agree with Eric Ries in pretty much every aspect and the value proposition of continuous delivery is exactly that you can get fast feedback from users and then pivot and make your product better and make more money, and delight your users. The other two things about continuous delivery in terms of the value proposition is firstly that it reduces the risk of release, and the third thing about it is that it’s the only real measure of project progress.
People talk about when being done with their software you are only done when you released so the shorter you get your release cycle the more you can actually get software done. There is never 80 percent done or 30 percent done; you are done when you have released. So, I think Eric Ries would agree with all we are saying and I think we are talking about exactly the same principles.
Martin Fowler: Except there is a kind of a sense of a distinction between continuous deployment and continuous delivery in the sense that continuous deployment implies "Well I am going to take everything that goes through the deployment pipeline and I am going to, pretty much automatically, put it into production." So, you are not typically going to put every single commit into production because not every commit makes it through the pipeline, simply because of the testing times tend to be greater than the cycle time that’s coming in at the beginning of the pipe. But you are typically in a situation making multiple deployments a day.
With continuous delivery, the slight difference is we say "Well, that’s perfectly OK and that’s great if your circumstances go with that and your business people are comfortable with that do it because you get all this rapid feedback and stuff that we talk about." However what we would say is that final step of "Do I make it live?" that’s a business decision, it should be no technical constraints tied to that, but it should be an entirely business decision. And some businesses prefer a slower rate of change. I mean the typical one that we tend to run into and we come into an organization is we take them from a case where they can deliver, maybe deploy into production maybe once or twice a year and it’s a high stress, weekend, overtime, sleepless night kind of exercise and we get them into a continuous delivery flow and they are happy with something like every couple of weeks, which is a huge step forward over once or twice a year.
But they are not comfortable with cycling the whole way and they feel every couple of weeks is a good cycle, but the point is that the decision about what that cycle into production is should be purely a business decision.
Jez Humble: So, for example with Go, we don’t want to release Go multiple times a day because we would have to support all those different versions of Go. We only want to support three or four different versions of Go a year. I mean we want to give our customers time to upgrade so they’re not forced into new versions. So, we only release to users three or four times a year. However, we still keep the software production ready at all times. And we release early access versions much more frequently every couple of weeks normally and we deploy to our internal servers every version of Go that passes the automated tests that automatically upgrades the version of Go that is used to build Go. We deploy it internally for ThoughtWorks users every week. So that’s the distinction between maybe you don’t want to release to users every good build, but you still want to keep your software in production ready.
Martin Fowler: And I think it’s an important thing that the users should decide what level of updates do I want. Anybody in the open source world is familiar with the fact that you can decide where you want to go. Do you want to go with Debian releases, which tend to be relatively infrequent but tend to have high confidence in them or you go for really touching on to the leading edge? And it should be your choice and it would vary depending on the kind of software -- things where you’re not as dependent upon getting the latest stuff all the time, you can go for a slower cycle. But other things - you are on this leading Edge Rails project you want to go to Edge Rails and get the latest stuff all the time and you are prepared to deal with the changes and results of that.
But I think it should be the choice of the customer which way they go, and the customer should always have a high level of quality so they and not going to be badly burned by being on the edge if that’s where they want to be.
11. You mentioned earlier automatic A/B testing. How hard would it be to implement automatic A/B testing through continuous deployment? So if I have a new feature and I want to deploy it I will just deploy it probably with a blue-green environment and then get the feedback automatically without any developer interference, roll back the decision or just promote it to everyone. How hard would it be to implement such a thing?
Jez Humble: I know they do that at IMVU I am just paraphrasing what I have hear from Timothy Fitz’s blog and other people who blog. At IMVU they have a system where they put stuff live. We have a pattern that we talk about in the book called "Canary releasing" which is used in a number of different organizations. But they use that at IMVU from what I understand where you put a new version live on a small set of servers and you route your small set of users to that and you monitor the metrics. If there is anything out of line in the metrics, outside the safe boundary then it automatically rolls back, otherwise it rolls out to the rest of the cluster and changes the routing so that all the users got to that version.
That is a pattern that is part of the continuous delivery toolbox. Again it depends on great automation and having good monitoring in place and good configuration management of your infrastructure but it’s absolutely doable. The people at IMVU and I think Netflix as well are doing that.
12. Which practices, which ideas should we be using at least in order just to say "OK, this is the minimum quality that I need in order to be able to do continuous delivery?" For example I need to have unit testing at least, or something that tells me "Ok, that’s the minimum that I need."
Guilherme's full question: In continuous delivery one of the things that I like the most is that it implies that you need to know that your software is in good quality and you can deploy it at any time. So it means you must have something that tells you: "Ok you can deploy it and there will be no problem in production." Actually it implies probably in some good for extreme programming or something alike. Which practices, which ideas should we be using at least in order just to say "OK, this is the minimum quality that I need in order to be able to do continuous delivery?" For example I need to have unit testing at least, or something that tells me "Ok, that’s the minimum that I need."
Jez Humble: I met at least one person who thinks that A/B testing means you don’t need to spend so much time on creating quality software because you can have your users do the testing for you. That might seem sensible when you first hear it, but the thing about A/B testing is you roll out a new version to a set of users, if it doesn’t work they will not use it and they will go and use something else. That is indistinguishable from there being a problem with the feature in terms of the value proposition. So it’s a really bad idea to rely on users to do your testing for you. Delivering continuously actually requires much better automated testing.
Martin Fowler: To me I say very definitely it flows from the extreme programming approach to testing, which has always been that you want to get your test suite up to the level that you can confidently release anything that passes all the tests. I was used to say the idea would be you may be doing iterations every couple of weeks or even longer but if somebody said "Hey that release I saw last night is really good, could you just put it into production?" the team’s reactions should always be "Yes, if it passes the tests, throw it into production." That requires that high level of testing and multiple levels. Particularly what the build pipeline helps to do is formalize the notion that the testing has to occur at multiple stages because some tests take longer and therefore have to go on a slower frequency. But you still have to do them and you still need the rapid frequency tests for more rapid feedback.
And so we align this whole line of tests along the pipeline and a way of looking at it is a series of hurdles and software has to clear each hurdle and eventually anything that manages through the entire assault course you are confident that that thing can go live. What you want to do is arrange your build pipeline so that you have that degree of confidence that anything that gets through all of those hurdles you are comfortable with pushing it live.
13. For the past 15 years we have been through copying files around in order to integrate the code that I have done and someone else has done into something as continuous integration and finally into continuous delivery which allows me to deploy as many times a day as I wish. What is the next step? What is coming next?
Martin Fowler: I don’t know, for most organizations it’s still hard enough job getting this stuff moving forward to be honest.
Jez Humble: I anticipate a big fight over the next several years, trying to get organizations to adopt more of this stuff. I wish I knew what the next thing was because then we could write a book about it and make lots of money. But I think the problem that we have at the moment is already going to be a pretty challenging one to solve. The one thing I would say is, actually I am not sure, this is more a question: "What do you think is new between continuous delivery and XP?" I am kind of springing this on to you.
Martin Fowler: I think it’s a better formalization and figuring out the details. I mean the principle was there from XP and indeed Kent would do this project for an insurance company in Switzerland where they released every night. I remember going and visiting them and they said "Oh, yes we release every night, we did once in a year of development have to roll back a version," but they just put everything into production every night. And they could do that because they had a well setup Smalltalk environment and all the rest of it. But it’s not as straightforward in all environments as we know because we struggled to do it - every time we have gone into a client it’s taken a lot of time and effort to get ourselves in the position where we can do that. So it really comes down to taking that vision that Kent had and was able to make work in a particular circumstance back in the 90s and say "Ok we can begin to so this on a much wider scale on a much wider range of organizations."
Jez Humble: So the idea has been there for a long time, I think what’s new is just the saying, "We have applied this to large organizations, it works, here is how you do it in these different situations." And also new tools are coming out - things like Puppet, Chef and cloud-based systems, AWS - that actually enable you to do this stuff a lot more cheaply.
Martin Fowler: I mean the Unix community would also claim that they have been doing this kind of thing forever. I mean Make has been around for as long as I have been programming. So this kind of stuff has been out there, there are just too many challenges to do this is in many circumstances. What we’ve done I think is made a significant shift over the last 10 years to running into these kinds of situations, understanding what the challenges are, figuring out how to go past them and capturing the knowledge of how to do it together with the tools to make it happen. The Cloud world I think makes this all the more apparent because one of the key things that I am hearing from people in the Cloud world is you can’t just put something into production and assume it stays there. The big thing about cloud is stuff is appearing and disappearing all the time.
So it’s really important that you can very quickly put something into action and you can expect that things are going to drop and so you have to be able to cope with that kind of change and that fluidity much more. And that I thought was quite interesting I mean I don’t know if caught that bit in the Netflix talk where they say that they basically have to rewrite all their applications in order to send them up to the cloud. And some of the things that they talked about in their rewrite was stuff that I would hope anybody would do anyway - it wasn’t specifically stuff about the cloud. But there was clearly this emphasis of "We have to make sure that we get the deployment process fast" because we are deploying all the time, because the whole point of cloud is you’ve got this elasticity and the elasticity isn’t going to be any use to you at all if you got a complicated deployment exercise and you can’t take advantage of it.
Jez Humble: Actually one of the things that’s happened in the last few years is that people have been using web-based architectures to create their internal services as well, a book that came out this year by Jim Webber and Ian Robinson and Savas Parastatidis, called "REST in Practice" was all about taking web based architectures which are loosely coupled and where you have small services that interact with each other that is how the web works. You have these - I can’t remember the right word for it - but relatively transient services which are very highly available. And that architecture works on the Web, which is the most highly scaled environment that exists.
Actually when enterprises say "We can’t use this RESTful stuff." Actually, that will cope with loads much more severe than you will ever place on your software, so maybe you should learn something from these kinds of architectures. I think that that is another change that’s taking place at the moment. The other thing that I would say about what will come next is and going back to what Eric Ries was saying. He comes from a start-up world where you have venture capital and you have organizations, where you have start-ups which get venture capital and they spring up and arbitrage business opportunities very quickly. I think actually I would like to see us move more in that direction in the enterprises as well where the business acts like a VC and the technology organization acts kind of like a start-up.
A lot of the time companies feel that they don’t want to take on that kind of risk, but actually a lot of the time businesses do deal with risk and development projects of any kind that are strategic are very risky. So not only can projects fail in development, but you can put something live and users can say "Well this is rubbish, we don’t want it." So actually even enterprises deal with the same ecosystem that start-ups do. I would like to see actually organizations behaving more in this way, and changing the way they deliver software, which means leaning more towards a product mindset where you have a product team that exists right from the beginning and goes all the way through operation through to service retirement rather than having the usual system you have at the moment, where people start up projects and then when the system goes live, the project team is disbanded and it gets passed on to the operations team and then deal with it.
That’s a very inefficient way to create and manage and deliver services. So I think actually the enterprise world has a lot to learn from the start-up world.
I know they've been working together for a while now...