Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Interviews Rachel Laycock on Continuous Delivery

Rachel Laycock on Continuous Delivery


1. Hi, I am here at QCon New York with Rachel Laycock[...]Can you tell us something about how that topic came about and what you were thinking about when you created this talk?

Graham's full question: Hi, I am here at QCon New York with Rachel Laycock who is lead consultant at ThoughtWorks in Johannesburg, hi Rachel.So you gave a talk on Polyglot architectures for rapid release. Can you tell us something about how that topic came about and what you were thinking about when you created this talk?

Sure, so ThoughtWorks has been doing continuous delivery for a few years now, I know that Dave Farley wrote the book with Jez Humble who still works with us and we sell that as something we can provide for our clients. And so I went to a client thinking “This is awesome, I am finally going to do Dev Ops stuff, maybe play with Puppet or Chef I really get to understand PowerShell, and when I got on to the client and we started looking at which parts of it we could break off, and start putting tests around and releasing in smaller pieces, we realized very quickly that the architecture that they had didn’t support that at all. So they’ve created this huge cyclic dependency mess that was going to be really really hard to pull apart, and the majority of the time that I spent on that project was just figuring out where we could create seams and really educating people in the organization on their architecture and how they can think about architecting not just for build and run but also for release, if you want to be able to deploy regularly, you need to break things into small enough pieces that you can deploy them regularly and you also need to have a good understanding of your dependencies and if you have things with long release schedules, can you break out that dependency or create your own component that does something similar so that you’re not tied to their release schedule because essentially you are going to be tied to the release schedule of your longest release schedule of any dependency that you have.


2. So the big ball of mud architecture was the thing that was stopping them from being able to adopt this? Or where there other factors too?

There were other factors so one of the other factors was that one of the reasons that they’ve created this architecture in this way was they treated their developers very much as fungible resources so they were just developers that were put on any project at any given times, so they created their architecture in such a way that it was all written in exactly the same way, which means that you were just creating tons and tons of dependencies across the projects, because one developer would be on a project, they would write something over here, and then be on another project and it was very much about build as quickly as possible, and so they would create some dependency to something that was completely unrelated and this would keep continuing as developers got moved around, and it was very much the decisions of the architects, that was it, and the developers just sat there and hooked things together and wrote the code. And a lot of it was generated code so the architects have created this generated architecture, and the developers had to just hook things together. And so because of also that context and that environment that they were working in, the developers weren’t really thinking about things in terms of how is this all fitting together or when I use this dependency what does it mean? They were just thinking and very much pushed in the direction of just build something, just build it.


3. So it sounds like changing the mindset of the developers and the architects that is going to be a key part in getting any kind of technological change out of this system.

It was pretty much the part, changing the mindset of the developers, the testers, the architects, and pretty much every manager that was involved in that in some way and changing the way they thought about it. They often used to ask me “Can we do continuous delivery without Agile” and I said “Yes sure, but you don’t have to do a small analysis just in time, but what you do need to do is you need to have your developers and testers working together, because in order to have continuous delivery, you need to have a CI, you need to have a build and release pipeline, and in order to trust that, you have to have tests in it”. And so you have to move away from the mindset of “Developers write some code, and the testers are going to check it at the end”.

That’s what I consider to be quality control, they just are going to tell you at the end how buggy you are or how good or how bad your code is and give it a mark out of ten and then come back and say you need to get seven to pass. With continuous delivery it very much relies on the quality you build into the product the quality you build in either through unit tests which in the majority of time I would say you want huge unit test coverage, but also with integration tests, whether it’s contracts between the system or contracts between your own components, and also functional tests and especially in the case of unit tests it’s better to write those tests up front to use the TDD approach and so when you are using the TDD approach you are also thinking about the design and you are really building that quality in and that’s what I consider to be quality assurance.

And in order to do that you need to have very good communication between the testers and the developers and you have to change the way the developers are writing the code so these are fundamental people changes, and not technological changes. The technological bit is sometimes hard but it’s often a lot easier to solve and especially with the tools that we have now, like we’ve got so many awesome CI servers, obviously ThoughtWorks has Go, but there are plenty of others out there, you’ve got awesome tools for virtualization, infrastructure automation and all that configuration stuff and we even have the capability to test those now, so tools like Puppet and Chef and PowerShell and there's things continuously happening in that space and even in the Windows space which is where I spend most of my time, the support on some of those tools is getting better and better all the time.

So the tooling is there, but they very much rely on that being reliable so the code be tested, do I trust what’s coming through? Because you can just put your entire ball of mud through the pipeline but it doesn’t mean it’s going to work at the end. So you need to be able to be assured of that quality all the way through, and that comes all the way back to the development process, which is how are your developers writing the code and how are they communicating with the testers so do you need to do every part of Agile, do you have to do a stand up, do you have to do this, probably not really but these are things that really help you, but for me fundamentally it’s that communication between the developers, the testers and then if you really want to do CD then you also get to get involved your Ops guys as well.


4. If you are going to move towards TDD then you are changing the power balances, of the system perhaps, so you’ve got now the developers writing the tests does this make the testers or even the developers feel uncomfortable?

I wouldn’t necessarily say that it’s changing their power dynamic, I think that it really depends on the organization and where the power existed in the first place. Because we like to forget that organizations are politically driven and they are all about people and we as developers often think that we want to focus on the technical solutions. Of course we should write tests, we are building quality into the code, but the power dynamic is really dependent on the organization, what I have noticed is the change is really in the resistance to change. So the developers are used to just writing the code and being done with it, and the testers are going to find their bugs and they can just move to the next story. But in the TDD approach they actually have got to think about how they are going to build it, and they are already thinking about how they are going to build it obviously, but they are actually thinking about that sort of they are thinking about what tests they can write, and use that to drive out the design.

So they do have to do more work and especially initially when they are learning, it does take longer, and they are resistant to that, because they used to I know that at QCon and ThoughtWorks a lot of developers here we love learning new languages and new tools and we get excited by that, but a lot of clients and a lot of developers globally they like to be masters of what they know, so when you introduce something that they don’t know they get very uncomfortable because suddenly they are not the master anymore. And some people are quite adaptable to that but others can be very resistant and they can be resistant in a lot of different ways, so depending on where they are politically in the organization they can pay lip service so if this is coming from a higher order like “We must do Agile transformation, we must do this project like this”, then they can be like “Yeah, yeah, I’ll do it, I’ll do it” but they don’t they just try to get you out the door.

And then others that might be kind of like nine to fivers or we just assume they are, they just want to get in there, do their job and want to go home. But often when they start to get it, or they start to understand it or they start to enjoy doing it, they can totally change. In terms of the testers, what I generally found is the testers usually get the worse end of the stick in the development life cycle anyway, because they are right at the end, and the testing always gets pushed especially if you are using a waterfall methodology, you are supposed to have two months of testing and now you got only two weeks, test as much as you can, so the are frantically trying to test stuff and then they come back with bugs to the developers and the developers are obviously not happy about that, and so they are getting pushed from both ends, so their lives improve because suddenly someone is talking to them about writing tests and usually initially we start by getting the testers to do acceptance testing and start automating some of their tests, and then we are trying to get the developers and testers work together on this, and the purpose of that is to improve those tests, because those tests are usually long running so you want them to be more performant and, guess what, developers are usually better at that.

But the second thing that I usually get to as teams mature, is to get the developers to see what the testers are covering in their tests, and to communicate what we are covering in our tests, and so that we are not overlapping, and we are writing usually, because unit tests are much faster, we are writing more unit tests, and less of those end to end tests. So I have not seen the testers be resistant, I have seen developers be resistant and for lots of different reasons, and again as I said it depends on the organization. In another example, the client was trying to get some software out of the door for such a long time and the developers were just fixing bugs all the time and “Can you guys come in and fix the developers?” and I am like “Ok”. And the developers were actually open to learning TDD or doing any of these things, but they said what’s the point because we are going to get pushed again and told: “We just need to get stuff done”.

And that was like “Oh, that’s an easy question to answer, your managers brought us in, so you might as well use us while we are here” and when the product owner and the manager the dev lead saw that the developers were improving so much and the bugs started to go down as opposed to just fix them and more get created, they actually started to ask “What can we do?” and we were like “Well you know when you keep disrupting things and making changes in the middle of the iteration, that’s kind of annoying, it doesn’t actually help and it puts a lot of pressure on people and people don’t do the best job”. So what I have discovered through consulting very quickly is that it’s never the technical problem, it’s never that we don’t have the right tools, like when I was saying in my talk if you are going to choose a tool, if you are going to choose a new language for you want to go polyglot for whatever reason, I am ok with it as long as you can test it and the tools mature enough to allow you to test it and put it into the build and deploy pipeline. Pretty much all languages as soon as they are written somebody else writes the tests for it.

So the tools are there, all the tools for CI are there, there are so many, I mean ThoughtWorks has one Go, which is awesome, but all the tooling is there, there are the tools for the build configuration and for the virtualization and those problems have been solved, the problem that you have to solve is the context of the environment that you are in and that’s the people, what are their goals, what are they trying to get out of it, how are you going to handle their resistance? And depending on that we will pretty much make or break the success of whether they can transform and learn some of these techniques or whether they can’t because if people want to they will learn it. If they keep resisting they won’t learn it because they have already decided: “I am not going to learn it” or “I can’t learn this” or whatever it is.


5. Are there patterns of developers resistance, are there particular categories, how can we identify people who fit these patterns and how to help motivate them to adopt these changes?

That is a really interesting question because I have never thought about it and the minute you said that I was like “I could write a book about this”. But I think it’s not a pattern of developers resistance, I am sure there are a lot of psychology/people related books out there that we can actually read and learn about in order to figure how we can help people change and how we can figure out what their resistance is and how to, maybe not “counteract” that, but how we can say “Hey you are resisting because of this, but this is the benefit”. I am sure there are patterns but for me it’s more instinctive it’s more just about people. I just look at them as people and say “Think about rather than they are just resisting they don’t want to do it, is the first question to ask is why? Is it because they are uncomfortable and they don’t feel like masters?” and it’s like “Hey your secret is safe with me, I am not going to tell anyone that you didn't know what you were doing for six months because that’s why I am here, I am going to shield you from all that and if the tests are rubbish then that’s my fault because I am the one that is teaching you”.

It really depends on what they are resisting as to how you can handle it, I am not sure if there are patterns, I am sure there are, but I am going with what does this person want? And that assumption about people being nine to fivers, when I make that assumption, or I made it in the past, oh that guy is a nine to fiver so he is not going to want to learn, then you dig in and you get a little history and it turns out the project that they were on last year everyone was working so hard that they were sleeping on their desks, and yes now he is on a nice easy project he wants to work nine to five, fair enough. That doesn’t mean he doesn’t enjoy what he is doing and want to do it, so for me it’s always a question to find out what motivates that person or why they are resistant because there is usually a counter argument or a way of convincing them to do the right thing. And most people especially if they want to be masters, telling them that if you start writing tests in your code you are creating the quality in your code, you become even more of a master, that can sometimes win them over as well.


6. You talked a lot about how perhaps the developers have to change, what they do, there is a famous trope in software development that your architecture will eventually come to reflect your org chart, so there are presumably organizational changes too, and does that lead to manager resistance?

Very, very, much so. Changing the developers and the testers is easier than changing the managers' mindset, but again that can actually be the context of the environment so a lot of the environments that we go into there is a key executive who looks after hundreds of testers and another one who looks after a hundred of developers, and never the two shall meet, and in order to get these teams to start working together as teams, we call them feature teams, because it’s basically a combination of the developers, the testers, the analyst, the PM, the user experience person, anybody that you need to build that team, they should be together if possible, and working together as a team, not working in silos. And that can sometimes be a really hard sell because you are asking these people to change the structure of their organization. And the only way to really sell it is to prove it so we always go with just “Let’s do a pilot, let’s take a couple of these testers and some of these developers from these different environments and get them to work together and prove that this will help us release quality software regularly”.

And usually when we prove that, that gets them into the mindset of “I can see this works”, but that kind of change, of changing the organizational structure takes a long time and even in order to get that team together you’ve got to go all the way up to the executives to each different group to convince them that this is the right idea. So there is a lot of explaining the benefits and trying to convince people or influence them to try and make some of these small changes just to prove that it works. Otherwise basically, and this has been proven, any point in which you have communication breakdown between teams is where you will find the complexity in the system. So I have worked on projects where it’s been awesome, Agile, we are doing lots of TDD and technical practices, and it looks like the code is awesome and then when we are going to release it, because the communication is broken down between Ops and the development teams, that suddenly takes forever and ever and ever. And that’s complexity that can only be handled by people communicating with each other, and in order to get that team together, you’ve got to span all different directions in an organization to convince them to let you do that. So that’s even harder, so this stuff is hard.

Graham: That sounds like a good message to end on, so Rachel Laycock thank you very much.

Thank you.

Sep 11, 2013