Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Interviews Mary and Tom Poppendieck on the Role of Architects, DevOps, and Diversity in IT

Mary and Tom Poppendieck on the Role of Architects, DevOps, and Diversity in IT


1. I'm Charles Humble, and I'm here at QCon New York 2015 with Mary and Tom Poppendieck. Can you introduce yourselves and tell us a bit about what you're working on?

Mary: Okay. My name is Mary Poppendieck and I started life as a process control engineer and I worked at University of Wisconsin and 3M for lots and lots of years and retired -- I don't know, before 2000. So, subsequently became author, speaker -- stuff like that. Go ahead, Tom.

Tom: I started out as a physicist: taught in the university for a while. Worked at Honeywell and commercial aviation, worked at GE, worked at a small consulting firm and joined Mary in retirement at the beginning of the last decade. Today we're speaking, teaching, writing, taking pictures.

Charles: I remember you talking in about 2004 about how long it took to deploy a change of a single line of code into production, and obviously since then continuous delivery has happened and Microservices have started to happen. So I'm just curious about how you see that; what impact that's had and...

Mary: Oh, well, one of the reasons I wrote it was because I was so surprised. Because when I did software in process control environments with my own minicomputer hooked on my own piece of equipment, I made a change and it changed. I come to find out in the world that I had moved into, it didn’t work that way. You made a change, and it would be weeks or months before it actually could get through and I was rather astonished at the concept.

We've done some research to show that at that time, something like 30% to 40% of any release cycle was spent after all the code was done, just fixing it; and release cycles were a year or 18 months; boy, six months was really fast. So a line of code -- little tiny change, that was needed now -- you could maybe patch it in in the operations area, but it took forever. So I said, "How dumb is that?"

Jez Humble wrote in the introduction of Continuous Delivery that he saw that and he took it upon himself as a challenge: what can we do to drive that time down? All of the practices in continuous delivery are focused on, from the time I need to make a change until I can actually deploy it in production, why don't we have it be proportional to the size of the change? For example, if it only takes me an hour, surely I should be able to get that thing in overnight. So he develops in that book a lot of the practices of continuous delivery which by now, five years later, have become extremely common in the industry - almost, if you're not heading in that direction, you got to ask yourself, "Why not?"

Tom: We gave that talk at ThoughtWorks around 2004 and Jez wasn't the only one that heard it. Other ThoughtWorkers also encountered it and ...

Mary: Fred George.

Tom: Fred George, and he said, "I was so frustrated because there was nothing I could do about it: it’s totally out of my control. We ran into Fred just a few months ago at the YOW! Conference in --

Mary: Australia.

Tom: Australia. He says, "But now, I can. I've got Microservices and I can release any time I want." So it has kicked off, irritated people, and is behind some of the biggest things in the industry today.


2. Are there any other desirable side effects of that? Presumably it helps remove hand offs between IT and operations and support and those kinds of things as well?

Mary: Its major side effect actually is more stable code, because the thing is if you make a large change to a complex system, you do not know the results: you cannot know the results. If you make a massive change, you are begging for unexpected things to happen. It will not behave the way you want it to behave in some small way.

If you want stable code and a big system, you have to make small changes. So not only do we get changes that are interesting and important in way more rapidly, so we're way more responsive to whatever need it was that generated the response, but the systems are much more stable.

So there was this thought, I don't know where it came from, that if we would just spend all kinds of time and take forever and test everything until it's perfect, we would have better, safer, more stable systems. That’s just wrong. If we have very small changes that we do frequently and we learn how to adapt rapidly to the results, that's what gives us stable systems. So when people begin to do this, they understand that their systems are much better off because of it.

And then the last thing is, there's much more rapid feedback to developers. The job is just easier, more fun, more engaging, and when the developers can see, “I've done this, actually it works”, well that's kind of a nice feeling. I used to have it when I did software; I could tell right away if what I did worked. Now we're giving developers end to end feedback on whether or not what they just did is right and that's a very nice place to be.

Tom: Organizations get irritated when their systems crash, but they get even more irritated when they don't achieve the business objectives that they were invested in for; and simply doing what an organization asks you to do, no matter how well and how reliable you do it, doesn't achieve the impact that they were after necessarily. In fact, often it doesn't.

How do you know? You try experiments. You try experiments in the market. You try experiments around the market and these experiments can't be expensive: they have to be quick and cheap. If you can try and experiment, make a change, see what the impact is in a few hours, you can learn from it and incorporate or not incorporate depending on the results. If it takes you six months, you can't experiment. All you can do is guess, and you'll guess wrong most of the time.

So I think even an equally important impact to better stability, better reliability, fewer crashes, is that you can identify the things that actually matter much more effectively by being able to try experiments quickly.


3. Are there any challenges to getting developers and ops people to work well together? I think they have quite different personality-types on the whole.

Mary: Well, dev people and ops people have different personalities. The theory is called regulatory focus theory by Tory Higgins and also a book on Focus by --

Tom: Heidi Grant Halvorson.

Mary: Yeah. Heidi Halvorson, Grant Halvorson. They write that there are two kinds of people in the world. One kind of person really wants to avoid failure. They don't want to do something that's wrong, are the kind of people that we want running our data centers and our nuclear power plants and the air side of an airport. Okay, we want them to just do the safest thing, which is what they're comfortable with.

And then there are those entrepreneurs and explorers that are looking for aspirational goals. One is prevention focused and these are promotion focused and that is, “Try anything: something's going to work."

Now, if you take a poll of generally ops folks and generally dev folks, you're going to find that developers tend to be promotion focused and ops people tend to be prevention focused. There's some tension there because the things that people like in promotion focus are lots of fast new stuff, great opportunities. Prevention focus is, "I don't want to do anything that's not safe." But it turns out that having both personalities on the team is the best approach, because you actually want to have sort of an argument between those two focuses. Yes, you want to be safe and yes, you want to have things that are interesting and adventuresome.

If you can strike a balance, and that balance isn't going to be struck through some rules or process, it's going to be -- because whatever the rules or process come from it's going to be one side or the other -- But if you have people with different focus having fair arguments with each other about what's the right thing to do right now in this case, and they have the same goal, “We want to keep this thing working: we want customers to be happy”, you're going to get the best results.

So dev and ops folks working together on an ongoing basis in continuous delivery is necessary and it also helps to have those two groups of people working together towards the same objective; making really good day to day tradeoffs that are impossible when you have a throw over the wall thing.

Tom: All of us have both aspects to our personalities and our preferences, both prevention and promotion. We can be primed to focus on one or the other depending on the context and by having the mix of tendencies from people focused on safety, the ops [KH - he says dev here, but it’s clearly a slip] people, the people focused on promotion, developers, but also analyst designers, business responsible people, all of these who are tending to be more promotion focused, finance people who are of course much more prevention focused, these can prime each other to sensitize the relevant aspects depending on what's at stake in any given context.Mary: Well, dev people and ops people have different personalities. The theory is called regulatory focus theory by Tory Higgins and also a book on Focus by --

Mary: Actually that's why a leadership team in a business will have the different functions on a leadership team, so that they can have discussions between the different functions as to what's most important in this case. Is it going to be development? Is it going to be quality? Is it going to be the financial issues? This is just taking that same concept and bringing it all the way down to the ground where those detailed decisions get made.


4. In the second talk you gave yesterday, the design versus data talk, you made a bit of a case for architects, who I think have got quite a bad press in the industry over time for being a bit divorced from reality maybe. So I just wondered if you could speak a little about that?

Mary: Well, I've never seen a tough technical environment that doesn't have to have a range of senior engineer people, more-experienced, and less-experienced people. When it comes to design of either a technical environment or of a user interaction environment, you need people who happen to be good at design. So I don't see why it would be a good idea to take a tricky technical design and just kind of hope that whichever team happens to be there has the expertise to make it work. So we need some view of, “What's our system architecture going to be and why and how is it going to happen?”

As I come from a family of architects, this is building architects. My father did -- my grandfather, my father, my brothers, my cousins, there are lots of architects. My opinion of architects came from that environment in which all architects were expected to be at the building site, all the time. They walk there, they ask questions. I saw my father walk on a site and it was, "Hi, John. How you doing?" He walked around, everybody knew him. Everybody could talk to him about the problems that they were having. He would write some down, he would give some advice. He'd bring the rest back to the office: and it's that concept.

My brothers were required to work as construction workers from the time they were old enough to get a job until they got a job as an architect, because my father said, "You want to be an architect, you got to know what construction's all about."

So when we have architects that have become architects because they have a strong feel for how the thing ought to work, and are deeply involved in the building process, and understand the issues of what it takes to build, then I think that's a really important role. But if architects -- unfortunately, some architects have gotten into the -- the architect tells everybody else what to do. Basically, when we thought we could just write everything down and send it off to a team in some other country and they could code it, remember those days?

Charles: Oh, yes.

Mary: Yeah, it didn’t work, did it? Similarly, architects can't have a grand idea and make it work. But at places that have good architects, I always hear the -- and the architect is the most respected technical person in the building because everybody likes the architect. I saw that with my father: everybody loved him when he came on site.

If the architect is a deeply respected person for the experience that person has, then I think when you see a good architect at work, then you see how come it's a really important role. But somebody that’s sort of trying to be an architect but doesn't have the depth, the experience, the vision of design? Maybe not.

Tom: There's architecture, and there's architecture: architects and architects. You've described one category. I don't think that many architects start up striving to become members of that category that you disparaged. So what drives it? I suspect that it is pursuit of efficiency and specialization. After all, if somebody's good enough to be an architect, why you should you waste their high salary on writing code? Shouldn't they just be doing architecture?

So the management thinking that utilization of resources, rather than continuing growth of resources by continued direct engagement, is what's important, drives people away from continuing to mature in their understanding of how to optimally structure systems, and forces them out of the implementer role into a purely, "tell other people what to do”, because that's more resource efficient. Resource efficiency is evil, if pursued for its own sake.


5. Do you think there's a similar trend towards taking senior developers and moving them into management roles, so away from coding, which again I know is a concern among many senior developers that I talk to?

Mary: So there's -- I come from 3M where we had a dual-track and there's certainly a need for managers who really get what development is all about, to be managers of development. In fact, I think a lot of the problems with software engineering in general is when managers who have no clue what's going on pretend that they're going to be managers of the environment. So that's a good career path for somebody who cares about helping others be successful and using their knowledge to do it. But then, there are other career paths where you become very skilled in an area and help others in that area.

When we had a dual career path at 3M, you didn't have to choose. You said, “I’m this kind of person” or “I'm that kind of person”. You could get just as high a level from a status, from an influence, from a salary, point of view in a technical track or in a management track. So we tended to have the right people in management and the right people leading the technical areas.


6. I think it's reasonable to say that there's been a general trend with some of the Lean and Agile methodologies and some of the other things we've talked about with DevOps and continuous delivery, to try and break down silos between teams and individuals in an organization. Do you think that's a fair characterization of what you're trying to do?

Mary: Yes, although I don't think Agile actually went far enough because Agile has always been about development and then they let the testers in, and now, we let the ops people in. But when you think about it -- I think about software as being a product and what you're thinking about is, “How do I put this product in the market?”

Now if you go into a non-software company, all of the people necessary to put the product in market sit in a line division and they -- in 3M, a manufacturing company, we had a lab in every division and those people developed the product, and they manufactured it, and did quality, and did the warranty support, and all of that sort of stuff. And we would have this sort of extra area that was helping us to design our products.

So in so much as software becomes part of a product -- let’s say you're a bank, let's say you're an insurance company, your products are essentially software. Since they're essentially software, you should think about "Why don't I have the software engineers in my line division?"

If you look at the -- well, let's call them the new companies, the companies started up in the internet era, starting in the late '90s, they don't have -- by the way, they don't have developers and they don't have IT departments,: they have software engineers, that are in the line business. In that environment, you don't have these walls; not even the ops wall, because the responsibility of the team is to do that end-to-end thing, starting with understanding what the customers really want and then making sure that it is properly deployed: with the responsibility across that entire span.

So if you look at a service team in Amazon, they call it a two-pizza team, ten people, but they have everybody from deciding what to do to ops on that team, having discussions about how is this service going to perform. So I'm not sure that we need to start thinking about why are these silos even good? Why do we have IT departments removing the software engineers that are there from their customers, and instead we have their colleagues who happen to be in the line business as an intermediate between engineers and the customers. I don't think we need to have it that way.

Tom: The worst mistake a development organization can do is to do what somebody asks them to do: because customers don't know what will be an effective solution, in the context of the available technology, to what hurts. They know what hurts. They have an often naive diagnosis of why it hurts and a design for, “If we just had this or that, maybe it wouldn't hurt so much”. But they're usually wrong.

Far better that the people who have the technical expertise, people that have all the expertise that's necessary, are given a problem, "We have this pain, we have this issue. How do we solve it?" and work together with the customer to explore solutions and figure out what is the best approach. The direction being "Help us make this hurt go away”, not “Give us this precise capability."


7. I'd like to change tack slightly. I think it's reasonably well known that IT has actually seen a drop in the number of female graduates that we have now, even relative to other STEM subjects which have a pretty low ratio to begin with. Do you think that's something we should be concerned about as an industry?

Mary: Well, when I got into software, there were not quite as many women as men but certainly a whole ton of women doing software, and after a decade or two I looked around and said, "What happened to all the women? Where did they go? Where are they?" So I've been asked that question again and again and again, and recently, Mikey Dickerson, the guy that worked for site reliability at Google that went and worked at and he now works in the government, he posted on Facebook a chart. It's a chart that's been around, it shows women in physics; okay, that's STEM. Women in lawyers, women in other areas like --

Tom: Healthcare, medicine.

Mary: Healthcare, yeah. They all went 20-40 in up to about 40% of people in schools, in graduate schools. Then there's a woman in computer science following them along up until about mid-1980s and then it bumps and then it goes down and it goes down, it's now at 20%. So being an ops guy, Mikey Dickerson says, "We should be paying attention to this, because when I'm in ops and something goes wrong the first thing you look for is, ‘When did it start?’” Because there's all kinds of things wrong but you've got to find the cause.

What you do is you say, "What happened when things went wrong?" So take a look at this chart, he says, "What happened in 1984?" So he threw that out to the world, "What happened in 1984?" Probably the most believable answer that came back was, “Not exactly 1984”. In 1984, the children that were ten and 12 when computers started becoming things in homes, toys that you gave to boys, got to college. Suddenly, the guys -- the boys that had computer toys -- knew how to program, and the girls — that wasn't the toys that they got — didn't. They then said, "Well, I'm not going to go on competing in an area where I'm behind and I've got a disadvantage, so I'll go to physics, or I'll go to any other place." And that's persistent.

I have a daughter who was born, who was 14, in 1984 and I asked her. And she says, "So when I get to college, Brian," her husband, "and the other guys they all had computers when they were kids and I didn't. Dustin didn't --" Dustin, her brother, didn't either, because we had a computer in the house but it was not a toy; it was for doing homework.

I went to the computer store, and I did the check-out, because the guys built the computers because they knew that from when they were kids. I think there's something to be said for the fact that what happened in 1984 was that we had started having a crop of people hit college that had experience with any kind of programming and stuff before that, and there was a very strong difference between girls and boys, men and women, when they entered college as to what background they had.

I can understand if there's an area where all those guys are skilled and I'm not, I'm just going to find a different area to go into. So it’s something that we have to think about from a really young age.

In fact Carol Dweck who wrote [the growth] Mindset, also wrote some articles on the same subject: why not women in Computer Science? She said that, again, there are these different personalities.

One is: I'm not going to do something if I think I'm going to fail. I'm going to be good because I've always been praised and when I fail, I don't like it. And she says, typically it's more common for girls to always have to be good and always get praised, and a little bit more common for boys to mess up and get out of it and not be so afraid of failure, because all through their childhood they've been primed to say, "Failing and getting up and continuing on is okay”, much more so than girls. Which is why when you get just passed middle school, you'll see girls who were good in math, and statistically all the way through elementary school to a certain grade level they will be good in math, and then it'll suddenly fall off; and it's true across the US for sure and I think other countries too.

It's because of the, "Okay, I don't want to tackle hard problems because I'm used to being successful, and because I'm used to being successful and I might fail in that problem, I think I'll go find something easier to do.”

So it's this not being brought up to accept the concept that good means failing and learning; instead, good means being successful. It's that talents-inborn, versus talent is something you learn through hard work. So the growth mindset -- when they prime girls in middle school with the growth mindset, the only way that you get good at anything is by working hard and failing, suddenly the math scores of the girls are just as good as they always were. Without that priming and control groups, the girls' scores routinely go down and the boys' scores do not.

Typically, we have not got the same focus with the children that we're raising. Is it okay to fail? Is it okay -- or do you want to just go towards and do something where you're going to be successful? And if we have a different way that we treat our children with regards to that question, then we're going to have a different result when they get to college and say, "Well, that one looks harder and there's too much competition over there, so I'm going to go to one where I'm more likely to be successful”; instead of, "I'm going to go to the one that I'm more likely to be challenged."

We have to get into the way we raise kids the concept that challenge and overcoming tough stuff is more fun than being perfect all the time.

Tom: Correlation is not causation, but the model that you have offered is very interesting and plausible. But there's something else that happened in the early '80s. Computers became important to organizations and when it becomes important, higher level executives pay attention.

Most of those higher level executives have come from marketing, or sales, or finance. So they want to manage more effectively this increasingly expensive aspect of their organization. How do they do that? They don't understand the technology; they don't understand computing. The only way that's left for them to control it, to manage it, is by the numbers, by cost, by optimizing efficiency. The result was waterfall: waterfall was not common before the early ‘80s.

Mary: That’s true; it was not.

Tom: The sequential, resource optimizing, ‘we have a specialist here, and a specialist here hands off to a specialist here, hands off to another specialist’ approach is supposed to optimize, in a way that's understandable to people that don't understand technology, the utilization of these expensive people who are so important. The consequence, we all suffer from still. That doesn't work, but the numbers look good: the input numbers. People are finally starting to realize that controlling the inputs does not control the outputs.

Mary: So how does this affect the -- how many women and men go into software?

Tom: Well, the real question is why do men put up with that? I suspect that the female population is more concerned about making a difference rather than just doing some interesting stuff, and they're less willing to tolerate the way that people end up being treated in the context of a waterfall process, practice, than guys are. I don't know why guys tolerate that.

Charles: I think that's a fantastic place to stop. Thank you both very much indeed.

Jul 25, 2015