Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Interviews Joshua Kerievsky on Anzen and Safety in Software

Joshua Kerievsky on Anzen and Safety in Software


1. Good morning, this is Amr Elssamadisy at QCon San Francisco, here with Josh Kerievsky. Welcome Josh! Can we start off by you telling us a little bit more about yourself?

I am the CEO of Industrial Logic. We are very early pioneers of the Agile and Lean space. We started in the late 1990s, working with lots of companies around the world, helping them to adopt things like Extreme Programming, Lean software development methods, Kanban, Lean Startup, all of these things, that have been useful to companies in the past. We have been early pioneers and practitioners of them. So, that has been our focus.

Amr: Josh, you have been in the Agile community a long time and you have experimented with many of the new things, one of the things that I know about Industrial Logic is, you are constantly at the bleeding edge in coming up with newer ways of doing things.

Yeah, we are really relentless in terms of trying to make software development better and we believe to do that, we have to experiment on ourselves as guinea pigs. So we build products, we keep our hands dirty, we constantly experiment with process. When Kanban was first coming out, there was as a birds of feather meeting I remember, the Agile conference in D.C., we were immediately playing with experimenting. So we are always trying to come up with faster, better ways to work, slimming down. So you know, Agile started out being called “light weight methods”, and we believe that was the heart and the soul of it, making these things lighter and lighter. So we are constantly trying to improve.


2. Excellent. So what’s wrong with the Agile world now and what are your experimenting with these days?

The Agile world, I mean my gosh, it has become so popular. There are so many people doing it and it is a world wide thing now. Certainly it has hit the main stream. What’s wrong with it? Well, in some ways you could say, this is a fantastic success, right? And if you look a little bit deeper, you can see that it is not necessarily providing the output or the outcomes, that we would like to see. Is it really having a significant impact on software development? You could say, in some places yes, in many places no. In many places it is only being adopted superficially, or it is being adopted in terms of a management change, but really not touching software at all. Or vice versa.

Amr: So these magnificent successes of the early days, even though we are all “doing Agile”, those results have not become widespread.

In general no. In general you don’t see those like magnificent changes happening too often. There is kind of this way, that has become popular, which I can’t stand myself, because it is really messy and mediocre, and that is roll out some really superficial kind of thingy and two years later realize it is not working and try to put some lipstick on it, to make it look better. And that is unfortunately the way that many folks approach this stuff. We are going around seeing the problems and the customers, that call us up say:” you know, we started with Scrum and you know, we are having all these problems with our software still and it is just…” and than you go in and you realize, well, a) they are not even doing Scrum, b) they have ignored all of the tactical issues, but they adopted certain ceremonies, so that they think they are Agile, because they are doing ceremonies. But in fact they are not getting anything real accomplished. This is very common and it is very unfortunate.


3. It is. That has been my experience also. So what are you doing in Industrial Logic? What’s new? What are you coming out with?

We have something, I am really excited about. You know, it has taken years to actually come out with this, because there is this sense I had and others like you have had, of this is not enough. There is something missing here. And for years I have been thinking, what is this next thing? You know Lean Startup came along and it was wonderful, I still think it is wonderful, it is a great thing. It provides a lot of what was missing in Agile. Because it really provides a smart approach to business. But it is not everything still. It is still missing something and I was trying to discover what that was.

So a few years ago, 2012, I came across a concept in a book about safety. It was the story of Alcoa, and if you don’t know the story, Alcoa invented the aluminium smelting process, became this giant successful monopoly in the early 20ies and 30ies, and eventually started to degrade in terms of performance; problems in the unions, problems in the quality of aluminium and so forth. So by the early 1980s they were struggling. A struggling conglomerate, a worldwide conglomerate. And it was that point, when they brought in a new CEO who made safety their number one thing. He made safety THE value of the company and observed, he had a very smart insight: “If we can make Alcoa the safest place to work in the world, we are going to be excellent.” We are going to be outstanding in what we do. Sort of, how does that come from? How do you decide, how do you see that?

Amr: That’s not intuitive, at the least.

It is not intuitive at all. What he saw was, that people were being injured regularly. Even though they had a good safety record compared to other companies, he said, it is not even close to being good enough. You know, so what happens is, you start with this thing about everyone’s job is to make it safer. And you develop a habit of excellence through safety, because you notice problems and you look for ways to solve them. That gets your innovation mind working, right? How can we do this in a better way? [For example] Instead of climbing a tower and shimmying over to some gauge to read the gauge, even in rainy conditions or snowy conditions, what if we just had a high-powered binocular to read the gauge from the ground, in complete safety? That was a fantastic little idea. It made it faster, made it safer, they could read the gauge quickly. Multiply that little tiny story, which happened at Alcoa, across thousands of people, in thousands of locations, all doing the same thing and now you start to realize: Wow, that company went through a monumental shift, in terms of how they work. So I was blown away with that story. Of course that’s a manufacturing story. And in the manufacturing field, safety is the big deal. So I said, there is got to be something we can do in software. And the more I started thinking about it, …


4. You are telling me about Alcoa and I understand it is manufacturing and safety, it was a big deal. But did that translate to profits, did that translate to anything other than a safer environment?

Yes, it did. In fact the safety initiative that Paul O’Neill brought in, the CEO of Alcoa, led to incredible profits. You can look at the stock curve from when he started in 1987, he left in 2000, 13 years later, to stock just goes like this (points upwards). I mean you’ve got a couple of dips during recessions, but it really just climbs and climbs. And meanwhile the injury rates and the lost work days, those numbers just went down and down and down. And they asked him, when he left in 2000, he became Treasury Secretary of the United States then, they asked him Paul: Were you successful? And he said: Well, I will let you know if I was successful at Alcoa, if after I left those trends continue, profits continue to go up and injuries continue to go down. And they did. They did for many, many years, they’ve continued. It was a lasting change, cross thousands of people in a culture, that was over a hundred years old when he started.

Amr: So that is huge.

It is huge.

Amr: And so, it is not just about safety. Safety somehow creates benefits in many other things.

Yes. It basically opens up some doors and helps your people start to access excellence. So it is just a beginning, but we think it is a beginning that is phenomenally important. And when you are ready I can talk to you about safety, safety in software.

Amr: Please go ahead. Yes, tell us about software.

Ok. So in software, we do things that provide safety, right? First of all, there is people and there is software, right? Safety is needed everywhere. You need to make your users safe to use your product. If they do not feel safe using your product, you are dead-meat, right? So if your users are not safe, no good. If you are selling a product to buyers and those people are buying the product and than giving it to users, if that is not a good purchase for them. If it is a failure, the buyer gets a black eye. Not a real black eye, but they get black eye politically. Oh, that was the person that spent a million dollars on software, which no one very worked with, or which was awful. Bad on him, right? That’s not a safe transaction.

Amr: So it’s bad for their career, it’s bad for their impression on how they work with others.

Exactly. It’s an injury, it’s a white-collar injury, right. So there is safety in that realm. Than you take developers and testers and the people, analysts, database people, all building the software and you can start to say, well, using the manufacturing metaphor, what injuries are they sustaining and how can we make it safer. How can we make the process of software development safer? How can we make the production system safer? Right, what is it that is harming us? If we start to pay attention to those injuries and start to be innovative and look for ways to make things better, like the binoculars, we can start to see much, much better results. So, that’s what I’ve posited, when I first read that Alcoa story a few years ago. But I said, well, you know what, we like to experiment in my company Industrial Logic, so let’s actually do it. Let’s make safety our number one thing.

Early on I started talking to one of Paul O’Neill’s top lieutenants, one of the persons, you know, his right hand man. And this is a fellow named Bill O'Rourke. Bill O'Rourke and I have become some friends on e-mail and stuff. I asked him, you know, my first thing was called it “prioritizing safety” and he said: Josh, no, no, it is not a priority. Because see, priorities can change. It’s a value. When you make safety your value, your values don’t change. In fact, what I have come to see is that we don’t have five values. That is kind of typical: the company’s five values, right? Bill O'Rourke showed Enron’s five values, right? Great, well, we have these values, you know. They mean nothing. So I said, I am through with the five values, let’s get one value. Let’s make it safety. So, at Industrial Logic we started to experiment with this. We said, well, where are we unsafe? And it turned out, on a technical side, we had opened up, we had basically started e-mailing ourselves whenever we had a severe error in our logs. Some people will tell you, there is a lot of hidden magic in your log files if you just go looking. Well we went looking and we are seeing, we started e-mailing these severe errors and there was a concurrency issue we were facing. We were not facing it early on, but later on in time we started to see this, when people had ten browser tabs up, all on our website, on our eLearning platform, hitting all these pages at once, you could get a concurrency problem. It’s unfortunate, how do we solve this? So we made it our number one thing to go after. We said we are going to solve this, we are going to silence these severe errors by solving, by getting to the root calls.


5. How does that relate to safety again?

Well, we were being injured by the fact, that we had started to doing some work and than a severe error e-mail would come in. And it distracts you. Well, I can’t ignore it, a student may have had a real problem here. So, ok, I am going to stop on what I’m work on and go work on that. That kind of mental task switching is not very good.

Amr: So it is an injury to the developer.

It is also an injury to the student potentially. In some cases we get these severe errors and the student does not see anything. It is just a false positive. That’s even more confusing, you don’t know, is it a severe error for real or not, right? So this is causing some potential injuries to our users, making their experience a little unpleasant and of course interrupting us. We started to talk about these in terms of injuries, or hazards in the codebase. And we started to build a culture around this. When there is an injury, report it. We had a little database of injuries. We would have meetings, we still have them, every week, where we meet to look over all the injuries that occurred. And again, it’s not just on the software. You could have an injury in some situation with a colleague at work. You could have an injury with respect to some piece of software, that screwed you up badly. And in fact we never want to use that software anymore, because it caused a huge issue with a client, cause there was some conflict that occurred.

Amr: You could be embarrassed at the client’s side for example, where something doesn’t work.


Amr: And both you and the client would be injured.

Exactly right. And you could say, well, where was our lack of safety there? Where were the hazards and how can we remove them? So it’s pretty much looking at safety everywhere, even financial safety. We restructured Industrial Logic to have three safety circles: financial safety, client safety and our own product safety. That’s the focus. So more recently I have changed how I talk about it. So it turns out, because some many of our clients they are doing manufacturing, they are building all kinds of things and so many different injuries, safety already is a pretty charged word.

Amr: So it would mean something different than what you are talking about.

It can often mean something different. And many of those manufacturing oriented companies that do software development, software is just a small component, or so they think, it’s becoming more and more important, but they haven’t brought that safety culture into software. And in fact the lawyers sometimes get a little worried, when they use the “safety” word with respect to software. I am not a lawyer, I don’t know what all these issues are, but we are starting to talk about it now with the Japanese term. There is a Japanese term, and this is not a game where we are trying to use a mystical Japanese word to make this sound more important. It ultimately comes down to looking at the Toyota production system and trying to understand it. Trying to understand the heart of Lean, to be able to see how it applies. So…


6. So safety is part of Lean also?

Safety is certainly absolutely part of Lean. I’ll explain how I see it is part of Lean. First of all the Japanese word is “anzen”, A-N-Z-E-N, anzen. I love the word. I have spoken about with my Japanese colleagues and it is a very important word in Japan. And in Toyota, “anzen” is basically, the anzen world you’ll see it all over Toyota. People need to be safe when they are building cars. But how else does it work with the culture? So most of us have heard of the respect for people, one of the pillars of the Toyota production system. Respect for people. So I ask you, if you are not being respected at work,, if someone is being disrespectful to you, how do you feel?

Amr: That’s got to feel pretty bad.

Right. Upset.

Amr: Upset. Maybe even unsafe, depending if that person is senior to me or can affect me in a negative way or not.

Right. So you feel unsafe. You feel injured if there is something particularly disrespectful, that is happened to you.

Amr: We use the word “hurt.

Hurt, yes, you hurt me. It is not a physical injury, it is an emotional injury, or it’s a cognitive thing. But Toyota has made respect for people one of their pillars. Without respect, without that respect, you have an unsafe situation. So I would say, because that is one of the pillars of the Toyota production system, that is Anzen.

Amr: Safety is the base of respect.

It is. Safety is the base of respect. Anzen is the base of that principle of respect for people. So it’s there, it’s there. And here is the thing: when I went looking for Anzen in Extreme Programming, in Kanban, in the Lean methods, often written about by the Poppendiecks, in Lean Startup, it’s there. It’s this common denominator, like you would say in mathematics, that is basically is present in all of them.

Amr: So all successful things that we have done in Agile, when they have been successful, safety has been a part of it, but we haven’t called it out specifically. We haven’t been focussing on it specifically.

I would say that, I go a little bit stronger, I would not say that Anzen or safety is part of it, I would say that Anzen or safety is the root of it. It’s the root of why these things are successful. When we fail at Waterfall, it is often we can look back and say, you know what, it was inherently unsafe to spend all this time doing upfront analysis and design, when we didn’t really even know, why we should be building the software. Who is going to pay for it and so forth. That was financially unsafe, politically unsafe and so forth. When you provide safety, you are allowing yourself to be high-performing. And becoming safer and safer and safer, helps to become even more high-performing. It’s this wonderful doorway. And once you see it, it is hard not to see it everywhere. You just see the common denominator. So, I am positing that yes, it is the thing that companies that are adopting Agile or Lean need to be aware of, or need to make it their number one, their value.


7. Has this affected the way you and your teams work now? Are you doing things differently with your clients?

We absolutely are. Every time we interact with a client, we ask ourselves: How can we make this safe for them? The other day we had a frank discussion with a client. We had spent the prior year a bunch of workshops. They had asked us for this help. Teach us these workshops. Give us these workshops on things like testing and refactoring. Very successful workshops, the evaluations were excellent, but we knew that that wasn’t enough. That if you just take some course and than go back to your environment, that doesn’t appreciate what you have just learned, …

Amr: Monday morning it doesn’t translate.

It doesn’t translate. You don’t actually do those things. Your manager hasn’t been trained, doesn’t know anything about this testing and refactoring stuff. What do you mean TDD (Test-driven development), we don’t have time for that. So this kind of throw-it-over-the-wall training, where you get trained and than you go back and than you are in an hostile environment, or in an environment that is not valuing the safety practices you have just learned, it’s not going work. The client was really clear about that, that they wanted to get back to team-based training. And boy, I was happy to hear that. Because you know, we used to do a lot of that and that provided a lot more safety. Because you get the team building software coming to get trained together and learning how to work together in this very safe way and than actually going back and doing it. So, yes, we are going out to more of our customers now and saying, you know, we could give you what you are asking for, but we can tell you how it is not going be very safe, for you.

Amr: It’s going to be dangerous for you, if you do it this way.

It’s an unsafe investment. And we don’t want to be in interactions with clients, that are unsafe. So we want to provide the safest possible services and products, or a mixture thereof, to our clients. And that’s our focus and it has been helping us. So financially it has been helping us. We see our clients care, that we care so much about them. They can feel it and understand it. Now, very interesting, I tell you one more quick story. One of our greatest safety practices, that I have started doing in 2001, was the readiness assessment. Why go into a customer that is not ready to do some Agile or Lean process, one that is just going to fail? We might make some money, but they are going to fail and no one is going to be happy. So we use these readiness assessments to help provide safety. Let’s discover a win-win. The other day I was trying to sell one of those to a new client, and it turned out, I found it out, that they didn’t feel safe to buy an assessment. Why is that? Well, they have already been doing a form of Agile, a very superficial form of Agile for a couple of years. And politically for them to actually buy an assessment makes it sound like they are starting over. So that is not politically safe for them.


8. They could get in trouble. What have you been doing for the last two years?

Yes, why are you starting over with an assessment. So we discovered, that we can’t sell an “assessment”. We can sell something that is like the term “kickoff”. We can do a kickoff. But we can’t sell an assessment, because that’s politically unsafe. So these safety things keep coming up for us.

Amr: They affect everything, not only development, but sales and marketing and your relationship with your clients.

Yes. If you make folks feel safe, it changes everything. And I want to be really clear about one thing. Safety does not mean not taking risks.

Amr: Because that’s unsafe.

It’s unsafe to not take risks. We talk about this in our own company, right. We could continue to do things that aren’t innovative, but make money and you could think, well, that’s financial safety, right? Well no, it’s short-term financial safety. What are the big things that we want to do, that …

Amr: If you are not innovating, you are going to die in our field.

That’s right. So we look at it, we want real safety, right? Not superficial safety. So that’s where we are still learning and we are constantly thinking about that. So again, what do you call Anzen, or you call it safety, it has opened my eyes and made me excited again about Agile and Lean, because I think that with this perspective, with this value, real change can actually happen.

Amr: Lasting change.

Lasting change.

Amr: Well thank you so much Josh for telling us about this. We look forward to hearing more. Maybe next year, after you have worked with this more. Thank you for your time.

Thanks so much Amr.

Dec 27, 2013