Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Presentations The System of Profound Knowledge

The System of Profound Knowledge



Ben Rockwood talks about Dr. Deming’s four simple ideas known as the System of Profound Knowledge, a modern system that sets forth a pattern of thinking that the society is still trying to fully implement. He urges people not to try to imitate LEAN Practices and instead learn from the underlying ideas themselves to improve the way they learn, lead, and operate.


Ben Rockwood is VP, Production Engineering at Packet. At Packet he's driving the development of the Bare Metal Cloud with an emphasis on Edge deployments to put computing power in more places than it's ever been before to meet the needs of the next wave of applications.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.


Rickwood: My name is Ben Rockwood. I'm the VP of Site Engineering over at Packet. We're a bare-metal cloud. If you love the idea of cloud, APIs, quick provision, but you're, "Virtualization, not so much," we're definitely the place for you. We love hardware, we love the capabilities that we get from it. We love security, we love firmware. We're a very interesting cloud company based out of New York. Glad to be here.

I've been involved in the DevOps movement for a long time. Remember with great pride back in 2009, we had the first DevOpsDays event down in Mountain View. The industry has changed a lot. We knew, back in 2009, that the industry was in the process of changing. More and more of that is coming to bear, and it's proved out where it ultimately was going to go. Throughout the DevOps movement, I've always viewed it as really a banner of change.

We knew that the circumstances around us were changing, and we needed to evolve with those times. We didn't quite know what that was going to mean, and a lot of the things we were talking about was going to come. Now here we are 10 years later, in 2019, things have changed quite a bit. There is a great body of knowledge that has been accumulated over the centuries, and particularly lessons learned through the 20th century that help inform how we're going to do things in the 21st century.

I'm very much interested in foundations. I'm very much interested in theory and in basis. When DevOps started to grow, Lean was something that we reached at. Something that really concerned me was not a lot of us knew a lot about Lean. It was something we grabbed on to, two bits and pieces that we liked. I was really concerned about the fact that there seemed to be a lot more knowledge in this area that we should learn about, take advantage of, and help learn the lessons that industry learned in the last century to help give us an idea of what was going to come next.

Once Upon a Time

Around the turn of the last century, there was a lot of technological change. The world was changing. A new generation was coming up. We were adapting to new forms of technology, and a lot of those technologies would set up the future to come. All those technologies were really bespoke. These were foundations that we were building on top of. It wasn't really until early in the next century that we really started to see somebody bring those ideas together. We started to see industry consolidation. We saw standards, we saw new technologies emerge, new methods of bringing all that together that really laid a foundation upon what we have in the world today. Any idea of which company was responsible for really that consolidation and setting the benchmark for what we have today? Turn of the century?

Participant 1: Ford.

Rickwood: Ford. Here's something that I find really interesting. A lot of people get really concerned when we talk about manufacturing. "Manufacturing? Look, we're not manufacturing. Why are we drawing all this stuff for manufacturing?" Because if you go back before the manufacturing heyday, as we moved from the 19th to the 20th century, automobiles were clearly coming, but they were things that were for rich people. Not many people had them. The majority of the world had horse and buggy, carts, and whatnot. Roads really hadn't been built yet that could handle it. Cars were actually built in small little shops. There were all bespoke. A lot of universities, in fact, were building cars.

It wasn't until we had Ford, and Ford saw an opportunity to do something. The thing that was interesting is Ford's real passion actually wasn't the car. He was always really passionate about making a lot of something. His father was a watchmaker. He, when he was a kid, would tinker around with watches and would learn how to fix the watches. He would be thinking in his head as a little kid, "What if I wanted to do 100 watches or 1,000 watches?"

He was really interested in how you do a lot of something. He actually took a number of different types of jobs, including a job at the Edison company so that he could learn how to use electricity because he knew it was going to be an important component in the car he was going to build. Then when he started his motor company and gave us all kinds of interesting innovations, namely the assembly line and a bunch of other innovations that would come out of that and formed the basis of mass production, we saw this explosion.

Really interesting, if you ever look at pictures of Los Angeles, at the turn of the century, and then 1930s, it's completely different. It's amazing how fast the world changed with the innovation of the automobile. Really, in a lot of ways, it redefined American culture, redefined who we were, a whole new generation that was defined by mobility and the car. There's so many parallels that we see at that time that we look to today where we had Bolton boards in the mid-90s. We had lots of bespoke companies that were small ISPs and whatnot.

Then we see where we come to now where we all have computers in our pockets. Networking is ubiquitous. We're still on the cusp of this journey and very early in this journey. I don't think we quite yet even know where this is going. We can look back at that parallel and draw a lot of interesting conclusions and look back at those thinkers and see what they were thinking to give us a little better idea of where it is that we're all going with this.

Interpreting the Past

The important thing about looking into the past is being really careful in the way that we interpret the past. A lot of times we look at the past with our modern eye, and I don't think it's entirely fair. If you're going to look back at Ford, you're going to look back at Deming, you're going to look back at any classic literature or whatnot, you need to think about "What was the problem they were facing at the time? What was the challenge that they had? What were they trying to do?" Then from that, extrapolate lessons that we can then bring forward and then apply to our modern eye. We're going to be really careful when we interpret the past.

I want to look at a couple of the things that came together that informed what would become Lean so that you can walk away with a much more solid idea of what this Lean thing is, what it means to us as an industry in this modern era, and help give you some better ways to extrapolate your own ideas of how to use those things. We're going to start with something called pragmatism.

Darwin changed the world, didn't he? In a very big way. He brought us out of an era where we believed in the Greek ideal and suddenly, evolution, and this idea that things can evolve and have independent causes changed everything. Out of that came an American philosophy, pragmatism, by Pierce, James, and Dewey, that really opened up this idea of testing everything, not necessarily looking for a universal essence, but looking for specific use cases and really applying scientific method very rigorously across the board to everything that we do.

Pragmatism was something that certainly appealed to Ford and to others around the turn of the century. There were some really great things about it: The separation of the philosophical from the practical, getting very practical. Breaking down to platonic forms of the ideal and starting to look at local optima, what really is happening in individual systems independent of anything else, not taking anything for granted. There were some things that were challenging out of it. People really value their ideological beliefs.

It had a tendency to create this dehumanizing utilitarianism. It allowed us to really break down and bring the scientific method into a lot of places that previously just hadn't really gone. Pragmatism was definitely very prevalent across all aspects of society as we approach the last century.

Another was Frederick Winslow Taylor, publishing "The Principles of Scientific Management." This is a great example of something where I think that Frederick Winslow Taylor gets a really bad wrap. Taylorism is generally seen as the bane of many people's existence. It really is the beginning of scientific management, and management in the modern era really does trace back to Taylor. Taylor, of course, is doing time and emotion studies, believed in looking at creating the maximum amount of efficiency out of a certain amount of effort.

This had some certain very good effects. It increased productivity, which tended to increase earnings. Most people at that time were earning based on piecework. The more work that you did, the more you earned, that was good. It also had a tendency to reduce injuries. There's a standardized forms for the way that things were done. However, it was very prescriptive. It tended to deny personal agency and was incredibly dehumanizing. I think when we look back at Taylorism today, we tend to see it for that primary reason. It's very dehumanizing. There were a lot of benefits, particularly in that time and place, and it really came in the face of a number of other social factors at the time that it needed to work against.

It wasn't, of course, just Taylor. When we look back there, Taylor takes the brunt of this. There were a number of other people who were taking this same idea based on pragmatism, looking at the work that we do, trying to be more scientific about the way that we do that work and improve its capability. The Gilbreths were particularly interesting one if you've ever seen "Cheaper by the Dozen," great movie to go back and watch that one and look for all the things that the Gilbreths did with regards to efficiency in time and motion studies. Very interesting stuff. This was certainly a thread as we brought science into industry.

The next step that came after was Walter Shewhart's work on statistical process control. He was at Bell Laboratories. As we got more in scientific, we were measuring more and more. We needed to look at something in particular variation; that in all systems, there are variations. That we need to account for those variations, and that we should be using data for the decisions that we make.

Some of the great things about statistical process control was being able to first make the step into data-driven decisions. We got increased consistency by controlling that variation, increased quality and reliability. This is when the quality movement really got going, was with statistical process control. Of course, it also had some negative connotations as well. This is where we get business by the numbers, definitely took bureaucracy to the next level and empowered it with some terrible things. It further challenges personal agency and the ability for people to be innovative.

All this data without context can lead to incorrect conclusions. This is a problem we deal with today. I think almost all of us like to be data-driven. We like to support things by the numbers. We also hate it when we take those numbers and hand them off to somebody else without any explanation. They start to draw wrong conclusions and start to make changes based on that data. Data can be your best friend or can be your worst enemy. All this started back then with Shewhart.

We saw how this played out in society, Charlie Chaplin's infamous "Modern Times" in 1936. You got to see a good example of how society was feeling about all this. The world is going faster. We have all these new technologies all around us. It's business by the numbers, and it's massively dehumanizing. We're all just cogs in a giant machine.

We got something of a balance through the explosion of psychology in the 20th century. Jung and Maslow are certainly two that stick out to me as being a primary interest. There were a great number of others that were there. The idea that we need to be fighting against that dehumanizing force, and thinking about the needs of the individual, the motivations of the individual, what really comprises that individual and their needs was a thread that was growing quite strongly. We have all these threads that are really growing through the first portion of the 20th century.

Then a crisis occurred. World War II happens, World War II ends. Throughout this time of all these innovations in growth, Japan finds themselves in crisis. The country had been devastated by these two explosions, and they had a need to rebuild industry. Also, the Americans, post-war, needed to do a number of things in a country that have now been crippled. Namely, they needed to produce a census, and they needed to get industry back on its feet in Japan. To do that, they drew on some of the best and brightest.

One of the people that they drew upon was Dr. W. Edwards Deming. Deming had gotten a doctorate in physics from Yale. He'd worked with USDA. When he was at USDA, he met Walter Shewhart. To a large degree, he became a disciple really of Walter Shewhart. If you look at almost anything Deming's ever written, he really harkens back to Walter Shewhart quite dramatically. When Deming was asked to go over to Japan after World War II, it was predominantly to take care of the census that needed to be done in that country.

While he was there, a number of people asked him to come and speak about statistical process control. They were very interested in taking advantage of that. As Shewhart's disciple, he was in a great position to go and do that. During his time in Japan, he provided a number of lectures. As a result of those lectures, and also his desire to not take any money for those lectures, they created a prize for innovations in industry, known as the Deming Prize, which is still a prize awarded today. It's considered one of the most prestigious awards to be given to anyone in Japan. He also received a commendation from the Emperor of Japan several years later.

The thing he was most focused on was quality, the importance of quality in building goods and services. That quality would have the ability to reduce the cost of goods, but improve the speed of production and also improve the lives of all the people who are using these products. He was able to really harness all these threads that we were just looking at and how they came together and be able to give that in one sitting to all the most prominent people in Japan at that time. It's interesting that the thing that he wanted them to employ the most was quality. Of course, what are the Japanese best known for, at least particularly up through the 80s? It was quality. They heard that message. They implemented it.

Taiichi Ohno at Toyota knew that if they didn't get production at Toyota up on par with the United States within a very short period of time, that they would be doomed as a business. In this crisis, he was willing to do whatever it took in order to put the company on good standings. He had a lot of great engineers around him, and he really took the teachings of Deming to heart. A lot came out of those earlier ideas, in particular, pragmatism was very important to Ohno. It was, how do we look at individual problems that occur and come up with the appropriate solution using the scientific method, using data to support the conclusions about what we need to do to push things forward? All these ideas grew into the Toyota Production System.

Toyota did a lot of good work. Japan grew in prominence, and they were doing really well. Japan really came to the forefront in 1973 when we had the oil crisis. The American mass-production system pioneered by Ford was producing massive number of cars every year that were rolling off the assembly line and expecting to be sold. We get this oil crisis. All of a sudden, these cars aren't going into the hands of buyers. They're sitting there. This created massive overruns and really shocked the American mass production system for the first time. The American manufacturing had been supreme predominantly because of the fact that it had a privileged status as a result of World War II and its capabilities after World War II. This really came to head in 1973.

On the other hand, the Japanese were able, because of their just-in-time production system, they only produced what they needed to with reasonable buffers, of course. They were able to slow down their manufacturing and be able to come through this period unscathed, and their quality was high. As people began to start looking at cars, they started looking increasingly at Japanese options, which were smaller, more fuel-efficient vehicles that really met the needs of the market. This catapults the Japanese really strongly onto the world stage to the point that in the 1980s, the Japanese were seeming to be unstoppable. Their lead on quality in production was vastly outweighing what we had in the United States.

NBC decided to do a documentary about why the Japanese were winning. In that documentary, they happen to learn about Deming and his trip to Japan. They thought it was kind of an interesting twist in the tail that the reason the Japanese are beating us is because an American went over and taught them a bunch of things that they seem to have used to good effect and now the Japanese are beating us. If that's the case, why don't we know about this American? At that time, Deming had been fairly obscure but now was catapulted instantly onto the world stage.

As a result of that demand, goes and writes a book out of the crisis. Of course, the crisis that we're predominantly referring to is the oil crises of '73 and '79. This book contained a real roll-up of everything that Deming had procured in the early part of the 20th century, talked to the Japanese, and then had perfected through his consulting practice. In that book, he proposed the 14 points of management.

14 Points of Management

What's interesting to me is that this list doesn't look nearly as dated as one would imagine. The first was create constancy of purpose for improving products and services. Why do you build the things that you build? What's the value that it provides to the market? There's plenty of companies today where we're all working very hard, and, to be honest, we're not entirely sure what exactly it is that we're really giving to the world, we're really giving to the market. Today, we would generally call this vision, but he calls it constancy of purpose.

Adopt the new philosophy. The philosophy that he is speaking to here is really adopting a scientific pragmatist viewpoint of testing hypotheses using data, working collaboratively, sharing with each other, all with the focus of providing higher quality to meet the needs of the customer in the market. Cease dependence on inspection to achieve quality. This, of course, is pretty specific to manufacturing, but where do we, in our industry today, have inspection. QA teams would be a great example of that. A couple other security and I guess a couple of other things. End the practice of awarding business on price alone. Instead, minimize total cost by working with a single supplier.When was the last time you bought software and you went to a whole bunch of vendors, and then you just bought the one that was the cheapest?

This is pretty powerful for manufacturing or other industries where you're buying a lot of components from a supplier. The idea here, underlying this, is to have a relationship with your supplier. That your supplier is an integral part of the product that you're providing, and that you want to have a real strong relationship with that supplier to provide the best possible product to your customers.

Yesterday, there was an interesting track about software supply chain. I think it gets to the same idea. When you think about implementing dependencies in your application, do you just grab whatever's hot at the time, or do you think about ecosystems and ways to really invest in those dependencies to make them stronger so that your product can be stronger? Improve constantly and forever every process for planning, production, and service. This is where we get that continuous improvement.

The Japanese implemented this as Kaizen continuous improvement. One of the things that we tend to get wrong out of this is that we tend to think of it in terms of really big chunks. One of the things that Walter Shewhart came up with was the PDSA cycle, Plan-Do-Study-Act, which morphed into Plan-Do-Check-Act. The idea here is that it is really scientific method applied to business. Unfortunately, it's become this abstract way of doing long [inaudible 00:25:50] If anybody's done ITIL or a number of other ISO 9000, whatnot, you'll actually see PDCA is a prescribed way of actually managing your products and your initiatives.

However, they'll talk about in terms of things that are years-long, multi years-long, which makes really no sense at all. When we talk about Plan-Do-Check-Act, we're talking about figuring out what is the problem that we're trying to solve? What's the actual problem today that that gentleman in the hallway is having? What is the result that we want? In engineering, I normally refer to this as the definition of done. What is the thing you actually want? It's all very easy to point out what a problem is, but what is the future state that you desire actually look like? Can we define that?

Once you can define where you're at and where you want to be in very clear terms, we can then do a root cause analysis, and there's a bunch of different techniques for doing that. Fundamentally, we're trying to figure out why isn't the world that way. If it was obvious the way things ought to work, why doesn't it work that way already? How can we understand the constraints that are implicit in the system, understand its place in the system? Then from that, we can produce experiments. When we do in PDSA, we're performing experiments.

I think this is one of those places where it breaks down. We want to have this big, long planning process, and then we want to go and actually execute all the stuff. After that, then we can go back and reflect on how it went. PDSA was really intended to be very small, very tight, something that you can use all the time. Improving constantly and forever every process for planning, production, and service.

When we talk about, in DevOps, that we want everyone to be involved, we want to empower people to implement change, this is what we're talking about, constantly and forever, every process. Institute training on the job. This is something that we don't have a whole lot of. I'm sure there's some of you here who have companies that actually have training. Frankly, a lot of us in tech right now, you hire somebody with equivalent skills, you drop them in, have them kind of sink or swim on the team. Normally they swim.

As we move further along in the next two or three decades, we're going to get to a place where we're going to be training more as things become more and more standard. Right now, we're still in that very young bespoke era. We all do things a little bit differently. We're seeing standards come in. Now, it'll be interesting to see which standard is it going to be. Kubernetes is not going to be here for the next 50 years, I don't think. In fact, I'll be surprised if it's still here in five years, but we'll see what happens. As we start to consolidate as an industry, training is saying it's going to be important to us.

Adopting Institute leadership. That's a funny one being so short there. Honest-to-goodness leadership, that leaders need to be responsible for where their organizations are and where they go, not just setting goals and leaving employees on their own to kind of flounder, but really leading an organization. This comes down to the way that we design organizations, the way that we assembled teams, the tools that we give those teams, and being an active part of the way that organizations grow. Deming really believed that all leaders in an organization really need to understand the customer problem. He was very frustrated with what he called the mobility of management, that people would just hire into a director role or a VP role.

We see this in tech all the time, people just hire into higher parts of the organization. They haven't worked in support. They haven't actually been an engineer. A lot of times, they haven't even used the product. Always an interesting question when you interview somebody, "Have you even used the product?" Generally, the answer is no, and that's not a problem. Probably should be.

He believed in a very long view of management. You see, Taiichi Ohno really glommed on to this, that if you have, say, a bread delivery company, that if the VP at one point drove a truck, when the truck breaks down, he has an intimate idea of what does that mean to the customer and people are not going to yell at him the rest of the day because he's late. He knows what that means. When that VP sitting in his office hears that a truck broke down, he comes from a position of empathy and understanding, rather than frustration and anger that now the metrics are slightly off. These are the sorts of input ideas that come back into DevOps when we talk about empathy and understanding and collaboration, I think often gets lost.

The other points of his 14 points of management, drive out fear. A lot of us have tried to implement efforts to have more collaboration in our organizations, and a lot of times they start very slow, and they can die very easily. A lot of that has to do with fear. Employees need to develop a sense of trust in each other, a trust in their leadership, a trust in the organization. When you start a DevOps effort or any other kind of collaborative effort, you're going to have people look for signs of it breaking down. It can be very slow to volunteer. Generally, they have lots of opinions, but they don't normally want to be responsible for that. Being able to take responsibility to put your ideas out there and to see that you're not penalized when they don't work out is important in driving fear out of the organization.

Breaking down barriers between staff areas. DevOps is all about that. Eliminate slogans, exhortations, and targets for the workforce. We don't need to put all these posters up that tell everyone to collaborate, work with others. It should be implicit in the way that we do things. Eliminate numerical quotas for the workforce and numerical goals for management. This is something that's been a big issue, I think, in tech and directly maps over to us. Whenever I see metrics about how long it takes for somebody to close a pull request. That can be really interesting in terms of making sure we have the resources to close those pull requests. If it's just, this team's failing because the pull requests aren't quick enough, then we're just back to the old world of counting lines of codes in calling that productivity.

Removing barriers that rob people of pride of workmanship, and eliminating the annual rating or merit system. Pride of workmanship is something that I think is really important to all of us. If we want people to put their hearts into their work, we need to allow them to be proud of the work that they do. That can be a whole talk on its own. Pride of workmanship is important, is implicit in the way that he saw this working.

Institute a rigorous program for education and self-improvement for everyone. Probably the clearest example we've seen of this in tech actually would be 20% time. Allowing and encouraging all of your people to think about, not just what they're doing today, but who are they, and who are they going to become in the future and allowing them opportunities to grow into who they're going to become.

Then finally, putting everyone in the company to work accomplishing the transformation. There are some of the things we've heard when people talk about DevOps transformations about which teams. Can I start over here? Can I work with just these? If we're going to transform, we need to put everyone in the company to work. I think, hopefully, you can see that one of the main thrusts we see here is that this is really a philosophical change in the way that everyone views a connection between the market and the products, the customer and the employee workforce making that happen. It's not just a couple of tips, tricks, and techniques.

Lean Is Born

All this interesting stuff is happening. Because of a desire to understand what was going on in Japan, MIT commissions of study. Womack and a couple of others are involved in that study. At the end of that study, they produce a book called "The Machine That Changed the World." That book creates the tag Lean. They can't call it the Toyota Production System. They need to take that out, so they have to change the name of Toyota Production System to something else they call Lean.

This is a story some of you may have heard, one of the other labels when they were trying to come up with a name for it as a possible name for what is this method and this system called was fragile. Somebody looked at all the different attributes of the Toyota Production System, they put it out there, and they said, "Yes, it's Lean." It gets to this idea of doing less with more and being very agile and adaptive. It depends on so many different factors all coming together. That if any part of this isn't working, it's all going to fall apart and break, so maybe we should just call the fragile method. That name, of course, wasn't with so many books, so they decided not to use that one.

It's really interesting that the very inception of Lean as something that becomes institutionalized, they're already aware of the fact that it's based on several of these pillars, many of which are soft pillars of this collaboration across the organization working together, goodwill for the customer shared experience and all this. They knew that if those elements weren't there, all these other tools, tips, and tricks were all going to just fall apart and be useless components. That happens in '91.

What's interesting is that while Lean took off and got a lot of interest, if you look at book sales of "The Machine That Changed the World", if you look at the amount of industry interest in manufacturing at Lean, you don't actually see a giant uptake or really big surge in interest in it until 2001. During the boom years when everything was good, Lean's cute, read a thing in a magazine about it, heard a thing at a seminar about it. All of a sudden when manufacturing demand slows down around 2001, suddenly companies are looking for a way to change and make things better.

The same thing happens again in 2008. Even though Lean has been around for a while, it's still something that if you go into manufacturing companies today and ask about their Lean program, it'll sound a lot like those of us in this room who talk about the DevOps movement. "We're looking at it, we're working on it, we got a team that's evaluating it." Lean is something that you would think has been around for a while that manufacturing would have considered this old news. They're still trying to adopt it today.

They're having remarkably similar reactions to what we actually see with the uptake of DevOps. It's because Lean fundamentally misses the point. To the point earlier, if you look at a bunch of books that have been written really since the late 80s, so about the same time that Lean became a thing, we've seen an explosion of books that say, "Ultimately, the problem is your culture. Ultimately, the problem is how you view the world." We see books like "Start with Why," "Extreme Ownership," "The Fifth Discipline," "How to Become a Learning Organization." "The Fifth Discipline" is a great book, it talks about a lot of the same things we saw Deming really advocating for, got missed through a lot of the implementation of Lean. Then we were looking to see how we could get that back. How do we connect different parts of the organizations so they're shared responsibility? Deming writes another book, "The New Economics," where he further distills this.

If you're anti-Lean, and you want to go and find a way to say, "Lean is all completely useless, forget this stuff," go get Jim Womack's book called "Gemba Walks." It's actually a collection of articles that he wrote, newsletters that he wrote, but it's all about him going out and his experience with different companies and seeing what's going on in his consulting trips. He really lays out pretty clearly his frustrations with how people are trying to cherry-pick and pull these different tools and techniques out, and they don't want to hear about how the executive teams actually need to go and spend time on the factory floor actually working with employees, actually in meetings between organizations. They got to come out of their ivory towers and actually be involved with the work and how that doesn't work. That's what the Gemba is. Gemba is where value is created. It's the Japanese word for where value is created.

The fact that Lean is still right with all the Japanese words that all come from Toyota, gives you a sense that no matter how much they say they want to free themselves from the tools and the tips and the tricks of Toyota, it's still at the heart of what they're all about. They've really struggled against that.

One of the things we can take away from all the experience we've seen in Lean as has been applied across a whole bunch of different industries is that you got to stop cargo culting. Just stop. Cargo culting really is, in a lot of ways, the organizational equivalent of works on my machine. It's what we need to be doing, and I think is what Deming was trying to get at the very beginning and what he taught to the Japanese, because the Japanese didn't ask him to come over and teach them a new philosophy. They wanted him to teach statistical process control from his mentor. "Just give us statistical process control. We already have Ford's books, we've got all those techniques. Let's put this together." He teaches them a different way. They adopted it with great success. They didn't just cargo cult those techniques alone. They developed new techniques based on those teachings.

That's what we have to do. We have to take the principles and the essence of these things, and we need to adapt it, not just adopt it. The SRE book from Google is great, but what we should be doing is reading that book, looking at the problems Google had, and the way that they solved those, take the principles out and then say, "What problems do I have today in my business where I'm at, and how do I take some of these different things that have been out there to learn from and adopt it or adapt it into my organization to fit my needs where I am today?" Cargo culting is not the solution.

The Answers Are Within Your Organization

Throughout the history of the DevOps movement, as has been the case with Lean, we've tried to emphasize that the answers you seek are already inside your organization. We all know that one of the most entertaining things about getting a consultant is a consultant comes in, does an analysis of your organization, looks at how everything is working, and then writes a fancy report that is generally exactly what a bunch of people have already said in the organization, except they don't want to listen to the people in the organization. They're just seen as whiners and complainers or whatnot, but they're going to listen to the consultant. Of course, they don't normally like what the consultant say so they put that report in a drawer and they just carry on business as usual. The answer is already existing in the organization. We've got to break down the barrier so that we can hear each other and take shared responsibility, not just point fingers up, down, side to side, all work together.

In the first book that Deming wrote, he presented his 14 points of management. He consolidated everything down a lifetime of work through this incredible 20th century. He consolidates it all down into what he calls the System of Profound Knowledge, which is four elements. First, appreciation of a system, second, knowledge of variation, third, theory of knowledge, and fourth, psychology.

These aren't just four things you do, these are lenses to look through. These four things in interacting together are a way of looking at different problems as you try and adapt things in your organization, try and experiment with new things in your organization to help guide you, and therefore timeless principles, I think, that can be useful. I have found them incredibly useful, whether I'm looking at a new incident management system that I want to put in place, changes to the way that we handle monitoring, new engineering structures. I can apply this lens as a way to help guide me as I move through decisions.

Appreciation of a system. Everything is a system. Systems interact with other systems. If you start with anything in our modern world, it is a system which is comprised of other systems as a component of other systems. A car is comprised of a number of systems. That car is a part of a wider transportation system, which is part of a culture and all that. When we use the lens of appreciation of a system and we see everything around us as systems, interrelations and relationships, there's a couple of questions we can ask ourselves whenever we're thinking about something. Whatever it is that I'm looking at, what does it interact with? What are the other touchpoints with other things out there, other processes, other teams, other people? Are there feedback relationships between those entities? Can a failure in X impact Y? What stakeholders are actually involved?

They are a part of the system. In fact, I would say probably the first fundamental contribution of DevOps was a reminder that we have technical systems of databases and applications that are connected to load balancers that are on networks, but also an integral part of that system is the people who are involved in building, and managing, and maintaining that infrastructure. They're just as important to that system as is the database.

What constraints exist within the system? Once we look at everything as things through having a strong appreciation of a system, we start to see all these relationships, and it gets a little overwhelming. This is where mapping techniques become useful. There are a bunch of different techniques out there for how to visualize these things, but it all comes from an appreciation of a system.

Second is knowledge. Knowledge really speaks to epistemology, speaks to pragmatism. Scientific method in what we do and being data-driven. Why do we believe that X is so? What is it that we're actually doing, and what do we really want thinking about current and future states? What is it that caused such and such to be this way? Particularly when you join a company, you should spend a lot of time doing an archaeological dig. How did the systems get to where they are? It's very easy to be like, "We were done. We didn't have time," or, "It's a stupid implementation." It's really easy to bag on engineers for not being very bright.

If you look back and have empathy, you'll realize that there were constraints on that individual. Some of those constraints might have been time, they might have been technology at the time. It's important to have empathy and look at "Why did it get to be the way that it is? How can we experiment with solutions and alternatives? How will we know when we've accomplished our goal, what is done look like? What we'll we do with all this new knowledge? Once we've produced knowledge, what do we do with it? Put it in a drawer, throw it away, or do we use it and change the way that we do things?" It's looking at things through this lens of knowledge.

Variation is one that I think that we've actually gotten a lot better at. When statistical process control was a new thing, the idea of having data and looking at periodicity of things was a new concept. Today, we're surrounded by data. We have graphs for everything. A big part of statistical process control was looking at the differences between common cause variation and special cause variation. Once upon a time, that would take several weeks to fully explain to people what those were. Today, you can look at any CPU graph. You see that it goes like this a whole bunch, and then every so often, it goes like that. That's common cause, that's just normal stuff going on. Then when it goes like that, that's something that probably wasn't supposed to happen.

In fact, I would say a lot of times, today, we love making things visual because we almost get to be intuitive rather than intellectual actually about data. A lot of times we're not really thinking hard about what our data is. We're just graphing in such a way that we can just point at the big spike, which gets back to the point of knowledge and thinking more deeply about these things. With variation, what does normal look like?

Remember, these are lenses that we can evaluate anything. What about you in your mood? What about your employees and how they feel? If you do engagement surveys across your organization to see how people are feeling about their company, you're going to notice that there's some common cause variation that's going to happen there. People are going to feel good, people are going to feel bad, people are going to feel good and generally washout. You will see special cause variation that will pop up some major event occurs and everyone suddenly changes.

Understanding the variation. Most of all, things do change constantly, and that's normal, is important. How much variation is acceptable is a good question to ask. What should be done in the event of special cause variation? CPU spikes? Do you freak out or do you carry on? What do those boundaries look like? How do we avoid overreacting to normal oscillation? For managers, a big part of this is if an employee shows up and is having a bad day and is just really pissy. Do you reprimand them or do you ask them how they're doing and chill out. Oscillation is a normal thing in all systems, including humans.

Then finally, psychology. I think when we talk about culture, we try to get into the psychology of it, but I think it's something that, as an industry, we haven't fully embraced yet, but we're getting really close to. When we're looking at any system and we apply the lens of psychology, we have to ask questions about how does it empower humans? How do humans feel about it? Their feelings actually are important. If you don't think somebody's feelings are important about REST versus gRPC, just wait and see how the projects turn out in the next year. How do humans use intuition to adapt the process to fit to changing situations? Is there broad consensus? Do we have a shared vision?

All these things come together because everyone is involved. It is a lens. We can put all these elements together and apply them as a lens that is incredibly instructive to what we do. You can see all those elements in these 14 points of management before. We can see that it was present in what was presented to the Japanese. This consolidated lens of the System of Profound Knowledge has the ability to be incredibly instructive. If you look forward at where we're going to be going in the next decade or two of what started off as a DevOps movement, I think that you'll have yourself a very profound sense of where it's going if you use these four principles in your evaluation.


See more presentations with transcripts


Recorded at:

Mar 03, 2020