Distributed Systems programming is not for the faint of heart, and despite the evolution of platforms and tools from COM, CORBA, RMI, Java EE, Web Services, Services Oriented Architecture (SOA) and so on, it's more of an art than a science.
Brendan Burns outlined many of the patterns that enables distributed systems programming in the blog he wrote in 2015. He and David Oppenheimer, both original contributors for Kubernetes, presented a paper at Usenix based around design patterns and containers shortly after.
InfoQ caught up with Burns, who recently authored an ebook titled Designing Distributed Systems, Patterns and Paradigms for Scaleable Microservices. He talks about distributed systems patterns and how containers enable it.
InfoQ: Let's get straight to it. Distributed Systems have been around for decades. What is new about the book? If I am a programmer or an architect in an enterprise, give me a good reason to pick up this book apart from the price tag (free)?
Brendan Burns: You know, I think it takes a period of time for books to emerge that codify the patterns that we've been seeing for years. And usually it's keyed off of some transformational technology that makes the patterns easier to talk about abstractly. If you look at something like the Gang of Four book, it emerges nearly ten years after the development of object-oriented programming, but clearly that book (and all of the patterns books of the 1990s) are inspired by object-oriented programming.
For me, containers and container orchestration were the transformational moment that inspired me to write this book. I thought "Finally we have common language and vocabulary where we can discuss these patterns" and that's why I wrote the book. Of course, we have all of these newcomers coming to cloud and distributed systems, so that's part of the motivation too. People need to learn the lessons via a book, not by building broken systems.
InfoQ: In your keynote talk at Kubecon 2017 you talk about how to make distributed systems programming a challenge as simple as getting through a CS 101 class. So, I have a two part question; first, did you ever imagine that Kubernetes would gain such widespread adoption so quickly as evidenced in Kubecon?; and second, how do containers and Kubernetes help in programming distributed systems?
Burns: I think as founders, Joe, Craig and I are really humbled by the level of interest and usage of Kubernetes. I think we always hoped that it could really help lead to adoption of cloud native patterns and practices, but it's still really amazing to see it come to pass. And to stand up in front of all those people who are betting their business or applications on Kubernetes, it's an amazing amount of trust, but I think it really speaks to the quality and commitment of the whole Kubernetes community and ecosystem. It's really a worldwide, community effort. When you think about all the things that divide us, to be able to come together around this software, it's pretty inspiring.
I think the goal for Kubernetes all along was to be an autonomous system for distributed systems. I like to draw the analogy between a manually flown plane and a quad-copter drone. The quad-copter is way easier to fly, because it's got an on-board controller to keep it level, and you just tell it where you want it to go. Kubernetes is the same way; you tell it what you want to be true, and it's responsible for keeping the right number of replicas, reacting to application crashes or machine failures. It's like anti-lock brakes for your distributed application. A computer in the loop to react automatically and keep you safe. Additionally, I think with things like service load balancers and the like, it really makes it easy to fall into the "pit of success" and implement your applications in a decoupled scalable way.
InfoQ: In your keynote talk at Kubecon 2017, you talk about democratizing distributed systems development with a cloud native runtime that you call MetaParticle. What exactly do you mean by this? Is Istio a good example of this?
Burns: Well, Metaparticle is really a way to re-think how we design and build our systems. Right now we write code for each machine, and then we use these YAML files to describe how all those individual programs fit together. Along the way we have to learn and use multiple different applications, file formats and languages to describe our application. It's too complicated and it's too hard for new developers to pick up and use. Metaparticle is an attempt to say: "Can we do this all in a single programming language?" It's really an experiment to allow more inexeprienced developers to translate their skills into cloud-native patterns. I think Istio is different; Istio is really about making existing systems more observable and dynamic. It's really a tool for people who already know how to build cloud-native systems, but want it to be easier to monitor and adapt them.
InfoQ: As suggested in your talk, if languages have to provide cloud native bindings to make development easier, they're probably years away from doing it in a standardized form. Are you suggesting that it's just a matter of time before this happens? In the meantime, will the ecosystem provide tools and platforms for solving some of the challenges associated with distributed systems development?
Burns: Yeah, I think there are really two different classes of solutions. There are things that smooth existing experiences and take away pain. Tools like Helm and Draft as well as all of the VS Code integration we are doing are great for that. I think we'll see a steady evolution of those tools to make Kubernetes easier and easier for people who are already cloud native developers.
I think there's a whole separate class of things that are transformational and empower a whole new class of developers to become cloud native. Think of it like a really great IDE for C++. It can't make you a good C++ programmer, but it can make a good C++ programmer really happy. Then there are languages like Java or C# that just empower a whole new group of developers. I think both things are needed in this space.
InfoQ: Almost everyone who has programmed distributed systems has been bitten by making incorrect assumptions, for example, as postulated in the eight fallacies of distributed computing. Do the distributed design patterns explained in your book help alleviate some of these issues, and if so, how?
Burns: I think that it helps you apply patterns that build more robust systems, and they definitely talk about the design considerations you should think about when building your system. I think the other thing, hopefully, is that it gives people a common language to discuss their systems, so everyone isn't learning the same lessons over and over again, but instead we're all learning and teaching each other about the same patterns. It's not magic though, people do still need to learn a lot through hands-on experience.
InfoQ: A lot of talks at Kubecon were aimed at making development boring (it should just work). Where are we as a community today? If you can let your fantasy run a little bit wild, how will this look about five years from now compared to where we are today?
Burns: Well, I really hope that talking about Kubernetes is like talking about Posix, or the x86 instruction set. It's something that's worth doing, and it's something everyone kind of has to learn to help get their jobs done, but it's not really a source of excitement. I hope it's kind of like talking about garbage collection, or object-oriented programming. It's an important tool to help get your job done, but it's not a novelty, not something where there are fans.
And I also hope we look back and we say "wow, look how much more we got done because we had this great abstraction layer to build on top of" because Kubernetes isn't the end; it's a great foundation, but we have so much more to build on top.
Code samples discussed in the book are available for download from Burns' GitHub repository.