All Things Containers From Solaris Zones to Docker
The 2016 Container Summit was held at the New World Stages in New York City. It was a multi-track, day-long agenda on everything containers and had about 400 attendees and speakers from the analyst, vendor, and user communities.
InfoQ’s Rags Srinivas caught up with Bryan Cantrill, CTO of Joyent, the day after the summit while he was still recovering from hoarseness.
InfoQ: Let’s talk a little bit about containers. Bryan, if you want to introduce yourself to the InfoQ audience, please go ahead.
Bryan Cantrill: Yes. I’m Bryan Cantrill. I’m the CTO of Joyent. We just had a great day yesterday at the Container Summit talking all things containers. It was a new experience for me as the emcee. I had a good time.
InfoQ: Can you provide a brief history of containers and your association with them, starting with Solaris Zones perhaps?
Bryan Cantrill: I was with Sun back in early 2000s and a bunch of us had different ideas for things we wanted to do with the operating system. For example, Jeff Bonwick had wanted to pursue a new file system. That became ZFS. I wanted to pursue a new way to debug the system. That became DTrace. A couple of other engineers, led by Andy Tucker and Dan Price, were inspired by Jails from BSD and wanted to take that to the logical extreme. And so they did Zones.
Zones, DTrace, ZFS, SMF (Service Management Facility), the Fault Management architecture — all these things happened in parallel. Zones was kind of happening off to the side and we were all inspired by one another and kind of working together.
Adam Leventhal and I, really led by Adam, did some work with Zones and DTrace and really began to appreciate the possibilities of OS-based virtualization. We did that work in 2003 and 2004. Fast forward a couple of years, and I was a huge believer in Zones. Just as a bystander, it seemed like it was really valuable.
I led kind of an entrepreneurial life working in Sun to build a new storage product. We shipped that in 2008 and it was very successful — too successful. We drew the attraction of Oracle to a degree.
In 2010, Oracle bought Sun and I realized pretty quickly that I needed to leave. As I was looking around, I obviously felt that cloud computing was taking off. Even though I had not been using Zones in the intervening six years, I remained a true believer. The company that stood out to me was Joyent because Joyent had been deploying containers in production all along and had deployed Zones in production starting in 2006. So I came to Joyent to help develop containers and Zones. It brought us up to where we are today. I would say I’ve had a longstanding love affair with Zones.
InfoQ: There are a number of different companies associated with containers but Docker popularized containers to an unprecedented level. What is it about Docker that makes containers so popular and so easy to use?
Bryan Cantrill: I think the key insight is that Docker connects containers to the developer. At Joyent, we were longstanding believers in containers but we believed in their operational advantages, for performance, for tenancy, which are important. I think the reason that containers didn’t catch fire until Docker is because Docker connected containers not to those things but to the speed and ease of development, making it easier to develop software. And that was the real revolution. I’ve quipped before and still firmly believe that Docker is doing to apt, the Debian Ubuntu packaging system, what apt did to tar. We still use tar, cpio, whatever, under the covers but people think in terms of apt. Well, now people are thinking in terms of Docker and Docker images, and it’s kind of taken off from there. It’s really the developer revolution that Docker has affected more than anything else.
InfoQ: That leads me nicely to the next question. Buzzwords in the developer territory are DevOps, microservices, and so on. Does the term "containers” help at all?
Bryan Cantrill: Oh, yeah. I think the interesting question is to what degree the rise of DevOps, rise of microservices, and the rise of containers are feeding each other. I think they’re all related. If you take away one, certainly if you take away two, I’m not sure you get the rise of the third. So all these things are important. I think that microservices force us to think smaller — it’s that small systems revolution that we’ve seen for so many decades and it forces us to think small. Docker is a good way to package software in a small way. And of course, the whole DevOps revolution behind this kind of cross-operational empathy really forces developers to think about production and vice versa. Docker and microservices are great tools for that. I think all these things serve to reinforce one another. They’re all very much related.
InfoQ: To use containers, microservices, and all that, there’s a lot of infrastructure to get running like Kubernetes, Mesos, Docker Swarm, or whatever — the whole ecosystem. On the other hand, PaaS (platform as a service) takes an opinionated implementation. Do you have any opinion as to containers versus PaaS?
Bryan Cantrill: Clearly the line is blurring, and actually I saw a funny tweet this morning that had the cliff between development and production and in development it has just a little Docker, and then in production it has Docker at the bottom and then libnetwork, Kubernetes, and all these things that you kind of ramp up on.
InfoQ: The things that you don’t need in development.
Bryan Cantrill: These are the things you don’t need in development. It’s apt and it’s a real challenge. There are different perspectives, different folks taking different approaches. Our approach has been that you need to compress those layers and make it as simple as possible. Production should be every bit as simple as development without being simplistic, providing the power that you need in production without giving up the simplicity and elegance that is the real appeal of Docker. But I think there are a lot of challenges about how these layers look.
The boundary of what PaaS is is blurring. I think you can deliver a PaaS using Docker as a vector. The power of PaaS is that the developers can develop high-level software with it but the PaaS often locks in developers to the abstraction of the PaaS selected. With the rise of Docker, I feel that developers can DIY their own PaaS. They can share that with other people. Traun Leyden from Couchbase calls this “software packaged as a service”, SPaaS, and I do feel it’s kind of spastic.
Take something simple like ZooKeeper. ZooKeeper is an infamous mess to get working. The word on ZooKeeper is that it’s great once it’s working, but a real bear to actually get working. I think even ZooKeeper proponents would acknowledge that it could be difficult to configure. And yet, you watch people with Docker Compose or what have you being able to immediately hydrate a ZooKeeper cluster. That’s pretty powerful. The ability to take a distributed system and then use this high-level tool and to hydrate it is blurring the boundaries of PaaS and I think that we’re going to have lots of people choosing their own PaaS, getting the power of PaaS without the lock in.
InfoQ: I think your opinion was, how do I put it… diplomatic. But let’s talk about the other part of it, which are unikernels. I know you have a strong opinion so go ahead and let the audience know what a unikernel is.
Bryan Cantrill: Well, this is the key because I think that people don’t really understand what a unikernel is. And I think that when you do actually understand what a unikernel is, you don’t actually need me to say anything.
There’s this idea that a unikernel eliminates the operating system. People have this idea like I’ve got my application and I’ve got the operating system sitting beneath it and then I’ve got virtual hardware and underneath that virtual hardware is the actual hardware. If I tell you I’m going to eliminate the operating system so that my application runs right in the hardware, it sounds faster. It feels like I’ve eliminated this bureaucratic layer. I feel like I don’t have a curfew. It feels liberating. But that’s because of the way I phrase it: “eliminating the operating system”. How does it sound if I instead say that you’re going to implement your application functionality into the operating-system kernel and that we’re going to pull that application functionality in kernel, and instead of being an application developer, you’re going to be a kernel developer? That doesn’t sound as good. You think, “Well, wait a minute. I’m not a kernel developer. I’m going to have to interact directly with the hardware?” Yes, you’ve got to interact directly with the hardware. That is what a unikernel is.
A unikernel is not eliminating the OS layer, in that you still need a layer that’s going to talk to the hardware. It’s just renaming that application to be the operating system. Your application is now the operating system. Your application now has the power but especially the peril that the operating-system kernel has. And that means that you need to know how to debug an operating-system kernel. You need to understand that the operating-system kernel is responsible for constructing the illusions upon which we have built modern computing. Those illusions don’t exist if you’re the operating-system kernel. There is no memory protection. There are no processes. I think I was a bit shocked to learn how many people were surprised that there are no processes in the unikernel. Of course, there are no processes in a unikernel. You are the operating-system kernel. You are the process. You are everything. As an operating-system kernel, you may create the notion of a process. We chose to do this in kernel development many, many years ago.
The idea of pulling applications in kernel very much upends 40 years plus of progress in operating systems. It takes us back to DOS days and I feel that these kids are too young to remember DOS. When you were running things on DOS, machines rebooted all the time for no good reason. An application had a memory bug and had corrupted the operating system itself, had corrupted another application, and things would reboot. Sending us back there is really, I think, irresponsible. So yes, unikernels are unfit for production, which is the title of my blog entry, which got a little bit of attention. But it’s not because I said so. I think there’s kind of this idea that I’m being mean to unikernels. Actually, I’m just letting people know what a unikernel is because they don’t understand what it is and what it means.
The thing that I find galling that I did not get into the blog entry is that some people even confuse a unikernel with a microkernel. The reason that’s so galling is because unikernels and microkernels live at opposite ends of the spectrum. The whole idea of a microkernel was that because it’s so difficult to develop software in the operating-system kernel, we want to put as much into user land as possible to use the protection boundaries as much as possible. Unikernels discard that protection boundary entirely. The idea that it’s just like an embedded system ignores why we use microkernels in embedded systems — because embedded-systems development is actually really hard. People who think that we solved this problem because of embedded systems make me want to flip a table because I think they don’t actually understand embedded-systems development. Anyway, suffice it to say that I don’t think that unikernels are going to be in production anytime soon.
InfoQ: The Docker ecosystem is evolving quickly. How can developers cope with this landscape, which is shifting so quickly.
Bryan Cantrill: There’s a lot out there. There’s no doubt about that. I don’t know that there’s an easy way to cope with it. It’s like going to New York City for the first time. It’s easy as a developer to walk around and look at everything going on. And if you spend significant time in New York, you figure things out. You’re overwhelmed at first but ultimately you’ve got get to work. You figure out the subways and you figure out the trains and the cabs and everything else.
Yes, it is overwhelming. There’s no way to sugarcoat it. There’s a lot going on. But I think that as a developer, you want to focus on making incremental progress. You can’t just wander around looking at skyscrapers all day. You have to get to the point where you actually figure this thing out. You don’t need to figure out everything. You need to figure out your little corner of the universe and begin to build that and understand it, and you can navigate the space better.
At the same time, it is incumbent on the industry to provide the maps to this place. It’s more difficult than it needs to be, for sure. I’ve been involved with Cloud Native Computing Foundation (CNCF). And my hope for the CNCF is that it can help developers navigate some of this space. The problem is not just Docker; it’s Docker and the substrate that it need to run on, be it Kubernetes, Mesos, Triton, or whatever. There are all these different technologies out there. The good news is they’re all open source. It’s a liberating era in many ways. You and I remember proprietary software where you couldn’t look inside to see how it works. Now, in the era of proprietary software, things tended to be a lot more polished because you paid for it. That cuts both ways.
InfoQ: There’s really no shortcut.
Bryan Cantrill: There’s no shortcut but it should be easier. I don’t think you can sit around and wait for the industry to make it easier. I think you’ve got to wade in but at the same time know that we as an industry are trying to make it easier. We are trying to make some tangible steps forward.
InfoQ: : I like the metaphor of Manhattan, which is small but daunting to master. Tell us what you learned at the conference and how containers are going to progress.
Bryan Cantrill: I think that we are seeing things settle. I think people understand that this is going to happen, that this is not a speculative future. The reason that it’s happening goes back to what we earlier said about Docker. It is about developer velocity, developer agility. I thought we heard that so many times yesterday in so many different ways. We heard that from the analysts — Forrester’s Dave Bartoletti had a great opening keynote to set the stage. We heard that from Hudson’s Bay Corporation and Saks Fifth Avenue. We heard that from all these panels. We heard from Jet.com and all these guys that need to move fast.
Andreessen was right: software is eating the world. I quipped that containers are the gullet by which software is eating the world. I think we’re seeing this democratization of infrastructure and many more people having to think in terms of infrastructure. We heard that again and again. We heard that there are cultural challenges. These organizations are effecting change. How do you do that? How do you do that successfully? How do you do that so that you are effecting change, not inflicting change? There’s a very fine line there. I think that, fortunately, organizationally, people are so accustomed to change now that I don’t feel that many really resist it.
I thought it was interesting that whether people are concerned about security or not for containers has no meaningful difference on their speed of adoption. Even people that are concerned about security are finding a way forward. I believe Canadian Tire said in one of the panels that their security folks are excited about containerization because it gives rise to immutable infrastructure and allows for a more secure future. This change is happening. People are asking how they can use this change to effect some of the things that they would like to see, be it from a security perspective, a performance perspective, or an operational perspective. The future is bright in that I think that we are going to get there although there’s still that middle ground of “But how are we going to get there?” and it’s so overwhelming. So we’re kind of caught in this interesting period when everyone knows it’s going to happen but we don’t exactly know how it’s going to happen. It’s overwhelming right now, which makes it an exciting time.
You and I have seen a bunch of these revolutions like Java in the mid ’90s, the rise of service-oriented architectures in the early 2000s, and the rise of frameworks in the mid 2000s. Whenever the right thing comes along at the right time, you get new energy that suddenly feeds into it. And there’s a little bit of chaos, for sure, but it’s exciting, and I think that’s where we are right now. It’s chaotic but it’s exciting and I think that everyone agrees that we have to get there and support developer velocity.
InfoQ: One last question I wanted to stay away from but you brought it up. Everybody talks about container security. Is the fear that a rogue application or something can break out of the container and kind of take over the kernel well founded or a little bit overblown?
Bryan Cantrill: The Linux kernel substrate that these things are built on, cgroups and namespaces, were never designed to meet security needs. I think Docker just this week released the first version that really uses user-level namespaces, that enables any kind of security and it’s not necessarily on by default. I do feel that there are issues. Our angles are very different and — full disclosure — we don’t use the Linux kernel. We use the SmartOS kernel, we use zones, and we run that Linux binary stack inside of a zone, which we’ve been running in production for a decade. So I’ve got a high degree of confidence in that. I think that it’s not just a matter of the ability to break out. It’s the ability to impact service as well. What happens if a container exhausts disk space? What happens if a container exhausts DRAM? What happens if a container consumes all the bandwidth? It’s resource management as well. I hope that the Linux kernel gets there, from the perspective of all of us. In the meantime, we’re not waiting. In the meantime, we’re taking a different angle and the substrate we know has all that stuff. We’ve added Linux binaries to and allowed people to deploy Docker in production that way so they can run containers on the metal and manage containers directly.
So I do think that the multi-tenant security is not there yet. I think it’s a challenge to get there because there is a “you first” attitude. Because you’re one vulnerability away from real devastation, everyone wants someone else to go first. That said, I think that we’ll certainly get there.
Videos of the Container Summit’s keynotes, panels and technical talks are available at its website.