Bio Amir works at Docker and was previously Programme Manager in the OCaml Labs group at the Cambridge Computer Laboratory. Most of his time is spent on open source efforts and he's a big fan of automation (testing, deployment, etc). He's previously been involved in a number of startups and has a diverse academic background with an MSci in Physics and a PhD in Neuroscience.
CRAFT is about software craftsmanship, presenting which tools, methods, and practices should be part of the toolbox of a modern developer and company, and serving as compass on new technologies, trends.
Hi. I work at Docker. I primarily work on things relating to Unikernels and I'm based in Cambridge, England.
Docker helps developers build, ship, and run applications and they do that by providing something called containers as a service. So they have multiple products there, containers as a service is something that sits between infrastructure as a service and platform as a service, trying to hit that sweet spot by giving people something that they can use but that is not too opinionated.
Essentially it gives people a bit more of control over how they want to set up their environment and the developers don't have the ability to containerize their applications. So dealing with infrastructure is sometimes too low level and dealing with the platform is sometimes too opinionated. If their opinions don't match your own opinions, it can be sometimes a little frustrating. So containers as a service sits in that sweet spot.
So yes, it's right. I'm still working on Unikernels. I'm very happy today. So one of the things that's important to think about is that essentially where do Unikernels and containers sit? We see them as sitting on the same continuum and that continuum is one of specialization, some degree of isolation. So containers is something you can use to help package up your application and share a kernel across the containers. And that makes it easy for me to develop essentially a great application and deploy it. But you can think about the progression from that, from a full traditional OS to containerized applications and what comes next, what is further along on that spectrum of specialization. And that's where Unikernels sit.
So it's like an extreme form of specialization which is something we have already talked about for many years. So you essentially take the components that you need, your application, the components you need, compile that down into an image that you can deploy. So given these things are all on the same continuum, it just made sense that the same tooling should work for all of them. So that's why it makes sense for Unikernel Systems and Docker, Unikernels themselves and containers to work together.
Unikernels are definitely in a way more efficient. So I'll describe briefly what a Unikernel is. So there are two broad categories of them. One is the -- I also call them library operating systems. So Unikernels are the artifacts constructed by using library operating systems and we can come back to what that means a little bit later. So one of the primary benefits is everything is a library. So you have a collection of libraries that's the same way that you have when you are building any other kind of applications. It's just that these are system level libraries. They represent things that the operating system would be doing for you like networking or file system stuff, storage and that kind of thing and you just link against only the components that you need from that OS. Compile that in with your application code and then produce a single image, a single purpose binary that is just your application code and only the things that needs to run and then you deploy that out. Does that make sense?
And something else, to finish the thought, was there were two broader approaches to creating these library operating systems. I talked about it yesterday. Essentially, there's one approach where it considers legacy, another approach that is clean slate. The legacy approach is to say let's take existing code. Let's say for example Rumpkernels uses NetBSD. So you take NetBSD, break it up and make those components available as user space libraries. So you can take a NetBSD TCP stack, for example, and use that as your code to produce your Unikernel. So that's one approach. There are several ways, very several projects that are trying to take that kind of approach.
Another category is clean slate. Clean slate is where you essentially say you are going to throw out all the old stuff and build things from scratch. It's a little bit scary but there are lots of benefits to doing so because you start to benefit from language specific features. So Ocaml, which is used for MirageOS, the one I'm most familiar with. There's also Haskell for HalVM, Erlang and LING, what you have to do there is you rewrite those operating system components in that language from the ground up and this process can take a while but there are lots of benefits of doing so. And then your application code then also has to be in the same language but you can start to benefit from optimizations throughout that whole stack and you can also do away with things like POSIX compliance because you then just present developers with a clean API to use and that's much more pleasant for them to work with.
Werner: So you don't have to use 50-year-old, 40-year-old APIs which might be unsuitable to your application.
Well if you don't want to.
There are. So I mentioned library operating systems. So everything is a library. So you are using libraries to construct the Unikernels but the libraries themselves are also extremely useful because you now have a bunch of stuff you didn't have before that can help you solve new problems. So one of the places that using these libraries has been really, really useful is one of the products that we released through Docker which is Docker for Mac and Docker for Windows. So Unikernel libraries, Unikernel technology underlies that product specifically -- like one specific example is the MirageOS TCP stack is part of that product because it helps deal with mapping the networking in the container and the machine and the Mac host side. That wouldn't have been possible without the collection of libraries, MirageOS has produced. So even though you're not necessarily building Unikernel artifacts everywhere, those libraries have been very, very useful.
Werner: What's an interesting point in that -- it's not just building new libraries for fun. It's useful for other things than Unikernels basically.
Exactly. And this collection of libraries is actually where a lot of the value and a lot of work also lies producing these libraries in a way that makes them reusable.
Yes, there are lots. One of the great benefits is that everything -- you have now a collection of libraries that you didn't have before and those libraries can be used with many different things. One would be the best example is that I like to point to now is Docker for Mac, Docker for Windows which are right now in beta that Docker announced back in March. And what we have done there is taken some of those libraries that are part of MirageOS and use them to help solve problems so that we can ship that product and these were -- we may not be able to ship such a product without these libraries. They are that critical. So one of the examples that have been pointed to is the MirageOS TCP stack which helps them deal with networking between what's going on in the container of the machine and the host side on the Mac. That's really exciting because it just demonstrates that once you do the work of producing all these libraries, they are actually usable for many, many different things.
8. So moving to Unikernels in general, is anybody actually using Unikernels in production yet or is it mainly, besides Docker on Mac, but have people started using them? Do they build their companies on them?
Ooh, interesting question. It kind of depends on what you mean by production. That means different things to different people. So I can argue “We totally use MirageOS in production. Look at our website.” Mirage.io is a Unikernel and a bunch of other people who are on the team and outside the team deploy their static websites as Unikernels. Plenty of other people would argue that that's actually not production, it's a hobby. I would say that I can say that there are production users with Unikernels. One that I know of from HalVM is a product called CyberChaff.
So I was chatting with Adam recently and he essentially told me yes that there are production users of that product, a security product where essentially you distribute a bunch of Unikernels on a network and it acts as literally as chaff. So if you have a network intrusion, all these extra services distract and confuse the person who is trying to attack and break their way into things and during that it will alert the network operator that an attack is in progress and that's possible because of HalVM Unikernels. So that's an obvious production use. They haven't really released much information about who is using it or where. [Editor's note: since the interview there's been more news about CyberChaff: https://galois.com/news/cyberchaff-reed-college/ as well as http://unikernel.org/blog/2016/halvm-cyberchaff ]
Well there is nothing to actually attack. When you have these little small services presented on a network that look like something else and someone tries a scan. Well why is anyone making a connection to that Unikernel? There is no need for anyone to be doing it because it's not really providing any other kind of service. So when some connection is made, that connection is probably coming from someone who shouldn't be in your system.
Yes, that's right. So Adam Wick is the lead developer there and based at Galois up in Portland.
Werner: So where can people start to get more information or to put it differently, is there a simple way for me to get started without having to build a lot of stuff and dig through lots of readmes etc.
Well there is a couple of things. One is we are putting together a site called unikernel.org and that's meant to be a place where everyone can go who wants to learn about this stuff to gather more information. It will also become, I hope, a place where the projects themselves will interact with each other, interact with the community that's growing and share what they are working on because at the moment we have multiple different projects with multiple different sites, multiple ways of building and doing things. So trying to coalesce the ecosystem side under one banner is going to be of help for everyone.
The other thing is I think Docker for Mac is going to be one of the best ways to try and get started. I mean Docker in general is very useful but now Docker for Mac, Docker for Windows. It's really easy to start using containers, the simplest way to start using containers on your native machine. One of the things that used to be awkward about our projects is essentially getting the environment set up so that you can actually start playing Unikernels. You can start building a MirageOS Unikernel. And we actually now have hub images. So Unikernel/Mirage contains images that people can pull that container down, everything they need is right there and they can actually build the MirageOS website with two or three commands.
So we can do away with the 30-minute setup process that involved doing brew install this and canfigure this file and then install this other thing. All that goes away with containers. It makes getting started much, much easier and it can literally become minutes. So I think looking at the different projects that are out there, using their containerized way of getting started and then basically playing around with that. So that's something I showed people in my talk yesterday was essentially using Docker for Mac to build the MirageOS website in a container and showing that that process takes minutes.
Werner: So I find that really exciting because that has kept me from doing too much with Unikernels because it's you say 30 minutes, for me it will be three hours of “Did I miss a slash here or there?” So I think our audience now has -- and I myself have no excuse not to check out MirageOS or HalVM or DockerHub. Thank you, Amir.