Transcript
Felipe Huici: My name is Felipe. I'm the CEO and co-founder at Unikraft. We build cloud platforms or we allow people to build cloud platforms on our software, that hopefully scale to a lot of things where security is pretty good and where VMs fly. That's the idea anyways. Let's see how far we got there. This talk obviously is going to be focused a little bit more on sandboxes because that's the cool thing. Sandboxes used to be a container primitive. These days it's about all AI. The term has been repurposed if you've been around long enough, but we'll get to that.
To get started, how fast can a virtual machine cold boot? Anybody want to take a guess? I put some options. I said I have time for lunch, I have time for coffee, a few seconds, and a few milliseconds. Of course, as I'm trying to preach that they can go fast, I'll say a few milliseconds and then we'll see if it's true. Another one, how many virtual machines can I put in a 48-core server? I wrote, 1 Ubuntu-based one, 10s of them, 100s if I'm feeling adventurous, or if I could somehow manage to scale them to zero when they're not being used, maybe I could get even more density. What provides the best multi-tenant isolation when you go to the cloud? Some options, virtual machines, containers, language-level runtimes, unplugging the cable to the server. Of course, that's the correct answer. Presumably, the cable does something useful for you, so you got to plug it back in, and at that point, I'll try to convince you that virtual machines are the way to go.
Long and Winding Road Down Cloud Scalability Lane - Isolation Primitives
I've themed this, the long and winding road down cloud scalability lane. We're in London, I'm a Beatles fan, so a reference to the Beatles. First, to give context and to make sure that what follows makes sense, I thought I'd take a little tour down isolation primitives road, because you have things like containers and virtual machines and unikernels and microVMs and isolates, and it's just basically isolation soup. To the point where people are just like, "I don't care. If it's isolating, I don't need to know the details." I thought I'd give a bit of isolation primitives in a nutshell with a few diagrams, because I do come from a research background and I have to do diagrams, so otherwise I don't feel comfortable. Basically, for a standard virtual machine, and obviously a bunch of you probably know this, but let's see, you start with a server.
On top of that, you install something called a hypervisor, which you can think of it almost like a very tiny operating system that does resource sharing, just the basic CPU RAM, so that on top of that, the virtual machines can come. You can have something called a host OS, which is typically Linux. For those of you who know a lot about virtualization, you'll call me out and say there's a Xen hypervisor, there's other architectures. I'm simplifying here. In the host OS, there's something called a VMM, a virtual machine monitor. You may have heard of QEMU or Firecracker. These are things that actually launch the virtual machines. Then on top of that, because we have a host, obviously we have a guest, and that's the actual virtual machine, typically split into a kernel, like the Linux kernel, and then the distribution, the applications, everything else that comes on top.
That's the VM world. How many people have heard the term microVM? A microVM is exactly the same thing, it's just the VMM, that little tiny box that launches VMs, it's Firecracker. That's it. That's the only difference. It's just the Firecracker team decided, because we created a VMM that launches things faster, we need a new name for the VM, and it's a microVM. Because they did that, you can have a VM that is a 16-gig Ubuntu, which is a microVM, because it got launched by Firecracker. How many heard the term unikernel? A unikernel is still a VM, it can be launched by Firecracker, so you can have a unikernel microVM, and the difference is what's inside. Inside, the idea is, if you have an application, like a web server, you build a custom operating system that just has the pieces the web server needs to run, and nothing else, and you mash the two together.
That's the main difference. Containers, different world, so there's no hypervisor, and what comes on top of the hardware is the host OS, so like the Linux kernel, for instance. On top of that comes a bit of a shim layer called a container runtime that hopefully prevents bad things happening to the kernel, but it doesn't always do that. Then on top of that come the actual containers, the actual applications. Then of course you can have fun, and you can combine all this stuff. First, let's talk about runtime isolation. Who's heard of isolates? Now you're talking about, you have something like Node or JVM, and that language provides isolation to the applications which are actually processes on top. Those are the three models, you have VMs, containers, and runtime isolation, more or less, I'm simplifying. Then you can have fun combining them. You can take a container and put it inside a VM, so the stack looks like the VM one until you get inside the VM, and then you have the container runtime and that on top.
That's what ECS, for instance, does, because containers, they're not safe enough, so you stick them in a VM to isolate them, so isolation within isolation. Then of course you can take it to the logical conclusion, let's start in the center, where you can have isolates inside containers, inside virtual machines, and you can have these massive stacks where you have redundant isolation that doesn't do much except slow you down.
Why Virtual Machines?
Why virtual machines? I went to Wikipedia. There's a definition of trusted computing base, which is basically saying all of your critical code is only as safe as the pile of software it's sitting on top of, and that pile of software is your trusted computing base. You want that to be as small as possible because there's usually a correlation with more code and security vulnerabilities. What does that look like for the models I just showed? For a virtual machine, the trusted computing base, it's what's common to all of the different virtual machines, all of the applications, that's the critical part. In this case, it's the hypervisor, and the idea with the hypervisor is that hopefully, it's smaller, leaner than an entire full-fledged operating system. In the container world, there's no hypervisor, there's the host OS, so now you're talking something like the Linux kernel, tens of millions of lines of code, and so that is actually common along with the runtime to every single of the different applications that are supposed to be isolated, so your trusted computing base just got a lot bigger.
Of course, you can see where that's going with language runtimes, where now your trusted computing base is not only the host OS, but the entire runtime of the language. Then the applications come on top of that. Let me ask you a question, what do you think the trusted compute base might be if I, for instance, stick a container inside a VM? It looks exactly like any VM because the trusted compute base, the layer that is shared by the applications is still just the hypervisor. Everything else is independent. Every VM gets its own OS. Every VM gets its own runtime. It looks inefficient, but it's very isolated, so the question is, can we make it efficient because we like that isolation? I'm being facetious here a bit. Containers don't contain, isolates don't isolate, and so VMs are the golden standard in the cloud.
Minimalistic VMs
Back to the story, rewinding. Several years ago, I was a PhD student at UCL. There was a UK research program that was supposed to build virtual routers, basically take software packet processing and make it go fast inside a VM. The idea was, ok, we're going to take a virtual machine, there's a hypervisor, of course, and we have a guest OS and a guest application, we want the application to do software packet processing, what can we pick? Of course, UCL has very close ties to Cambridge, and Cambridge was the Xen hypervisor, if you know what that is and you know those people. They started telling us, we have a very small reference OS called mini-OS, you could take that. In our case, we took something called the Click Modular Router software out of MIT, and you can mash them together, and now you can have VMs that go fast and are small.
Fast forward a few years, we published a paper called, "ClickOS and the Art of Network Function Virtualization," in which we started showing you can go at 10, 20 gigs per second in software out of a virtual machine with this. Of course, that was fast then, these days you should probably add a zero for that to be impressive. We also started doing measurements into the fact that the virtual machines themselves, they were small, and they could boot in a few milliseconds. Then you could have more than the 48 Ubuntu ones, this was in the hundreds. It was giving a taste of what could be possible. Fast forward a few more years, and the Docker wave came over, and Docker came to save us all. It was the silver bullet, perfect isolation, everything milliseconds, and we're off to the races. Except anybody in the security field started obviously sounding the alarm of, this isn't so secure, and maybe it's not so fast.
Of course, we did a hot take white paper, "My VM is Lighter (and Safer) than your Container," in which we hacked Xen a lot to try to push it to have a lot of VMs in a server, 8,000 in this graph, and try to have the cold boot times be in milliseconds. Then we compared against the container runtime, and you can see that we're trying to say, you can have the isolation. If you're very careful about the engineering, you can also have the performance. That paper did pretty well. It got about 28,000 quotations, which in research world is like TikTok level of buzz. Then it hit Hacker News some years later. Then from there, we moved on to building other custom virtual machines. This was packet processing. We did one that was called Minicache for doing caching, Minipython for Python. Obviously, we were lacking in the creativity department for naming, but that was going well, except we were going to conferences and people were going like, that's great, can you run this other application or this other service?
We were handcrafting all of this. The answer was always no. We started thinking, should we be doing a bit of an SDK for building these specialized virtual machines? That took us to the creation of an open-source project from the Linux Foundation, again, through our Xen friends who had an incubator program, who said, you guys should apply for the incubator program. Then through that, eventually, you can launch off as your own standalone open-source Linux Foundation project. That's what we did. It's been seven years of building an operating system since then, more or less. I'm not going to get into the details of it, but partly because we wanted it to be Linux API compatible or as much as possible so that unmodified applications could run on it. Fast forward several years, we did another paper to summarize the work of that open-source project called, "Unikraft - Fast, specialized unikernels the easy way." Only 8,000 citations this, so small buzz. Then Hacker News again. That was the end of the past history.
Key Requirements
Now moving forward to more recent history, 2022 or so, we started thinking, that's great. We've gotten those images to be small and lean, and we've proved some proof points. What does a cloud platform look like if we were to apply all of these principles over? We wanted efficiency, scalability, and security as first-class citizens. We didn't want to compromise on any of those. We had some key requirements. Obviously, the theme of the day is we wanted strong isolation for all workloads, but we also wanted things to start really quickly. Why really quickly? So that people don't notice that it was ever off. When things are idle, though, we should be able to shut them down because they're not doing anything useful, and then wake them up just in time. If you can wake it up quickly, maybe the users don't even realize that it was put to sleep.
When they do get resumed, we probably want to resume from where they left from. They shouldn't lose any state because otherwise the users will notice the hiccup. Another thing we wanted to do is run a new workload. If you can define it in a Dockerfile, it should be able to run. Functionality is king. Nobody will listen to you about performance if it cannot run what they want to run. Functionality first, performance second. Then, of course, high server density. How many of these things can we cram onto a server? We'll come onto that. The first thing we did is we took a Dockerfile, and we have a CLI, which is open source, which is Unikraft, and that extracts out of a Dockerfile the file system and the binary, and turns that into a virtual machine, an AMI in this case, because we tried on AWS first.
That creates an instance, so a virtual machine. Then we were expecting 10 milliseconds because that's what we were getting on our local servers. Instead of 10 milliseconds, who wants to guess how much we got when hitting the nice dashboard button on AWS? Start fast, please. We got about 30 seconds. Obviously, that was a very naive approach, and probably most people in the room were like, why did you even bother clicking that button? It turns out, obviously, that a cloud platform is not just a fast image. It's a lot of other components that are in the middle. There's a load balancer when you send a request first. There's a proxy. There's a controller that does VM lifecycle management. There's a virtual machine monitor, like Firecracker, that actually launches the VMs. Then there's the images. What we wanted to do is make sure that that entire chain and back to the user would be in milliseconds.
We had to start applying some of that unikernel, let's be efficient about everything, to all of those components, start building components from scratch as well. The idea is you get a request, there's a proxy, and the first thing that happens is the proxy will just buffer the request. Will tell the controller, I have an ID for a connection. Do you know anything about it? The controller will say, for instance, yes, I do. It's something that used to be running. I have put it to sleep. I need to resume it. Firecracker, wake this thing up. If you know what QEMU is, we didn't use that because that's slow and chunky. If you know Intel Cloud Hypervisor, we didn't use that because when we started putting thousands of VMs on it, we started to get a lot of jitter on the CPU and we didn't like that.
Then the actual VM wakes up, and then we put a signal all the way back to the proxy, and the proxy says, ok, it's ready. I'm going to unbuffer that request. Request goes to the instance, and then the instance replies to the user, and hopefully, if you did that really fast, the user has no idea all of this mess just happened. It just sent a request, and the thing that was supposedly running just answered.
On the road to having end-to-end millisecond performance, we just said the load balancer is quick, the proxy is quick, the controller is quick, the VMM. What about the images? Because sometimes the applications, those pesky applications take 2 seconds to start or 10 seconds to start. The JVM, that's driving me nuts. Why does it take so long? How do we make it so that the application doesn't ruin our just freshly achieved millisecond semantics. The idea is to use something called snapshots. How many people have heard of snapshots? A snapshot is just a copy of the internal memory of the running VM at any point in time, and any virtual machine monitor you can request this of. This is the REST API call that you can ask Firecracker to create a snapshot of. It'll just take a picture, put it in a file, and then you have all of the contents of that VM at that one point in time.
What can you do with this? Imagine you have a CI/CD release that happens once a month, and then out of that, you send your image out, you start a VM out of it, and you're going to let that application take its nice time with seconds or minutes to actually start. When it's actually done and ready to service requests, we'll take a snapshot of it. Then what we're going to do is we're going to publish the service as ready, but now the actual VM is going to start from the pre-initialized state of the snapshot. We've masked all those two seconds, five seconds, whatever it may be, and now from the snapshot, the thing will start in about 10 milliseconds. Another thing that we can do is we can turn that snapshot into a template, and then out of that template, you can launch any number of virtual machines at once.
Snapshot power doesn't stop just at this. Obviously, it's the thing that allows it to do stateful scale to zero, so when you put something to sleep, it wakes up exactly where it left from. You can do forking, where you can have a virtual machine running, and you can say, fork me, so you can create things like sub-agents. You can do checkpointing, so you can have a machine, you can say, checkpoint it now, and now, and now, and then you can rewind to a sane state, for instance. Of course, you can do migration. You can have a virtual machine that's running and move it over to another server. Of course, the last one, application masking, is the one I talked about already. Images works as well, and now we want the millisecond cup. What about generic workloads? Unikernels, I said, are specific to applications. Things like sandboxes can run anything, headless browsers, and for that, we had to go and take the Linux kernel, but apply unikernel principles to it.
What's the tiniest, smallest Linux kernel we can grab. Then, can we do a distroless distribution, meaning there's almost nothing in user space, going to the extreme of the init process, so the first process that starts once the kernel is done booting. Use that to actually launch the application that is in the Dockerfile that people want to run, and so you get the tiniest possible Linux-based distribution and now you can run anything, and it'll still start very fast.
Going back to requirements, we can cross all of those off. We haven't talked about high server density yet, so let's do that. The idea with server density is, in a standard server with standard VMs, you're going to have not that many VMs. Some of them are green, they're doing useful work, and some of them are gray, doing no useful work, they're just idle. That's the standard pattern. The idea is, if we can scale them to zero milliseconds and wake them up, we can cheat, and we can say, anything idle, put it to sleep, and then we can cram a lot more things into a box. There are a lot of services where you have 1,000 users, but there's only 10, 20 active at any one point in time, and so the other 800 whatever users, you can scale them down to zero and stuff all of that into a server, and you can get much better density out of that.
How do you scale that density up, even if you're using scale to zero? We had designed a lot of the components to be scalable, so we were happily going up to 5,000, and then 10,000 of these things scale to zero, and then they would wake up whenever needed, but everything breaks at some point. The Linux host was pretty upset that we were using one TAP device for each of the VMs, and now the Linux kernel's like, what are you doing to me? Why am I having 50,000 TAP devices? The kernel lock was freezing up, and then services such as NTP and SSH were really unhappy. We were also running tail scale, and it was crashing when it was starting. We had to do a lot of tweaks to fix that. The number of ports you can put in a Linux bridge is also limited, so we had to do tweaks around that.
Then IPv6 lock contention and other things. Snapshots as well. You can take a snapshot, but that consumes disk. We tend to use NVMes because we want to have fast access to those snapshots. Storage is not infinite, so, of course, you want to do what's called differential snapshots, so not take a full snapshot every time, but just take the little differences as the instance is waking up and going to sleep. We had to develop a way to do compressed snapshots, but then when a snapshot is compressed, make sure that it still wakes up fast, and things like reference links to templates and so forth. Then, I'm not going to go into a whole lot of this, but the platform and the way the different components, the proxy talks to the controller and so forth, was getting really slow going through the network, and so we had to do all communications based on shared memory and not using any of the network protocols.
A little bit in numbers. This is a graph of what happens when you measure cold start times for 100,000 VMs. Don't worry too much about the fact that there's a lot of lines. Mostly, the purple line, that's the cold boot time, and then the top line is the time to first packet, let's say, just to simplify things. The main point is, to get started, it takes about 10 milliseconds. It's fairly constant as you ramp things up. A little bit higher for first time to packet, but still fairly constant. We also measured CPU, memory utilization. Of course, everything is scaling. It starts, it does stuff, it scales back to zero. In that state, almost no CPU is getting consumed. Then the controller is using about 12 gigs. That's because the controller sizes out a million VMs ahead of time, and then it probably uses about a few kilobytes of metadata per sleeping VM.
Then, of course, snapshot storage utilization is linear. We did all these tricks, compression and differential snapshots and so forth. When you do that, you would need about 12 terabytes if you wanted to get to a million. That's definitely feasible. Most servers these days have about two NVMes with 2 terabytes, the cheap ones, and you can also add more. Back to requirements. We knocked off the last one.
Use Cases
You may be wondering, what could this be useful for? Of course, sandboxes, and I'll talk about that in a moment. Headless browsers, how many people have heard of headless browsers? This is like the workhorse of an agent. When an agent needs to go retrieve something on the web, they use a headless browser, it's usually Chromium. The problem with Chromium is it takes forever to start. It can take 30 seconds or a minute, and so the only way to be reactive and have fleets of them is just to leave them running the whole time, except the headless browser consumes 4 gigs, 8 gigs, 16 gigs of memory. You can imagine now, how many of those can you stuff in a server when each of them needs 16 gigs? Not that many. The idea with this is, if you can scale them down to zero, they can just wake up in 10 milliseconds, they do the job, and then they go back down.
Functions, obviously. Build and test environments, same thing. If you can eliminate cold boots, a build environment sometimes takes 30 seconds and minutes to get started, and then maybe a minute to build. That's not really a good ratio. Then these other things. Sandboxes. What's a sandbox? Essentially, it's just a virtual machine, where you have full access and you can run anything in it. Obviously, you require strong isolation because who trusts agents these days. Those things are crazy. They can be used intermittently. You could be interacting with an agent, or an agent interacting with an agent, and then for a while it doesn't, and then it comes back up, so it'd be good to put them to sleep when they don't. As a result of that, you want to resume them fast, and you want to keep the state of what happened in the last session each time. Of course, if you're a sandbox company, you want scale, because there's a lot of these things going around.
Demo
Let's do a demo. Everything I'm going to show is based on Dockerfiles, so you can see it's just from nginx. I'm going to use nginx first to show how it works. I optimized a lot of what should go into the image, so it's small, but you could have used any standard Dockerfile. Then what I'm going to do with that is I'm just going to create one, create a virtual machine out of that, and I'm going to tell it to scale to zero. I'm going to give it the name nginx 0, you can see here. I'm going to put a watch on it. It's not ready yet, of course, so let's just start it first. Put a watch on it. You can see it says that it's on standby. Standby means it's scaled to zero, so it's consuming no memory, and it's consuming no CPU. What I'm going to try to do is access it.
It replied right away, and maybe a little too fast. Keep an eye on the standby. You should see it for a fraction of a second go to running, to reply, and then go back to standby. Every time I do it, it'll just go up, reply, and go back down. Let's try to do it a bit quickly, it should stay on running, and then eventually it'll just go back down. That's just the basic workflow. That's just nginx. That's a little bit boring maybe. We want to see sandboxes. Of course, sandboxes mean Claude these days. Let me just get this guy. What I have here, you can see on the right-hand side, the box is loaded with a ton of OpenClaw. Each of those is a virtual machine. All of them are scaled to zero. What I'm going to do is I'm going to call one up.
In this case, you can see it's number 42. You can see it's running. This just resumed. I had actually asked it this. Let's do this, is it sunny in London today? Weather skill. It went a bit overboard. I just wanted you to tell me that it's not sunny. It created a skill for me. That's pretty cool. You can see that it's running on the right-hand side. It's the green one. Then I just killed it, and then it just went away. Let's take another one. I'll just wake that same one again. You can see that it resumes. Let me just go to 41. 42 went to sleep. 41 is up. Let's see. What's QCon? This one is missing an API key. Let me just try a different one. What's the best UK food? Better say fish and chips or Indian food? What do you guys think?
Fish and chips comes first. Amazing. Super cool. Then, of course, as you saw, it restores. Then the thing I'm going to do is I'm going to try to wake them all up once, so I have a little script that just will go through all of them. It's just doing little queries, and they'll wake up, and they immediately go back to sleep when they're done. You get a little snake effect. That's how it scales. Then, final part of the demo. Here I wanted to show you on the box. That's 1024132 scale to zero virtual machines that are on that actual box. That's running a bunch of nginx instances. That's instance 939933 saying QCon Rocks. Yes, let me just pick it all, I'll say 111111, I'm sure I get the number right. I need to dash. There you go. That's the instance number 1 millionth running nginx and answering QCon Rocks.
Kubernetes
Kubernetes is chunky, non-millisecond, whatever. We were like, but Kubernetes is actually very useful. Many people use it for very good reasons, HA, self-healing, scalability. Super easy to get going, you just drop a Helm chart, you have infra, but Kubernetes does not operate in millisecond scales. Was there a way to have our cake and eat it to have all these amazing properties of Kubernetes and the fact that everybody knows how to use it, but retain our millisecond semantics? What we did is we built a virtual kubelet, so it drops in just like any node into an existing cluster. What we do is we tell Kubernetes, you can schedule pods on me, no problem. Kubernetes says, please schedule this pod, and we'll take that pod, and instead of scheduling a container, we'll actually schedule, of course, a microVM, because we are using Firecracker underneath all of this.
Kubernetes is happy, you scheduled the pod. We're like, it's scheduled. Then what we're going to do is we're going to tell Kubernetes it's running. Then when Kubernetes asks, is it running now? We always say, yes, it's running all the time, don't worry about it. Then under the hood, we're just going up and down, up and down. Because for all intents and purposes, if we can reply within those few milliseconds, nobody cares whether it's actually sleeping. That's how we try to pretend that we're going fast, or we actually are going fast, but still getting all the benefits of Kubernetes.
Takeaway
The final takeaway is in cloud infra, it used to be the case you have to pick speed or scale or strong isolation or two out of three. The idea is if you're very careful about how you do your engineering, you can actually pick all three at once.
Questions and Answers
Participant 1: How do you handle when there's multiple tenants, that is more than the resources your actual hardware is spinning up immediately in the same time?
Felipe Huici: Of course, everything has a limit, and the server may have 48 cores, and when the 48 cores are running at 100%, there's nothing you can do other than queue the requests. The idea is you can now dimension your system for the max number of active concurrent users rather than for your peak all the time.
Participant 2: I was wondering if you want to run AI agents in a sandbox, you want to have isolation to make sure it doesn't get root access, and I can imagine this approach would achieve that. Is there also a way to avoid your private code or your credentials being exfiltrated from such a microVM? Is there any network monitoring or restrictions possible?
Felipe Huici: First thing, it is possible that the bad agent maybe gets root access because it still has the Linux kernel user space divide, and so if it can crack through that, it'll get root access. It'll only get root access to its own kernel and not the kernel of anybody else. That's one level. Then the second level is, yes, you should never stick very useful, very good credentials. Don't give your credit card to the agent. I can't believe people are doing this. Like, the first posts I read this, I was like, is this a meme? People are like, no, this is for real. You don't want to give your Anthropic key to the agent. The best practice these days, and this is changing super-fast, like every month, is those credentials should live outside of the agent in probably the host that has a proxy that injects those as the requests come in with firewall rules to decide whether the request is sane, but the agent has no access to that proxy.
Participant 3: In Kubernetes, you hear a lot about KubeVirt, which was their attempt of using for VMs, and when you did what you did, I assume you didn't do anything. You just worked off the semantics of a pod.
Felipe Huici: Yes, pretty much. Exactly. We conformed with a Kubernetes pod scheduling API using a kubelet. We did do a lot of work in the past with KubeVirt, but it's slow. We started measuring the different parts and what was contributing to the slowness and then talking to people about maybe we should modify this code and that code, and it just got messy very quickly.
Participant 3: Did you also have to tap into the CNI part?
Felipe Huici: Not for this, but some customers sometimes want a CNI, and for that, there's a side CNI that's possible to plug in as well. I didn't want to get into the product stuff.
Participant 4: In one of your graphs, the netboot, I believe, there was a peak really early on. Any reason for that?
Felipe Huici: I don't know, because one of my really good field engineers produced that. What may have happened is he upped the batch size, meaning instead of firing like, start 100 VMs in a millisecond, start 200 or 500, and that caused the spike. That's usually what happens.
Participant 4: Because you expect a flatline, same as the rest?
Felipe Huici: Because it takes a long time to load 1 million, and the demo was coming up, I just told him, just load it as fast as possible. He was really hammering the thing. It wasn't a 1 second, one VM, 1 second, one VM. It was just going in batches.
Participant 5: How transparent is the snapshot restore? Do we have to customize the application to handle being woken up? Are there any obstacles there?
Felipe Huici: No, it's totally transparent. These days, you can vibe code Firecracker over a weekend. I recommend do a little fun Firecracker project this weekend. It's REST APIs. Literally, there are people, it's like, I can just vibe code Firecracker, and then I'm an infra company. The trouble begins later when you go at scale, and you go in production, and you start getting all these really tricky issues. In principle, literally, that REST API will get you the snapshot, and then there's other REST APIs to restore from that snapshot. That's transparent. Firecracker doesn't know or care what's inside the VM.
Participant 5: If you're basically dumping the memory of the snapshot to disk, any bandwidth problems if you swap them out a lot?
Felipe Huici: Yes. That's why you don't vibe code it over a weekend. Sorry, don't do that project over the weekend. It's not going to work. This is why I said differential snapshot, compressed snapshots. We had to do a multi-tier snapshotting system from NVMe down to SSD, and other tricks and pieces. Deduplication, wherever there's freed-up memory, make sure you don't snapshot that. There's a lot of things that go into making snapshots go at this scale and speed, yes, for sure. Of course, having multiple NVMe drives also helps.
Participant 6: I was thinking Firecracker, I know mostly as the microVM that's also underneath AWS Lambda. I was wondering, how does running a serverless function compared to running a Unikraft mainly on your own machines versus in a hyperscale cloud, what are other fundamental differences?
Felipe Huici: We can run on your own cloud. We can do on-prem and all that stuff, that works. Getting back to the technical bit. Yes, Firecracker started in AWS Lambda, correct. As you can see, it's a generic virtual machine monitor. Generic. The specificity of it is that it can only start Linux VMs. Because of that, they were able to make it go fast and trim a lot of the code that something like QEMU can't really do. Other than that, at its core, it's not that it can only run functions. It just launches VMs, stops VMs, and how long you want to run them for, that's all attributes of how the Lambda product was built and limitations with scale. There's nothing that says that Firecracker, when you do your weekend project this weekend, that it's going to say I've run for 30 seconds, I'm going to stop now. That's not an intrinsic limitation of Firecracker itself.
Participant 7: Where is scheduling happening here if you've got more demand than you've got cores?
Felipe Huici: Of course, imagine you have 48 cores and you have more than 48 Ubuntu's now, but you get 100 requests for them. There's no mystery there. The performance is going to degrade. The boot times are going to get longer. Sometimes you tend to do a calculation of how much degradation you're willing to accept, after which you basically queue the requests. Or you can horizontally scale. The other thing that we can do is, if you're in environments like AWS, we can take EC2 instances that are in its stop state, pointing to an AMI, and whenever the peak starts to go beyond the base servers, you can actually kick those on in about 15, 20 seconds, and then that gives you another platform that can run thousands of VMs again.
Participant 7: If I follow on from that, the early days of running containers inside of VMs, there was this claim from VMware that their scheduler in vSphere was more efficient than the scheduler in the Linux kernel, and so you could actually have greater density and higher efficiency running containers inside of VMs, and you could have containers on bare metal. The scheduling behind this, where would you place it in a ladder of Linux kernel scheduler versus vSphere scheduler.
Felipe Huici: At the very top, of course. It's the Linux kernel doing the scheduling, because Firecracker is processes. Obviously, the Linux kernel has gotten a lot better, and Firecracker works pretty well in the Linux kernel, so you can schedule a lot of things and they run ok. Ultimately, it's tricky because you don't know what's going to be running in the virtual machine as well. Some virtual machines are going to be CPU-hungry or not, and so the mileage varies. You need to get the base scheduler as good as possible. That's also why we didn't use Intel Cloud Hypervisor. We did a bunch of benchmarks, synthetic benchmarks, and we noticed that the scheduler was less good, and it was consuming a lot of CPU at idle, and we're getting jitters as well. Beyond that, at some point, your system is not going to go infinite when it actually has a ton of load on it.
Participant 7: Are we going to be able to move towards a live feedback loop between observed production performance and a desired priority of schedule in terms of which workloads we're wanting to give priority to?
Felipe Huici: I think Granulate and some of these companies do that. What we try to do is you monitor it. One server gives you humongous capacity because of the scale to zero dynamics, and when it doesn't, we can launch these additional VMs on EC2 or other providers so that in 15 seconds, you have another fleet of more hosts that you can then bring down. That's a different engineering track because that's VMs within VMs, and we had to do a lot of tricks so that the performance wouldn't be terrible.
Participant 8: Kata Containers, how do they play into this?
Felipe Huici: The idea with Kata Containers is building a virtual machine distribution that is built to run containers. It's almost like I know that my application is the container runtime, and how can I build a distribution that only has the needed parts for only that? It's a good idea. Basically, if you're ECS, you're doing something like that because you're sticking containers inside VMs, and you might as well make that as efficient as possible. Obviously, my opinionated take on that is you don't need the container, you just need the application. You put it directly in a virtual machine, that'll just be more efficient. I love Docker and containers for local and development, and we use that all the time. In production, a bit less.
See more presentations with transcripts
