Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Presentations Panel: Secure Isolation of Applications

Panel: Secure Isolation of Applications



Applications have been isolated by lots of different means: processes, virtual machines, containers, and new methods are appearing such as SGX and in-process isolates. What is secure? Have Spectre and Meltdown changed the landscape? What should be used?


Justin Cormack is developer at Docker, working on unikernels. Jessie Frazelle is an infrastructure engineer at GitHub. Per Buer is CEO of IncludeOS. Allison Randal is a software developer and open source strategist. Kenton Varda is the architect of Cloudflare Workers, a "serverless" compute platform.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.


Cormack: Welcome everyone to the second thing in the "Modern Operating Systems" track. We've got a panel discussion now, which gives you the chance to talk to a bunch of our speakers, ask all your questions. And really, one of the themes that came across in the talks this year was really about process isolation. Isolation is one of the basic jobs of the operating system, is to stop things conflicting with each other, but it's become suddenly an interesting area again. I just want to let our panel introduce themselves. I'm Justin Cormack, I work in security at Docker, and I've had a longstanding interest in operating systems, and systems, and internals.

Buer: My name is Per Buer. I work for a company called IncludeOS, so we do unikernels. I'm generally interested in infrastructure things.

Randall: I have a long history in operating system development, but at the moment I'm taking a break to do a PhD at the University of Cambridge, specifically in the area of secure isolation for virtual machines and containers. And, I do some consulting on the side through a company that I own here in the UK.

Varda: I am Kenton Varda. I'm the tech lead for Cloudflare Workers, which is experimenting with a new way of doing isolation using v8 Isolates instead of using processes or VMs.

Frazelle: I'm Jessie Frazelle, currently unemployed, and I like putting things into boxes.

Trusting Isolates

Cormack: I was going to start off by talking about software-only isolation, which is a really attractive idea. We can just write code that isolates itself, that runs without any support from the hardware, and generically it runs everywhere. It's a really nice idea, and it’s the model that Cloudflare have been using. But it's also the model that containers have been using, rather than virtual machines where you just get the kernel to isolate bits of software separately.

With kernel isolation and containers, we know that there are kernel exploits. This can break and it's a problem with much more restrictive forms of isolation like Isolates. At the moment, we trust it a bit more; is that a reasonable trust? Should we trust everything, trust nothing? What should we do? Maybe Kenton would like to start. Also, Kenton is talking later, so if you want to hear much more about this, he'll be on later on today in this track.

Varda: I don't know if I would say we trust isolates more. There's certainly a lot to a debate there. On one hand, virtual machines for instance, have a much smaller attack surface, versus v8. When we use v8 JavaScript engine for isolating, v8 is big and complicated, and there's a lot of ways that there can be bugs, and I'll go into some of those bugs later.

But, you described containers as software-only isolation, but I wouldn't quite describe them that way; they are relying on the MMU, and on the hardware support for privileged instructions versus unprivileged operation, and with Spectre and everything that's happened lately, we're starting to see that the hardware can have bugs and the bugs, so far, we've been able to mitigate them, but will we always been able to when new bugs come out? I'm not sure. The further away you are from hardware, probably the more opportunities you will have to mitigate a new hardware-level bug. So, that's something I've been thinking about lately that worries me.

Randall: Unfortunately, the answer is, there is no hardware or software isolation. It doesn't exist, that you can completely trust in all circumstances, and Spectre and Meltdown did really punch a hole in our assumptions. For a long time now, we've been assuming that VMs were pretty secure, that we had a pretty good idea of the software security model that we needed to run on top of the hardware. And what Spectre and Meltdown and the related vulnerabilities showed us is, actually, all of our assumptions about what the hardware was doing underneath were wrong.

When you hit that, then you realize that virtual machines are not as secure as we thought they were. The kernel is not as secure as we thought it was, containers, and it does go all the way up to the level of what I would call language-virtual machines like v8 or the JVM, or Google App engine was a Python-based-language virtual machine, that language-level isolation. So if your hardware underneath is just allowing any other tennant on the same hardware to scrape your memory, that's a level of not transparency, it's a level vulnerability that no one ever expected that we would have.

But, in a sense, that's kind of the wrong question; asking if there is any software solution that's going to be perfect and protect us from everything is like asking for a car that will never get in an accident and prevent you from driving off of cliffs. There will never be perfect isolation, but there is such a thing as good-enough isolation for particular use cases.

So, the really important thing is to understand the isolation that your software provides. Whether you choose something like isolates, or, containers or VMs, really understand what it provides and what it doesn't, and then, understand your own workloads, your own applications that you're running, and the risks that you, that your business, that your project can tolerate, and then make your decisions that way. Because, it's never going to be perfect, but it can be good enough.

The Future of Isolation Models

Cormack: Allison, you're saying we should move away from one-size-fits-all solutions. Are we going to have a lot more different kinds of isolation in future then, do you think? Are we going to have separate kinds of solutions for business applications, and cloud providers, and different sorts of users, different kinds of threat models? Is this going to be a really exciting field with lots of new software? It's been, kind of, historically limited. VMs and containers have been the two things that people have talked about. They've had most of the mindshare, had most of the development work. Should we be peeking out, and spending more time building different isolation models for new things?

Randall: Not necessarily. One of the exciting things about containers is, they aren't a single security feature; they are a combination of a whole bunch of security features, from the Linux kernel primarily. But, you can add on seccomp, you can add on AppArmor, you could add on SELinux, you can add on a bunch of different things to make them more secure. So, as new technologies are developed, they don't have to be not-containers. They could be containers using the new technologies as they're developed.

And, I kind of see that as the direction things are going. Actually, VMs are starting to borrow some technologies from containers. So, there's this hybridization between VMs and containers taking the best from both, and then also evolving some new isolation features to add on top. I think that's more what you're going to see, partly because people are just so sold on the name containers that it's like "magic sugar" sprinkles on top of everything. So, just to add better security in with that, and you can sell it.


Cormack: How do unikernels fit into the isolation world?

Buer: So far, we've mostly taken the underlying systems that isolate us. We've taken them a bit for granted. I do actually believe long-term in hardware being able to supply a level of isolation that I think is quite superior. Hardware is, at a very fundamental level, very simple. If you managed to create these narrow-waisted systems that have these very narrow interfaces, you can spend a lot of resources securing those. It's like securing an out instruction on an Intel system; that's the thing that allows you to send something over the PCI bus. It takes four bytes and you making sure that there's nothing funny going on. Such an instruction isn't really that complicated. And since setup cost for a CPU architecture is so humongous, I think companies like Intel can invest in at least getting closer to formal verification that the hardware is more or less according to specification.

If you look at the complexity of software, most software doesn't have these narrow-waist Linux kernel with its 380 or 390 system calls. It's really an unending task to secure all those interfaces. How many privilege instructions do we have on Intel? I think it's 15 or 16, I think we use 9 of them or something. It's a doable task to make sure that those are reasonably safe. I'm not entirely sure if that answered your question.

Cormack: Yes. Anyone else have anything to say about simplicity? Kenton, you mentioned the complexity of v8 and I'm sure Jessie has opinions on the complexity of Linux. I mean, is it a problem that we need to fix?

Varda: Well, just a quick comment on what you said. It's not just those privileged instructions that need to be skewered, it's all of the instructions as we've seen with Spectre and Meltdown. It's a little bit more complicated than that, but on the other hand, with something like v8, it's a huge like shit compiler, garbage collector, all these different things that are incredibly complicated. Small bugs can allow a breakout there too. So, everything is complicated, and everything will have bugs and we have to work through it. Do you have anything?

Frazelle: Yes, I definitely agree with that. Everything will have bugs. Now having seen a lot of how places and big companies deploy containers, not a lot of people put in thought into actually dropping capabilities and writing seccom profiles, or even using the default. They don't even do that, it's absolutely horrifying. So, yes, it would be nice if there was just have a one-button click, like, "Go do the right thing." But, at the end of the day, like everyone has been saying, if you like it, it will still be insecure in some way.

Cormack: Jessie, you're talking about SGX in your talk later, which is new hardware mechanisms. Are new hardware mechanisms going to help us?

Frazelle: Not right now. It's more harm than good as you'll see from my talk. I'm not an expert or anything, I just read a bunch of papers, but yes. The weird thing with the SGX, in my opinion, is that it started as just for DRM, which is a pretty good use case for DRM, but then people started executing code in there, and then all hell broke loose with Specter and Meltdown.

Methodologies in Writing Software

Cormack: One of the things about SGX and those types of relations is a question of how are you supposed to use them? Are you supposed to use them just to run small security critical bits of code, which was the original use case? You talk to people who think that you should be running your entire code based on SGX. Are we supposed to carry on writing software the same way, and using it the same way, and get extra protection, or are we going to have to change the way we write software in the future, to keep it isolated?

Frazelle: There are two ways to go about this, and two different methodologies, and then they both have trade-offs. You could shove a whole OS in there, which is crazy, and that's from the Microsoft Haven paper, which was the first one that was done. If you do that, then if there's bugs in your OS, like, still have bugs inside the enclave. Then, there's another way of doing half-in, half-out, and then you're passing I/O calls back and forth, but then you have to encrypt everything that's going back out of the enclave, and then back inside the enclave. It's also just slow, horrifyingly slow, because you're just doing I/O and passing syscalls back and forth.

There's no good way right now, and I think a lot of that has to be on the fact that SGX was originally just for DRM, like you're going to shove a cert in there or something, and just make it do that, versus actually running things in there. So yes, it will be interesting to see how the technology evolves, but right now, it's not great.

Lessons Learned from the History of Virtualization and Isolation

Cormack: Talking about evolution, Allison, you've been looking at the history of these things, and I'm interested in the lessons we can learn from history of virtualization and isolation.

Randall: I spent about a year, almost a year and a half now, going over the history of virtual machines and containers all the way back to the '50s. It started with multiprogramming, which is the first time you had more than one process on a machine at the same time, and they started trying to isolate that. From that, you split out two tracks, VMs was one track, but the other was capabilities, which are kind of like modern containers, they're just another name for the early form of modern containers.

I suppose the biggest thing I learned from that is, all along the way it's been trade-offs. People think, "Oh, VMs are so secure, but containers and VMs both have always been trade-offs between performance, and security, and portability, and the complexity of the code base, and they're entangled in a sense; less complexity can make things more secure, and better performance. But, we've never had a perfect answer, we've always just, kind of been trading things off.

Another thing I learned, surprise, surprise, in the '60s, Intel had this amazing memory-protected architecture that did not succeed well in the market, and so they chucked it, and that architecture would have actually completely prevented Spectre and Meltdown. So we're where we are today through a series of unfortunate compromises between market forces and technology choices, yes.

Cormack: And performance? Performance was a driver for those?

Randall: It has, well specifically that highly memory protector architecture, Intel didn't bother. Explicitly, if you read all the papers on it back then, they intentionally didn't target performance, because they said, "We can fix that later." The performance was terrible. It was like 95 times slower than any other hardware architecture on the market at the time. So, yes, those are all important trade-offs along the way, and you can't just say that any one thing is king, and you can't say that the best technology will win either, unfortunately.

Untrusted Code and Cloud Providers

Cormack: One of the things I've been wondering is, is running untrusted code just going to be something that the large-scale cloud providers do? Because it's going to become really difficult, you're going to have to be instrumenting stuff, and you're going to have to have an on-call team that affects the next Spectre bug. Is it going to be just a really expensive thing to do? And the world will divide between the people who run untrusted code at hyper-scale and everyone else who is only safe to run trusted code, and you shouldn't run services for anyone else, and should audit your developer's code very carefully and so on? Jessie, you've worked in more cloud providers.

Frazelle: Having looked behind the curtain, there's a lot of untrusted code there too. So yes, I don't know, honestly. I almost am now of the thought that people in finance or life sciences that have a lot of data centers as it is, I don't see them taking their really high-trust components and running them in the cloud anytime soon, especially because with SGX you still have bugs and stuff like that, so there really isn't a good way to do it. And you still have to trust Intel, so they might as well just keep their data centers, but yes, I'm not really sure.

Buer: There are so many billions of dollars riding on people's ability to run untrusted code that I think we just have to figure it out. Even if it means new architectures, if that's what it takes, then that is what will happen. We just can't live without proper isolations, and we will have it, one way or the other.

Varda: Yes, as someone who ran a startup that ran a lot of untrusted code in containers, I certainly hope that that that continues to be an option for people in the future, because we need innovation. I think it can be done, I think you need people who know what they're doing on the team, and hopefully, we'll see more tools, like off-the-shelf tools. Taking V8 off the shelf is nice because it has already been beaten up by people attacking Chrome for 10 years. So, let's have more of those.

Randall: There is one possibility that a few of us have been talking about, and there's nothing going on yet, but so in mainframes, there is this concept called, "Logical partitioning", where at the very fundamental hardware level, they split up the CPU, and the memory, and the cache, and the at the software level, at the level of Linux kernel, it has no access to modify that. Some of us have been talking about with RISC-V, because it's much more open to modifications, trying out hardware partitioning as a way, specifically for these multi-tenant, untrusted-code systems, so that you could have large-scale hardware with very hard, very low-level separation between them. You wouldn't want to run like an individual container in one of these hardware partitions, but maybe an entire pod, like your whole collection of containers delivering an application. It's a set of things that you're sort of mutually trusting, you've deployed all of them, and maybe there are bugs but it's yours, you own it. But, you may be on physical hardware with other tenants that you can't trust, but you'd have that hardware guarantee of separation.

And, obviously, it's not certain, but at the moment that's an idea of where to explore, where you don't have to rewrite the kernel and rewrite your entire software stack through the system and operating system and applications, but get that promise of isolation that we thought we had from x86 but actually don't.

Buer: I remember when Sandy Bridge came out, I think there were traces of those same ideas there, and I believe that Intel has worked on that since then. I think, these partitioning hypervisors like Jailhouse and Bareflank which allow you to split up systems either on CPU level, or on core level, and you get that PCI otherwise, and that memory bank. Have you looked at those, are those a step in the right direction, then?

Randall: It sounds like a step in the right direction. I haven't seen much published on it, but we might be able to pester them and see if they'd be willing to release more information on it.

Buer: No, I think a lot of that has been happening with people wanting real-time capabilities, because there's to some extent real-time and side channel, free or go hand-in-hand.

Randall: The noise of having other tenants. So that it's more about the guarantee of performance, but it could still have security benefits as well, yes.

Changes and Hopes for the Future

Cormack: What else would you like to see change in the future? What would be the most exciting thing? What would you like to work on? What things do you think are important?

Buer: I remember the day I discovered SGX, "This is so great.” I could actually protect stuff on my system from the system itself. I could take my TLS keys and I could lock them in there, and hardly the app, not relevant. When I worked on the Varnish project, we never had TLS support. The reason was because we had no proper mechanism to actually protect the keys. We didn't want to be liable for protecting those TLS keys. A properly implemented SGX means that the system could boot up, keys could be stored fully, encrypted on this, loaded into it. The system doesn't know what it is, it just loads in, and whenever you make a TLS connection to that system, that secure enclave signs the connection.

To some extent, it's not really that performance-sensitive as well, because it only adheres to the TLS handshake, I believe. Cloudflare does these Keyless TLS or something where you basically do the same thing, except that you do it on another system over the network. But, on this thing you could do it in the same system with marginal performance cost. I think that would be really great. Yes, that's my wish list, at least.

Varda: Now I forgot the question. Oh yes, what do I want to see? I would love to see someone take another crack at the capability-based CPU. I assume what you were talking about with the incredibly slow Intel CPU from back then, but yes, it would fix things like Specter and it would make it harder to screw up.

Buer: Could you explain what it is?

Varda: I don't know a whole lot about the specific implementations that have been tried, but basically the idea that the CPU knows the difference between data and pointers, and if you can't forge a pointer out of thin air, you can only follow the pointers that you have.

Randall: Also every sector of memory is owned by a particular user, and you can't access any memory that you don't own or have permission to access, which is the whole Spectre and Meltdown thing. Although, what they do is cache. So, you'd also have to have ownership of cache as well, and then prevention. You absolutely cannot access cache that you don't own or have permission to access. There is one project, Cherry, they were based on MIPS, they're switching to RISC-V. Spectre and Meltdown didn't exist when they started, so they didn't consider it, and they are actually vulnerable to it, to certain aspects of it. But there's definitely potential there if you pushed it even further and really considered those vulnerabilities.

Cormack: Allison is doing a talk later on hardware and software coding and with RISC-V, which she's going to talk about that stuff, so do come along.

Randall: Yes.

Frazelle: I'll second the thing about SGX. Actually, one of the papers that I read was written by someone who, to prove that they could work on a secure enclave for RISC-V, they had to prove that SGX was bad. So, seeing whatever they come up with with RISC-V will be really cool.

Randall: And I'd say I'm interested in the hardware level, but really all the way through the full stack. I think we're at a point right now- it's kind of exciting, it's like being in the '60s again- where all bets are off, and in some ways we have to start over from scratch, but that means it's a great opportunity to rethink, reexamine all of our assumptions top to bottom, and then think about where we need to be 20 years from now. Because, it could take that long to really dig ourselves out of that hole, but it's totally possible. We just have to find a new direction.

Cormack: Have you lot got questions you want to ask each other before we open up to the audience, because I think you're all people who are interested in these things. Have you got more questions? Let's have some questions from the audience.

Who is the Enemy?

Participant 1: We've been talking a lot about security, but we haven't been talking about who we are protecting ourselves against, and how is that affecting the way we plan the next steps, because 20 years is a long time, and they are different kinds of enemies. I wrote down a couple, such as criminals or state actors or political forces such as Cambridge Analytica, but who is the enemy, because I don't really know, and how does that affect how we do this?

Randall: Yes, that's a really good point. Part of it is, it's different for different people who are deploying software in containers or VMs. If you're a government, then you're going to be concerned about other governments, If you're a bank, then you're going to be concerned about thievery, but also privacy. If you're just running a pet food startup, you probably don't care too much, but you do care about credit card theft, so then you care about that. I think, at the end of the day, there's a general class of people who have the skills to break down whatever isolation we put in place, and there are different sets of people, but their skills are all very similar. So that sort of analysis is very important for the people who are deploying code to do their own risk assessment, and do their own analysis of what their tolerance is for risk.

But then on the software development side, we can actually abstract away from that a little bit and think about those techniques, because at the end of the day, we want to protect against all of them, and we may not be able to in all cases, but we can at least provide the opportunity for mitigation, provide the tools for mitigation that people can use. It doesn't mean companies always will use them, unfortunately. They have a bad habit of not deploying the most secure software even if we give it to them, but at least we can at least try to aim to provide the tools that are powerful enough to accommodate whatever risk tolerance level. If someone has a really low risk tolerance, they might be willing to take a lower performance for it, whereas somebody who needs higher performance might be able to tolerate more risk, so you might end up with a spectrum of options in there.

Hardware Diversity

Participant 2: Do you see an upcoming split in hardware for more of, I don't know, consumer grade and enterprise grade? Still, maybe even enterprises, but I worked in some companies where you don't really care about Spectre and Meltdown because you install npm packages. You have no idea where they're from. You say, "Hey, I need this, I just import, does make it work?" Spectre and Meltdown, they're not even on the radar. There are so many easier ways to get to your code, to be honest. But, do you see a split there, because of the performance cost, or will that just be options on the hardware, or do you have any idea of where this is going? Because currently, it seems Intel is just monkey patching everything.

Randall: I feel like I might. Traditionally, there was a split, right? For a long, long time, big servers were like Sun or something, they were like a completely different set of hardware, and it's only in the past 20 years that we kind of had this big shift to we just use x86 in everything. Even the big Google servers are just running the same hardware is your laptop. It's different, but it's kind of the same architecture model anyway, even if it is a very different architecture around the whole server layout, the motherboard and everything. But I think we do already have a diversity of hardware. You still have people who run giant mainframes. They have the x86s, you have the ARMs, and I think maybe we'll just add a few more options into the mix, that's what I suspect.

I also suspect that Intel will have to shift. They won't throw away the existing hardware, but they'll have to start introducing some new hardware that takes more of this into consideration. They already are doing what they can to try to mitigate Spectre and Meltdown at the hardware level, but they can't do much, because a lot of that effect is not a bug, it's actually a feature. It's actually the intentional design of super scale of processors, is that behavior. So, you can't just throw it out and start over.

But I hope that whatever we manage to solve will appear on laptops as well, because Spectre and Meltdown do actually affect your desktop applications as well as it does servers. Not as much, because you don't have quite as pathological a level of multitenancy, but you have your browser, yes, and you have little apps that you've downloaded onto your phone. So however we solve it, you want it to make it all the way through into the desktops and the phones as well.

Varda: Yes, specifically on the browser point, though, obviously, Spectre and Meltdown matter a lot on the browser side, and if it's the case that all I need to do to get access to your systems is trick someone who has access to them into clicking a link to some website, and then I break out and I grab all their cookies, and now I can log in as them, that's pretty scary too. You can't just focus on the server, the client is just as important.

Where Will Innovation Happen?

Participant 3: You touched on hyperscale cloud providers all inside. You also mentioned the RISC-V bits and pieces. Where do you think the innovation will happen here? Do you think it will be in the open or do you think it will be value added very single user proprietary systems? Or, will it be a bit of both and a mess?

Frazelle: I think maybe a bit of both, because it seems like a lot of the cloud providers, or a few of them are making their own silicone now, but then there's RISC-V, which I really hope that that takes off. It would almost be nice if Apple started using it or someone that's really big, so that adoption picked up really easily. So, it would be nice if the two converged, but right now it seems like it's split.

Virtual Machines, Containers, and Language Level VMs

Participant 4: A bit of a broader question for, I think, mainly Kenton Varda. What's the big difference between something like isolates, something like v8, and something more than Docker itself, stuff like Zones and Jails? Basically, I also need a bit of introduction on Zones and Jails and their main difference against Linux.

Varda: Let's say, how fast can I write a talk? Zones and Jails. You're talking about container-based sandboxing, right? Which I did a lot of at my previous company. So, with container-based sandboxing, you're trying to take native code binaries off the shelf, and run them in a sandbox. With isolates, we don't take arbitrary binaries, we only take JavaScript assembly, things that were built for the web platform basically. It gives us a few extra layers of interaction, so in terms of security, there's more chances to mitigate a problem. It's also a lot more efficient for us. We actually run multiple Isolates in a single process, which is important for us, because we'd have so much context switching otherwise, which I'll talk a lot about in my talk later. I'm not sure.

Buer: I think I could take a stab at it. Both the v8 engine and Linux, and free basis system, have some native virtualization built into them. For v8, it's the Isolates and for Linux and free basis systems, it's containers and the Jails. It tries to virtualize the interface between the application and the operating system, and it does it natively. Both of these are very efficient ways of doing it, because doing it is that perfect level. If you want to do virtual hosts with Apache, you likely want to use the virtual-host function, because it just does it at the perfect level instead of doing at the VM level, which would mean that you have to replicate a lot of the infrastructure up. I don't know if that was a sufficient answer? Yes?

Participant 4: Yes, sure.

Varda: Quickly, another thing is, a lot more of the implementation - well, come to my talk later - but we can try it at much higher level API, a lot more sharing between the isolates than is possible at a a system call level, basically. But yes.

Randall: To really simply break it down, virtual machines, containers, and language level VMs like v8, they're all in approach to multitenancy. Containers take an approach to multitenancy where the kernel basically manages it, so each container is a process. Virtual machines take an approach to multitenancy where the hypervisor manages it, so each VM is equivalent to a container, is a thing that runs on the hypervisor, so the hypervisor manages it. Language level VMs like v8, and the, JVM, and some that I worked on, in that case, it's actually the language itself that provides the multitenancy. It's sometimes called a thread, so it's lighter weight than a process, but the language engine, the language virtual machine is the one that manages slicing up the multiple tenants, and managing the isolation between them. Yes.

Kata Containers

Participant 5: What's your view on Kata Containers? Which are basically lightweight VMs? Are they more secure in practice than, say, Docker, or have we just moved the problem from kernel bugs to QEMU bugs?

Randall: Do you want to answer?

Cormack: They're an option where you replace the kernel isolation with VM isolation. So, historically, that has generally had better security. It comes at a cost of things around memory allocation. There are still issues, right? The interface between, and the security boundary of VMs is complicated, because VMs have to actually be able to talk to the host and talk to each other, and so, those communication channels become your security boundaries I think that we've seen a lot of effort now with things like Firecracker, that is basically a virtual machine designed around actually building a secure infrastructure for doing those things, whereas previously QEMU, which was what everyone tended to use with KVM, has had a historical set of vulnerabilities, they’ve been quite large, because QEMU was really designed for totally different purposes and not about isolation tool, it was designed as an emulator, hence the name.

I think that we're really seeing some proper engineering in VMs now, and I think that's quite exciting and important, and that the work that's going on with the rust-vmm staff is actually really foundational work for making secure VMs that we really should have seen earlier. It's nice that it's actually happening now. And so, if you you're running containers and you can deal with the inconvenience and extra overhead of running VMs, you do definitely get some incremental security improvement, and it's up to you whether that trade-off is important for your application, and you have to understand the isolation properties, and it's not a total cure for all security issues, but it can definitely help if it improves your threat model.

Buer: Would you say that the interface is simpler and better-defined when you have these clear containers?

Cormack: I think it was Firecracker, you can choose which interfaces you have, and it doesn't implement very many. I mean, it implements just what you need to beat the machine and run the file sharing that you want to do, and that's definitely an improvement over something that has all interfaces you might conceivably need. Firecracker doesn't bother to emulate PCI, and PCI is as you know kind of complicated, and there can be bugs in the implementation, so it removes that and so, there's definitely advantages from that point of view.

Buer: Yes, I know, but I was thinking more in terms of, traditional containers versus the Clara containers?

Cormack: It's complicated. There are some interesting papers that try and measure the difference. At the moment, I think that the ways we have of measuring the difference are bad and they need improving. So, I think there's an interesting research with how you measure differences other than counting the number of vulnerabilities in the past, which has been what people do, which doesn't give you a very good guidance for the future. I think, spending more time trying to actually understand how to measure vulnerabilities. But, measuring vulnerabilities is always a difficult thing, because you only need one to escape, so saying that there aren't 399, where there's one really bad one doesn't really help either.

Varda: And sometimes the number and frequency of vulnerabilities being larger as a good thing, because that means that people are actually looking at it. If you see zero vulnerabilities, do not trust that.

Choices in Hardware and Software

Participant 6: You talked about the tools, maybe, that we’ll analyze in the future that will help those certain governments protect themselves from other governments, and users from maybe stealing credit-card data and stuff like that, and how the one-size-fits-all approach that we have today isn't really working. Do you see a future in which there will be choices in hardware in the future in differentiation, where users can pick out on an informed level what their hardware means to them, and there are different trade-offs that they'll have, such as performance and low-risk tolerance, and stuff like that? Versus the software level, basically?

Randall: Yes. Maybe not at the consumer level, because I think, maybe in terms of laptops and phones, the manufacturers will tend to make the choices for you, and that's probably okay, because they'll make pretty good choices, and they'll typically understand the threat model of their customer. Apple really thinks about the threat model of their customers. At the server level, yes, I think you're going to see more and more diversity at that level, and maybe even a combination of hardware. SGX is a good example, you can add extra hardware onto an existing piece of hardware. MIT's Sanctum as another example. There are possibilities for combining RISC-V chips with existing Intel chips, with ARM chips, all in the same server board. And I think that will start to get us approaching the solution that we want, yes. And it won't even necessarily be massively more expensive, just a very different model than we're used to at the moment.

Cormack: Definitely come to Allison's talk if you're interested in that, because she's going to be talking about some of those hardware, software, code design things.

Running on Amazon or Google Cloud Hardware

Participant 7: As someone who is working on getting my company to use more containers, and Kubernetes, and better service orchestration tools, but not necessarily someone coming from a security background, you're all security experts. What is your one piece of advice for someone who is running on Amazon hardware or Google Cloud hardware? What hardware choices would you make, or what choices would you make running Kubernetes? We're probably already doing some of those things, but just one sound bite from each person, your piece of advice. Thanks. It's a big question.

Cormack: One of the things you can do, which I think Amazon has introduced last year, which is interesting, is you can run on bare-metal. If you're running Kubernetes, you don't need to share infrastructure with other people anymore. Previously you could only get it from other providers like Packet, but the idea that you can just commission the whole thing. If you're running something like Kubernetes, it's probably fine to have a big machine. You don't need it, because you're just running lots of containers on it. There's no actual need to buy your hardware pre-packaged into small VMs, and then run individual Kubernetes. No. It's just run one big Kubernetes instance.

Then, you've also got the choice of say, running parts inside VMs with cache containers or something, well you can't do it if the VMs are already being used to cut your machine up into small pieces. So you've got options now, and I think they're kind of interesting to explore from that point of view.

Frazelle: Yes, I would agree with the bare-metal thing. And, yes, most of the cloud providers right now, it's all just Intel chips, other than Amazon launched ARM on AMD, so they're breaking out into other things. But you don't really have much choice today, but I'm sure that's going to change in the future.

Varda: Yes, mostly you have to leave it up to the cloud providers, but they are working very hard on these issues, so hopefully they've figured it out.

Cormack: We might have time for one more question.

Participant 8: There's been quite a few high visibly threats in recent years, like Spectre and Meltdown, Heartbleed and Crackdown, for example. But have we really seen any orchestrated attacks against these threats in recent years, or in the recent months? Because at least, I don't remember a single one that reached news level for Spectre and Meltdown that has really manifested itself.

Randall: Part of it is that you wouldn't hear, because companies are not legally obligated to announce when they've been hacked. Also, even when you do hear highly public attacks like emails being stolen that completely swung the U.S. election, you won't necessarily ever hear the mechanism that they used. So, I haven't personally heard of any major attacks using Spectre and Meltdown, but that doesn't mean they haven't happened.

Cormack: I'll say that, I think, Spectre and Meltdown were so high profile that there were eight months of work before they were publicly announced, and doing fixes, and everyone had pretty much implemented a version of the fixes that would help at least mitigate some of the problems. If it had been announced with no notice, there probably would have immediately been attacks.

Randall: There might have been attacks. I mean, there could be continuing ongoing attacks using variants that we don't know about and in fact, it would be to their advantage to not tell anyone about those variants, because they're profiting from it in the meantime.

Varda: Well yes, also, there's frankly often way easier ways to attack people than Spectre and Meltdown, is the thing. So yes, I haven't heard of anyone actually doing it in the wild, probably because they found easier things to do, and we're being told to stop.


See more presentations with transcripts


Recorded at:

May 19, 2019