BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News AWS Nitro Sandboxes Put Each EKS Pod in Its Own MicroVM with One RuntimeClass Field

AWS Nitro Sandboxes Put Each EKS Pod in Its Own MicroVM with One RuntimeClass Field

Listen to this article -  0:00

AWS has launched Nitro Sandboxes, which run an opted-in Kubernetes pod, ECS task, or EC2 workload inside its own lightweight virtual machine with a separate guest kernel, carved out of a running EC2 instance by the Nitro hypervisor. On Amazon EKS, the whole change is one pod field: runtimeClassName: isolated-sandbox or confidential-sandbox.

The feature is available in all commercial regions at no additional charge, and AWS positions it for workloads where the tenant or the code cannot be trusted, with coding agents named first. In an AWS blog post, Carlos Santana and Trishi Mehta write:

Each pod gets its own operating system, completely separate from the host and from every neighbor.

According to the EKS documentation, a containerd-shim-nitro-sandbox-v2 process on the parent instance launches a Firecracker microVM as a first-level guest with hardware-assisted virtualization, after which an agent inside the VM runs the pod’s containers through its own containerd and the two sides communicate over virtio and vsock. The result is a sandbox that shares neither kernel nor memory mappings with the parent instance or any other sandbox, while the containers themselves start unchanged. How the nodes get there depends on the cluster type: on EKS Auto Mode the two RuntimeClasses ship with every cluster and sandbox-capable nodes are provisioned on demand, whereas on standard EKS a team builds a sandbox-enabled AMI for its node groups or Karpenter.

Diagram of a Nitro Sandbox. On the parent EC2 instance, kubelet calls containerd, which calls the sandbox-shim, which drives a Firecracker process with virtio-vsock, virtio-blk, and virtio-net devices over the parent kernel. Inside the Nitro Sandbox, the eks-sandbox-agent uses its own containerd to run the workload over the guest kernel. Device Memory Buffers (DMBs) form the device data path between them, the Nitro Hypervisor sits beneath both and carries sandbox lifecycle and runtime notifications.

(Source: Amazon EKS User Guide)

What separates the two modes is which side of the boundary is trusted. Isolated mode trusts the parent instance and not the guest, which is why kubectl exec, logs, and process listing keep working, whereas confidential mode treats the parent as untrusted and moves enforcement inside the VM: a Cedar policy generated from the pod spec pins image digests, arguments, environment, mounts and capabilities, exec and stdin are denied, the Nitro Secure Module measures the image and policy into PCRs, and a KMS key policy can release secrets only to a sandbox whose measurements match. Image layers can be encrypted as well, so the parent moves ciphertext it never reads.

That chain only holds if something checks the measurements, and the documentation says so: without a KMS policy or external verifier checking the PCR values, "the attestation measurement is recorded but never verified, and the confidentiality guarantees of confidential mode are incomplete." Separately from either mode, a network-isolated annotation boots the sandbox with no network device at all, which also removes DNS, NTP, IMDS and Pod Identity and breaks HTTP and TCP probes.

Sandip Das, an AWS Container Hero, read the documentation the same way and drew out what isolated mode leaves to the operator: the protection "only runs in one direction," so a compromised parent can still observe or tamper with unencrypted traffic and DNS, writes the pod's /etc/hosts, and can mint Pod Identity credentials for any pod on the node. He also lists prerequisites the blog post does not, among them Kubernetes 1.36 or later, a launch template with Nitro Enclaves enabled on managed node groups, without which "sandboxed pods sit in Pending," and Pod Identity rather than IRSA for credentials. On AWS's description of performance as near-native, he advises treating it "as a vendor claim" and benchmarking startup latency, I/O throughput and tail latency.

Beyond agents, AWS lists multi-tenant SaaS, regulated workloads, CI builds and remote IDEs, and points to the open-source agent-sandbox project for warm pools that hand out a pre-booted VM in about two seconds.

The constraints are on the considerations page, and the largest is GPU and device passthrough, which is not supported at launch because Firecracker cannot pass host devices into a guest, so GPU pods run without the sandbox class. Density is capped twice over: total sandbox vCPUs may not exceed five times the node's vCPU count, and neither may the number of sandboxes, halved on CPU-overcommit types, which gives a 4-vCPU node room for twenty 1-vCPU sandboxes or five 4-vCPU ones. Simultaneous multithreading costs more than the arithmetic suggests, because core scheduling stops two sandboxes sharing the sibling threads of one core and leaves roughly half the advertised vCPUs usable; Graviton, with one thread per core, is unaffected.

The nearest comparison is GKE Sandbox, which has offered a gvisor RuntimeClass since 2019 and more recently a microvm class built on Kata Containers and Cloud Hypervisor. gVisor intercepts syscalls in a user-space kernel, starts in under 200 milliseconds and supports CUDA on a subset of GPUs; Google's microVM option carries a fixed 250 mCPU and 130 MiB per pod, starts in one to two seconds and does not support GPUs. AWS has taken the VM side of that split and expressed its overhead as density and threads rather than per-pod resources.

On EC2, Amazon Linux and Firecracker are supported natively and patches are available for other virtual machine monitors. The EC2 User Guide carries the supported instance list and notes that snapshot, restore and live migration are not available.

About the Author

Rate this Article

Adoption
Style

BT