Comparing Virtual Machines and Linux Containers Performance
IBM Research Division has published a paper comparing the performance of container and virtual machine environments, using Docker and KVM, highlighting the cost of using Docker with NAT or AUFS, and questioning the practice of running containers inside of virtual machines.
The paper authors ran CPU, memory, network and I/O benchmarks against native, container and virtualized execution, using KVM and Docker as virtualization and container technologies respectively. Benchmarks also include sample Redis and MySQL workloads; Redis exercises the networking stack, with small packets and a large number of clients, while MySQL stresses memory, network and the filesystem.
The results show that Docker equals or exceeds KVM performance in every case tested. For CPU and memory performance KVM and Docker introduce a measurable but negligible overhead, although for I/O intensive applications both require tuning.
Docker performance degrades when using files stored in AUFS, compared to using volumes, that have better performance. A volume is a specially-designated directory within one or more containers that bypasses the union file system, so it does not have the overhead that the storage backends may have. The default AUFS backend causes significant I/O overhead specially when using many layers and deep nested directory hierarchies.
Docker's default networking option, --net=bridge also introduces overhead due to NAT packet rewriting, noticeable with high packet rates. Network performance can be improved by using --net=host, instructing Docker to not create a separate network stack for the container, allowing full access to the host network interfaces. This option should be used carefully though, as it lets container processes open low-numbered ports like any other root process, and access local network services like D-bus, which can lead to processes in the container being able to do unexpected things like restart the host.
KVM performance has improved considerably since its creation, although it is still less suitable for workloads that are latency-sensitive or have high I/O rates because it adds some overhead to every I/O operation. This overhead is significant on small I/Os but negligible when performing large I/Os.
Given those results, the paper questions the implementation of IaaS using virtual machines:
Conventional wisdom (to the extent such a thing exists in the young cloud ecosystem) says that IaaS is implemented using VMs and PaaS is implemented using containers. We see no technical reason why this must be the case, especially in cases where container based IaaS can offer better performance or easier deployment. Containers can also eliminate the distinction between IaaS and "bare metal" non-virtualized servers since they offer the control and isolation of VMs with the performance of bare metal.
Although it is common practice to run containers in a virtualized environment, the paper recommends running them in bare metal Linux servers, otherwise they are bound to the performance overheads of the VM with no added benefit to using containers directly in a non-virtualized Linux.