Cilium is open source software for transparently securing the network connectivity between application services deployed using Linux container management platforms like Docker and Kubernetes. It is a CNI plugin that offers layer 7 features typically seen with a service mesh. On this week’s podcast, Thomas Graf (one of the maintainers of Cilium and co-founder of Isovalent) discusses the recent 1.6 release, some of the security questions/concerns around eBPF, and the future roadmap for the project.
Key Takeaways
- Cilium brings eBPF to the Cloud Native World. It works across both layer 4 and a layer 7. While it started as a pure eBPF plugin, they discovered that just caring about ports was not enough from a security perspective.
- Cilium went 1.0 about a year and a half ago. 1.6 is the most featured-packed release of Cilium yet. Today, it has around 100 contributors.
- While Cilium can make it much easier to manage IPTables, Cilium overlaps with a service mesh in that it can do things like understand application protocols, HTTP routes, or even restrict access to specific tables in data stores.
- Cilium provides both in kernel and sidecar deployments. For sidecar deployments, it can work with Envoy to switch between kernel space and user space code. The focus is on flexibility, performance, and low overhead.
- BPF (Berkeley Packet Filter) was initial designed to do filtering on data links. eBPF has the same roots but it’s now used for system call filtering, tracing, sandbox, etc. It’s grown to be a general-purpose programming language to extend the Linux kernel.
- Cilium has a multi-cluster feature built-in. The 1.6 release can run in a kube-proxy free configuration. It allows fine-grain network policies to run across multiple clusters without the use of IPTables.
- Cilium offers on-the-wire encryption using in-kernel encryption technology that enables mTLS across all traffic in your service fleet. The encryption is completely transparent to the application.
- eBPF has been used in all production environments at Facebook since May 2017. It’s been used at places like Netflix, Google, and Reddit. There are a lot of companies who have an interest in eBPF being secure and production-ready, so there’s a lot of attention focused on fixing and resolving any security issues that arise.
- 1.6 also released KVstore-free operation, socket-based load balancing, CNI chaining, Native AWS ENI mode, enhancements to transparent encryption, and more.
- The plans for 1.7 is to keep raising up the stack into the socket level (to offer things like load balancing and transparent encryption at scale) and likely offering deeper security features such as process-aware security policies for internal pod traffic.
Subscribe on:
Show Notes
Intro
- 00:21 In September 2018, the InfoQ podcast featured Justin Cormack and discussed Decomposing the Modern Operating System. That episode included a discussion of Cilium.
- 00:38 Cilium is an in-kernel, high-performance service mesh that leverages eBPF. It's often been called "iptables on steroids."
- 00:53 Today we're talking with Thomas Graf, one of the maintainers of Cilium, and a co-founder of Isovalent, the company behind Cilium.
Overview of Cilium and eBPF
- 01:24 Cilium 1.6 was released last week. It's probably the largest release ever, in terms of commits.
- 01:42 The velocity is driven by the very active community, with just over 100 total contributors, and between 20 and 40 contributors on every release.
- 02:04 The simple explanation of Cilium is that it brings eBPF to the cloud-native world. It provides networking, security, and other service mesh values to Kubernetes.
- 02:23 Cilium doesn't just stop at Layer 3 or Layer 4; it goes all the way to Layer 7, and understands API calls. This is something people are used to when using a service mesh, but is completely new to people who are used to dealing with networking.
- 02:42 Cilium started as a pure networking plugin. But just caring about port numbers wasn't sufficient from a security perspective. The ability to understand application protocols was a feature added very early.
- 03:01 We wanted to go further than saying an application can talk over HTTP on port 80 or 443, and, for example, allow a request to /public, but not to /private. That's where Cilium has an overlap with service meshes.
- 03:36 Cilium provides both in kernel and sidecar deployments. For sidecar deployments, it can work with Envoy to switch between kernel space and user space code. The focus is on flexibility, performance, and low overhead.
- 04:23 The sidecar model works with any proxy, such as Envoy, Linkerd, or HAProxy.
- 05:03 Facebook is not using Cilium, but is co-maintaining eBPF.
- 05:08 The Cilium user community has grown tremendously in the last 12 months. 1.0 was released about 16 months ago.
- 05:38 The Cilium release cycle follows the Kubernetes release cycle, to take advantage of all the latest features.
- 05:57 BPF was initially very network-specific; that's why it's called Berkeley Packet Filter. eBPF has the same rules, but has grown much further.
- 06:28 eBPF is a language with at just-in-time compiler that produces a binary. The user writes BPF bytecode, and loads it into the Linux kernel. It is verified, then JIT compiled and runs natively.
- 07:06 One use case that eBPF excels at is a flexible networking solution, whether on bare metal or in the cloud.
- 07:26 We have integration into cloud provider specific networking, such as AWS ENI.
- 07:33 Because eBPF is flexible and programmable, it isn't limited to known, existing Linux network constructs, and can create new ones.
- 07:47 From an overhead perspective, eBPF allows the most efficient implementation. A typical Kubernetes implementation will use kube-proxy, which is iptables-based. eBPF use has tables instead of a linear list of rules, meaning the latency is significantly reduced.
Features in Cilium 1.6
- 08:13 The 1.6 release was focused on policy scalability. The scalability focus in 1.5 was on non-security policies.
- 08:35 1.6 allows policy enforcement across many clusters. Cilium's multi-cluster feature allows multiple Kubernetes clusters to be linked together and treated like one, big cluster. This allows policies to scale across hundreds or thousands of pods.
- 09:03 The 1.6 release can run in a KVstore-free configuration. Cilium is now fully backed by Kubernetes CRDs, in terms of storing state. Small-scale users can use the CRD implementation, then bring in a KVstore to optimize state when necessary. This means being able to benefit from KVstore, without depending on it.
- 09:58 With 1.6, Cilium is a 100% kube-proxy replacement. You can run Cilium instead of running 8kube-proxy, and have a completely iptables-free Kubernetes cluster.
- 10:28 If you run the default kube-proxy configuration, you will get an iptables-based implementation. As your cluster scales up, the list grows, potentially to thousands of rules, and must be searched for every new connection. Cilium uses eBPF to implement the same functionality, so the latency doesn't change whether you have one or one-thousand nodes.
- 11:16 Socket-based load balancing is a new feature that's just getting started. This hooks into the system call layer, so redirection happens before any TCP connection, avoiding IP lookup.
- 12:38 The goal is to move a lot of the functionality in Cilium to the socket layer. This is more efficient and more transparent. If a policy violation occurs at the networking level, the packet is typically dropped, which looks like a timeout to an application. By handling violations at the socket layer, it means a specific error message will be seen by the application, including what went wrong and why.
Additional new features: Generic CNI chaining, native AWS mode
- 13:44 Cilium does much more than CNI, which is basically just a networking piece. In managed Kubernetes, cloud providers have their own CNI plugins that work well with their hardware. In those scenarios, Cilium can run on top of an existing CNI to provide all the features that are not networking specific.
- 14:42 Customers with large-scale deployments in AWS will be interested in the new native AWS mode, developed with Palantir. The operator-based approach, rather than the per-node ENI approach, scales much better, in particular when combined with auto-scaling and hundreds of nodes. High-scale can mean either adding 500 nodes in five minutes, or running several thousand nodes. If you're one- or two-hundred nodes, that isn't high-scale.
Security and transparent encryption
- 15:57 Transparent encryption, being able to encrypt on the wire, east-west or north-south, between services, was in a previous release. This helps requirements for all traffic to be encrypted, like those in GDPR. The traditional option was to handle this at the VPN network level. The other extreme is to enable MTLS (Mutual TLS) across the entire service fleet. If you don't want to do that, you can use a service mesh to provide MTLS.
- 16:49 Cilium provides transparent encryption, which is similar, using in-kernel encryption. The advantages include encryption that is completely transparent to the application, and it guarantees all traffic is encrypted.
- 17:48 Because Cilium is doing the encryption, it is able to see everything before it is encrypted. Cilium's extensive APIs can be used for observability needs.
- 18:49 eBPF has been used in production at Facebook since May 2017, so they, and other companies are very interested in making sure eBPF is secure.
- 19:18 The vulnerabilities that have been identified require being able to inject BPF bytecode. By default, this requires elevated privileges, so the vulnerabilities would only apply if you allowed unprivileged processes to load BPF. For Spectre and Meltdown specifically, they were less of a BPF bug, rather BPF made it easy to exploit. It was mitigated by several Linux kernel patches.
- 20:20 Cilium warns you if you're running in unprivileged mode. None of the attacks were related to Cilium.
- 20:55 When a security vulnerability was found in Cilium, three patches were quickly released for three CNI plugins, following best practices.
- 22:08 Cilium provides nine months of support for the open source releases, which defines how far back they will go to offer security patches.
- 22:40 The commercial version of Cilium offers additional management tooling and longer-term support.
- 23:04 Future versions of Cilium will include more functionality moved up to the socket level. While the product is taking on more service mesh values, it is not a full service mesh, so some service mesh features that users are asking for will not be part of Cilium.
- 23:56 Another focus will be offering deeper security features, such as process-aware security policies for internal pod traffic.
- 25:45 Cilium is not trying to replace service meshes. The differences may be in when a sidecar is not the answer.