Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Presentations The Service Mesh: It's about Traffic

The Service Mesh: It's about Traffic



Oliver Gould talks about the Linkerd project, a service mesh hosted by the Cloud Native Computing Foundation, to give operators control over the traffic between their microservices. He shares the lessons they've learned helping dozens of organizations get to production with Linkerd and how they've applied these lessons to tackle complexity with Linkerd.


Oliver Gould is co-founder and CTO at Buoyant, Inc.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.


Gould: My name's Oliver [Gould], I'm the lead of the Linkerd project, and I founded a company called Buoyant about when we started working on that. I'm on the internet, if you want to get in touch with me.

First, I want to go back to the last time I spoke at QCon, which was November 9th, 2016. That date ring a bell for anybody? It was the day after the presidential election, so it was an awesome, stressful day for me. I hadn't gotten my slides prepped, so I woke up at 5 a.m., super stressed, got my slides ready, go to QCon, stand on the stage. As soon as I stand on the stage, the whole side of my pant leg ripped, down the seam. I spent the whole time hiding behind the podium, so this talk's already going amazing, and I'm amazed they invited me back. This is great.

I really want to talk about three big things. There'll be some live demos, and I definitely want to have a bunch of time at the end for questions, because I assume you all have questions about the service mesh. I want to talk about why we even started working on this, how Linkerd came to be, and how the service mesh came to be as part of that. Then, I want to talk a little bit about the hype cycle, the trough of service mesh disillusionment. I think you'll relate to that a little bit. At the end, I want to summarize with what I think the important things to take away as you approach a service mesh journey.


This all starts back five or six years ago, when I was a lowly engineer at a company called Twitter. Twitter had a very large microservices problem, we didn't know they were microservices, we called it SOA. That picture back there is the topology of Twitter's call graph, taken from some zipping data. That led into what became and I’ll go through that story now.

Twitter had a service mesh, they didn't know they had a service mesh. They had a service mesh, now, they do know they have a service mesh. It's like a proto-mesh, the way I describe it is that Twitter writes all their applications with a library called Finagle. Finagle's a Scala library, so it uses a strongly typed system, it's very composable and flexible. It's how Twitter wrote all of their services. Every service that Twitter uses, uses Finagle to do its serving, and being a client, and all of its network communication.

As we dealt with production issues at Twitter, in the infrastructure space especially, a lot of the lessons we had out of those post-mortems ended up in Finagle, so things dealing with service discovery, and outages, and latencies, and timeouts, and failure. All the things that you have to do when you're doing network communication, we started by solving them on an app by app basis, which is fine until you have 1,000 services and that no longer becomes a maintainable thing.

Putting this all into a library was a great approach. Phil here once said, "Any sufficiently complicated microservices architecture contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Finagle." It's true, Finagle is massive, it does a lot of things. That really got us thinking, as I was leaving Twitter, "Well, this is a really valuable piece of the puzzle. If people are going to go into this microservices world, which I think we all should be going to for a number of reasons, we're going to have to recreate the same value that Finagle has. I don't expect you all to write Scala, unfortunately, so we're going to have to get it out of the process.”

Another big insight we had working on Finagle at Twitter was really looking at the OSI model, and thinking, "There's nothing at Layer 5. Nobody talks about Layer 5, but really that's where all the value is." This is a slide from my last QCon talk three years ago. We used to talk about rpc, and now we talk about service mesh, and all these things have changed slightly. Still, we're dealing with networked request-response protocols. HTTP/2 is a great example, because it's multiplexed and actually has a session layer. You can pull other things like HTTP/1 into that as well. The point being, we have all this infrastructure at the lower levels of the stack that deal with power, cooling. We outsource that to our cloud providers now.

We now have systems like Kubernetes that allow us to provision things. I can provision pods automatically, and schedule things. The job of one of those schedulers, or orchestrators, is to get a process running and listening on some ports. That's basically it, make sure it's healthy, maybe. There's a lot left that we still have to tackle, and that can't all live in the application code. We want to pull this down into another layer of the stack that is the most common concerns for dealing with microservice communication.

That leads us to a service mesh. Here's some arbitrary service topology, it's a weird circular one, but that's fine. As we bring a service mesh, what we do is we add a proxy onto the sidecar. At Twitter, that was a library, it wasn't outside of the process proxy. What we're doing in the service mesh is to take all that value and to put it into a proxy sidecar. Then, we enhance that with a control plane, so that's a thing that you can interact with to ask questions about the mesh and to change its behavior, etc. Just this pattern, service mesh is a pattern, fundamentally. You can have service meshes that do security really well, you can have service meshes that do other things really well, but it's a pattern.

An Abridged History of Linkerd

Going back to 2016, is when we initially launched Linkerd. We were calling it Twitter-style operability for microservices, which is still true. It was this big Scala JVM thing, and it was really, really configurable. What we had was this amazing library, and, "We have to expose all this library's features as configuration of users, because we don't know what you're going to do." If you remember, in 2016, Kubernetes was still a new thing, we didn't really know what that space looked like. There was no Mesos, and Aurora, like all the Mesos ecosystem, there's Swarm, Nomad, legacy stuff, Kubernetes, and it's complex. A lot of people we were talking to were running multiple ones, because they were trying to move, or trying to migrate. It was very a complex time in the world. That flexibility and power was very important, because we couldn't make assumptions about anyone's operating environment.

However, it also meant that it was very difficult to learn that configuration, it wasn't easy to get started. You probably had to spend a week or so just getting your head around a Linkerd configuration to even get started. A high barrier to entry, which is no good if you're trying to solve problems today. The other detail here is that because Kubernetes wasn't really the foregone conclusion when we started, we didn't really have the pod model in our mind. We were thinking a lot more about a per-host model. In a per-host model, a JVM sidecar isn't really that big of a deal. When you're dealing with a pod that has a 10MB Go process in it, adding in 100MB JVM sidecar to that - I don't want to do that. You don't want to do that, nobody wants to do that, so we really needed to rethink a lot of our assumptions.

Then, the other really important detail here is that that breadth of configuration meant it was impossible for us as a product to actually help people debug their things, because there were so many variables in anyone's config. They say, "Oh, this doesn't work." We really have to replicate a lot of different environments to be able to reproduce these things. Helping people got harder and harder the more configuration we added, so it was terrible. That said, Linkerd 1 is still used a lot of places, and is still actively developed. There are people in this room who've committed code to it not so long ago. It's very alive, but it was getting complex.

Linkerd 2

We took a new approach, and launched Linkerd 2 about a year ago, and that got forked out of a project called Conduit, which was a bit of an experiment to see if we were going to be able to do this new world. We wanted to revisit our fundamental assumptions here. One, being that Kubernetes is now a foregone conclusion. Two, it has to have an out-of-the-box experience that just works. If you're trying to be more effective with your microservice or Kubernetes cluster, you don't want to have to go learn a whole other thing to just get the value that you can get out of the service mesh. We want to reduce a lot of that time to get started. Because we're assuming the pod model, we need to be much, much lighter weight. We cannot be in a JVM, we need to be in a native language, and we can't have GC, so that led to choose Rust to rewrite the proxy, which has been a wonderful journey over the past couple years. I'll talk a little bit more about that.

Also, it can't be magic, simplicity can be scary, if we do lots of things for you, if we're interacting with your Kubernetes cluster in ways that you don't know, or that we're not telling you clearly what's going on, that's scary as a user. I wouldn't run a tool that was doing lots of things that I couldn't reason about or debug. We need to make sure that Linkerd documents itself well. Also, we bundled in things like Prometheus and Grafana, which are other cloud-native projects, so we can have that out-of-the-box experience without having you to go configure all of those things yourself.

The architecture starts looking something like this now. This is all Kubernetes at the moment, we have a microservice of our own running the control plane. In fact, it runs Linkerd so we can use Linkerd to debug Linkerd while you Linkerd. Most of that is written in Go, because it interacts with the Kubernetes API. We don't have good language bindings outside of Go for the Kubernetes API right now. I've written them in Scala, it's not fun, I don't want to ever support that ever again.

Go has rich libraries out-of-the-box. It also makes it much easier for folks to get contributing in the control plane, which is where we think, actually, most of the contribution should happen. We want the data plane to be pretty locked down. We don't want to have lots of optional features being shoved in the data plane, which is something we learned in Linkerd 1. We had a great plugin model, it was very flexible, but it means you could put a plugin in the data path that could totally ruin your performance or your GC characteristics. We can't help you with that, so we wanted to be really conservative about what goes in the data path.

A quick aside here, we spent quite a bit of time rewriting the proxy in Rust. There are a couple motivations for that. One, being Scala aficionados, we really value a strong typing system. We want to be able to catch as many bugs as we can at the compile phase of our project and not at the runtime. That takes out things like Go, which has lots of runtime exceptions, null pointer exceptions, etc., if you're not very careful. We wanted to use language that was virtually impossible.

Two, we don't want any sort of garbage collection in the data path. Rust has this nice feature called RAII, Resource Acquisition Is Initialization. This is a nice idea that when you open a resource, you can drop it, you have a handle on it. Once the program drops that thing and no longer has references to it, anything associated with that resource goes away. You can have customized drop handles. This one is in our metrics stack, when a response body gets dropped, because we're no longer processing it, we now record a latency for the response, and we can classify it as a success, or failure, etc. This means that we can have things that are tracked through the lifetime of request. Then, when the request is done, all the resources associated with that request can get torn down.

Similar for caching, we do lots of caching a proxy, so that we cannot do lookups every time you send a request through it, and those sorts of things. Those caches get evicted, and when the service gets evicted, all the resources associated with that resource get dropped automatically. It's a really nice feature of the language.

Anyone dabbled in Rust here? There's a Rust talk simultaneously, so I assume they're all in there. I'm guessing in a year or two that's going to be very different, Rust is really hitting a maturity point right now. When we started, it was not, now, we have very good networking libraries, gRPC, HTTP/2, all the things you actually need to build a real application. I strongly recommend you give it a dabble.

What Does Linkerd Do?

What does Linkerd actually do? We've talked about a bunch of how it came to be and our decision process, but why would you use this thing ever? I think the most important thing in the microservice world is visibility. We no longer have one log to go read, even logs are no longer sustainable, because we have to go across teams, across boundaries all the time. We need a really uniform set of metrics that I can ask about any service and know generally about its health. Then, we can have further metrics off of that, but we want goal metrics, which are a success rate, some latency idea, like how long requests are taking, we want a latency histogram of distribution, and we want rough throughput. How many successes? How many failures? Etc. This, I think, is the most important thing.

If we did nothing else, Linkerd would still be super valuable. The point being, you're running a Kubernetes cluster, it's hard to know about any of the traffic going through it. You can ask lots of questions of Kubernetes about the resources themselves. "How much memory is this pod using? How much CPU is this pod using? How many things are on this node?" You can't ask Kubernetes about any of this network-level or applicational concerns here. A service mesh can give you that as just a layer in itself.

The other big reason, especially with Linkerd 1 that we saw people adopt a service mesh, was for load balancing. Load balancing cloud providers, and doing client-side load balancing especially, is difficult. You can do this in your own code if you want to. If you want to do this in a generic way that's latency-aware, you really do want something like Finagle or Linkerd 2 in that path, or Envoy for that matter. These load balancers use latency information about its request so we can keep it EWMA, I'll talk about that in a minute. We have security, if we're in the data path on both sides, we can just give you automatic TLS.

Linkerd: Observability

To go into this a little bit more for the observability, we ship with Grafana by default. That's just so you have a dashboard you can go look through and browse through. You don't have to use a Grafana. In fact, in an upcoming release, we're going to make that an add-on feature that you can add on after the fact. We do ship with Prometheus, so we have all this great data.

We have another feature called Tap, which is unique to Linkerd 2. A nodelet Tap allows you to basically query the proxies at runtime and say, "I want to see a sampling of request metadata that looks like this," so paths that match this Regex, or these verbs, or to this IP address, or whatever. This is an on-demand, you can think about it like a logging system in a way, and right now it's very coarse. We can only give you request-level HTTP envelope metadata, no headers, or payloads, or anything like that. Because we're working on the security model around this, if we started exposing payloads from a system like this, or even headers, we can start exposing auth data and all sorts of things just by querying the proxies. This is now getting integrated with the Kubernetes RBAC model so that you can set RBAC policies on who can Tap which things, which will allow us to enable those deeper introspection features in the data path.

Linkerd: Reliability

I think load balancing is the sharpest tool we have in the shed. There are lots of other things we can do, retries, timeouts, circuit breakers, but latency by load balancing is super important. Here's a great example, this test is just a simulated test with a one-second timeout. What we see here is that just by changing the load balancing algorithm, we can change our success rate from 95% in the round robin to 99.9% in the peak EWMA, the Exponentially Weighted Moving Average. Almost every system I've been on call for, 95% success rate is a wake me up success rate. You can sleep through the night just by changing your load balance algorithm, is basically what I'm saying.

Linkerd: Security

On the security side, we've really focused on integrating this closely into Kubernetes. The way this works is every Kubernetes pod by default has a service account, which is a way it can communicate with the Kubernetes API. We use that to authenticate to the control plane so we can get a certificate. Every pod generates its own private keys, the private key never leaves the memory of that pod. As soon as the pod goes away, the key is lost. It issues a CSR to the control plane, which checks the service account token against the Kubernetes API, and it all comes back.

There are some issues with this right now we're still working through. One, being the service account token model in Kubernetes is broad, so if that service account gets taken, you can forge identities all you want forever. In Kubernetes 1.15, I think it's stable, I don't actually know. There's a new feature coming online, though, that will allow us to have audience-bounded service account tokens. We can say, "This token can only be used to get Linkerd's identity, and nothing else." If it's lost, you only lose this one leg and not the Kubernetes API, vice versa. Also, we're going to have time-bounded service account tokens, so they can be rotated every 10 minutes. If you lose one, it's ok, it gets invalidated, and we can keep going on.

The other big thing that we want to do here is make it by default. If you have to go think about identity and configure all this yourself, that's a lot of extra work. We want this to be like the iMessage thing, where you get the blue bubble and it's good. If you're both using iPhone, you're both secure. We've done something similar here, where if we have the mesh on both sides of things, if we have a proxy on either side of a connection, we make sure it's TLSed, mTLSed. We don't do any TLS right now for anything other than mesh communication.

If you want to get TLS traffic into your cluster, you want to use an ingress. If you want to use TLS traffic out of your cluster, you probably want to get your application to actually have that trust built into the application of what it's talking to. We really just want this to be mesh mTLS. The big value we get of doing mTLS besides the security is we now have identity. I have a canonical name that I can associate with a pod, that is not the pod itself, but is the pod's service account, which is really the only identity we can establish in Kubernetes at this point.

That said, we've been in production for over two years, both Linkerd 1 and Linkerd 2, a ton of users. In the last six months especially, Linkerd2 has gotten a lot more external contributions. I'm going to ask at the end as well, but if you're at all interested in this, I really encourage you to come to our GitHub and get involved. We have issues that are easy to get started with, and it's a great community. We do stable releases about every two months or so. Edge releases we do weekly, so those are very iterative and end up being a stable release, so please come try it out.

The Trough of Service Mesh Disillusionment

I'm going to talk a little bit about all the problems you might have. This may look familiar to some of you, this is Gartner's hype cycle graph. It talks about how we overestimate the usefulness of technology at the outset, way overestimate it. Then, everyone's like, "Oh, man, this is useless. It's garbage. It's too much," and everyone falls off. Then some of the people who stuck around end up bringing real value out of that thing and become productive. After that, it becomes a boring technology that no one's talking about. I think we've been through this with container orchestration in the past couple of years, or Mesos, and there was so much stuff going on. Now, Kubernetes is boring, honestly, which is good, and people are being productive with it. We're up on that slope of enlightenment now.

I think in the service mesh space we're in that trough, or somewhere in that trough. It's because it's gotten a lot more crowded, when we started three or four years ago with Linkerd, we were the only game in town. People would look at us like we had three heads when we were talking about adding sidecar proxies to things. Now, everyone's doing it, we have Istio from Google, we have Consul Connect from HashiCorp, App Mesh from Amazon, a whole army of these things. That means that there's so much noise in the space, and so people are getting quite confused, I think. What I want to impress upon you all is that it doesn't have to be so noisy, it doesn't have to be so complex. We can have a much simpler conversation about this, I think.

What Can Go Wrong?

There are three things I've really seen go wrong as folks try to adopt a service mesh. The first one being they can't even get it working, it's just too complex, "Oh, I've got to learn all this new stuff." For instance, with Linkerd 1, and Istio, and Consul Connect as well, they have their own set of abstractions that you basically have to learn to get started. You have to learn what a virtual service is, you have to learn what a Dtab is in Linkerd 1. We really wanted to eliminate that, and just make everything Kubernetes-native so you don't have to learn new things. Also, we added a bunch of check commands like this one here, which make it possible to check your cluster to see if it's even ready to get Linkerd installed. There may be some configuration issues where your install is going to break. If you install and it breaks, you're probably not going to keep at it too much longer, you're going to move on to something else, so you can move forward.

The other conversation we have a lot with folks, or people come to us and say, "I want to use a service mesh." We say, "Great. What problem are you trying to solve?" They're, "I don't know. What does a service mesh do?" Or the other one is they've seen a blog post, as we all have, and there's a big checklist, and they're, "Oh, I want all these 10 things." We're, "Ok, do you want to prioritize?" "No, I just want them all. I want security, and I want tracing. I want this, and I want that, and the other thing." "Ok, great." Those projects almost always fail. It's because if an organization can't actually list their priorities, they can't make trade-offs about how these things are deployed.

The other part about this is that we see people try to install it everywhere. To use a service mesh, it has to be running on every single pod on the cluster. It has to be totally ubiquitous in every service. That's a really good way to upset most of your engineers all at once. What we've tried to optimize for is an incremental approach, where you have to get your cluster admin basically to grant you permission to be able to install Linkerd. Once you do that, you can just install Linkerd on your pods, and not other services, and that's fine. You'll get limited value out of it. It's definitely better when everyone's participating in the mesh. You will get the metrics, you'll get a bunch of the reliability features. You'll probably get less of those security features.

The other huge challenge that's set up in both of these is that it's always the mesh's fault. When you add a mesh to the equation, it's the first half of the network, it's the new DNS. Every time you're going to push a request through this thing, something can go wrong. Before, if something went wrong, you'd be like, "Oh, the app's broken," or, "Kubernetes is broken," or something. Now, you're, "Oh, it's the mesh. It's the proxy," which might be true. What we've learned is that we have to ship with a much better alibi around that.

It's All about the Traffic

I want to take a little interruption here to show you how Linkerd works, what it's like to use Linkerd, just to give you a taste of it. This is the Linkerd dashboard, Linkerd dashboard is a command. It will open the dashboard via port forward. We can also do things like "linkerd stat -n linkerd po." This is doing metrics queries for the Linkerd control plane itself. The reason there are so many pods, I'm running this in HA, so it's a little bit more prediction. You can run this with a much smaller footprint than that. If we look at the Linkerd namespace itself, Linkerd is running in a proxy.

We can see here that we have three of three mesh controllers, etc., and we have a whole bunch of obvious goal metrics for each of them. If you click into these, we can get topology information. Another point I want to bring up is that we frequently talk to folks who want tracing. They say they really need tracing, "It's really important that Linkerd support tracing." We say, "Why?" The reason we have not really prioritized tracing as a Linkerd feature is because it requires application change. We have to actually modify your code to be able to increment the spans, or at least pass them through. If you have to do that, why not just use a tracing library in your code? Then, Linkerd doesn't get in the way of anything and you're fine. We will add it eventually.

The real thing we hear is that people have seen the awesome octopus graph, or the big spider web, and they're, "I want that. I want topology for my service, and I want to be able to do it automatically." Well, we can do that from stats. Because we're using Kubernetes metadata in our metrics, we can actually reconstruct call graphs. We can't track individual requests through a call graph right now, but we can give you topology information just for metrics.

Here, we're looking at the Linkerd controller. The website we're viewing is calling the controller right now to do these metrics lookups. In turn, the controller is calling the Prometheus endpoint. We also have these Tap queries, and these Tap queries come in here.

Here, we see actually a live view of requests going through the controller. We see the requests going to Prometheus. We see them coming from the web, etc. We can click into these things if we want to, and they give us more data there. We can also do ad-hoc Tap queries in the UI if we want to, or we can do it on the CLI as well. Also, we have a new feature here which shows off the identities of the services. Here, we see we have service counts for each of these. There's another application as well that will have more services. We can click through this. For instance, the web. This is boring, because there are no failures.

If we wanted to quickly look at something that had failures, we'd see this Emojivoto app. Here, we see something that's a little more interesting. We have two services, one that is perfectly happy and successful, the emoji service, and the voting service, which Phil over here wrote. We can actually look at that via Tap.

We can get a quick view of this, we see these services coming in. Everything is green except for this one endpoint. We know that Doughnut endpoint is broken, the last code we did there is bad. Again, this is no configuration, I have not configured the Emojivoto app at all to know about Linkerd. All we've done is added the proxy to it when we install it, and we can get this great debugging information.

This is what I'm saying, it's about the traffic, if you go to the Kubernetes dashboard, you don't have anything about the traffic between the services. You only have the pods and resources themselves.

On that point, Kubernetes services are a really important concept. I was shocked when I looked at Kubernetes and saw service discovery as a primitive in the actual API. It's a really weird API, I don't know how many of you have dug into the service and endpoints API. What services allow you to do is basically select over pods. Not necessarily just pods, but you can select over pods and add them to a service. Then, there's readiness and liveness checks, so you can configure a pod to fall out of the service if it doesn't pass its liveness check, etc.

One of the important things I've learned recently, in the past few months, is that deployment replica sets, which are the de facto way of running things in Kubernetes right now, are actually the lowest common denominator tool. Most folks we talk to who are going into larger Kubernetes environments, or more mature organizations with more mature workflows, are writing custom controllers for their own workflows, to schedule their own pods. When we started building Linkerd 2, we made a lot of assumptions around deployments and things like that. We really wanted to talk about those high-level constructs. We've learned that those are not good assumptions.

We actually have to work on the lowest common denominator, which are pods, and then be able to reference back to the Kubernetes APIs to figure out what controls those pods, what the parents of those pods are. Services are totally disjoint from that, so it's a really nice way to plug in the traffic. Services just select over pods themselves, so I can make a connection to a service and just send a request. It responded, I don't have to know anything about how it's deployed, or what workflow drives it, or anything like that.

The Service Mesh Interface

This leads us to SMI, which is the Service Mesh Interface. Anyone here familiar with service mesh interface? A month ago at QCon EU, HashiCorp, Istio, and we at Linkerd did some integrations with Microsoft, who's pushing the SMI, basically, as a set of the most common standard APIs you need for a service mesh. Linkerd has its own CRDs and APIs internally, Istio has its own. We don't want to make folks who have to integrate with a service mesh, especially people building other projects or products like Flagger, or Datadog, or any of these things, we don't want to make those folks have to pick one. These integrations don't have to be so tightly coupled to such a large project.

For instance, the traffic management API, which we're calling Traffic Split, is a way that you can hook into services to split services. It's slightly different from Istio's model, but you can think of it similarly. That's a nice thing that we can implement over Kubernetes services, and it really doesn't have anything to do with any implementation of what the mesh is behind it. Some folks have talked about SMI as a way to run multiple meshes at once in your cluster, or switch between them freely. I don't know about that, I'm highly skeptical, but you should think of it as the common layer for integrators. It's an integration point where we can all participate. Even if we have our own internal details and our own APIs that are better than theirs, we can still integrate through these APIs. That's the goal there.

We started with three, one being traffic splitting. That's for canaries, and blue-green deploys, and things like that, the other is called Traffic Policy. I prefer ACLs, because all they are, are basically RBACs for, "Can I call this thing?" We have a generic telemetry endpoint, so this works a lot like the custom metrics API, for those familiar with that. This is an API extension we can add, we have one that talks to Linkerd's Prometheus, and can hydrate that full of rich data. We hope that folks like Kiali and all the dashboard and monitoring folks will integrate around this API, so they don't have to understand our metrics format specifically, or Istio's, or any of them. There is an InfoQ article up right now, actually, about this. Lachie probably does a much better job explaining this than I am, I would suggest reading that.


Looking forward, as of 2.3, which we released about two months ago, we have all the things I've talked about here today. Two-four, which we're hard at work on when we're not giving talks, is really focused on Traffic Split, and starting to bring some of the SMI work into Linkerd. We've also done a lot more work in terms of splitting up the install. This is setting up for a bunch of the RBAC-type changes we want to make in the controller. This is so you can have the cluster admin basically grant you permission to install Linkerd and manage it yourself. After that, we have a number of things we're considering working on. If you care about this, definitely chime in on the GitHub. That's how we prioritize things, when people tell us they want things.

We're really focused on completing the SMI integration, so that would be the ACL side of that. We think that API has to move forward a little bit before we integrate there, so we'll be doing some work with those folks. Of course, distributed tracing is something that's on the table for Linkerd to participate in, but doesn't need to drive it. Mesh expansion, or multi-cluster might be a better way to think of this. Most people running Kubernetes are actually running multiple clusters. Frequently, they say, "I want the mesh to work on multiple clusters." I say, "Great. What do you want?" They say, "I want it to work on multiple clusters." The point is we're still figuring out what it actually means.

At a starting point, we expect identity to span clusters, so you can reference identity across clusters, and do validation, and RBACs, and things like that. Whether we're doing federated metrics and that sort of thing, I'm much more skeptical. We will find out.

If any of that is interesting, if any of the philosophy is interesting, we have an InfoQ article that Daniel and William wrote a couple months ago now, but it's still up there, and a pretty good overview of what I talked about today.

I want to implore you to join the community. It's a really great place, I love it. I'm thrilled that I come to a conference like this and run into people from the community, it's a fun thing. I hope some of you, at least, can get involved.

Questions and Answers

Participant 1: There are a lot of things about this that are exciting to me, but some of our services are serverless. Is there either a way to work that into this same dashboard, or, alternately, is there still value in only getting the things that we can get out of this?

Gould: I think there's definitely value in just getting what you can get now, is my opinion. I do think, depending on how you're doing serverless, there's a lot of ways of doing that, there's more opportunities for merging that. For instance, for Knative, and OpenFaaS, and things like that can run in Kubernetes, I think there's an obvious way for us to integrate there. That just needs some blog posts, probably. I don't think there's a lot of technical work that needs to happen. For things like Lambda and special cloud providers, things get a lot squirrellier. We've been talking to some folks who have that want, and we're working through requirements. If it's important to you, definitely get involved in the GitHub and tell us what would be important to you.

Participant 2: Great talk, and plenty of things to think about. What's your take on events rather than RPC? Because a whole bunch of stuff we're doing now is event sourcing, CQRS. Is the mesh going to play a role there, do you think?

Gould: One, it depends a lot on the protocol, you can do events with gRPC, for instance, and that is not inconceivable. You lose some of the value, especially in load balancing and things like that. We are very interested in starting to look at Kafka support and other protocols like that, that will at least allow us to get metrics, and hopefully TLS, and some of those things. A lot of the reliability features fall down in that world, because you have to have so much knowledge about the actual streams of the application itself. I think, yes, it'll be a smaller subset of the features. It's something we're actually talking about.

Participant 3: Can you talk a bit more about the overhead that Linkerd introduces, in terms of latency, concurrency, bursts, memory, everything?

Gould: I actually took out a slide I had on that, unfortunately. A sidecar proxy generally uses about one or two MB if you look at the [inaudible 00:42:08] Top output of memory, RSS rather. I generally see it around 10MB for reasonably trafficked proxies. The CPU side, right now, we only use one core for data processing. That's to keep the resource [inaudible 00:42:23] low. We don't want you to have to allocate multiple cores to the sidecar proxy.

The latency overhead right now is quite low. Once we've warmed up the caches and all that, I think the P99 is under a millisecond, which is awesome. As far as bursts and things like that, that's a good question. We try to do back-pressure into applications, wherever we can, we try to use HTTP/2 semantics, and things like that, to push back-pressure so we don't have to buffer things in the proxy. When we buffer things at the proxy, we have to decide what to do with them if they timeout, etc. I don't have a great general answer there. I will say that we can get to tens of thousands of requests easily on that one core. I haven't seen anyone saturate that core yet, really, so help us try. We can make that a multi-core thing. There's no technical reason we haven't, other than it's simpler the way it is right now.

Participant 4: If we're building a high-throughput, low-latency system, would you recommend working with Linkerd, or just try it and see if it works? Did you run this in high-throughput, low-latency environments?

Gould: We definitely see this used in low-latency situations. In fact, the load balancing can really improve latency and reduce CPU usage in a lot of cases as well. I'm not going to give you a blanket recommendation that it will be great for you. I will suggest that you try it, and help us. You should do your own testing, you should never trust the testing or the numbers I give you on any of these things. Please test before you go prod, people will come to us and they're, "Hey, we're about to go to prod. Is there anything I should know?" I'm, "You should probably know." I don't see any reason why you couldn't use it. I would encourage you to test it yourself.

Participant 4: Thank you, great talk. Do you see service mesh being used for only online requests, or also for data pipelines?

Gould: I think it's mostly important for online requests. That's also my background, so I'm probably very pushed in one direction on that. I don't really think there's a lot of value on the data stuff, but I'm sure someone could find a reason to use it there.


See more presentations with transcripts


Recorded at:

Jul 22, 2019