Transcript
Ruth Linehan: If you've ever heard anything about Rust, you might be familiar with these preconceptions, that your code in Rust is going to be faster, but that the Rust learning curve makes it slow to develop in. These were things that we thought at Momento when we started talking about moving our services into Rust. We'll get performance improvements in Rust, but the engineering cost is going to be 5 times or more. What we learned from the past three years is that these two preconceptions weren't entirely true. That's what I want to talk about today.
My name is Ruth Linehan. I'm a software engineer at Momento, working on high-performance caching for about the last two years in Rust. I have about 15 years of backend engineering experience. Prior to Momento, I worked at GitHub on Rust and GraphQL APIs, as well as on webhook delivery. Then prior to that, at Puppet on infrastructure as code. I'm a polyglot. I am not a Rust expert. I have experience with Ruby, with Clojure, with Golang, with Kotlin, and with Rust. I think notably, I don't have any C or C++ experience. I'm not a Rust expert. I really like Rust. I do not consider myself an expert.
I work for Momento. We do high-performance caching. We run daily performance tests. We're benchmarking for the daily tests at about 60,000 transactions a second, with 3 millisecond latency at the 99.9th percentile on decently large instances. Then we also try to push those instances, or larger ones, as far as we can. We're a startup. We want to use our resources as well as we can, so getting up to 600,000 transactions a second on a larger instance or beyond. We started out with our services in Kotlin. Over time, we have moved all of those to Rust. I'll give a shoutout to my coworker, Ramya Krishnamoorthy, who gave a talk at QCon about our move from Kotlin to Rust. We started out with our most performance-sensitive services. The motivation was performance. Then eventually, this past year, we moved the most complex service to Rust. This is a workflow server.
This is not systems programming. It's not performance-sensitive. Rust was the right choice, first of all, because now everything else is written in Rust. We're very familiar with it. Also, because we could express the complexity more clearly in Rust and have more certainty in it. The ergonomics of Rust made it the right choice for this project. Those ergonomics are what I want to talk about first. I also want to make it clear that I'm not saying you should go out and rewrite your code into Rust. That's not the point of this talk. It's just, go out, rewrite your code. If you have other reasons to rewrite a service, or if you have a greenfield project, maybe consider Rust.
Developer Feedback Loop
Let's talk about the developer feedback loop. As developers, we are more productive when we get quicker feedback on whether what we are doing works or not. From the moment in time that you've made a code change, how long does it take for you to validate whether that change was successful or not? I'll throw another shoutout to a former coworker, Chris Price. He's been a great mentor to me and heavily influenced how I think about code. He also gave a talk at QCon on a similar subject, although more cross-language focused. If you're interested in overall language features and many different languages that help us save time as developers, I encourage you to check that out. I think of the developer feedback loop a little bit as two parts. If you're putting up a pull request, how long does it take you to write code that you know works?
Do you have to deploy it to a dev environment to manually test it? That's going to take a while and it's going to require context switching, which means you're distracted, you're less focused on your code, you forget what you're doing. Then once your code has shipped, how long does it take you to find a bug? The longer it takes, the more impact it has, the harder it is to fix and to find. This is from an internal survey from Google from 2022. I think it's an impressive quote that 85% of respondents were confident that their Rust code is correct. When you're more confident that your code is correct, that's going to speed you up a lot.
Rust's Learning Curve
Let's go back to the first preconception I mentioned, that the Rust learning curve is steep. I'm not going to really argue with that. I had that experience. The conclusion that is often drawn from that is that developing in Rust is slow. Maybe if you're brand new to Rust, you're doing a proof of concept that you just want to get out as fast as possible, you're hacking something together, Rust is probably not the right choice for that. If you're writing code for production, I'd argue that while your initial experience might be a bit slower, your overall time to get into production will be sped up. Let's talk about the Rust learning curve. Here's some pretty basic code. It gets an endpoint from an environment variable and uses that to build some client. We just have a function to read a variable from the environment. We have a client that takes an own string as an endpoint.
All that's going on here. We have a string. We build a client with that string. I'm running this code, and I want to make sure I'm getting the expected value for this endpoint. I want to log that. What happens? I get a compiler error. It's a pretty big error, actually. There's a lot going on here. All I wanted to do is log something, and now I have this compiler error? I think this is where the Rust learning curve comes into play. I wanted to do something that I thought was extremely basic, and now I have an entire seemingly massive compiler error. Let's look at this error. What is it actually telling us? First it says I have a borrow of a moved value. That endpoint was consumed when we build the client, but we're still trying to use it as if it wasn't moved, and we can't do that.
It tells us about ownership. That build_client takes ownership of the string passed to it. Gives us a hint. Maybe you want to change build_client to borrow a reference to the string if it doesn't need to own it. Then it also suggests another possible solution. We could consider cloning the endpoint if the performance cost isn't too great. Then, finally, it tells us where to find more info about this error. Let's try that. This is cool. It took a long time for me to ever actually try out these error explanations, but they're actually pretty great and very well written. I highly recommend making use of them. There's also a lot of text on this slide. I'll highlight a few things here. A variable was used after its contents have been moved elsewhere. This is fundamental to Rust's ownership system. A value cannot be owned by more than one variable.
It goes on. It has a whole bunch more about references. It even goes on from here. A lot of really great examples. What I want to call out is that using a reference, we can let another function borrow the value without changing its ownership. Rust's ownership model. Every value has a single owner. When the owner goes out of scope, the value is dropped and the memory is deallocated. This is key to Rust memory management. You can never have multiple owners for a value, otherwise, you can't safely deallocate it. We have this concept of borrowing, where you get a reference to a value without transferring its ownership, which uses the ampersand syntax. There are some specific rules governing borrowing that ensure safety. We'll get into these a bit more. When we're just getting started, this ownership model and the compiler errors it produces can feel pretty arcane and like we're just getting in our way. This is how Rust does memory management safely and without garbage collection. Rust's ownership model makes you think through what should have sole control over data versus what can reference it. I think these explicit decisions help eliminate bugs before they happen.
Let's take a look at what this looks like in another language. Ruby is known as a language for quick prototypes with pretty easy syntax. Here's a basic example. We have a hash of string to integer. We have a second variable. Ruby is pass-by-reference. That's going to be a reference to the first hash. We print them both out. They're the same as expected. Great. Let's do the same thing in Rust. We have a HashMap from string slice to integer. Ruby is pass-by-reference, so I'm going to follow that. Here in my Rust, I'm going to explicitly have hash_2 be a reference to hash_1. Print them both out. As expected, they're the same. In Ruby, I want to update hash_1. I'll change the value of foo. Again, hash_2 is a reference to hash_1, so it's updated 2. Again, let's do the same thing in Rust as in Ruby.
We'll mutate hash_1 to change the value of foo. Except here, we get a compiler error. It says two things. That we cannot borrow hash_1 as mutable as it is not declared as mutable. That makes sense. We have to specifically tell the compiler what can be mutated. Then also that we cannot borrow it as mutable because it is also borrowed as immutable. This is an important principle of the borrow checker, that something that's been borrowed as immutable cannot be borrowed as mutable. Let's make this code work. First, we'll declare hash_1 as mutable. Then we can do the mutation on hash_1 first. Then borrow it as immutable in hash_2. In this case, they print out the same. Or alternately, we can borrow hash_2 and print it first, and then insert into hash_1 mutably. This also works, but it has different behavior. Since hash_2 was printed before we mutated hash_1, it still has the initial value, or it still prints the initial value. Again, in Rust, we have two options that work, but with different behavior that we have to decide between explicitly. Then we have this third one that does not work, because we can see that there's an immutable reference outstanding when we do the mutable borrow.
I like explicit code. I really like when I can read my code and I know how it's going to behave. I don't like unexpected behavior. Code surprises lead to bugs. Here's an example of what I would call surprising behavior in Ruby. I have a hash. I'll call it inner. I have another hash, again, hash_1. Inner is a nested value on it. In my previous examples, my second hash was a reference. In this one, it's going to be a clone of hash_1. Clone, I expect it to be divorced from hash_1. I'll change its name first so that it shows up when I print out the hash as hash_2 and hash_1. I've cloned it. It's very unexpected to me that if I mutate inner on hash_1 to change the value of foo, it gets changed on hash_2 as well. What's actually going on here is that although hash_2 is a clone of hash_1, within it, inner is still a reference.
Mutating inner, even on hash_1, also mutates it on hash_2. In Ruby, we can inspect the object IDs. You can see that although hash_1 and hash_2 are different, that inner on both is the same. I would consider this unexpected behavior. I was surprised the first time I encountered this in Ruby with HashMaps. It took me quite a while to discover that this was the root of a bug I had. I lost a lot of time trying to figure out what was going on. What happens if I try to do this in Rust? For simplicity and code conciseness, I've turned the outer hashes into structs. Otherwise, this is the same idea. I have a struct with a reference to a nested hash, and I clone it. Then I try to insert on inner in struct_1, and I get a compiler error because it's already behind a reference.
I could try to make inner mutable, but once I've put it in the struct, it's been borrowed as immutable. I can't mutate it. There's a few other ways I could try to contort this code to work, but it's just not possible. There is no way for me to make this work. Rust just doesn't allow the unexpected behavior here. If I did want to do something like this, the inner HashMap that the struct has can't be a reference. I can no longer mutate inner itself because it's been moved into struct_1. I explicitly have to choose whether I'm mutating inner on struct_1 or on struct_2. When I go back months later and read this code, there's no surprises. I can tell exactly what it does, and the compiler prevented us from writing unexpected code.
We talked already about how you can't have a mutable borrow if there's any outstanding immutable borrows. Another key feature of the borrow checker and the safety it provides is that only one mutable borrow is allowed at a time. You can't do this. You can't have a HashMap that is passed to two threads that are both going to mutate it. Any time we have two threads mutating shared data, that's going to cause data races, unless you have some synchronization. In many languages, you just have to watch out for this and catch bugs in code review. In Rust, the compiler just won't let this happen, which is pretty powerful. We talked about how the borrow checker ensures that only one reference that is going to mutate data is allowed at a time, and that also, if the data is going to be mutated, that nothing else is currently referencing it.
To put this another way, the compiler ensures that a mutable reference to a piece of memory is a unique reference. Nothing else in the process has a reference to that piece of memory. I should perhaps have a note on this slide and more accurately say data races are not possible in safe Rust. You can write unsafe Rust, where this won't be true. Sometimes you do have to do this. I haven't had to in anything I've done, but you sometimes have to interface with another language. Then you have to explicitly mark your code as unsafe. Sometimes you do need to deal with mutable data across multiple threads. To do that correctly, you have to involve synchronization. Mutexes are one way to do this. They're not unique to Rust. They guarantee mutual exclusion to the data by a given thread at a time. In a lot of languages, a mutex is just the lock.
Its API is that it is just locked or unlocked. You have to remember specifically to lock the mutex protecting a piece of data. Here's an example from Kotlin. You can see we do mutex with lock to access the data within a lock. I could also forget the mutex and still access the data. Rust deals with this a bit differently. It has types that themselves have internal mutable state. Because of that, the compiler continues to enforce safety. Continuing our mutex example in Rust, a mutex actually contains the data it's protecting. You can't get access to this data without going through the mutex. Once we lock the mutex, we get a different type, a MutexGuard, which gives an exclusive mutable reference to the data. In this way, the compiler enforces that this is the only way we can access the data. Since it's a mutable reference, that it's exclusive access.
Therefore, we're guaranteed to avoid data races. We don't have to remember to lock or unlock a mutex. We don't have to remember if our code is thread safe or not. We're guaranteed that. This is a small example about Rust and concurrency. I'm just using this to show another way that the Rust compiler guarantees safety. For more on concurrency in Rust, I want to call out the book "Rust Atomics and Locks" by Mara Bos as a great resource if you're more interested in that.
In order to provide the safety we've talked about, the compiler, and specifically the borrow checker needs to know how long references are going to live for. That's how it can say when it's safe to borrow something mutably, for example. When you talk about the Rust learning curve, something that people will inevitably bring up is lifetimes, as this confusing thing. We've been talking about how long things live already. In all of our previous examples, the compiler has been able to figure that out on its own. Sometimes the compiler can't tell how long things live on its own, and so you need to help it out. That's where we get into lifetimes and lifetime annotations. Here we have a function that is either going to return the value for a key in the HashMap, or it will return the string reference unset. The compiler gives us an error, missing lifetime specifier.
It tells us that the return type for this function is a borrowed value, but that the signature doesn't say where it was borrowed from. There's several possible options. The compiler doesn't know how to figure it out. Again, here's our function. The HashMap keys and values are string references, as is the key that is past it. In order to enforce safety and not have Use-After-Free errors, it needs to know which of these references the return data is referring to. The compiler suggested quote A as the annotation, and in its suggestion it put that annotation on every string reference. We don't need it on every one of the string references in the type signature. When annotating the lifetime, we need to tell the compiler which data from the input the returned data is going to live as long as. In this case, we're returning the value from the HashMap, so we need to annotate the same lifetime for the HashMap value and for the returned string reference.
The compiler suggested A, but the lifetime annotation doesn't need to be that. It's just an annotation. You can name it whatever you'd like. I could name it something much longer and clearer than val, but then it wouldn't fit. We say that our lifetime annotation is val and put it on those string references. My initial experience with the error missing lifetime specifier is, no, I don't know how to deal with this. What even is going on? I'm just going to redo my code to not have references at all. If you think through the lifetime of the borrowed data, you can get there. This is an important part of the borrow checker and allows it to enforce safety. Sometimes you just have to help the compiler along.
We've talked a lot about the borrow checker and about Rust's ownership model. Something I haven't explicitly mentioned is Rust's type system. I like statically typed languages. I think Rust's type system is a pretty nice one. It helps with compile time safety, but it has a reasonable amount of type inference so you're not having to specify types on every single variable when the compiler is able to figure it out. I've done a lot of work in languages like Ruby and Clojure that are dynamically typed. I just generally found that there's a lot more to keep track of. I think Clojure, for example, is a really fantastic and cool language. It's a Lisp. Writing it taught me a lot about immutable code, about functional programming and avoiding state. I also spent a lot of time when writing Clojure and reading Clojure and working on a large codebase trying to figure out, what gets passed into this function?
There's more archaeology when reading and writing the code. Statically typed languages help with that, but up to a point. Here we have a struct Account. It has a few fields that are all strings. These include an account ID and also the user ID of the owner of the account. I go to instantiate myaccount struct and I accidentally flip these two IDs. All these fields are strings, so nothing catches this. You can probably catch something like this with a test, but a test is a good intention. Maybe you forget to write a test and then you don't catch this until it hits production. When the compiler catches bugs for you, that speeds up the feedback loop. Let's lean into the compiler. Rust has something called the new type pattern, but you have to do a lot of boilerplate with that. It's ugly. Rust also has macros, which we can use to write some code that's going to write other code for us.
Here we define a macro, typed_string, and that just wraps our string in a struct. I've simplified this somewhat, but you can do more complex things like include serialization for your API formats, for example. One thing, though, is that we purposely don't have a two-string implementation here. If you're more familiar with Rust, we don't derive the display trait, because we specifically want developers to have to decide whether they want to borrow or clone the inner string. We want you to be explicit about it and think through it. Going back to my example, now let's declare our three different typed_string types. We have account name, account ID, user ID. Now if I accidentally swap the IDs when I instantiate the account, I get a compiler error. This bug didn't go out to production. I didn't have to write any tests. My code just didn't compile. This thing where a basic type is wrapped in a custom type so that the compiler catches mismatches is not unique to Rust. You can do this in many other statically typed languages. Kotlin, you can use data classes, for example. Having a macro that does the boilerplate for you is really nice. I also wanted to call this out specifically because this is a bug I shipped. Now that we have our typed_strings macro, I cannot ship this bug again.
Safety Features of Rust
We've gone through a few different examples of how while, yes, the Rust compiler can feel like we're jumping through a bunch of extra hurdles, it also provides a lot of safety. To review a few of those features we've talked about, the ownership model and the borrow checker, so less manual memory management, no dangling pointers or Use-After-Free errors. Only one reference can be mutated at a time. Then part of this is the notion of a lifetime, that data has a lifetime. Safer concurrency. Less unique but still great, the type system and compile time type safety. A result of these features is that the code is more explicit. There's less surprising behavior. That it's elevating wrong code into uncompilable code. You're catching bugs at compile time rather than runtime, which helps you to be more confident in your code and more productive by decreasing the feedback loop.
Rust is Fast, but Performance Isn't Free
I started out this talk with two preconceptions. As I said early on, these were things that our team at Momento expressed when we started out with Rust. We'll get higher performance but with the downside of reduced velocity and higher engineering cost. So far, I've talked about how I don't think that that downside played out and that long-term the engineering cost has been lower to write Rust. What about the other preconception? Actually, when we first rewrote our most performance-sensitive service into Rust, it wasn't actually faster than in Kotlin. In general, Rust makes it easier to write faster code without contorting yourself, you don't have to deal with garbage collection or complicated memory management. You're not going to get all your performance benefits just from writing Rust. You still have to write good code and architect it well. Like with what we talked about earlier, Rust is going to make you think through what you're doing and write explicit code.
That's code where you can find and make performance improvements. Here's an example. This is a diff from a pull request I made. We had been calling format for our metric names multiple times per request. Format is a macro that does concatenation, interpolation of strings. It produces a new string. In the case of our metrics, that was happening in a request hot path. In fact, several times a request, which is a lot when we're running hundreds of thousands of requests a second. Each format, each new string requires a dynamic memory allocation. The compiler doesn't know the size of the string, so it has to go out to the operating system to coordinate that. That takes time. In comparison, the size of a static string is known at compile time. It's on the stack, so it's faster. Even though the previous code was drier, there was less repeated code, in load testing, this change got a millisecond of latency improvement for the 99.9th percentile of requests. This isn't a complicated change, but it is something that didn't just come for free. Once we found this, once we thought through this, Rust gives us access to the primitives to find and make changes like this fairly easily and to be confident in them.
A lot of the hard part of improving your performance is figuring out where are the performance problems, or which possible change you have is going to solve them. There's a bunch of different tools you can use. I'm going to give a few examples of things that I think are cool. These are some of the things we use at Momento, but there's many more options out there. Let's go back to the start of this talk, to our very first code example. We had this code that didn't compile, and the compiler gave us two possibilities for how we could fix it. How do we decide which one to use? In the real world, there's probably some functional considerations here around whether the client should take an own string or a reference. For the sake of this talk and having examples that can fit on a slide, let's ignore that for a second and just look at the performance implications of these.
One tool that's available is micro-benchmarking. We can use a crate called Criterion, which will run our different possibilities a bunch of times and get data about how long each one took. Criterion has a specific way it wants to lay things out. We make a directory, benches, and put our benchmarking code there. We define our two different options, the one where we have a struct that takes a reference and the one where we're going to clone the string. Both of them compile. We'll define our bench function. We'll say we want to benchmark a group. That is, we want to compare these two functions and give them a name. In this example, we don't have any inputs to our functions, but if we did, we could also include that and look at how the function performance changes based on the input. We run cargo bench, prints out some data on the command line showing how many iterations it's running, samples it's collecting, some of the measurements.
The really nice thing is it gives us an HTML report that has a few different graphs. I just have pulled out one of the graph types that it has. This chart shows the average time per iteration. The shaded region shows the estimated probability of an iteration taking a certain amount of time, while the line down the middle shows the mean. This shows that the version where we borrow the endpoint is faster. Not a super big surprise. It's not having to do dynamic allocation for a string every time. We can see that in the cloned example, the tail is longer. The median is shifted over to the right there. Because when we do dynamic allocation like that, it's interacting more with the operating system, and so there's more variation, depending on what else the operating system is doing. Obviously, this is a very simple example. Criterion is a cool tool. Micro-benchmarking in general is very cool. Especially when you have a few ways you can implement something and you want to get some measurable data quickly around how they behave in different scenarios. Because sometimes what you think is going to be fastest isn't actually. It's helpful to find that out quickly.
Let's talk a little bit about other tools we have, like profiling and flamegraphs. In these examples, I've used the flamegraph rs, Rust crate. There's other tools out there that will do this. Again, this is not unique to Rust. Flamegraph is a tool that can be used to visualize where a program is spending its time. Here, I'm running on Linux, so it's using the perf profiling tool. Then the flamegraph visualizes the data that perf provides. Many times a second, the program that you're profiling is interrupted, and a stack sample is taken, showing the current chain of functions for each thread. The flamegraph takes all of these samples and combines them so that the common functions from each sample are added together. The y-axis shows the stack depth with the most recent function on the top. The x-axis does not show time. It can be alphabetical, but it generally doesn't have meaning.
What does have meaning is the width of each box, of each function call. It shows the total time that function is part of the call stack samples. It's combining all these different samples together. You don't necessarily know if the function was called more times, and it showed up more because of that, or if it took more time, and so showed up because it took more time, or both. It just shows the percentage of the total of CPU time sample that was spent on this function. Let's talk about a slightly more complicated example. Let's say I have an HTTP server that supports key-value datastore semantics. I have a route that allows me to set a key with a value, and I have another where I can get the value for a key. There's a bunch of other code I'm not showing setting up these routes. I'm using the Axum web application framework, Tokio async runtime.
That's less important here. I'm calling my datastore, SleepDb, because under the hood, that's what it does. On set, on the write path, it sleeps for 10 milliseconds. This is not great. Maybe I'm writing to a separate datastore. Maybe in some other world, I have a separate datastore that has increased latency, and I don't really have control over it. That write path is taking a lot more time. Let's say also that I care more about the latency of my read path of my gets than my sets. I can't really do anything about the set latency, so I'm just going to try to improve my gets and see what's going on here. In the first iteration of my code, I'm using a mutex to coordinate access to the HashMap with that data. Any time I access it, either to read or write, I have to acquire the lock. The set path is holding that lock for 10 milliseconds.
We can look at the flamegraph for this. Flamegraphs are not great for slides, because they're not just flat images. The format I have is an SVG. I can load it in my browser, and when I hover over an individual bar, it'll show me the full name of the function call, as well as the percentage of the total CPU of the whole flamegraph it took. I can search. I can zoom in on specific areas. I cannot do that on a slide. Try to narrate it. In the example code I wrote, I said there was a bunch going on, HTTP server, TLS, all of that stuff that the Axum framework is doing. In this example, we're saying we're really just caring about what is going on in this get path. How can we make it better? We can start out by searching for the get function. Here's where it shows up.
We can hover over it and get the full name, and also see in how many samples it showed up and what the overall percentage that was of all the samples taken. Here's where that is on the whole flamegraph. Again, so we're only going to look at this small portion of this flamegraph for now. When we zoom in, we see that there's roughly three different chunks of things going on. On the line above the magenta highlighted one, we see there's three boxes. Then each eventually break into some other things. We have a portion on the left that was involved in unlocking of the mutex when it was sampled. We have a small portion in the middle involved in an actual get on a HashMap. Then we have a much bigger part on the right that is blocked on the lock. It's trying to acquire the lock and completely blocked.
Let's talk a little bit about mutexes in Rust. You have a thread waiting on the mutex, waiting to acquire the lock. It spins something like 400 times checking to see if the lock is released. That goes really fast, like microseconds. If it's not available, then it jumps to the kernel. I should also say here, I'm specifically talking about the Linux implementation for this. How it behaves in Mac or Windows, a little different. Linux, it jumps to the kernel, and it does a futex_wake. The kernel parks that thread, and then later it gets notified that it can get access to the lock. Even if it only goes to that futex briefly, there's a bunch of coordination that has to happen here with the kernel. Any time you see lock contended at all, that's not good, and might have a higher impact than what you're really seeing in a flamegraph. I know there's some other things we can see here. We can see in the unlock, there's a swap going on in an atomic, and then there's a syscall there. Mostly, a lot of our get is waiting on getting the lock, on acquiring the lock. We should consider what other options do we have rather than this mutex.
Performance Feedback Loop
My initial plan for this talk was actually to go through a few different versions of this read path and different improvements, different types of locks or other things, and see what the flamegraph looks like. It was getting more into the weeds about finicky details and less about Rust. I've already gone on for a while, and so I'm not going to go further on that for now. I think we can see the lock is contended in that read path, and that's where we should invest some time. I did end up going through a few different things, with my little example here. I was able to make quick iterations to this code because I was confident that it worked when it compiled. The Flamegraph-rs README, so I talked about that, was the tool I was using, it's great. It has good pointers about how to use flamegraphs.
One of the things it calls out is that humans are terrible at guessing about performance. Some other point is that the Rust compiler is actually quite good at making optimizations that are not necessarily obvious. Just in general, we're not good at guessing about all of what's going on and how that's going to play out in terms of the performance of our code, which is why experimenting is great, and then getting data about the performance and being able to change your code quickly and with confidence and repeat. That's another feedback loop that we experience as developers. We do a lot of performance work at Momento. I want to give a shout out to the folks at IOP Systems, and especially Brian Martin. Among other things, they make a benchmarking tool we use to drive load, called rpc-perf. We use rpc-perf to run performance tests and then look at the client metrics that rpc-perf has, as well as the server-side metrics that we have, the system metrics, tools like flamegraphs, htop, and say, did this code change have the impact we thought?
Did tuning the number of worker threads make a difference? What if we combine these two options? Does that improve things or make it worse, and come to real performance wins. It's this constant feedback loop of experimenting, getting data that gets us eventually to improve performance. One other thing I'll mention. I said that when our team rewrote our code from Kotlin to Rust, it didn't get any faster. In the world of performance, fast isn't everything. Our latency stayed the same, but we went from being able to drive 20,000 requests a second to 100,000 at the same latency on a single instance. Performance is throughput also. Rust can help you get higher throughput with less resources and save money, which for a small startup we care about a lot.
Takeaways
Let's go back to our starting preconceptions. For our team, when we started, there was an estimate of a 5 times engineering cost for writing our code in Rust. We're now over three years in. I feel pretty confident saying that that estimate has not panned out at all, in part because of the safety features of the language that have helped us to move faster overall. If you're trying to hack together a proof of concept and get it out the door really quickly in a couple weeks, Rust might not be the best choice. If you're writing code that's going into production, Rust will allow you to express the complex code explicitly with greater confidence in your code, and the safety benefits of the compiler will help your team move faster. You can write slow Rust. Your code is not guaranteed to be fast. If you put a sleep in the middle of it while a mutex is held, it's not going to be fast.
If you can write fewer bugs, if you're more confident in your code, that gives you more time to write systematic improvements to architect your code well, to use performance tools to run experiments, and to make your code faster. It's likely that you're going to save money because you're doing better with your resource management as well. You're not dealing with garbage collection or manual memory management. You're doing less contortions to make fast code. When you improve the performance of your code, you can be sure that it works. We can talk about performance as well in terms of your team's throughput, your ability to avoid bugs, and ship safe and hopefully fast code to production. Rust helps with the performance, resource management, and speed, not only of your code, but of your team too.
Conclusion
Obviously, I really like Rust. Even if you don't end up using Rust on your team, I highly recommend you check it out. I think writing Rust has made me a better programmer overall.
Questions and Answers
Participant 1: I have two questions actually. One is about compile times. Number two, I also saw that you have a few async examples. That's its own learning curve, I feel, especially in Rust. What did you feel about those two? I also see that you've done a lot of Clojure. Maybe comparisons to Clojure would be good.
Ruth Linehan: Compile times, the first one. Sometimes you have to wait for it to compile. I think that the Rust project has done a lot of work on that over the years, so it has improved. We have a monorepo for our Rust code, and we have several really big dependencies. I was going to say my dislike for the AWS EC2 Rust SDK because it's massive, and it doesn't need to be that big. It can sometimes take a while to compile. I don't feel like that compile time has been any worse than in other languages I've dealt with. I've talked about writing Kotlin. I feel like waiting for our Kotlin code to compile and then the linter and everything else, and then compared to my Rust code is just the same. We did have to do some work in our CI, making sure that we were caching some of the dependencies, caching some of the stuff that was compiled to make it not run too long. It wasn't very hard when we invest, like I think one person invested a day on that. Yes, definitely a puzzle.
Your second question was that async. Yes, it definitely is its own learning curve. We used the Tokio async runtime. Some of it is I had the benefit of a coworker. Having one person on your team who does have more Rust experience, I think that we would have had a harder time overall if we didn't have that. We had one person who does have Rust experience and then most of the rest of us were not Rust developers at all when we started. Having that one person who did have experience and could say, here's how we should set this up, here's what we should be doing, was really beneficial. Because I think, yes, figuring out some of the how you should set things up is a little bit more challenging. I think now that we're there, once we got some weight, here's the patterns that we should use.
Tuning it has been another thing. We've been doing a lot of performance work, and a lot of like, what are some of the settings we should be using with Tokio? How can we get more performance? That has been a thing. That's where I was talking about, I've watched a bunch of my coworkers recently just constantly, if we do this, then what? If we do this, then what? A lot of that is very specific to what exactly we're trying to do. Yes, tuning it has been hard.
Third question was about Clojure. It's been about a decade since I've written any Clojure. It's been a while. The thing that I always felt the most with Clojure really was the lack of a type system. We were using at the time a project called Schema. You can, on all of your functions, on everything, define, here's what the inputs and the outputs should be. It was not compile-time safety. It was bolted on. I talked about that code archeology. If you're working in a big codebase, and you're trying to find a bug, and you haven't really read this piece of code before, having the type signature is very helpful on that kind of archeology. I think that's the thing that I wish the most that Clojure had had. The actual writing functional code, that was early in my career. Writing a Lisp, and writing functional code, and learning about state, and how to avoid state, and avoid mutating state, how that makes your code more readable and more understandable was really valuable to me.
Participant 2: I often see Rust being compared to C++. Now where I work, most of our code is in C++. There is some interoperability that we've built in internally. I wanted to ask you if you had any strategies, or if you've ever had to migrate from C++.
Ruth Linehan: I have not. I have not written any C or C++. My perspective is maybe a little different from a lot of people who come to Rust, because I don't have that background at all. We don't do any interoperability there.
Participant 3: When you made the decision to move the first service from Kotlin to Rust, what was the knowledge level for Rust of the engineers in the team that made the port? How did you convince business to do this? Because it's like, they have to trust that this really moves business forward, and not just do it because Rust is cool.
Ruth Linehan: Yes, definitely. I mentioned that we had this quote of 5 times engineering effort. I went and found this Slack thread from a couple years ago when we were first discussing this, of like, should we do it? There were a few things. We did have one engineer who had mostly experience playing around with Rust, and was like, this is really cool. Where our business was going was, what's the latency SLOs that we need to achieve? Part of it was also, can we reach those, and also add a higher throughput with less resources? Having to use fewer instances. That was part of the business consideration is like, we're using AWS, what is our spend on that? Then, long-term, is it more worthwhile to invest in continuing to optimize our Kotlin and going really even deeper on that? Or, are we going to probably eventually have to go to Rust anyway for what we want to achieve?
Now is the time to do it. I think the one other thing, we did have a very small Rust project. I think the one engineer who was into Rust and had made an explicit decision, let's go write our internal CLI tool that we use for all of our operations as operators, let's write that in Rust as a first push of like get a little bit used to it. That was a case where it's like, it doesn't matter if it panics, it doesn't matter if it has any performance issues, it's just the little internal CLI tool. That's the first place I wrote any Rust, was in that. That was a good experience before moving to this async runtime world, and like, what's going on there.
Participant 4: What editions of Rust or version of Rust to use? Do you modernize your codebase basically with numerous features being allowed, like async stuff? Do you actually do work to keep it up to date?
Ruth Linehan: Yes, we keep it very up to date. In our CI, we don't set the version. It's whatever is current, whenever it's released. Another thing I really like about the Rust tool chain is that the linting is just built in, it's immediately there. We go back to Kotlin again, like you have KtLint, it's a nice project, but it's separate. You have to install it and everyone has to have the same configuration. That's not the case with Rust, it's just all right there. That does sometimes mean, because we keep our CI on the most recent version, sometimes you'll put up a PR and you'll be like, I didn't change, what is happening? It's not compiling, or Clippy the linter is telling me stuff. Then it's like, someone will just go and quickly make a PR and do it, and it doesn't. It's like, ok, cool. I think we generally try to like, this new feature came out in Rust, that's really cool, let's use it. I've had a couple of times where someone puts up a PR and then I'm like, wait, my code isn't compiling locally. I need to rustup update. We stay pretty much right on top of it.
Participant 4: When you moved from Kotlin basically to Rust, you not only change the language, also debugging experience changes, everything changes. How did you go from teaching people doing Java debugging versus native debugging, basically going in this direction?
Ruth Linehan: A lot of it was that it was still the same tools that we were using. We were still mostly using our logs and our metrics. We have an internal metric system. We had mostly all the same metrics and then we ported those over, and it's like, ok, trying to measure the same parts of what's going on in our stack. That wasn't particularly different as far as the tooling. We generally have to care less about our memory graphs a little bit. I think the overall actual debugging, we did a concerted effort of making it easier than our Kotlin to run locally and to be able to iterate locally. We had some like, here's the recommendations for how you should set up your IDE. Here's how you should set up your local environment for Rust. I don't feel like it was that much of a big switch there.
See more presentations with transcripts
