Transcript
Spencer Judge: Spencer. I manage the SDK team at Temporal. Today I'm going to share with you some pretty practical advice about how you can potentially adopt this particular architectural pattern that I implemented at Temporal that has really served us very well, where you have a shared core written in Rust, and then a bunch of language-specific layers built on top of that. That's what we're going to talk about today.
A Messy State Machine Diagram
This first slide represents something called a local activity in Temporal. The important thing to take away from this slide is that it's complicated. This state machine diagram is backed by about 800 lines of code. Who would want to write this seven times in seven different languages? Any takers? No? I was hoping at least one of you would raise your hand, because then I'd be like, good, you like suffering. I too enjoy suffering. No, realistically, you obviously don't want to do that. To put it into context, here is the rest of the state machines. That's about 7,000 lines of code that this whole thing represents. In an overall repository, that's about 10 times that big, about 70,000 lines of code. This logic can't live on the server side. It has to be in the SDKs that our users take as dependencies. Again, we want to provide SDKs in a bunch of different languages. You don't want to write that seven times. That's not going to work.
Why Does Temporal Have This Problem?
Why does Temporal have this problem? Why am I showing you a bunch of messy state machines? Like I just mentioned, we want to offer SDKs in a bunch of different languages. We need to meet our users where they are. We can't really expect them to adopt some unique language that we would make up just for the purpose of writing workflows. You probably are familiar, there's a lot of products on the market that do exactly that. They have some graph-based DSL or whatever, and you inevitably run into limitations with that, and no one wants to learn them. We offer SDKs in a bunch of different languages to meet people where they are.
What is Temporal? We're a platform for durable execution, which is to say you write some code, we take it, we make it resilient in a bunch of different ways. In order to enable that capability, our users, they write some code in their language of choice. They write it against our various APIs, and we do magic and make that durable. Providing all of these durability guarantees is the reason that there was so much code in those last slides, and all those complicated things, is like, a bunch of logic has to exist on the client side. They have to be super fat because they need to do a bunch of complicated things that wouldn't scale if we made an RPC for every single little bit of logic that has to happen on the client.
What Did We Need to Solve For?
With all those kinds of restrictions in mind, what did we need to solve for as a business when we decided we wanted to offer SDKs in a bunch of languages? The first and most important thing, certainly for our business and probably for most of you, is reliability. These things need to work. There can't be a bunch of bugs. The minute we violate that guarantee in any kind of way, you immediately lose customer trust, like game over. That's priority zero. We needed the SDKs to be consistent. We can't have divergence among different languages and have them doing slightly different behavior. That would also cause all sorts of really bad problems. That's another huge priority. They needed to be maintainable. At the time when I started this project, we had five guys, four guys on the team. Now I've got 10 guys on my team. We support currently seven languages. We've got 10 guys working across 7 different languages. I'm super impressed with my team. I think that's mind-blowing. We need something that was going to scale with not a ton of people.
Another very important concern for us, we wanted to provide an idiomatic experience for our users in the language that they're used to. That doesn't mean, ok, we just write everything in Rust and we pipe it through in the most obvious way possible, and what you get is what you get at the language layer, and too bad. No, we want to provide SDKs where you use the things that you're used to using in that language and it feels natural. It's a good experience. Last, but probably not least, this thing had to be performant. It's actually not a particular concern of ours. We have relatively good budgets client-side for performance stuff. At the same time if you're asking your users to run library code that you ship every incremental extra bit of resources that you use, is like shows up in their cloud bill somewhere. We still want to be reasonably efficient.
Why Rust?
Obvious conclusion here, writing this multiple times in all these different languages, not a viable choice. We needed to pick some shared code solution. We needed to pick the language to write that in. What language did we pick? It was Rust. Why did we pick Rust? I think for the typical reasons people choose it these days: it's safe, it's fast, it's expressive, it's portable. This one is quite important for us. Our users are using Linux, OS X, it's called macOS now, Windows, they're on ARM, they're on x64. We got to be able to support all those different platforms. Very importantly, it is not C. No one really wants to be writing C. It is C compatible, which is very important because there's basically no other way to call across a language boundary in memory these days. You could do everything over IPC, but maybe that's not ideal. The C FFI interface is a well-established standard that people have been using forever. Rust, in fact, doesn't even have a stable binary interface. There is no way to export Rust functions and call them directly.
I'm not necessarily here to sell you on the virtues of Rust today, although I do think it's a great language and I really like it personally. In this case, I think it was just the right tool for our job here. We could have just written everything in C. I think probably pretty clear why we didn't really want to do that. We could have picked Zig, which is another modern C replacement language that is also cool. At the end of the day, we chose Rust because of its focus on safety, memory safety specifically, and for its expressiveness, and for the fact that it has a pretty good community behind it still, certainly compared to something like Zig. That's the setup.
The Architecture
What's the architecture that we landed on? It looks like this. Over here on the left, we have the Temporal service. It's in the cloud, whatever. We don't care about that for the purposes of this talk. Over here on the right, we have some user's worker, which is what they run to interact with Temporal. The fact that it's a worker for the purposes of my talk to you guys is not really relevant. This could be anyone's code using something that you've built using this pattern. Here that happens to be a Temporal worker. For you guys, if you are interested in adopting this kind of architecture, it's whatever thing needs to be deployed that way. There's all this code over here on the right, and all this stuff that's in the light purple is code that I've authored, my team has authored. Then there's this user code at the top. The stuff I'm talking about today is really the two layers at the bottom. We have the Rust Core, which is the core shared logic. Then we have this bridge layer above it. There's one of these bridges for each of these different language layers, although in the case of Swift and C#, they share one. I'll talk a little bit about that later.
A lot of what I'm going to focus on goes on in that bridge layer. Then, above that, you have the language-specific SDKs, which expose the bits that people are going to actually program against. They don't see anything below this line. Ideally, they should never have to know that it even exists. In practice, maybe they might for various reasons, but the third thing, the language layer, is what they really interact with. Again, this layer is the part most of my talk's going to be focused on, this bridge layer. It forms the glue between the language-specific layer and the Rust Core.
Rust and Language Bridges
Let's talk about them, Rust and language bridges. As I just mentioned, each language has their own one, notable exception in one specific case. These are also written in Rust. They are defined as Rust crates that live in the language-specific repo. We have all these language repos that have the language that they're written in and then also some Rust in there, which is this bridge. Ultimately, all of them are calling into the core through C FFI somehow. Depending on what language we're talking about, that fact may be more or less obscured. Some languages have these pretty nice helper libraries specifically for this kind of use case where you want to write some stuff in Rust and call into it from the language layer. In a lot of those, you can paper over the fact that there's C calls going on.
Not all of them have that. Some of them don't have anything at all, in which case you're directly exposing C-compatible functions and you're calling them the hardcore way.
In an ideal world, these layers are thin. You don't want a ton of stuff in here because at the end of the day, this is just distraction. You have the core logic in the Rust library. You have the user-facing stuff in the language layer. This is pure mechanics. There's no other reason for it to exist. You want to keep them small, but they're not trivial. There's a lot of stuff we got to do in them, as we will see.
Speaking of those bridge helpers, our first bit of practical advice in the talk, here's the current ecosystem of this kind of thing, at least as far as the languages we're using are concerned. Python and TypeScript both have these very nice helper libraries called PyO3 and Neon, respectively. The TypeScript, JavaScript one, that's for Node specifically. This is if you are writing an application that runs inside of Node and uses Rust. If you're not using Node, this can't help you. For Ruby, there's also a pretty decent helper library called Magnus. Doesn't do nearly as many things as PyO3 and Neon do, but it helps a little bit. For our .NET and our Swift ones, the Swift one was actually recently authored by Apple. They published it. We didn't write that directly, but they used the core that my team and I have written, which was really cool to have an external contributor come along and do that. In those cases, for Swift, I don't think there was anything. For .NET, there was some stuff, but it didn't have the maturity level that was going to be necessary for us. In those cases, we just had to do it all by hand. It's just handwritten representation C exported stuff. That's the helpers.
Moving on to the concerns, things we have to deal with in the bridge. The first and perhaps most obvious is just type conversion. You have things you need to represent, and you need some representation of them both in Rust and in the language layer. Then they need to pass through the bridge somehow. There's a lot of lines of code here. We're going to talk about each of the things I'm about to list specifically. We'll talk more about this later, but this can be quite verbose. The second concern I want to talk about is async stuff. In Rust, you have futures. In TypeScript, JavaScript, you have promises. In Python, you also have futures. In Ruby, you have fibers. Conceptually, these are all basically the same thing, but they don't exactly map to one another. You have to figure out ways to bridge this stuff together. Lastly, memory management. This might not really be that hard if you get to use one of the nice helpers. This probably just gets taken care of for you. If you have to write the C stuff by hand, now we're in a situation where we have to be writing unsafe Rust. This is very much like writing C. It's going to be tricky. The advantage compared to if you had done everything in C is, it's just restricted to this layer. At least you have that benefit.
Type Conversion
Let's talk about type conversion. As I mentioned, much blood is spilt dealing with this. My first practical piece of advice here is that you can use an IDL to do a lot of codegen for you. For example, Protobuf, which we use within Temporal. I'll talk about why that's the case in a little bit. These can help, but they can't do everything for you. Part of the reason for that is that there are many representations of types that you might want to have. Starting from the top and the bottom of this list are ergonomic types, what I would call the types that you actually want to have to work with when you're writing logic. At the bottom, your users, you want to give them types that are nice to use and work with. At the top, types you're using just in your own core, in your own work, you want these to be nice. The thing is, most IDLs don't actually generate very nice stuff to work with. They are great for putting stuff on the wire and moving it between computers, but they're not necessarily a great hands-on experience.
Then, say you do use an IDL for some of the reasons I'll talk about, you might have those types. Then also, perhaps, you have some just handwritten types in the bridge for certain situations that didn't match the other ones. There's a whole bunch of different things you might have. You've got to write all this code to convert between them. It just takes up a lot of time and energy. There's a lot of code going on here, but how do we deal with that problem? First let me talk about why I happen to use Protobuf. This is really just because, if you remember in that architecture diagram, like the Temporal service we talked with it via gRPC, there's a bunch of stuff already defined in Protobuf that I needed to reuse. It just wasn't going to be practical for me to rewrite all of that. That's why we use Protobuf. I might have made a different choice if I had the opportunity to, and I'll also talk about that a little bit later.
There's a lot of code here. I mentioned a good technique for addressing that, particularly in Rust, has a very nice macro system. A lot of this stuff is fairly obvious to the point where you can just write a macro and do a bunch of these type conversions that way. Instead of writing thousands of lines of this field goes to this field, this field goes to this field, this field goes to this field, you can write a couple macros and just generate that code for everything. Another very practical thing I would do here is avoid complicated types in the bridge interface. It's tempting when you're in Rust to make some fancy enumeration with generics and complicated type bounds and stuff. Then when you try to pass that over the C bridge, what you'll find is that some very non-obvious things might happen. You get in a situation where your fancy type, while it was nice to use in the core, is not nice to pass across the bridge, hence why you might want to make one of these sorts of simpler representations and make some conversions on either side.
Fun little pop quiz section of the talk. Speaking of IDLs, Protobuf, for example, is what you would call a serializing IDL. It has to serialize to a wire format. That has some performance cost. Which do you think of these two options would be faster? Make a JavaScript object from Rust using Node's built-in C APIs for doing that kind of thing? Or do you think it would be faster to take that Rust object, serialize it to JSON, read it on the Node side, and turn it back into an object? Seems obvious. It's got to be the first one. There's no possible way that serializing to JSON and deserializing again would actually be faster than just directly moving memory and making the object.
Wait, not what you thought? The guy who wrote me on this helper library here in some Slack thread is helping someone, and this person's like, why is this so slow? The guy's like, actually, the JavaScript object creation APIs in Node are super slow for some reason. It actually might be faster for you to serialize to JSON and serialize back. Which I was like, that's nuts. That seems crazy. It's funny, because if you think about it, it makes some sense. JSON serialization in V8 is probably one of the most optimized pieces of code on the planet. That is the hot path for a double-digit percentage of software that exists in the world. It's going to be optimized. We were like, cool.
Then I went and ran some benchmarks and actually it was the thing you would assume. It turns out that serializing to JSON and back wasn't actually faster, at least in our very specific use case. The direct object creation thing, which you would think was faster, was faster, but by not a lot. These are pretty small numbers here. I did two different tests where I made just a simple object and then I made one with a bunch more fields. You can see the advantage started to disappear. I think if you have some very complicated big object with a whole bunch of fields, the JSON thing actually starts to become more true, which I'm not going to get into that. It's no details that I don't really know. The relevant part for the talk here, which is advice that is not new, is, measure stuff.
In particular, in the context of what I was talking about with IDLs, if you come along and you're like, I don't want to use one of these because these two things are in the same process. Why should I be serializing? That seems so pointless and wasteful. On a principal level, that's true. On a practical level, it just might not matter at all. The savings you get from not having to write a whole bunch of types by hand might be totally worth the couple extra CPU cycles that it costs to serialize back and forth in the same process, even though it's pointless. We'll come back to that choice again a little bit later.
Async Concept Bridging
Onto the next topic for now, the async code stuff. You're listening to me, you're thinking, maybe I want to adopt this architecture. If what you need to do inside of that Rust Core is just pure logic, this might not apply to you. If you do need to do things like make network calls, write to disk, whatever, do async stuff, you're almost certainly going to have things that exist in the Rust Core that are going to be taking advantage of futures and you might need to wait on those things from the language side. Each language, like I mentioned, has its own different representation of these things. You might need to wait in both directions. You might have a future in Rust that you need to wait on from the language side and you might have a promise or a task or whatever on the language side that you need to wait on from Rust. You need some way to connect these two concepts. That's not always easy.
In the cases with the helper libraries I mentioned, PyO3 and Neon, they have some pretty nice stuff that'll do a lot of this for you. If you can't use those, you're going to need to do it by hand, so I'll show you a couple options there. The first one is to just use our old friend, callbacks. Maybe this seems obvious, but I think it sometimes cannot be when you have all the noise of everything else that's going on in the bridge here. If you need to do something like I was talking about where you need to wait on some Rust thing from the language side, you can end up doing something like this. You define some function in Rust, it accepts a callback that the language side passes in, and ultimately, this is some pointer, this is like a pointer to some function address or whatever because it's coming across the language boundary. Then inside of here, you can do your Rust future thing. Wait on it, invoke the callback, pass the result, you're good to go.
This actually doesn't always work. You might need to deal with some other weird concerns. I have a different technique that involves using event loops, but first a little bit about why you might not be able to use the callbacks thing. Some languages can only do things single-threaded, and in fact, even when I was talking about Neon on the last slide, that's true for V8. Like the async event loop in JavaScript is serialized on one thread. The same is true for Python because it has a global interpreter lock.
The example I'm going to give here specifically is Ruby, which also has a global interpreter lock, it's called the GVL there. All execution is serialized, but in Ruby's case, this is particularly annoying because I can't do the callback thing I showed in the last slide, because in Ruby, all invocations that are going to go back and touch Ruby code need to happen on a thread that Ruby itself created. Like if you have this code and here you see it's inside of Tokio spawning a task. I don't know how familiar you guys may or may not be with like Rust async ecosystem, but what this means is this little bit of code inside the spawn thing, that could run on some arbitrary thread that the Tokio executor spawned, like some thread somewhere created by Rust. I can't invoke this Ruby callback here because Ruby doesn't own that thread and doesn't have all the threads, and stuff doesn't line up.
In that case, we have to do something totally different. These callbacks, every time you would pass a callback in, you instead wrap that function, you turn it into some version of that that pushes a little event into a queue, and you instead do something like this, where you define a function that's like, run this loop or whatever. Ruby is expected to call this at the beginning of program setup. Then you just have this single thread here that just spins and pulls callback fulfillment requests off of this queue. That way they always get executed inside of Ruby land. Quite a bit of hoop jumping. I don't have code to show you compared to the PyO3 thing. The PyO3 thing, it's just like, this is a Python future now, and you're done. Here it's a much bigger pain. Just something to be aware of, like depending on what language you're doing this stuff in, it's more or less difficult.
Memory Management
The last of the bridge topics here, memory management. Again, theme here, those helper libraries are quite nice. Use them in languages where you can. If you can't use them, you're going to need to be writing unsafe code. My advice would be to be careful with that. The premise of Rust is that even very smart people screw up memory management code. That premise is true. I can tell you as a manager that you will have people on your team if you do something like this where like, "It's cool. I got it. I'm not going to mess up the memory management. I'm a super-good engineer," but they will mess it up. It's just really tricky to get it right. Don't pretend that you're going to nail that. Take the time to really make sure that you're doing this type of thing properly. It goes back to an earlier principle I mentioned, which is, the thinner that you keep this stuff, the easier it is to not screw up. Keep the absolute minimum amount of code and logic that you can in these bridges because then you have less opportunities to screw these things up.
What does that look like? To some of you this is going to sound really obvious, to some of you it might not at all. Memory must be created and destroyed in the same place. If you're on the Rust side, you're allocating some memory, and then it needs to go over to the language side. What that in practice looks like is you need to turn it into a raw pointer, pass it to the language side, stuff happens there. They pass that raw pointer back to Rust, you reconstitute it into something that you made the raw pointer from, which might be like a box, or an arc, or whatever. Then you free that memory. Vice versa, on the language side, you might allocate some memory, and then a lot of these are garbage collected languages, so you need to tell the garbage collector, don't garbage collect this, pass it over to the Rust side, do stuff you want to do with it, pass that thing back. Then tell the host language, ok, turn the garbage collector back on, and maybe you do or don't free this, and life proceeds as normal.
Limitations of the Architecture
That's what I wanted to talk about as far as the bridges are concerned. Change gears a little bit, talk about some of the general limitations of this architecture, lessons that we've learned. One of them is like injecting behavior into the core layer. For example, we do all that gRPC stuff that I mentioned earlier. If you have some situation where people in the host language want to introspect that, or even modify the behavior in some way, so say they want to do something like, every time you make this specific RPC call with this specific parameter, I want to go tick some metric. If you don't have some totally generic mechanism from that from the very beginning, you end up in a tough place. What do I do when they come along with that ask? Do I add some really specific option that's like enable specific metric that you asked for that only applies to one user, and 99% of people are getting no value out of that. That's a difficult position to be put in.
The lesson here is like, plan ahead for that. If there are places where you imagine that users might want to do this kind of like introspection or behavior modification, try and incorporate that into your design so you can have some very generic callback. In this case, what ideally I would have done from the very beginning is, every time core does anything with gRPC, it would go through some kind of like make a gRPC call callback. It would go all the way back through the language layer, language can do whatever they want, and potentially even actually have the language layer be responsible for executing the gRPC call and not have the side effect be in the core layer. I'll touch on that again later. The main idea here is like, try and plan ahead for these spots, because otherwise you get in this position where you have to do very specific fiddly knobs, which is not fun.
Another challenge, like shipping native code in all these languages is not always pleasant. For some of them, it's easy. The Python ecosystem is very used to this idea because they do all sorts of data science, but Python's super slow, so they need to have native extensions. The PyPi package management ecosystem is good at this. You can upload for all sorts of different platforms and architecture, and when the user takes your dependency, they only get the right one, and like, cool. Some of them it's not so good. npm doesn't work that way. For my npm package, every time a user takes a dep, they download all the binaries for every architecture, because npm just doesn't support this. Too bad.
That stuff's actually relatively minor though, compared to some of the real problem situations. We have Java and Go SDKs. These are actually the only SDKs that aren't based on top of the Rust Core I'm talking about, and that's just because they predated it, they existed before we started this project. One of the things that prevents me, aside from the fact that it would just be a crazy investment, from porting these to the Rust Core is that people really don't like to run native extensions in these languages a lot of the time. Like Java with the JNI, and to an even greater extent, Go with cgo cause a bunch of operational concerns for people, because they're really not used to writing stuff that uses native extensions. Go with cgo in particular, you literally have to set different build flags, they have to change their build process. That just sucks. You don't want to force that on your users. This is a real limitation. I'm going to talk about a way that we can maybe address that, which is coming up now.
Key Learnings from the Limitations
Enough about the downsides, the upside of downsides is that you learn something from them. Let's talk about some of the things I would do differently if I had the chance to do this all over again, and that maybe my team will still do in the future. First is WebAssembly. WebAssembly is super cool. It's really a promising technology out there if you're in the business of needing to ship portable code, which is exactly what we're doing here. I'll give you a brief overview of what WebAssembly is. WebAssembly is a bytecode, like the JVM, JVM is a bytecode. In many ways it has some similar goals. It's very fast. It's portable, which is to say there's no platform-specific compilation step. It's constrainable. It's very good for running untrusted code, which doesn't really matter for the use case I'm talking about today, but is a cool point. You can restrict all the side effects that might ever happen. You can restrict the compute, like cool properties that way.
It's not only for the web. Web is in the name, and the original conceptualization of WebAssembly is that it's faster JavaScript, basically, but you can run it outside of a browser context. It's useful for a lot more stuff than just stuff that runs in a browser. How could WebAssembly help the whole architecture I'm talking about? One, it can solve the problem I mentioned earlier, like you don't ship a bunch of blobs for a whole bunch of different platforms and architectures, you just ship one bit of bytecode, so that cool. Rust is probably the best language for targeting WebAssembly from. It has very good support for it. You just pick a backend in the compiler and you're good to go. There are some caveats there, like if you need to use operating system stuff like disk access, that's going to work or not depending on exactly how you're executing the WebAssembly VM, so some nuance there.
You can possibly avoid native extensions entirely. I said I was going to give you a potential solution to that Java and Go thing, here it maybe is. There are a couple implementations that are what I would call like pure, they're WebAssembly VM interpreters that are written in these respective languages. Java has one called Chicory. Go has one called wazero. These don't have native extensions because they're written in Java and Go respectively. They appear to be quite complete. I haven't had a chance to really use them in anger yet. If you're doing just pure compute stuff, they will work. If you need to do things like the operating system level filesystem network access things, they may or may not work. There's other really cool stuff you could potentially do here. Say you don't want to have to have your users get a new version of your stuff and redeploy it just to get an update, you could potentially push dynamic updates of the core logic out and just push a new WebAssembly bot and start executing from there. Depending on your use case, that might be a really bad idea, but if that's something that suits what you're doing, I think that's really cool.
Different topic entirely, another thing I would do if I could do it over again is use an IDL that doesn't involve the serialization step. I mentioned I had to use Protobuf because that's what the service uses. The whole thing I was talking about with the performance, the little graphs and things, you could just not have that problem. There are IDLs that don't involve serialization all. The two most popular ones are FlatBuffers and Cap'n Proto. These IDLs don't have a serialization step, which is to say that they generate code in multiple languages that uses the same physical memory layout. You don't take some memory, turn it into a representation that's going to work in both places and then do that again. Both of them know how to read the same thing. I think on Cap'n Proto's site, they say infinity faster at the top of the thing. It's like, yes, there is no step. This is something I would do if I could because if you profile Temporal's core, it spends 90% of its time serializing, so that's quite a lot.
Last thing I would do is route side effects back through lang. This is connected with a bunch of stuff I said earlier. I would just take everything that does side effects and probably pipe it back through the language layer, which means network calls, logging, metrics, applies to a bunch of stuff, anything users might need to customize. This serves a dual purpose. One, it gives users all those customization hooks that we said they might want to have. Two, if you are doing this, if tomorrow you're like, I'm going to do everything Spencer said, and you're doing this with WebAssembly, now you don't have those concerns about like, can it do the right OS calls or whatever, because you just don't make OS calls. They all go back out to the language layer and the language is responsible for doing that. You can potentially kill a bunch of birds with one stone here.
Summary
I can confidently say that engaging in this project, like writing a Rust Core was worth it. Not having to write all that crap over and over again has been a huge win for my team and I. Specifically, in my estimation, this cut the new development time for new languages in half, and probably more importantly, it has ongoing payoff when we need to add new features. We still have to touch the language layers, because we need to provide that nice white glove experience to our users. We need to write good APIs that they want to use. The logic part can be in the Rust Core and we just write that one time. Also, there's definitely fewer bugs. This is difficult to quantify, but it is obvious that there's just no way that we have more bugs with this architecture than we would with the other one. There's just dramatically less code, therefore, there's less bugs.
Key Takeaways
Rust is a cool language, and it's particularly nice for if you need to write a bunch of shared logic. If you're adopting this architecture, I would use those nice helper libraries I mentioned because they do quite a lot of heavy lifting for you. Additionally, I would use some kind of code generation. It's not going to do everything for you, but it'll certainly help. You have a lot of options to choose from here. If you're free of some of the constraints that I had, use one of those no serialization ones. This is more of philosophy than practical advice, but when you're designing stuff like this, take the time to make a very nice idiomatic experience for your users. It can be tempting when you're doing this kind of thing to just auto-generate everything and just plumb the whole thing through, and you have this crappy experience at the language layer that just doesn't really feel tailored or cared for. Then, lastly, plan for hooks. Planning for these places where behavior might need to be injected, and threading that through. I think this one's quite hard to get right. You have to have a lot of forethought here, but it's worth trying to think through the options.
See more presentations with transcripts
