Transcript
Meryem Arik: Most companies are overpaying for their inference by 2x to 5x, sometimes an order of magnitude for some of their highest token use cases, because they haven't really thought hard enough about what their inference stack looks like and whether their inference stack matches their use case priorities. Today I'm going to walk you through what it takes to design inference systems from scratch when your only goal is to produce tokens at the lowest possible cost. How this kind of focus can really outperform popular general-purpose setups by over an order of magnitude.
I think the first thing that I should say is I'm incredibly cheap, thanks to my father who's Turkish. I never buy anything without haggling for it. What we do at Doubleword is we're an inference company. One of the highest token-producing use cases that we've come across, especially over the last 6 to 12 months, and it's growing pretty quickly, is these non-real-time use cases that produce a lot of tokens. We've been spending a lot of time thinking about how to produce much cheaper tokens, and that's what I'm going to be sharing the learnings of that for you.
Inference Tradeoffs and Token Economics
I'll take some time to set up inference economics and how we think about charging for tokens. Inference is a tradeoff game, like most things. You can only really have two of the three, and that is between low latency, that is, how quickly can I produce my tokens? The cost of my tokens, and then the quality of those tokens that I get as well. I'll walk you through what each one of those examples looks like. I could say I'm going to trade off cost. I'm not going to care about cost. Then how would you make this tradeoff? I'll do things like run at low batch size, use very expensive hardware. I would do more latency optimized inference optimizations. This is really useful for some use cases, like a very popular one is coding assistants. Another example is chatbots, where you might care about this very low latency, high quality outputs.
I might make another choice where I want to trade off the model quality. I want my responses really quickly and I want them to be cheap. I might use a smaller model. I might quantize the model. The real-time use cases of this might be guardrails or a routing model, or I haven't actually got it here, but on-device models as well. If I actually want to run something locally on-device, I would make this tradeoff. The tradeoff that I'm interested in and I'm going to talk about today is this trading of when I'm not going to care about latency. All I'm going to care about is having high model quality and very low costs. How do you make this tradeoff? I'm going to talk about it more in this talk. I would do things like batch-specific optimizations, queue reordering, smart scheduling. The exciting thing for me is this type of tradeoff, it's actually a really popular tradeoff.
Use cases include like data processing, offline agents. I have an agent running overnight doing a very long complex workflow. Summarization, data labeling, data generation. Synthetic data generation if I'm doing fine-tuning or RL. There's a lot of use cases that fall into this bucket. Something that's exciting is as models get better and better, the number of use cases in this bucket will continue to grow as we can trust models to operate independently without always needing a human in the loop at every possible step. This is the tradeoff we're talking about today. In the inference space, there's been a lot of energy on these low-latency use cases. You have this tradeoff space between my cost and my latency. When I have really ultra-low-latency inference, there are providers that provide that really well, like Cerebras and Groq. The way that they do it is through specialized hardware. We then have the everyday chatbot-style inference, vLLM, SGLang are popular frameworks here. Also, most of the inference providers sit around this fairly low-latency space. Then, for these very cheap high-latency workloads, that's where our research team's been focused for the last six months to a year or so.
Making Cheaper Tokens
The way that I'm going to structure this talk is I'm going to start from very fast tokens. I'm going to talk about what tradeoffs we make to get very fast tokens. Then say, how do we get from those fast tokens to very cheap tokens? What are the tradeoff differences that we make? I would be super happy if anyone gets the reference of the next slide, but I'm going to go from like a Jeremy Clarkson Top Gear to a Monty Python, we've got to haggle for this. Does anyone actually get these references, or are they too English? I have a U.S. appropriate equivalent. I had a backup in case. We're going to go from like vroom cars to The Art of the Deal with these really cheap tokens. Let's start by looking at the leaderboard of providers on latency. For the purpose of this talk, I'm going to consider just one model, which is gpt-oss-120 billion.
This is a public benchmark that's available from Artificial Analysis, where they benchmark the latency, the throughput, and the cost of many popular inference providers. On this, you can see that there's pretty big range. I wouldn't recommend, for example, currently using Cloudflare for their inference. You can see here that I have Cerebras as producing very fast tokens. If I can take another graph, which is the output speed, you can see Cerebras is way faster than pretty much anyone else. Cerebras is where I'm going to start as this perfect example of making the tradeoff of very fast tokens, but pretty expensive tokens. Cerebras make really fast tokens, but you can also note here that they make really expensive tokens as well. They make really expensive tokens. On the other end, as we go down, these tokens that are a little bit slower get a bit cheaper as well.
Layers of Optimization
There are a couple layers of optimization I'm going to talk about. I'm going to talk first about the hardware. There's no surprise that Cerebras is actually the most expensive and also the fastest tokens. I'm going to talk about the tradeoffs we can make at the hardware level. The model level, you can do things, but I'm not talking about that because we're assuming a fixed model. We'll talk about the runtime. Then we'll talk about the scheduling and orchestration. We can make optimizations at these different levels to make our tokens much cheaper.
1. Hardware
Let's start with the hardware. It's no surprise at all that Cerebras makes really fast tokens because they've made a completely different tradeoff to the tradeoff that I'm interested in. They are just like, we are going to be super-fast, and that's the only thing that we care about. If I just move to GPUs, which are actually a bit better when it comes to unit economics, I can see that I get an instant reduction in that GPU cost. There's a bunch of providers that are about this kind of $0.6 rate for output tokens, and most of them are just running on GPUs. I would note that all of these are vendor-supplied prices, and so you can't necessarily trust them. I know, for example, a bunch of the providers on this are actually running at losses. Instead, we can talk about InferenceMAX by SemiAnalysis. Has anyone heard of SemiAnalysis?
It's a really fantastic resource and a fantastic thing to subscribe to or get the blog for if you're interested in GPUs. What SemiAnalysis has done is they've released this project called InferenceMAX, where they've benchmarked a number of different GPUs and looked at the theoretically possible throughput and latency and cost tradeoffs. This is taken from SemiAnalysis InferenceMAX project. Just by moving hardware, so in this case, moving from a H100 to a B200, which is a slightly newer generation of hardware, I'm able to get a similar latency. My latency is roughly about 4 seconds for the request that they measured, but my cost actually goes down by about 75%, which is really significant. The reason for this is because, essentially, these newer GPUs are just better. I can run more concurrent requests for the same latency, which is really impactful. As a very easy thing to implement if you want to care about reducing your costs is just think about moving to the latest generation hardware.
It's almost always cheaper. Note, it's almost always cheaper if you're able to fully utilize it. That's a key consideration as well. If you have the workloads, let's say you're running very frequently big batched workloads, and you're able to utilize that GPU 50-plus percent of the time, it's almost always worth moving to the latest generation GPUs if you can find them.
A slightly more advanced technique than just moving GPUs is disaggregated serving. There are two things that matter in AI inference. There are the flops and the bandwidth. Flops matter in your encoding stage and bandwidth matters in your decoding stage. On the very left, I have a bunch of different GPU types. This is actually a screenshot from something that we have internally. A bunch of different GPU types and a bunch of different providers. All of these GPU types have different specs. They have different specs for the memory. They have different specs for the flops, and different specs for the bandwidth. What I'm interested in is what those flops and bandwidth numbers are relative to the cost. What are the flops per dollar that I can get compared with where I'm able to get it from? For example, if I get a reserved instance, I know it's going to be cheaper than if I have an on-demand instance.
You can take the profiles of different GPUs having different things that they're good at. Some GPUs are really good at encoding and some are really good at decoding. You can use disaggregated serving to run your encoding on one type of GPU and then your decoding on another GPU. If I zoom into that a little bit more, I can see some of these have dramatically better flops per dollar and bandwidth per dollar than other GPUs. Then I might choose to run my encoding on one versus another. Prefill tends to be compute bound. My decoding tends to be memory bound. I might decide to try this combination and see between this combination, am I actually able to produce a cheaper system? I would note that this is super non-trivial to implement, and I wouldn't do it just for fun. It is something that if you are able to play around with, you can get pretty significant cost savings.
I would note this does destroy your latency because the moving of the memory between the two GPUs is not trivial. At the hardware level, what can we do? Different GPUs have different latency and cost tradeoffs. Be very wary of vendor supplied token costs. They are often fairly different to what the actual cost might be if you're self-hosting. If you are able to get good utilization, you should use the latest generation GPUs. There are advanced techniques like heterogeneous disaggregated serving where you can combine different GPUs for the prefill and the decode stage.
2. Inference Engine
The second layer is this inference engine or runtime layer. By far, the most important thing that you can do if you want to trade off latency for cost or cost for latency is play with the batch size. That is, by far, the most impactful thing you can do. This is a graph from SemiAnalysis from their InferenceMAX paper. In this case, I have the B200, so I'm just looking at the tradeoff between B200s. On the x-axis, I have the latency. On the y-axis, I have my cost per million tokens. Because in this room, we're all really cheap, we're trying to reduce that cost per million tokens as much as we can. We're fine taking that hit on latency. When I say that inference is a game of tradeoffs, this graph really encapsulates that. The way that InferenceMAX works is they test different combinations of parallelism and then the concurrent requests by the GPUs, and they plot it, and it looks like this.
If I take the two ends of that curve, on one end of the curve, so if I care a lot about latency, I end up with a setup that looks like I have eight GPUs running in parallel, and I can only service four users with that, which is really not a lot of users. Which is why my cost per million tokens ends up being so high, because every user is getting two GPUs, which is a lot. If I crank up my concurrent requests, so my batch size, to 128, then I'm able to get a much cheaper token cost, because those 128 are sharing the GPU with everyone else. The reason why batch size is so impactful is because during this decode stage, the thing that you pay for is not the actual token generation of every single request. The main thing that you pay for is actually the movement of weights, and the movement of memory.
Once you've paid that, then whether I generate tokens for one request or 128 requests, the difference is actually pretty marginal, because I've paid for that movement of memory. That's why cranking up the batch size is really impactful, because I'm able to take advantage of the fact that I've moved the memory for a couple requests, and then I just keep inferencing. It's not as trivial as more is immediately free, but it's certainly not if I double my batch size, then my latency goes up by two. Essentially what you should try and do is have your batch size be as high as possible.
This actually gets really interesting for long-context workloads, and a lot of batch use cases are actually long-context use cases. A use case might be if I'm trying to answer questions on deep research of really long documents. This is a long-context workload. Let's take the example of I have a KV Cache size of about 1 million tokens, and my context length for each request is about 200K. I have a theoretical batch size of about 5, which is much lower than my ideal, which my ideal is my batch size as high as physically possible. In these instances, what you want is you want your request to finish processing as quickly as they can, and a really good technique to make that work is speculative decoding. This essentially means that the large model, instead of generating every single token, can check tokens in parallel. With one pass, it can maybe check or approve two or five tokens.
In every single pass, I'm actually able to generate far greater tokens, and so the whole request length is processed much quicker. This is something that I can do to compensate for the fact that I've got a lower batch size is by trying to implement techniques like speculative decoding which allow me to process that whole batch much quicker and get onto a new batch as well. For long context, speculative decoding is really impactful. For lower context, high batch size is really impactful. From my inference engine perspective, high batch size and techniques like speculative decoding can be very impactful.
3. Scheduling
One of the most obvious places that you can save money is via your scheduling. If I'm running a low-latency inference service, I have to handle spiky load like this. This might be my load simulated throughout the day. If I'm running a real-time service, I need to make sure it's always real-time. I probably need to provision my GPUs for the 95th percentile of load before I want something to start queuing. Let's say that's roughly 500 users. If I want to provision up to 500 users, that means I need roughly 8 GPUs if I'm running at a batch size of 64. If I'm in a batched situation or I don't care about latency so much, so I'm happy for it to wait a minute or two, I really don't need to overprovision and I can just have it queue instead. The difference is going from 8 GPUs to roughly 2 GPUs.
If my boundary condition is that I just want all of this served within 24 hours, I can probably get away with 1 GPU, maybe 2 GPUs. This kind of basic scheduling and queuing is really impactful. There are other very interesting techniques that you're able to do as well when you don't have to be super latency sensitive. One is request reordering. If I have a situation where I have a single model, my gpt-oss is satisfying 10 different use cases. I have requests coming in pretty constantly to that model. It goes in roughly random orders and hits the model. I don't get any KV Cache hits. Let me give you an example of what that looks like. I might have one use case where my use case has some preamble, which is very common, which is like, you are a helpful assistant that's going to help me do X, Y, Z, and here's some information about the task.
Here's some examples of how that works. Then my actual question or the specific ask for that use case is this tiny thing, but I've got this long thing coming before it, which is giving context to the use case. I have this pretty much for every single use case. I'll have these really long prompts, these really long context we're giving to the model. If instead of just processing them randomly like I have to when I'm running a real-time service, because I need to process them as soon as I get them, if I have a bit of time to think, I can instead do request reordering. It doesn't actually look like a queue per use case, but you would order this based on the prefixes. I have each of my use cases being processed together in batch. For example, I'll build up a bunch of use cases for use case 3, and they'll all get processed together.
Why would I want to do that? The reason why I would want to do that is because every single time I process the request when it's happening individually, I have to process not just the specific ask, I have to process the whole prompt, I have to process the whole prefix. What I can do when I have this queue reordering is I can have a use-case-specific prefix that is processed once on the first time that the model sees that use case. Then the only thing that I need to process in my prefill stage is that specific ask. Essentially, what you end up with is if the specific ask is short and my big preamble is then cached because it's only processed once, I get my input tokens for free in a lot of use cases, or near-enough free, which is really impactful, especially for these long context use cases.
Because the only thing I need to process is that specific ask and that's fairly short. I would say though that the impact of this kind of technique is really dependent on the use case that you have. If the use case you have, it's a completely different prefix every single time, then this is not going to be useful. Most use cases don't actually look like that. There's another technique, which is bin packing. Given that I have a bit of time, I can also order my request based on the size and length of the request, the length I'm expected for input and output tokens, and process similar length requests at the same time. I don't have my GPU, for example, idle while waiting for a really long request to finish. How can I make tokens cheaper via scheduling? Simple things. I don't have to overprovision, I can just perfectly provision. Also, I can use techniques like queue reordering that I can use if I have a bit of time to maximize KV Cache hits and get my input tokens for near-enough free.
The Results
Let's talk about the results. I've talked about a bunch of different layers of the stack, but let's see how it actually plays out. From my different layers, I've got my hardware, making smarter hardware decisions, making different decisions in my runtime engine, either doing things like speculative decoding or increasing the batch size, and then scheduling and orchestration there. The results. I've compared these results with the Cerebras price, because that was the most expensive lowest latency price. Actually, the price of Cerebras wasn't that different from Azure, or Bedrock, or anything like that. They're near enough the same. For my input tokens, I can actually reduce my input token price from about 35 cents per million tokens to less than a cent, which is really impactful. Then, for my output tokens, I can reduce it about an order of magnitude. These are actually very impressive results, and this is the theoretic possible.
It's obviously different for individual use cases. For example, the input tokens can actually be much cheaper than this if you have a use case where you have a lot of those shared prefixes, but I can end up with a very significant cost decrease. This makes a bunch of assumptions about use case setup. Costs can be more or less. I think what I'm trying to demonstrate is that they can be very significantly less, and this was based on pretty sensible assumptions that we see in the world. What that actually looks like, and I've put this on the end, is it literally is about an order of magnitude difference, and the price that I have here is not a vendor-supplied price where they could be undercharging or overcharging. This is based on what is possible to get out of your GPUs, based on assumptions on how much GPUs cost to rent every year. That's a pretty significant difference.
Case Study: Deep Research
The last thing I'm going to talk about is a case study that is actually a really impactful demonstration of when all of the stars align, and this works really well. This is a case study with a financial services company, and they wanted to be able to give their analysts a lot more information about companies that they were surveying and interested in. There are about 250 questions that analysts would very commonly ask about the businesses that they were looking after. There were about 50,000 companies and businesses that they were surveying. What they wanted to be able to do was run deep research-type tasks to be able to ingest everything new that's happened on the internet, maybe internal information they have, maybe information that they get from analyst reports and things like that, and answer those 250 questions. Taking the Azure prices for the model that we've been talking about, gpt-oss, that came out to be at roughly 350K a year, which is a really expensive process.
This was, as you can see, a situation where they had a lot of input tokens, not very many output tokens. Most of these real-time APIs also offer batched APIs. These batched APIs are a classic 50% off, which is still massively overcharging you. If they just switch to batched API, it's still around 200K every single time they wanted to process this, and this was a roughly weekly process. However, if they were using a specialist batched API or were self-hosting with batched in mind and making all of the tradeoffs that we've talked about, trying to get The Art of the Deal, then you end up with a situation where it cost me about 13K. I go from a position of this use case being completely unviable to this use case being totally viable. An interesting observation that I'll share is, I often sell clients cheaper tokens and saving money on the token price, but what's interesting is they hardly ever end up reducing their inference bill.
The reason why is because as soon as you give them cheaper tokens, they just find a bunch more use cases. They're just like, great, we can do so many more things. You can go, from 50,000, we're now going to do 250,000 or whatever it is. I think the nice thing about offering cheaper stuff is it often doesn't just become saving money, but it often becomes being able to do more or unlock use cases that otherwise weren't viable.
Conclusion
To finish off, making cheaper tokens. The first thing that you should think about is, what is the need of your use case? If you are building a coding chatbot, this is probably not super relevant for real-time interaction. Think about if you actually have use cases where cheaper tokens are the tradeoff you want to make. Do you need faster tokens, smarter tokens, or cheaper tokens? You can optimize for cost at the hardware level, inference engine level, and the scheduling level as well. If you self-host and do all the optimization you talk about, you can get a 10 to 20 times cost reduction depending on the exact specs of your use case.
See more presentations with transcripts
