Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM

Transcript

Olimpiu Pop: Hello everybody, I'm Olimpiu Pop an InfoQ editor, and I have in front of me today, Edoardo Vacchi, one of the two Italians that usually talk about boxes inside of boxes. The other one is Andrea Peruffo. Eduardo worked for a long period of time in the WebAssembly ecosystem, he did some spotting around other types of VMs, and today he's back at Red Hat and he's trying to make things easier in terms of Kubernetes and inference. Is that correct?

Edoardo Vacchi: Yes. I mean, on my team. It's a few people working on this, and I'm just one of them, but fair description. We're working on distributed inference on Kubernetes specifically, and that's called llm-d, and it's part of the VLLM family of projects. So, yes, it's super exciting to be back at Red Hat and work on the bleeding edge.

State of WebAssembly on the Backend [01:35]

Olimpiu Pop: Yes, that sounds nice. But before we go there, because that's the gist of your current work, tell us what happens currently in the WebAssembly space. And I'm asking you because, besides working on the Java side of WebAssembly, you had exposure to Wazero, if I remember correctly.

Edoardo Vacchi: That's right.

Olimpiu Pop: So, maybe just have a short refresher. I know that the WebAssembly space is moving fast, so maybe if you can just provide a short summary where it is, what changed, and what's to be expected in the upcoming period.

Edoardo Vacchi: Right. So, for the benefit of our audience, I don't know how much all of them are familiar with it, but you will know that Wasm was born for web browsers, but more recently, and by recently now it means a few years, people were also interested in running it outside the browser, and for the backend specifically. So, running software on Wasm. And in that sense, it could be seen as a sort of a similar thing to the JVM in spirit. And that's why I guess some of us that were working on the JVM were interested in working on Wasm as well. So, Wasm has changed fast in some senses, it evolved quickly in some senses, it evolved just slowly in others. So, there's been a lot of discussion around the component model, which is this new modular system that's built around WebAssembly. That is probably where a lot of the development happens these days, I'd say.

But I'd say that the space for WebAssembly on the backend has changed a lot in the last few years. I think that all of our development landscape has changed. So, things have shifted a little bit. I think it's very meaningful and relevant today, even more possibly than it was when there was much more hype, because it has a lot to do with sandboxing. And as you know, sandboxing is all the rage these days.

Olimpiu Pop: One good scoring point for WebAssembly was earlier this year I was talking to the local first community, and what was interesting was that they were mentioning that one of the libraries they built, the name is eluding me now, but what they said is that rather than just thinking about providing implementation in different programming languages, they built it in Rust, and that meant that they can just compile it to WebAssembly and then use it via WebAssembly to different programming languages. And that was the nice part from their side, and they were quite happy with the result. And I think that's where the power of WebAssembly comes. Or I think the previous year I spoke to a couple of guys that used to work on infrastructure and they spoke about Firecracker. And what was nice, is that they said that most often than not, the harder that you have in the server acts is idle.

So, what they did on top of those idle space, they just had very small WebAssembly machines... Well, containers, I think it's a better wording for that. That was doing very small things that were quite useful for other people. So, I feel that there are all those different scenarios where people are using WebAssembly.

Edoardo Vacchi: Yes. I think one interesting aspect that connects to what you were saying right now is even migrating workloads somehow, like running stuff in the browser, but then you also have these WebAssembly run times that can run, even on devices or in general outside the browser, and you have the same workload that runs on your browser, and we have demos of things that are migrated, even live migrated, from the browser to an end user device without breakage. That's really fascinating. And I think these days when we talk about agentic workloads, that's an interesting space. That's where I think Wasm is still relevant. I mean, Wasm is still relevant, especially for browser related work, and many other, but these scenarios for the word that's changing a lot these days for developers, those are the use cases with really good shine.

Olimpiu Pop: Yes. As you mentioned, isolation is important and is getting more and more important these days. So, I think that's quite useful. What is bugging me for at least one week now, and probably I'll try it the following week, let's see what you are going to say to that, I'm just thinking to run a small language model inside the WebAssembly in a browser, is that feasible? Is not feasible?

Edoardo Vacchi: Yes, I mean it is feasible, there are examples of that. The thing is, I've been on the receiving end of this question a few times now, so it depends is the short answer.

Olimpiu Pop: Perfect.

Edoardo Vacchi: WebAssembly is meant to run on the CPU first. And as we know, we can do inference on the CPU, but generally it works much better. And for the way LLMs, modern LLMs, I mean, the modern architectures for LLMs are designed, they are meant to be run with highly parallel architectures. And the CPU is parallel, but not as parallel as a GPU. So, long story short, smaller models, which basically make sense to run in a browser, for instance, do make sense to run in WebAssembly, much bigger workloads... Well, those in general, we probably want to run on a GPU anyway. However, there are experiments that I'm aware of to translate WebAssembly into GPU kernels.

So, yes, possibly. But the short version of it is for all those workloads that are meaningful to run on a CPU, so relatively small language models, Wasm can be used, and there are many versions of these workloads that exist. I think that Hugging Face for some of the smallest models will run them on your browser, I think. And there are a few examples, on the Wasm I/O conference, there were quite a few talks of people showing this, running these workloads in the browser especially to do inference. So, yes, it does work, it depends on your workload.
TornadoVM: Translating Java Code to GPU Kernels [08:26]

Olimpiu Pop: Yes. Well, I'm going to say it anyway, it's simple, it's more about just trying and seeing how things evolved, and fingers crossed that something will happen and not just smoke. But in order to work and consolidate the point that I stated about an expert into putting boxes into boxes, you had the posts, I think last year or the previous year, about TornadoVM. I know that you just had a small exposure to it, so maybe just have a short summary what's TornadoVM and when should we use it?

Edoardo Vacchi: Yes. So, TornadoVM is, we could say a flavor of the JVM or an add-on, a plug into the JVM, depends on how you see it. TornadoVM allows you to run Java code essentially on the GPU. This is the bottom line, the selling point, the very short version of it. It's a little bit more nuanced than that. But yes, basically you can, speaking of boxes inside boxes, yes. It's called the VM because it allows you to translate on the fly your Java code, and compile it on the fly into GPU-specific kernels. So, PTX for NVIDIA, SPEAR-V, whatever. And it handles for you this translation. So, you can basically write Java code and then it will translate for GPU backend. Now, the code that you write there, it is Java code, but you have to write the code in a specific way. You might have to use some libraries that TornadoVM provides you.

You cannot run basically any code... You won't be able to run Spring on TornadoVM. I mean, yes, but it will run in Java mode. You won't run Spring on the GPU, that's the point. But yes, you can write GPU kernels using Java, and that's the beauty of it. And the blog post you were referring to, I think it was last year's, I'm walking through the reader... One of the examples in the code, I think there is an example of writing a simple kernel and running on the GPU, that's one thing that we show in the blog. But then we run GPU Llama 3, which is an implementation of the Llama 3 model that runs on GPU. And this implementation is completely written in Java. So, that's fascinating. The blog post walks you through how this works.

In the latest DevOps Belgium, if people want to catch up a little bit, they had demonstrated, I think there was definitely a talk about TornadoVM, there were talks from the GraalVM team of the similar projects to run Llama and other language models using GraalVM that will do the acceleration for you, and then there was also former GraalVM team member that spun off his own startup. I think his name is Alfonso Alfonso, I remember it because his nickname is Alfonso Squared. And he started his own startup branching off some work that he did originally in GraalVM, again to run Java code, I think, on the GPU and LLM.

So, yes, fascinating stuff. And so, bottom line, the reason why I started to look into this is fan of boxes inside boxes, fan of languages and compilers, so it felt like a natural evolution of the things that I did in the past, including compilation to WebAssembly to learn about compilation to the GPU.

vLLM vs llm-d [12:11]

Olimpiu Pop: That sounds quite interesting, and it definitely makes sense these days. For instance, I saw that a lot of the big companies are just using Java for inference, and that makes perfect sense to just have this closing the loop. Today, you are part of the team at Red Hat, back again at Red Hat, that is working on llm-d. And in my understanding, what llm-d is, taking a step back, if VLLM is making sure that everything runs perfectly on one box, llm-d is trying to make sure that everything is efficient if you have a Kubernetes cluster. Is that a correct affirmation or did I understand wrong?

Edoardo Vacchi: No, no, it is correct on a high level. More specifically, so VLLM is our inference engine. So, you can think of it, if you will, as the JVM of language models. So, it has support for several language models, there's an interesting aside here, it is a platform to run LLMs. The interesting aside here is you would expect that this is a general sort of VM, and the LLMs are sort of software that you load on top of it. But unfortunately every LLM architecture is slightly different. So, VLLM has to basically include the implementation of every LLM that gets released. So, when people release LLMs, they do not release the software to run them, they release the weights. So, all of these inference engines, such as VLLM, what they do is implementing basically the software, the kernels, the components that actually load the weights and then actually do the inference.

So, VLLM is our platform to run the core and effectively VLLM is able to serve, it's a server... I mean, it's an inference engine, it's also server that exposes endpoints that you can call to do inference, and it does so efficiently. It was originally VLLM, I think the V in VLLM originally stands for virtual, as in virtual memory allocation. Because it was the first engine to implement page detention, basically a mechanism inspired by good old systems programming to do paging of memory, memory pages, but for memory that gets allocated on GPU, this is in a very high level way. So, VLLM is able to do high throughput inference. So, you can do several requests at once and VLLM will pack them and try to serve them together, and do inference in one pass for several requests at one time, this is called continuous batching. But this gets you to a point.

Essentially, at some point you have your GPUs on your machine, and your inference engine and that's it. You will not be able to scale further than the resources that you have on that machine. So, how do you scale further? Well, you can scale by having more VLLM instances, right? So, you have several VLLM instances and you want to serve your inference requests using these few VLLM, these few VLLM instances that you have. Now, you have another problem. How do you route requests across your VLLM instances? So, if you are on Kubernetes, what you will do... And most people are on Kubernetes these days, it's the standard for platforms for serving services in an efficient way. So, you can scale your VLLM instances and have several VLLM instances that serve the possible the same request... I mean that gets selected by the scheduler, by the VLLM scheduler.

If you do it the traditional way and let the Kubernetes normal router to pick across the VLLM instances that you have replicated, you won't get necessarily good performance. The reason is inference does not behave like a traditional service, a traditional HTP service. Basically, if you send a request to one specific VLLM instance, especially if you are doing conversation, if you're doing a chat and if you're doing agentic coding, it's still a chat basically. There's a lot of exchanges back and forth between you and your server, between your agent and your server. You want to stick for as much time as possible to the same VLLM instance. This might be counterintuitive in a way because for microservices, we know we don't want to hammer the same service on and on because otherwise it will get overloaded. We want to do sort of round robbing or random robbing across the replicas.

But the thing is microservices or service in general, these are usually in a way stateless. The state is usually stored somewhere else and it's not part of your request. I mean, the request can be served independently by any instance. Unfortunately for LLMs, in order for LLMs to do their inference in an efficient way, you want to stick for as much time as possible to the same instance. But at the same time, you cannot stick forever to the same instance because at some point you will reach some limit. You are not the only user of that VLLM instance. You want to serve and distribute the load across your VLLM instances. So there's a balance that you have to strike between staying with the same instance and benefit of the caches, the KV caches, the prefix caches that are in place, the systems that allow you to do inference in an efficient way and lower costs.

But at the same time, you don't want to overload the same instance too much. So you also have to keep this into account. This is not the job for a traditional Kubernetes schedule. And that's where llm-d comes into place. That's what llm-d does. llm-d is not just a router or a scheduler. It's actually a family of projects and they are all meant to give you a set of tools to do distributed inference so that you can leverage all of the GPUs that you have and use it at their maximum as much as possible so you get the best value, the best bang for your bucks because these hardware is expensive whether you loan it or you own it. So you want to use it 100% as much as possible. And that's why we provide scheduler. We have something that we're going to release. I mean, in the open source ecosystem, it's already been released.

We're going to release also as part of product of Red Hat. This is called batch inference. Batch inference is using your cluster even when it's under load. So you can schedule request to do batch processing in the background when your cluster is underused because basically this hardware is so expensive that you want to keep it at 100% all the time. So even when there is no live inference going on, you can schedule batch processing to run when you're sleeping when there's less work to be done live. So that's what llm-d does. It uses your hardware as much as possible so that the money you spend gets you the best value and you do it on your own hardware so you can do inference without leaking data to a third party, which is important these days.

Olimpiu Pop: So, what you're saying is that we will be able to use our previous Kubernetes clusters. So, what we still have from the cloud native era, because everything was native these days, and just make sure that we can rebuild it to build our privacy also for the AI native era.

Edoardo Vacchi: Yes. It allows you to make cloud native inference and evolve your infrastructure in such a way that you can do AI on-premises, in a similar way as you did cloud native development before AI.

Olimpiu Pop: Well, that was always an issue, but now it feels that we are discussing in the cloud-native; most of the times we spoke about stateless. Whenever you call something, you didn't always ring on the same doorbell because that was one issue. Another issue was in the Kubernetes space, it was the part with I/O, well, most particularly persistent data, like database and stuff like that. Whenever you spoke to infrastructure guys, they told you, "Well, we can have everything in Kubernetes except for databases because then those things become more complex." But let's take them one by one. How do you bring in a stateless ecosystem like Kubernetes, worked a lot in Bing, something that theoretically should be stateful?

Because if I have a conversation, I would like to continue my conversation about, I don't know, whatever stupid thing I speak to Gemini or Cloud or OpenGPT or whatever stuff, I don't want to have my responses coming for what Eduardo was speaking to his LLM.

Edoardo Vacchi: Yes, it's a complex problem for sure. As I said, one part of the story is that... Well, I mean, if people that are hearing that the audience know how LLM generally work, essentially the stateful nature of the LLM in a way is fake, in that they are really stateless. The way they do inference is that we send them the entire history of a conversation every single time. So, maybe I jumped at-

Olimpiu Pop: At the conclusion.

Edoardo Vacchi: Yes, to the conclusion earlier. And I said a lot about prefix caches and KV caches, but yes. So, essentially the way an LLM works is that essentially you always send the entirety of the conversation to the LLM. And what really happens is the LLM completes the conversation. I don't know if people recall how OpenAI and the GPT Lab playground worked a few years ago. Basically, it was the completion API. You wrote a sentence and then the LLM wrote the continuation of that. And the cool thing was that someone discovered if you wrote that prompt as a conversation, as a dialogue with quotes, user says da, da, da, computer says... The LLM will continue the conversation. And that's how ChatGPT basically was born. People notice this, and notice this pattern.

Now, LLMs are being trained with this pattern, but that's how basically originally it work. So, if you think of it this way, every conversation really is a very long prompt to the continuation of that conversation. The way LLM works, you send the history of the conversation and then they continue with the next answer, and then you get back the answer, and then you continue with your answer, and so on, so forth, so on and so forth. If you are using LLMs, you will know that pricing for these things, depending if you're using token-based spend or a subscription... Well, if you're using a subscription, that doesn't really matter. But if you're using token-based pricing, you will know that there's a difference between cached responses and responses.

The reason is, if you don't do a cached response, then you are sending the entire history of your conversation to the LLM. The LLM does not have that conversation in cache, so it will have to do inference on the entire prompt, and that's why that will be particularly expensive. But if you've been going back and forth for a while without long pauses and the cache on the server has not been evicted, then you've been reusing stored cache estate to continue the conversation. That does not mean necessarily that you haven't sent the entire history to the LLM, it just means that even though you are sending the entire history, the history is recognized as part of a prefix of a cache, and so the LLM can compute only the continuation instead of the entire prompt.

So, we have these mechanism for caching in place so that basically LLMs, if the cache exists, can only infer the last few tokens that continue the conversation instead of have to re-compute the entire state from the prompt. llm-d does that for you. So, it maintains this cache both in the form of a KV cache, with one going to the deals how that works, and a prefix cache. And both these caching mechanism allow you to persist the state across the exchanges in your conversation, persist as in volatile memory because these are caches... Usually they're not persistent for a long time. And that's the reason also why you want to stick usually to the same VLLM instance, even though llm-d also gives you ways to move around across instances. It's a lot, we cannot explain how all that works. But in general, llm-d allows you to do this, allows you to have an efficient use and even for stateful conversation of your hardware.

Olimpiu Pop: Okay. So, llm-d, it's also donated, if I remember correctly, to the CNCF Foundation, right?

Edoardo Vacchi: Yes. It recently reached 1.0 a few days ago. Yes.

Olimpiu Pop: Okay, congratulations. So, we are right in time. In what state is it? Is it production ready? Should we wait for it?

Edoardo Vacchi: Oh no, it's definitely production-ready, there's a number of companies that are already using it. At Red Hat, we're actually partners with NVIDIA for a bunch of things. We have a lot of success stories already, people using it in production. And yes, you should definitely take a look at it. In fact, sometimes I go to conferences and I realize people are already using it, or maybe they're using VLLM on their infrastructure and they're not aware of llm-d. So, VLLM allows you to do inference up to a degree in a very high performance, high quality way, and the llm-d and VLLM projects are sister projects in a way. I mean, there's a lot of exchanges across the stack, so we are part of the same team in a way. But yes, llm-d gives you that edge. For instance, VLLM does not do automatic flow control, even if you are only running a one VLLM instance, llm-d gives you control over quality of service and prioritization of HTTP requests.

So, that is something that VLLM, at least until a few days ago, I don't know if things have changed recently, could not do. So, you can get benefit from llm-d even if you're not necessarily running in a very highly distributed situation.

Olimpiu Pop: So, what does that mean? Even if you don't have a cluster, use llm-d?

Edoardo Vacchi: Even if you don't have a lot of GPUs, even if you don't have to distribute the load across different GPUs, llm-d might still be useful. And yes, you can also run it bare metal. So, you might want to evaluate llm-d even if you're not using Kubernetes. I mean, Kubernetes is the first target, but it is supported also on bare metal.

Olimpiu Pop: Okay. So, it's focusing on inference, even if you don't have already existing infrastructure. So, it just tries to provide the best usage of your resources, even if you have or you don't have an infrastructure already existing. It seems that at the point when we are recording this, it's end of October, and it seems that llm-d started at the end of October in 2024.

Edoardo Vacchi: Yes.

Olimpiu Pop: So, it's a good moment to have the first major version out in this period. So, congratulations.

Edoardo Vacchi: Yes.

Observability and Hardware Heterogeneity in AI Clusters [29:34]

Olimpiu Pop: But staying in the Kubernetes space, one of the reasons why people prefer to use things that they are already familiar, Kubernetes is not that old of a solution comparing to other types of solution that were out there, but it's still one of the most mature in the cloud native space. And one of the things that are very important these days, or they always were important, but now they are more than ever, it's observability. What should we expect from that perspective in terms of observability when we're discussing about llm-d? I mean, as you mentioned, everything related to LLMs is more complex than everything else, how can we make that complexity easier to digest through the observability solution already existing?

Edoardo Vacchi: Yes, so I mean, it depends. It's a very wide question. So, the way llm-d actually works, it leverages a lot of signals that are metrics-based. So, there's a lot of insights that are being exposed by VLLM, metrics and data that explains essentially what's going on inside the inference engine. And llm-d uses this information to actually do what I explained earlier, that is this inform the routing across instances. What this means is essentially when you use llm-d and VLLM, you don't just have a system to do efficient inference, you also are getting for free, if you will, a rich system to get insight on what's going on, not just on your cluster, but in your inference engines. You can see how many tokens are being produced, how many failures you're getting, what's the throughput, you can get alert on situation where you might want to intervene.

For instance, you have control over the saturation of your cluster, which means saturation means essentially you have so many requests that you are struggling to serve them. And then at that point you can add more machines, you can throttle the incoming requests. So, you have a lot of knobs that you can work on. The observability story for llm-d is pretty interesting as well, and you have all the tools to get insights of what's going on in your cluster from a GPU and inference perspective as well.

Olimpiu Pop: Well, you already mentioned that it was complicated to get it right, but which thing was the hardest?

Edoardo Vacchi: Ah.

Olimpiu Pop: Everything.

Edoardo Vacchi: That's an interesting question. Yes, everything. Well, one thing that is interesting in this ecosystem is that everything is very new. I would say one of the things that are challenging, maybe not the hardest is that, well, NVIDIA is probably the biggest, most important vendor, but there's also AMD, there's Intel, there's Google's GPUs, and there are more. There are also a lot of startups that are building their own hardware. And while for the PC, we benefit for a lot of time now of a standardized set of hardware components, with a very well-defined instruction set, for these accelerators and GPUs, the languages are very different. So, one of the challenging part for VLLM itself and inference in general is that, and the reason why we need a framework on top of that, is that you can't write the kernel for a GPU computation once and target every single accelerator there is because everything is different.

It's like having Intel, but also ARM, but also RISC-V, but also Power PC, and you have to use all of them depending on your vendors, and you have to support all of them. I mean, in our case, you want to support as many as possible. So, that's probably one of the most challenging things that happen in this case. The other thing that is also possibly challenging from a hardware perspective is that there's a lot of data that has to be transferred. LLMs are large language models, and we hear about trillions of weights of parameters, these translates in gigabytes, terabytes, petabytes of data. And this data have to be moved across machines. You don't just have expensive GPUs, but you also have to have expensive connection between just GPUs. We're talking about RDMA. So, you have to connect this hardware with very expensive type of connection that allow you to have very fast data transfer.

So, I would say in general, from another perspective, these are things that are very challenging. And at the same time, it makes this space very fascinating and interesting.

On-Premises AI and Semantic Routing to Right-Sized Models [34:35]

Olimpiu Pop: Yes, that's for sure. But one, you did mention that the beach of possibilities is quite wide when we're discussing about GPUs, which were intended for something else, but now people, like usual, they adapted and they used it in a different perspective. But now we can see that there are companies that they started building their own hardware, we're discussing about DPUs. How will that change the game?

Edoardo Vacchi: Yes, that's an interesting question. I don't know. Hopefully it might make things more affordable. I mean, competition is always good. The one side is that this makes things challenging for people that are building infrastructure and software on top of these things, it makes things challenging for people that want to source this hardware because it's still very expensive. But at the same time, if there is more competition, hopefully prices eventually will go down. I think one thing that is very important, and I think this is also the message that we want to bring across with llm-d, VLLM, we're getting to a point where first of all, you don't really need extremely expensive hardware to do inference to a certain degree of quality.

Obviously, frontier models will always be more capable than older models, but open models are getting better and better, even relatively small models that can run on consumer hardware. I have a relatively beefy Mac myself, with 48 gigabytes of RAM, this is a company Mac, so I didn't pay for it, and it was probably bought before the price rises, so it was a good time. But it's a relatively small amount of shared memory because it's shared across CPU and GPU, that's the point. And I can run 27B, I think, 27 billion. Which is not huge, but it's pretty capable. And people have talked recently about Qwen 3.8-27B, this does not run very well on my laptop, but it runs on laptops still, and it runs on a laptop even though it's not super fast.

So, if you get a server that is only slightly bigger than my laptop, you will be able to serve a lot of requests in a very efficient way. And this model has been compared to Opus 4.6. And Opus 4.6 is only one year old. And we were super excited and impressed by Opus 4.6 last year. So, let's not get too carried away by all the announcements we read on social media about how all of these models are super capable, and the old models are no longer useful.

The old model are still useful, and they're even more useful if you can run them on your hardware because this is very important, you don't have to share your data with a company that you don't know you can really trust after all. And this is, I think, the message we want to bring across. You can own your own AI, you can own your own token inference on hardware that now it's relatively... Relatively is the word here?

Edoardo Vacchi: Yes, affordable. And this is especially true for companies. I'm not talking about consumer, this is true for consumer to some extent, but it's very true for companies. They can source the sort of hardware, they can do inference for these models, that are not as big as the frontier, but they're definitely quite something, and they can solve a lot of interesting problems already.

Olimpiu Pop: Yes. What I think it's important for people to understand is that do we actually need frontier models for everything.

Edoardo Vacchi: Exactly.

Olimpiu Pop: Because in the end, as you mentioned, it's becoming tangible that you can run them on even laptops.

Edoardo Vacchi: Yes.

Olimpiu Pop: I was mentioning it's something quite basic. It's ordering, I don't know, something. And then you don't actually need anything else but details in terms of human understanding. And then a very small model, it's more than enough, but it's very niched. And then that's the thing is that yes, we do need a perspective where we do that stuff, and we need, I don't know, the Argons, the Opuses and all the other frontier models out there, but now a lot of people are doing experimentation.

And at the point where you're getting into the production space, you have to do the refining, you have to do the efficiency part, and then definitely get on the right spot. And I saw something similar even on the edge side, where they had very tailored models that were just being focused on a very thin, narrow use case, and that's it. But now a lot of us are exploring, and we're measuring stuff that actually don't matter.

Edoardo Vacchi: For sure. I think another interesting aspect, this is another project that we are incubating in llm-d, it's called llm-d Semantic Classifier. It deals with routing, not just efficiently, but to the right model for some definition of right. So, right might be cost-based or it might be topic-based. And so, this system, there's another sister project that's called the VLLM Semantic Router. If you're interested, llm-d semantic classifier is sort of a sibling, if you will, with different targets. But the goal is the same. You have a prompt, you have a conversation going, and you want to route this conversation to the right model, which might mean right sized. And that's exactly what you were saying. So, it doesn't really matter, sometimes you have a very large model that is able to answer every possible question, it might be useful in many situation, of course, but you have also to bring to the table the cost efficiency.

So, you might have specialized models that are trained or fine-tuned on specific domains, and I think that's where we should go where the next thing should be. Because that really brings LLM inference, AI in general, agentic stuff to the people, it allows you to run this stuff on your hardware, with things that are attainable, goals that are attainable, not just super expensive hardware that nobody can afford.

Olimpiu Pop: I think we are adept, let's say teenage years, of what LLMs and AI will mean for the future because everybody's priding themselves on how much horsepowers do our models have. But actually we are stuck in traffic hours and it doesn't actually matter if it's there or not.

Edoardo Vacchi: Yes. Teenage years is a good metaphor, I'd say.

llm-d Roadmap [41:36]

Olimpiu Pop: Maybe just to close up with what's on your roadmap. So, version one is out, what's in the next period? So, upcoming months, upcoming years, what's the target for llm-d?

Edoardo Vacchi: Yes, I mentioned a few already actually. So, llm-d itself is able to run because on this hardware, you have to use it efficiently, it's able to run very large models, mixer expert models by sharing them across machines. But things that we are experimenting and we are getting more and more refined are this batch inference concepts so you can leverage your hardware more efficiently and use those dead moments in your traffic to serve and do inference, even scheduled inference, without the live interaction, so you can 100% reuse that hardware that you have.

And the other is obviously, as I just mentioned, semantic classification so that you can not just use the right machines, but also use the right model at the right time. For instance, let's say you have a very big model that is expensive to run, and smaller models that are more efficient to run and you can run on hardware that is much cheaper. Depending on the conversation that you're having on the prompt that you're sending, you might want to use the larger model in some situations or the smaller model in some situation. So, these are the things that we are working on currently, there are more, and I cannot even list all of them. But it's an exciting space and I think you're going to see a lot of interesting stuff in the next coming months.

Olimpiu Pop: Okay, thank you for your time.

Edoardo Vacchi: Thank you.

Mentioned:

• llm-d
• vLLM Inference Engine
• TornadoVM
• GraalVM
• Qwen