BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Presentations Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads

Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads

42:37

Summary

Sundara Raman Ramachandran shares how LinkedIn productionizes LLM ranking under strict P99 latency SLAs. He discusses distilling a 7B model down to 400M parameters, compressing job contexts into single-token embeddings for a 10x throughput leap, and eliminating hidden system bottlenecks - from custom flash attention masks to Python GIL workarounds, gc.freeze, and non-blocking CUDA memory copies.

Bio

Sundara Raman Ramachandran is a Lead Engineer on LinkedIn’s LLM Inference team, where he plays a key role in designing and scaling the infrastructure powering LLM-based ranking systems for search and recommendation. His work centers on building latency-critical, high-throughput LLM serving platforms that operate reliably at global scale.

About the conference

QCon AI is a practitioner-led event focused entirely on the engineering discipline required to scale these workloads safely. It provides direct access to the architectural playbooks and failure metrics that peer organizations use in production.

Transcript

Sundara Raman Ramachandran: I'm Sundara. I'm a tech lead at LinkedIn. My current focus is on LLM inference and using LLMs for ranking and recommendation workloads. I'm also an open-source contributor in SGLang where I have primarily contributed to the scoring API where we use LLMs for ranking. I have close to 10 years of industry experience. I have worked with Microsoft in different products like Microsoft Office and Azure. Today I'm going to talk about how we scale the LLM based ranking systems at LinkedIn, what were the challenges we encountered, and what are some learnings and takeaways you can take from the talk.

LinkedIn Search

I hope all of you would have used LinkedIn at some point, that you might have used it for searching for some job or you're looking for someone in a network. The reason I ask is LinkedIn Search operates at a massive scale. If you haven't read it already, maybe after the talk you can take up the LinkedIn app and maybe search about me. LinkedIn right now has close to 1.3 billion members worldwide across 200 countries, and we are operating a very high QPS for the ranking and recommendations. Search itself is evolving. Earlier we used to search based on specific keywords, and there were specific search filters that you put when you're entering any search query. Nowadays we are moving into more of a natural language to specify our intent. In LinkedIn's use case, you might be searching for a job where you enter, find me design roles in the entertainment industry.

Or you're searching for someone in your network where the search query could be, who in my network can help me file a patent for my business? All these are raw natural language texts which are complex to understand. If you think about this, this is where LLMs can be really powerful. They help us understand the user intent and then help us provide them better results. Search and ranking at LinkedIn scale is already a complex enough problem, and if you bring LLMs into the mix, it makes it even harder. For engineers like me, it's a really fun problem to solve.

Architecture

Let's look at the high-level architecture of how these things work end-to-end. It all starts with a search query where you have the find me design roles in the entertainment industry, and the search query goes through the first layer called query understanding layer. This is where we help figure out what the user intent is, what they're actually looking for. Then comes the retrieval layer. LinkedIn has millions of jobs posted in this platform. The job of the retrieval layer is to figure out the top-K most relevant jobs for your search query. Then all these results go to the ranker layer, which ultimately decides the order in which we want to show the results to you guys. In this talk, I'm going to quickly give an overview of the query understanding and the retrieval layer to add more context to the overall talk. My specific talk is going to be focused on the ranking layer.

Let's look at the query understanding layer first. The goal of the query understanding layer is to turn the raw, ambiguous natural language text that the user has entered and convert that into more of a structured, normalized format, which the further systems can understand. Let's take this example where we have a member who's a software engineer in the Bay Area with 5 years of experience. The search query they have entered is, find software engineer jobs near my location. We take this query and then the first step here is the query planner, which identifies the search criteria here. In this case, it's a job title and location, and figures out a plan on how to execute it. The next comes the query tagger. This step enriches the query with more rich information, which we can pull from the member profile, like what is their job title, what is their location, and also some search filters they might have in their LinkedIn.

Next comes the query rewriter, which is essentially the important piece here. If you look at a search query, we have something like a self-referencing, ambiguous text there, find me software engineer job near my location. Someone has to go and really understand what the user location is. That's where the query rewriter comes into play. It's going to replace the near my location with Bay Area, which is the user location. All of this gets packaged into a nice format, normalized enough, and then passed on to the retrieval layer.

Now when we move on to the retrieval layer, the goal here is for the user query, we have to find the top-K semantically relevant jobs. We have millions of jobs posted out there. We cannot go take every job and rank them. That's where the input starts with the query understanding layer output, and we take that and convert that into a query embedding. We then do a vector search with the embedding and do a KNN search on the index, which was pre-built using all the pre-computed embeddings from the jobs out there. From this step, we get the top-K most semantically relevant jobs for the search query. In this step, we primarily focus on optimizing for recall rather than precision. What we exactly mean by that is we would rather have a few extra candidates out of the retrieval layer than discarding something which is highly relevant.

The output from the retrieval layer then is passed on to the ranking layer. As mentioned about previously, this is where we figure out what order we want to show the results to the user. We want to show the highly relevant one as the first item. If you haven't already guessed, in LinkedIn right now, we are employing LLMs to do this job for us. As with any LLM, the first step is to provide enough context for the LLM to make the right decision. That's where the output from the previous two layers, the query understanding and the retrieval layer becomes the context for this LLM. Then comes the system prompt. You have to build the prompt. You have to tell the LLM to do a specific job. You'll be surprised what we have in production. It goes by this, like we prompt the LLM, given a search query and given the member profile description and a job, can you output yes or no if the job matches the query?

It's literally that. That's what we really have in production. At the end of this, we take the probabilities of model predicting yes or no token, and use that to compute the final scores. We do this for all the candidates provided by the retrieval layer, and then finally do some business logic to show what is the first result we want to show to you.

Part Zero - LLM Ranking is a Different Problem

When you're operating systems at this scale and you're serving billions of members out there, you want to optimize all the layers underneath. That's where we have three main levels. Before going to that, I'm going to specify why LLM ranking is a very different problem. Usually, when we think about LLMs, we associate with chatbot text generation. Ranking is a very different problem all together. If you look at the generative LLM workloads, you have a prompt which you give to the LLM and it goes through a phase called prefill phase. Prefill phase is the one that the LLM understands the context you have given to it. Next comes the decode phase, and we call it decode loop here because we are generating text token by token. It goes through several steps like sampling, next token prediction. You want to get the probabilities of all the token and the vocabulary and figure out what is the token you want to output to the user, which ultimately gives you the generated text token by token.

The latency here scales with the number of output tokens required. If you take a ranking use case, let's take the job ranking scenario. We talked about the query here is software engineer jobs in Bay Area, and the system prompt is simple, like output yes or no based on a job match. The list of candidates or the list of items to be ranked here is the job description. The main difference here, let's say we want to rank 50 items, we are constructing 50 prompts to the LLM for each item. You'll have a query item 1, query item 2, like that, 50 items. All of this goes through a single batched prefill phase. Another important difference here is all of this share a common long shared prefix in terms of the query, in terms of the system prompt, and all other context you feed to the LLM.

These are shared among all the jobs you want to rank. The third one here is there is not necessarily a decode phase. We are not generating any text per se. All we need is the final probabilities of the yes or no token. The major challenges when serving a system like this is we are dealing with strict P99 latency SLAs. Imagine you go to LinkedIn and search something and we take 10 seconds to show the result. That is not a good experience. We are also dealing with high concurrency where it's not enough if we just show one job or one person in a network, we have to show a bunch of candidates, usually in the hundreds. These challenges require us to optimize on all possible trends. That's where these three come into play. In an LLM system, we have the model, we have the context, and we have the inference.

Whenever you're working with these kinds of systems, you want to make the model as small as possible and you want to make the context as short as possible, because token here is the currency. Inference, you have to make it as efficient as possible. In the rest of the talk, I'm going to dive deep into each of these sections where I'm going to share what we did, what worked for us, and what are some learnings that you could take away from us.

Part One - Model Optimizations

Let's start with the model optimizations. The goal is simple. Like you have a model, you want to make it as small as possible. What are the techniques we can employ for this? The first thing we did is something called multi-teacher distillation. We started with the larger model, around 7-billion parameter size. This model is really good in understanding the context. It can understand the product policies and it can understand the relevance between the query and item, but it was never intended to be deployed in production. The whole purpose of this teacher model or the larger model is to provide high quality training signals. We then transfer this knowledge into two separate smaller teacher models by a process called knowledge distillation. We have two models which are optimized for separate metrics, one for relevance, another for engagement. These are metrics in ranking and recommendation workloads. The relevance here basically denotes how relevant is the search result for the query.

The engagement here basically specifies, if I show this result to the user, how likely it is to drive engagement, whether the user will click, whether the user will read, and all those things. Now we have two smaller teacher models, we call it, which is optimized on these two metrics. We then use this and then train another small model, SLM ranker is what we call it, small language model, which learns the objectives from both of these teacher models. We essentially started with a larger 7 billion model and we were able to knowledge distill all this in a multi-step, in a multi-teacher way, and then compress all the context into the smaller model. You will have some quality loss, but it should be acceptable for the business use case you're solving. We didn't stop there. We looked into how to make this model even smaller. That's where a technique called pruning comes into play.

It's essentially, you take the model and you do some trial and error and figure out, can I remove some layers? Can I remove some MLP layers or any attention heads? Then see if there is an acceptable quality loss that you can still serve. Obviously, pruning doesn't come for free. Whenever you're removing something from the model, there will be some quality loss. That's why we have to again re-distill, re-train and fine-tune the model so that the loss is within the acceptable range. Just to recall the model optimizations, we started with the 7 billion larger model with the multi-teacher distillation. We got it to 0.6B. We took the 0.6B and then removed some layers and finally got this in the size of 400 million parameters. Now the model is super small that we can now already serve this in production.

Part Two - Context Optimizations

Next, I want to talk about the context optimization. If we take the job ranking scenario and a particular job description can easily be 2000 tokens. When you want to serve a ranking request, so you have a request and you have 100 jobs you want to rank. This could easily be 100 into 2000, like 200K tokens that you want to process in one shot. If we have very few members, let's say in the order of thousands, we can serve this easily. At LinkedIn scale, when you have 1.3 billion members, this is not possible. That's where we started looking into how can we compress this context. The first step we came up with is to use a summarizer, use another LLM to summarize this job description from 2000 tokens to 150 tokens. We do all this work offline. We never compute all this in the online path.

In the online, we just fetch the summarized text from the database and then pass it as input to the ranker. This helped us to bring the item prompt length all the way from 2000 tokens to 150 tokens, which is a 16x reduction in the prompt length. The interesting thing to note here is we were able to get all this thing done with a less than 2% quality loss. This is where we took prompt engineering serious. Prompt is the currency, you have to make them as small as possible. In the next question, we didn't stop there. We were looking into opportunities, how we can make the item representation even smaller. Is it possible to just have one token instead of 150 tokens? Literally not one token. Can it just be one dense vector representation? Or in simple terms, can it just be one embedding? That's where our research comes into play, where we take the full text description and then pass it to an LLM, which basically compresses into just one embedding.

LLM does a really good job. It can also be used as an embedding model, where you can pass in a full text and you can get one embedding out of it. Now for all compute purposes, we have compressed the item token length from 2000 to just one token. Then, again, we do all this offline. Nothing happens in the online path. We just come up, fetch this from the database and then pass it to the ranker.

Now you can ask me, "Sundara, this is really great. You have context compressed, went all the way from 2000 to one token. How does it perform quality-wise? Does it give you relevance result?" That's where I'm going to discuss about this graph. On the y-axis, we have the ranking quality. On the x-axis, I have the throughput measured in terms of items per second per GPU. The leftmost data point over there is the baseline metric where we use the full job description to get the results. This is the one which gives the highest-ranking quality and also lowest throughput. We were able to get only 290 items per second. The second data point is the data point for summarized text. When we use summarizer to bring down the token size from 2000 to 150. This helped us to almost increase the throughput from 290 to 2200, close to 8x, 9x, with a less than 2% acceptable quality loss.

The third data point on the right is the one I talked about in the previous slide, where we take the full job description and then context compress into just one embedding. You can see the x-axis where the numbers run from 2200 to almost 22k. This is like a 10x improvement in the throughput which you can serve. Even more interesting thing to note here is, if you see the quality, you were able to get a slightly better quality compared to the summarized text. The key takeaway here is, the best representation for your workload is not necessarily the longest one. It's the smallest one that can give you the acceptable quality your use case needs. It can be the full text, or it can be a summarized text, or it can just be even one embedding. My biggest recommendation for people serving models in production is to get the model and context really tight. Make it as short as possible. Before even you do any hardware estimates or capacity planning, get your model and context right.

Part Three - Inference Optimizations

In our use case, once we got both of them right, we started looking into inference. Inference is a really interesting problem. We already have great open-source engines out there like SGLang, vLLM, TensorRT-LLM, and they already provide state-of-the-art throughput for most of the use cases. It doesn't necessarily mean that for your use case, it's going to work very well. That's how we approach the problem. We didn't treat inference as a solved problem. We started looking into what is different in our workload and what are all the optimization opportunities we can look out for. In the next few slides, I'm going to dive deeper into the inference side of things. The whole idea here is there are a lot of bottlenecks, not inference related, but system related, runtime related, maybe related to the engine you are serving, maybe related to the language you are serving, which I am going to call out.

These might be hiding in your stack as well. The first bottleneck was on the CPU side. When you are serving production systems using LLMs, the first step is something called a tokenizer. In our code, what we had was we were taking the batch of items. Let's go back to the ranking workload. You have a request where you want to rank 50 items, let's say. You have like 50 prompts as a batch input to the LLM. In our code, what was happening was the tokenizer was happening sequentially on the CPU side. Basically, this was happening in the for loop one by one, it was stalling the GPU and the CPU was just sitting idle. In order to fix this, we replaced this with a batch tokenizer. What it means is Hugging Face APIs provide a fast tokenizer API, where you can send a batch of prompts and then it will parallelize the work across multiple codes.

Internally, it's implemented using REST, but for our use case, all this is available in open source. You just have to leverage these APIs. A natural extension of this was, can we do something on the server side so that we don't have to repeat this client-side batching for workloads where clients cannot naturally batch and send a batch request. We did have a lot of embedding generation workloads where either the client was not able to batch on the client side, or we had a shared model where we had multiple clients. If you want to do this batching on the client side, you have to repeat this logic across multiple clients. That's where we introduced something like server-side batching, which is going to batch across the request and then do one batch tokenization using the same fast tokenizer API, which I mentioned before. The important callout here is, if you are having any production systems, make sure you are doing batch tokenization. All these APIs are available out there in open source. Just make sure you are leveraging them.

The next bottleneck was even more subtle. When you are having an LLM based system, as I mentioned, you will have the tokenizer process, which is going to take the prompt and then figure out the token IDs. There will also be a scheduler process, some sort of scheduler, which is the one which is preparing the work and then dispatching the work to the GPU. When we looked into our system, we were sending a batch of 50 prompts. All of this can fit into the model context. What we saw in the logs was, this was getting split into smaller batches. Then we profiled the system and then figured out the root cause here to be the inter-process communication between these two processes. Essentially, from the tokenizer process to the scheduler process, we were sending the prompts one by one in a separate ZMQ message. In order to fix this, again, the fixes here are really simple.

You just want to maintain the batch integrity throughout. You just send all of them in a single ZMQ message, so that the entire batch is passed on from the tokenizer to the scheduler in one shot. A key takeaway from this slide is, if you are having production systems based in Python, you are definitely leveraging multiprocessing for some reason. I'll discuss about why in the next few slides. Make sure when you are using multiprocessing, you are using the right set of communication protocol between them. Make sure you're maintaining the batch integrity when you're sending the request.

On to the next one, I talked about this in the initial part of the slides, where ranking is completely different from text generation. Let's walk through it again. Text generation goes through the prefill phase, and a decode phase that is where we are going to generate token by token, and we predict the probabilities of all the tokens out there in the vocabulary and we are going to pick the one which is the top or based on some heuristic. That's the sampling phase. In order for that to happen you are copying all these probabilities, all these floats from GPU to CPU. There is a lot of work that is happening which are not required for ranking. If you think about the ranking or the scoring use case, you just have to do one large prefill, and then you have to copy the yes or no probabilities at the end.

This motivated us to introduce the dedicated scoring API in open source where we bypass all this unnecessary work and then just get the logits at the end. The scoring API was already really good for us, we were able to solve most of the CPU side bottleneck. Then we deploy the system in production and we see the GPU utilization metrics, and the metrics were not good. We were scratching our heads like, what is happening? Then we go profile it, use the PyTorch profile and everything, and then what we see here is there was a lot of idle gaps between the GPU kernel launches. I am going to walk you through this graph here. On the first line we have the tokenizer, as with any LLM you will first have to tokenize it. This is happening on the CPU side. Next comes the scheduler process which is again on the CPU, and this is the process which is preparing the batches to dispatch to the GPU.

The third line is the GPU compute where you have the kernel launches, all the matrix multiplications happening. In our case, for ranking, we are scoring 50 prompts at the same time. What we were doing in our code was we are copying the yes or no probabilities one by one for each request for each prompt in a synchronized blocking way. This essentially stall the scheduler from launching the next kernel launch for the next request. In order to fix this, we replace this synchronous blocking call into an asynchronous non-blocking copy, and also instead of copying the yes or no probabilities of each and every item, we replace with a batched copy. This helped us to better overlap the CPU work of the previous request while we dispatch the GPU work of the next request. The key takeaway from this slide is if you have Python systems and if you are working with tensors and all sorts of things, just make sure you are using asynchronous non-blocking copies. For example, like when we are using some tensor and copying the tensors from GPU to CPU, a simple dot to list Python call made the copy blocking. All these are subtle things which can affect your overall GPU utilization, and we have not even gotten into kernel yet, everything is happening on the CPU side.

The next thing, once we solved all the CPU side, and all the CPU bottlenecks were handled, we were looking into what work we were doing redundantly and trying to figure out if we can optimize for it. Let's revisit the ranking use case where you have the query and the items. When you are ranking 50 items you are constructing 50 prompts. If you look at the query path, the query path is repeated across all items. We are redoing the query work every time. That is where we researched a bit and trying to see how we can reuse that work, and that is where we come up with an idea to concatenate all the prompts into just one single prompt to the LLM. The blue blocks here denote the query and the green is the first item followed by a delimiter followed by item 2. We just concatenate all of them, and it is just the one single prompt to the LLM.

The key challenge here is, let's say we have a query job 1 and job 2, when the LLM is processing the job 2 and making a decision for it, it should not bring the context of job 1, or else it is going to give you wrong results. That is where we went a little bit into the kernel and then we introduced the custom Item-Aware Attention Mask, inside the flash into the kernel, so that when the LLM sees job 2 it is going to forget about job 1. By doing this optimization we were able to score n items in single pass, and we were able to naturally reuse the query KVs because we only have it once in the prompt.

I know I went a little bit deeper into the kernel and all, but the next few slides will be really interesting because these are something we didn't expect when we deployed in production, and these are brought by Python. We might have a lot of Python code. The first interesting thing that we encountered in production was, we deploy the system, we make it fast enough, and we deploy in the model server. We go see the production metrics, we were seeing spikes every few seconds in P99 latency. Just to reiterate, for search and recommendation, we are operating at a very strict latency. Usually, the P99 has to be less than 500 milliseconds. We go see the graph and we see every few seconds the P99 spikes by 300 milliseconds or so. The reason here turned out to be the Python garbage collector. Before we even got to that, we spent some time, the usual suspects would be like, there was a surge in QPS, or there was one request that had a longer prompt.

These are like user culprits. You go rule them out, and then you have to spend more time debugging what's happening, and the root cause turned out to be the garbage collector. In model server, when you load the model and there are a lot of long-lived Python objects that get created that lives through the lifecycle of the overall process. What was happening is this Python garbage collector was kicking in every few seconds and was scanning each of these objects to figure out which one it should free. This contributed to almost 300-millisecond spike every now and then. In order to fix this what we did was we started the model server. We warmed it up using some warm-up request. Then Python gives you a beautiful API called gc.freeze, which tells the garbage collector, up until this point, all the objects that got created are going to live throughout the life cycle of the model server, just to ignore it.

After this, whatever temporary object that gets created during the request path is the one the garbage collector will be interested in. With this fix we were able to get the P99 in a stable way. The next bottleneck was Python's infamous Global Interpreter Lock, which effectively makes Python bytecode executable by single thread only once at a time. It's different from how Java and C++ and all those things can give you true parallelism when you are working with threads. Python is a different thing all together. In LinkedIn's system, we take the open-source engines we have, and we take the SGLang engine, and then put it behind a gRPC servicer process. The request comes as a gRPC protobuf and the servicer process is the one which is going to deserialize the request. It's also going to do some observability, logging, metrics, and other stuff, and then dispatch the request to the SGLang worker process.

Because of the Global Interpreter Lock, and when we send large QPS the single servicer process became the bottleneck. In order to fix this, we replaced a single process with multiple processes, so that all of them can do this work in parallel in separate CPU cores. Once we solve the bottleneck on a servicer with multiprocessing, the SGLang worker process which is effectively preparing the work and dispatching the work to the GPU became the bottleneck. The fix here is again the same, instead of one process you will have multiple processes. The callout here is, each process has its own model copy and it's going to have its own CUDA stream and do the kernel launches and MemSync and whatnot, and it's effectively going to time slice the GPU. In our case we were able to do it because the model was smaller. Nowadays I think we get enough HBM in the GPU, so you should be able to deploy this as well. The callout again here, in production systems based on Python, is, worry about this Global Interpreter Lock and this can be your single fix which can help improve the throughput.

Part Four - Learnings at LinkedIn Scale

I know I went a little deeper into the inference side of things. I'm going to keep it really simple in the last four slides. It's pretty simple. These are the major learnings that you could take away from this talk and from our work. The first one is what I call benchmark what you serve. You will be like, "Sundara, that's so obvious. What are you doing otherwise?" I will tell a small story of what we did. We do all these beautiful optimizations in model, context, inference, and then we go and do an offline benchmark. We figure out the throughput we were able to get out of a GPU and then we know how much QPS we want to serve. We do the estimates, and we go inform our leadership, this is the amount of GPUs we need. They were happy and we were happy they were happy.

We go deploy this in production and we send some dark traffic, that's where we see an entirely different result. We were not able to realize the offline benchmark numbers we were able to get. We are scratching our head like, what is wrong here? Is there anything wrong with the hardware? Is there anything wrong with the specific GPU, or is there any communication bottleneck between the client and server? Something happening over the network. It turned out, we were doing our benchmarks incorrectly. When we are simulating synthetic traffic, we were effectively using even distribution to send the request. What it means is within a second interval, you are estimating based on QPS. You're going to say that my throughput is 1000 jobs per second, 1000 items per second, or something of that sort. When you're simulating the synthetic traffic, the requests are getting sent in an equally spaced manner.

All of them arrive in a uniform distribution with equal spaces. In production, when you're serving real time user traffic, you cannot expect the users to follow the same patterns. Within a second interval, you might have a surge in requests so the queue depth increases. After that, there could be some quieter periods. The major takeaway from this here is if you are serving systems in production, the closest possible result you can get is if you can record and replay your production traffic. If you don't have that option, the second closest thing you could get is if you can simulate the synthetic traffic using a Poisson distribution instead of an even distribution. Just make sure you're doing the right thing there. The second learning is what I call offline must match online. What do we mean by this? When you're working with a model-based system in production, there is an engineer who's going to work on fine-tuning the model.

There'll be prompt engineers who are designing the prompts, and they have their own set of code. The system that gets deployed in production is entirely different. You might be leveraging an open-source system, or you might be leveraging your own model server. There are a set of defaults which is different from both these. Like for example, a simple tokenization config like add end of sequence or beginning of sequence config, can change the entire prompt in itself. You have to make sure there is absolute parity between the offline and the online system. Don't discover all these disparities or discrepancies in production. Disparity has to be a gate without which you cannot launch in production at all.

The third learning is the crux of the talk. There is no single fix here when you are serving systems in production, models, LLMs, anything of that sort. You have to squeeze in throughput from all the possible places. You have to make the model as small as possible, and you have to make the context as short as possible and make the inference as efficient as possible. The final learning, if there are any tech leads, managers, and leaders, this is for you guys. When you are dealing with production systems, there are multiple actors here. You have the inference engineers. You have the model engineers. You have the prompt engineers. You have the product managers who are defining the product policies. It's very important that we bring all of these actors into one setup, one meeting which is recurring. You can have a standup or a meeting every week so that each of them can share their idea, cross check, cross validate everything.

You will be surprised how many issues you can catch in these meetings before even you get to production. We often think about optimizations as, if we have to optimize the code, we have to optimize the CPU, GPU, Python, Java, C++, Rust, and all those things. The first and foremost optimization we had to do is with respect to bringing the right people together, you remove the communication barrier and make sure you allow them to share their ideas, and then effectively improve the overall system in play.

Questions and Answers

Participant 1: Can you tell me more about that single token embedding?

Sundara Raman Ramachandran: The idea here is you can use LLMs as a cross encoder. What it essentially means is you can give full text and then get one single embedding representation out of it. There are multiple techniques how you get this embedding. You can do some last token pooling is what they call. You make the model to do a complete forward pass and then just take the embedding from the last token. We expect the last token to have the complete context of the entire full text. That is one way. The other way is you can take the embedding for each token and then do an average of all these embeddings to get the average template. There are multiple techniques out there. What worked for us is the last token embedding. We take the full text, give it to an LLM, use it as an encoding model, and then get the one single job embedding representation.

Effectively, if you think about it, like from 2000 tokens, this has become one token. In LLM systems internally, this is how it works. You give a list of tokens, let's say 100 tokens. All this first gets converted into one embedding for each of these 100 tokens. The embedding is the input to the overall matrix multiplication that goes into the model. Then it's finally going to spit out another embedding, and then you convert this embedding back, map it to another token, and that becomes your output. Embedding is the one which is the input to the matrix and everything is multiplied between.

Participant 2: You talked about pruning the models themselves, the visible layers that weren't needed. How did you go about identifying the parts that weren't needed and pruning them?

Sundara Raman Ramachandran: The question was more about the structured pruning, how we identified the layers and the MLP layers we have to prune, and what are the techniques we employed.

This is where it comes in a bit of a trial and error, in a research way. Where we take the model, the 0.6B student ranker which we got, and then we do some SFT based techniques where we mask out certain layers. You start by removing certain MLP layers, one before the last layer, before we prune it, and then do some quality analysis to figure out if the current quality is good enough for the use case. You predict this quality and then you do again a post-training or a pre-training approach to improve the quality again. There is a lot of RL-based fine-tuning you can do further to improve the quality. Those are the techniques we employed.

The question is how much efficiency we were able to save with using the system.

We are optimizing for two things here. One is we want to drive the user engagement. When you think about all the social media websites, we want to make sure two things here, one is the relevance. We want to show the most relevant possible result. If you are searching for someone, the first result has to be the exact person you are searching for. That is number one. The second thing, if you're thinking about job ranking and all, it's really the job engagement which we are looking for. The most important thing is to improve the product metrics here. We were just comparing with the earlier techniques, what we employed versus this, and how much engagement we are able to improve, how relevant results we are able to give. The throughput on the other things come next for big companies like LinkedIn. Once we make sure we are able to see significant product policy improvement, then we go back and try to optimize on how much throughput we can squeeze.

This is an iterative process. Right now, we are here, we have already optimized good enough, but if you are asking the question about previous retrieval models or previous ranker models, the previous ranker models were just small MLP models where you can just do a couple of multiplication and get the cosine similarity or something. Obviously, that's going to be faster, but is this the thing which is giving you the right answer? No. If as an end user experience, if I give you the wrong answer by doing that, that's not helpful. That's why we compare the product quality and then comes the throughput and cost savings and everything.

Participant 3: What was the product quality difference that you saw between using this LLM based re-ranking versus the previous re-ranking methods?

Sundara Raman Ramachandran: I've discussed all this in the blogs, and there are a couple of associated papers also. I think we were able to see, if I'm not wrong, close to like a 5% to 10% increase in the auction metrics, is what we call it, the relevance metrics and all.

Participant 4: It's not related to the ranking, but for the retrieval, how do you handle the security part when you retrieve the information, the compartmentalization of the data?

Sundara Raman Ramachandran: You're asking, when we retrieve the information about the jobs and all, how do we handle the security part of it?

Participant 4: When the user asks a question, how do you make sure to retrieve the results only that the user allowed it to? With all the pruning and all other models, how do you handle this part?

Sundara Raman Ramachandran: Let's look at the job ranking scenario. The job description or job text is posted in public. There is no PII data which is available in the job. All those things get pre-filtered if there is something. For example, when we are working with people search or member profile, we're going to mask all the public information data or the private data, which is not exposed in their LinkedIn profile already. If there is something, for example, the date of birth or something, that's going to get masked out there. It's not even fed into the model at all. Only the publicly available information goes as input to the model.

Participant 5: Since you're saying this is in production, and I believe there's millions of requests every second happening for this kind of request search, I was wondering how this LLM infrastructure integrated with the existing service or server to handle this huge amount of requests?

Sundara Raman Ramachandran: I believe the people search and job search is right now rolled out to all premium users. It, I think, gradually will get rolled out to all users by end of this year. In terms of the actual server in production, I show a block diagram in the end. This is what we actually have in production. This is the entire model server high-level architecture. We are using a Python-based gRPC server, and we are leveraging an open-source engine, SGLang. It sits behind the Python gRPC servicer. It's all part of one application per se. It's not split into multiple, I believe. It's not like a multi-servicer and all, it's just one Python application. All these are just Python processes in the end. This is the model server. Essentially, this model server is the one which will fit into the ranker block, which we saw in the architecture diagram. This is the piece which is taking the query and the list of candidates, and it's going to give the scores for each of the candidates.

 

See more presentations with transcripts

 

Recorded at:

QCon AI is a practitioner-led event focused entirely on the engineering discipline required to scale these workloads safely. It provides direct
access to the architectural playbooks and failure metrics that peer organizations use in production.

Nov 17, 2026

BT