Transcript
Sudeep Das: I'm the head of AI for new verticals at DoorDash. In the last week, how many of you have ordered something from DoorDash, restaurants? How many of you have ordered grocery or something other than restaurants from DoorDash? This is the work we have to do. The talk that I wanted to give you today is about how grocery planning, shopping is like multi-state and it requires planning, reasoning, going back and forth about your preferences, and things like that. Personalization in the old way, the legacy way of one-shot predictions no longer work. With AI, we're in this unique juncture where you can actually build such systems that can reason, adapt, plan for the consumer and make a shopping mission, like grocery or something come true. I want to tell you about a few things that we are building at DoorDash that are the raw ingredients of that transformation, from one-shot prediction systems to systems that can plan and understand the consumer at a different level, and the items and the intent.
A lot of people still think of DoorDash as a restaurant delivery business, but we are actually shaping ourselves up for a time now to be your local commerce delivery system. To that end, we have been expanding beyond restaurants to grocery, convenience, alcohol, retail, flowers, and so on. We want to capture every shopping moment that's around you and deliver the things that you order at very high quality without defects. This requires ML and AI at every touchpoint. I'm going to talk about a few components, as I said, that we're going to build. Before that, I want to really drive this point home, which is the classical playbook based on your clicks and ads. We do more of that, like people who buy X also buy Y. Those kinds of things, they work for a while. They are still working. With generative AI in the mix, we have this unique opportunity to change the way these systems act, react, plan, reason for you.
Roadmap
The components that I'll go through at a high level in this talk that we're building to layer on top of this traditional ML layer that we already had are the following. I'll talk about how we create a fingerprint of the consumer through consumer memory. Similarly, for every item that we carry on DoorDash, how do we create a fingerprint for those items? How do you know everything about the item? I will introduce this concept of semantic IDs. It's a new way of thinking about taxonomies. Then, at the moment when you're shopping at DoorDash, we need to understand your intent. Intent can be very hard. I'll give you some examples later, but consumer intent is something that we really do need to understand before we go into your shopping journey. Then, under the hood, we have all these ranking systems that is like understanding you and ranking things.
That's where relevance is extremely important. Basically, if the thing that I'm ranking for you and showing you at the top of the page is not relevant to the query, then we have missed that moment. I'm going to talk about how LLMs and generative AI is helping us actually refine that relevance. Then at the end, I will end with how all these systems come together and how we're imagining a future where agents will do the shopping for you, agents, agentic assistants, and how we are actually building infrastructure to harden those agents to do the shopping for you without making mistakes.
Reading the Consumer - Consumer Memory
The first part is about consumer memory. If you have been doing traditional machine learning, typically the way we have been understanding the consumer is by compressing all their shopping, clicking, behavioral history into these things called embeddings. Embeddings are these dense vectors of numbers that encode everything the consumer has done on the platform. This is the proven and old way of doing things. Agents, if you actually have an LLM, do recommendations for you, they are not going to understand what an embedding means. These are just numbers. The shift is from these numerical expressions of the consumer to language native snippets about the consumer. This is semantic language native understanding of the consumer that other LLMs and agents can understand. This is a big shift. If you think about memory as like a timescale type thing, then there are three timescales that come in in the memory.
Because you have been on DoorDash for a long time, you have a long-term memory. You understand your dietary preferences. We understand your brand affinities and things like that. Those go into long-term memory. Then as you're shopping on the platform, you have this thing called real-time context. Today I'm shopping for a party of 16, but normally I shop for a household of 2. That's real-time context that these systems need to carry. Then, as you're chatting with an agent, you're probably stating things that we should record and put it back into the long-term memory. Let's say I say, I just got a pet and then started ordering pet foods. That becomes something that is stated inside an agentic session that can now be relegated back into the long-term memory. These timescales of different memory systems, they work together. One learns from another. We also learn how to erase long-term memory.
Let's say you did not have an allergy, now you have it, that has to go back into long-term memory. Sometimes we will also recover from allergies, so that needs to be erased. This is a living, breathing system where we are expressing the consumer as language native memory blocks. I'll talk about only the long-term memory, which is where the blocks come in. Essentially, what we do is we look at your entire history as a consumer and we break them down into well-composed memory blocks. These blocks can be things like dietary narrative, dining patterns, the brand affinities, the taxonomies that you buy, store preferences, and things that are cross-cutting. Like say this is the kind of wine that you want to pair with your steak and stuff like that. Here's a little snippet of like, one consumer, which is me, is saying strong affinity for organic produce, prefers premium brands in fresh categories.
Weekly bulk shopper. When it comes to packaged goods, this is the type of price range I buy. Average basket size is such and such. This is a language native understanding of the consumer that you could not get from an embedding on which now another LLM can act. This is a little bit technical of like how we do this in the sense like, these are strict, versioned components. You cannot just arbitrarily erase and update this memory. Also, if you don't have versioning, you cannot do an A/B test. What we do is we have strict versioning. We can do time travel. Basically, say, I want to just compare the memory you had until this date versus this date. Then to do A/B testing, you need this versioning, because sometimes what we're doing is to actually enhance these memory blocks. We do prompt optimization. We run GEPA. Then I want to know whether the new version is better than the old version. Without versioning, we cannot do A/B test. All these things are baked into the way we structure and define these memory blocks.
What do we do with these blocks? Once we have these, there's various things you can do. There are various downstream applications. I don't want to go into every single one of them. I want to showcase three. One is obviously you can now use these memory blocks to create carousels or basically merchandising for me in the moment. I'll give you some examples later. The second, which I already talked about, is like if I'm building a grocery shopping agent for you. That agent can interrogate the memory and say, for Sudeep, whenever he says chicken, it has to be chicken thighs, and organic, and from Mary's brand, things like that. Then, the third thing, which is where the traditional ML and this layer blend together, is you can actually have these memory chunks then again embedded as embeddings and then added back into the traditional ML layer. It's amazing because now there is this semantic understanding of the consumer that did not exist before.
Now you understand things like when someone says organic, plant-forward, there's many other things that you can actually know about this consumer, that you can infer about this consumer, that this person is probably going to also be interested in premium single origin beans or something like that. Those understandings of like from the what to the why is actually changing the way we're doing even our traditional ranking through these memory blocks. Going back to the first application, which is generating these carousels. Basically, I can create a ton of these carousels offline for all our consumers because LLMs are expensive. You don't want to run them for every consumer at every moment, live. What we do is we create blueprints of these carousels in an offline batch LLM way, by interrogating the memory blocks of the consumers. Then, when it comes to actually serving them, we hydrate in real time.
Essentially, there is a Milvus vector database against which this collection will go and pull in the metadata. Then we have rankers to rank the items in real time. None of that creation of the carousel happens live, and that saves a ton of compute and tokens. It can come up with things that are very contextual in an in-the-moment for you. Like, for example, if we see from your in-session memory or cross-session memory that you have been browsing a lot of noise cancelling headphones recently, then come Black Friday, I can automatically create this carousel which is extremely tailored to you. Of course, from the memory blocks, I know that this is your price sensitivity, this is the range you're looking at, and then create this Black Friday tuned-to-you carousel. This is the in-the-moment meeting the consumer. It is generative, but it's not live. Not everything in the stack is live. It's a very smart way to do this so that we are not burning tokens as we're building the carousel in real time.
The second application was agentic recommendations, which I'll get to at the end of the talk. The third application that I talked about was how these embeddings are being used inside our traditional ML models. What we're doing there is essentially creating this asymmetric two-tower embeddings. By asymmetric, what I mean is we put a little bit of instruction in the consumer side before embedding it, saying that given this consumer's preferences and brand affinities, retrieve items that match. This little trick of putting this asymmetry actually makes the model learn that it's not just about the consumer similarity, but also, given what the consumer has consumed so far, what are the other things that are relevant to the consumer? Essentially, after you have these memory embeddings learned through these asymmetric two-tower models, what you have actually done is expressed your consumer and the items in the same space with this natural language understanding of how consumers connect to the items.
Therefore, these embeddings can be then used in downstream use cases like retrieval, which is where we get into the generative retrieval side of things, and then multitask ranking. You can add them as features in our ranking models. We are going one step beyond this. These memory blocks are actually connected, not as like a flat record, but they are connected via a graph situation. Because the consumer can be described by how big their household is. They prefer some cuisine. They prefer certain taxonomies. Those taxonomies are described by some keywords and brands. It's a complex graph structure. Because these are natural language snippets, it's very easy to have another system go and build this graph for you. That's exactly what we did. I won't go into all the technical details of what we did. If you know GraphSAGE, it's one way of actually learning a deep learning-based graph model.
We did that, and we added that to our existing recommendation stack. We are already seeing 10% lifts in our MRRs. This is actually currently an A/B test. I cannot tell you how much it led to a lift in real business metrics. Offline, we saw enough of a lift that this means that our existing recommendation systems are getting a ton of complementary information by adding these memory blocks inside them.
One thing to take away is like, we started this journey towards using these textual snippets over the consumers from the point of view of, ok, because we're building agents, agents will need these textual snippets. It's beyond that. Memory is a semantic matching primitive. It's not just for the agent's context. It's telling you things about, how do I substitute things? What does this consumer prefer, like cross items, or like, here's a pairing they like with their steak, and things like that. It can all be subsumed into these natural language snippets and then added to downstream models. Extraction and encoding needs to be decoupled because it's like different cadences in which my memory change. Then the encoding is really for using the downstream applications. Then, we found that no single encoding wins. You have to do your old school embeddings plus you have to do these memory embeddings plus these graph embeddings. Every single time we add these, we see compounding improvements in the system.
Reading the Catalog - Semantic IDs
All I talked about right now was about how to create a consumer fingerprint. Now we know our consumers very well. We have all their memory and everything stored, which you can use at any time. What about the items? Traditionally, e-commerce systems have relied on IDs, like the SKU IDs, which are completely random numbers. It's like social security numbers, only belongs to you. No one knows how to interpret them. Or, taxonomy. Taxonomy tells you like, roughly this item belongs to this class. In this case like, hot sauce is the example. There are tons of hot sauces out there. It's a very coarse way of understanding what hot sauce is, which is just a taxonomy node. What if actually we could build something that understood what type and subtype of hot sauce this is. This is where semantic IDs come in, which is a very cool idea, came out of Google and Spotify, YouTube.
They use it quite a bit recently. We are also probably one of those first people who are using it for e-commerce. If you only had the SKU ID and the taxonomy ID, you'll run into a lot of roadblocks. If something is out of stock, if my favorite hot sauce is out of stock, what do I substitute with? What's in the neighborhood? Is it the entire hot sauce taxonomy, which goes from Caribbean Scotch Bonnet sauces to Korean sauces to everything? Or is it a small space in there, which is in the neighborhood of things that I'm looking at? Cold start, if you introduce new items to the catalog, how do we know where it belongs in the neighborhood of things that has a lot of signal on it already? Obviously, random IDs and taxonomy, they don't work. Then if you wanted to actually train an LLM to speak your catalog natively, basically spit out these IDs, there is no way that this random SKU IDs will do. Because LLMs learn from words and language, they don't understand what these random IDs are.
This is how a semantic ID looks, and I'll tell you how we get there later. Semantic IDs you can think of are tuples of numbers. This could be three or four numbers. In this case, there is a bunch of hot sauces. There is this Frank's RedHot cayenne, then there is the Sriracha. You can see that the first two, they share the first two prefixes, so 237, 483 are shared. The last bit is what is differentiating between the fact that one is a Thai chili hot sauce and the other one is a cayenne. Then, 237 overall means the coarse cluster of all sauces or condiments. There you may have something like a barbecue sauce. The way it's differentiated is through the second ID. The second ID is 12 in case of the barbecue sauce. That 8 at the end might just mean that it's sweet. Then, if you look at a completely different thing like basketball, all IDs are different.
There is no relationship. We have 55 million SKUs, all those SKUs expressed in some high dimensional space, there are neighborhoods in which similar things reside. If you think of this as like an analogy of a map. Essentially, you're trying to get down to what is like a postal code. All these people live in this one postal code. That's the analogy through which we actually learn this. The way we learn this is by learning what we call an iterative residual learning. I'll actually drive it through the analogy. Let's say you have this entire world map to start with, which is where you start with. Then you zoom in and you want to actually find where I live as some unique identifier. What you will do is first cluster all of the population in the world. You will actually come up with these very coarse clusters. Maybe they're like countries.
Then, you subtract the distance of the center of that country from where I live and you are left with these residuals. You do another clustering on the residuals and then you get other smaller clusters. These clusters, let's start with country, these are maybe states. You keep doing it until you get to a neighborhood, and then you get to a street, and then you get to a house number. Postal code is a good analogy here. Instead of all these random IDs, which is like SKU IDs, which are like social security numbers, or the backend taxonomy, which is like the Dewey Decimal System which is very coarse. You actually get down to things that are like postal codes for your items.
The way you do it is through this technique called RQ-VAE. Autoencoders is a way of learning smaller squeezed embeddings of the bigger LLM generated embeddings we start with. The details are not important. My analogy already told you how we learn it. After we learn it, what happens to this hot sauce cluster? As I said, the hot sauce like node in taxonomy had thousands of items. There's no way for us to know what the cluster means. They're all hot sauces. Now after doing this SID, if you do a clustering on top, you will start seeing there reside like 327 subclusters that no taxonomist, no human would have time to go and tag and create that level of granularity for you. It's bottom up learned to how these items are expressed in text. After you have these kinds of taxonomies, now I can do things like, ok, if my favorite sauce is missing or out of stock in a store, I can go in the neighborhood of that sauce and find another one which is highly substitutable.
The applications are even beyond that. I can say like, this person likes champagne, but only brew, only the dry ones, non-sweet ones. That's where your consumer personalization gets to the next level, which is amazing. There was no way for us to learn if you only use the taxonomy nodes, what is the sub-granularity of things that I like? Now through semantic IDs, you can do that. One unique way we have used this is for query reformulation. What we did was, and staying with the hot sauce theme, someone searches for sriracha. What we can do is we can go into that space that I showed before and point out where sriracha is. In the neighborhood, I can see all the different things that exist. Also, for the cluster of sriracha, I can look at other clusters that are close to it because they are geometrically similar. Conceptually or semantically, they're similar to each other.
Now what we can do is we use an LLM to actually give names to these clusters. It comes up with things like, it belongs in this chili garlic sauce area. For every single cluster, we can start giving them names which would be good queries to show the consumer to actually take them along the journey of, you searched for sriracha, but then what if you want to also look into sambal oelek and gochujang, which are like similar concepts. Essentially, coming from this hot sauce family. This lets us create these builds on the app where after you search something, we show these things which say, people also search for these other things. Previously, people used to do this in different waves. They looked at what people really search for after one search and stuff. This is giving you yet another way to add to that signal, which is extremely rich compared to the old way of doing this, because now every single of this cluster is extremely semantically connected and rich.
Here's an example. Someone is searching for cream. If you use this SID, you can come up with other queries like this. Then you have to condition it by the type of business. In grocery, the other query suggestions would be heavy cream, full cream. Then if you go to alcohol, it's actually strawberry cream, peach liquor, like cream liquor and those kinds of things, because cream means semantically something different in a different business. SIDs, the semantic IDs are letting us do this level of customization of what the next queries could be. Query formulation, we're already seeing a huge amount of lift by doing this new way of understanding our items. Then the other thing, as I said, because through the SIDs, now I understand that Sudeep likes only brut champagne, before, our systems would only understand that I like champagne and it will recommend me all kinds of champagnes.
Now that I understand this granular level of what subcategory I like, I can add this to our existing rankers. We are seeing huge uplifts by just adding this as a feature. It's getting much better at doing personalization now that it understands the nuances of what I like. We talked about query formulations. We talked about more granular recommendations. Another thing that happens is cold start is cheaper, because if I know I like certain kinds of things and you introduce new items in the catalog, because they're in the same semantic ID cluster or neighborhood, I can actually also push those items to the same consumer. Cold start can be faster. Then retrieval can also be faster because you're not retrieving these humongous embeddings, but you're retrieving four numbers. It's a tuple of four numbers. The two other ways that SIDs enable us to do something magical is we can now train small language models to directly speak these SIDs.
This is what YouTube, Spotify, they have done, but they have trained a large language model or an open-source small language model to start speaking these IDs, these four numbers, because these numbers carry semantic meaning. You can say, if user X watched or listened to this podcast, SID1, SID2, SID3, what would be a good recommendation? The language model will directly come out with an SID4, which is the recommendation. Instead of saying, they will like this band or this artist, and then you have to do a search against your system, there's a two-hop way of finding out exactly what the LLM means, versus it directly natively speaking your catalog. The other thing we can do, going back to our memory blocks, is we can now encode them with these SIDs, because again, as I said, it understands granular nuances about what I like or dislike. I can actually take those and put it back in the memory blocks, which makes future recommendations much more refined.
Reading the Moment - Consumer Intent
Now we have understood the consumer, now we have understood the items, but in the moment when you're using the app, you also need to understand the consumer's intent. Here's a classic example of how something can go wrong. Someone types wildflower in Phoenix, Arizona. It can mean that you're looking for actual wildflower bouquets. This could be these dry wildflower bouquets that you're looking for, or it could mean a restaurant chain called Wildflower Bites. These are made-up examples, but this happens a lot. Without understanding the intent, and this same person was not in Phoenix where this restaurant chain is, let's say they were in San Francisco where there is no wildflower restaurant, they probably mean actual wildflowers. How do we get to the level of understanding this intent? Again, this is a well-studied thing in machine learning. I worked at Netflix before, we did a lot of work on this.
We did not have LLMs back then. What we used to do is create these classifiers. Basically, of all these possible intents, create a classifier based on that. The problem with these classical classifiers is that they have this winner-takes-all situation. If you have a soft max, it will try to push all the probability towards one answer. In many cases, multiple answers are valid, and we need to keep them around. Then, contextually, given it's in San Francisco or Phoenix, we can activate one or the other. Then you say, now we have LLMs, so let's just use the LLM to say what is this wildflower query or this query 450 north? LLMs have all the world knowledge, so they will solve it. The problem is, if the LLM is not grounded in reality, it will just hallucinate. In this case, 450 north seems to an LLM a very plausible name for a restaurant, because it's like an address.
Actually, the ungrounded LLM says restaurant. Really, actually, it's an alcohol store. It's actually a craft ale store. Better chew, even if I ask humans, they would say something to do with pets, because it has the chew in it. An LLM says, pet product. Actually, it's a farm, which should go to grocery. Ungrounded LLMs do not work. Classifiers do not work. What do we do about it? What we did was we actually fused the two concepts, but added a few more things, which is like grounding the LLMs in our catalog. If the catalog fails, then we ground it in web search. This is a whole new agentic way of doing query intent understanding. Someone types wildflower, we look inside our catalog first, like what matches. It will probably bring up some things. Then, if nothing comes back, we say go and do an agentic search. Then we have all the set on which an LLM can do reasoning.
Then it can say, these are the two or three intents that makes more sense. Then you can add things like, what are the popularity priors and things like that? What have people actually clicked on in the past? You can have business rules and everything in the disambiguation layer. At the end of the day, you can figure out that for Phoenix at 8 p.m., if you're searching for wildflower, it's actually a restaurant. That's the highest intent thing.
I talked about how this works. I don't want to repeat what I just said, but just to give you the examples. Better chew, if you did a query against catalog, a semantic retrieval, then it will come back with things like Better Chew Farms. It will say Chews, Ultra Chewy, and all kinds of stuff that match semantically. Then what we do is we do a fuzzy refinement. We look at, for all the approximate nearest neighbors, what matches the query the best. We keep the top few. In some cases, like in this example, 450 north, there was nothing in our catalog that was matching. It went out and did a web search. It figured out that 450 north is actually a hazy IPA craft brew from a brewery. Therefore, this query should go to alcohol. Essentially, we do the catalog grounding. We do an external grounding. We take the union of the two.
Then we let an LLM reason over it. Then basically figure out what the actual intent is. One kickback we got from this is we built this system, which is agentic grounded search, to solve this problem. Now we have actually platformized it at DoorDash, and it is solving any other problem that requires grounded search. It's also saving us a lot of money because you have the choice of what search to use now. You don't actually have to be beholden to Gemini or something to do these searches. These kinds of things that we are building are becoming platforms, which is great to see. At the end of this whole thing, you can actually have these multiple intents. Then, as I said, you can activate them given the context. That was a huge win for us. Like on the long tail of queries, we see a 13% improvement, which, if you haven't worked in the problem space, you don't know. It's extremely hard to get that kind of an uplift on the accuracy of understanding the real intent on the long tail queries, which typically do not have a lot of engagement. Again, there's a lot of numbers here. The main thing to take away is like, on the long tail queries, we improved accuracy by 13% points.
Refining Relevance - Modernizing Ranking and Retrieval
We have all these things now. We have understood the intent. We have the consumer. We have the item. The last bit is like, ok, so now I can recommend a lot of things to you. How do I know that the thing that I'm recommending is relevant to the query or the consumer? There we broke down the crux of the problem into these things. Essentially, what happens if you search for organic almond milk without any understanding of relevance, because people just click on stuff. If the system right now is geared towards popularity, it might show dairy milk at the top because milk matches. I'm not saying our system does it, but this is a made-up example. That's a very irrelevant thing that's getting kicked up to the top just because it's very popular. Most people buy dairy milk when they search for milk. Then it might also bring in lexically relevant things like almond bites because you're searching for almond milk.
Then it's also off intent. It's not relevant. The only thing that's relevant is the last entry there, which is organic almond milk, which is highly relevant to the query. This messes up all our ranking systems. Essentially, because people, sometimes they are searching for bananas. If you show them apples, and they had the intent of also buying apples, they will go and click apples. Then the system will learn that, ok, for bananas, apples are relevant. It happens. What we did was we basically said, we are going to label everything into three classes. Exact relevant, moderate relevant. Another plant-based milk like oat milk is fine if you're searching for organic almond milk. What is irrelevant is this dairy milk. We need to push down that top result that I showed, to the bottom, actually. The problem we immediately face is like, how do we create these labels?
There are millions of queries that's happening, and there are billions of items. Then we need to label them for our machine learning system to learn. We need to create a big enough training set. Option A is human annotation. You need to pay a lot of humans and you need a lot of time to create 100 million query item pairs and label them. Behavioral signals, as I said, people click on random stuff if you show them. They're not very reliable. A lot of highly clicked things are not highly relevant.
What we did was we actually applied LLMs to this problem in a way which is like a tiered LLM application. Essentially, we start with 600,000 pairs that we paid humans to annotate. Humans made a lot of mistakes, actually. Then we used a very expensive big LLM to go and audit that and reevaluate. We created this very clean three-tier, relevant, medium relevant, not relevant labels. Then we had a lot of conflicts where we use this model that we have, which is called query-to-taxonomy. In some cases, different taxonomy items come and pollute your results. We actually use that to clean it up. The final, the big thing here is like we could then take that as a teacher and train a student model that is much cheaper and open source. In this case, we use Gemini 4.0, but we also are training Qwen models to do this problem now.
We could scale up our label pairs to 100 million plus at this extremely high accuracy, where basically within one accuracy is 98%. Three-class accuracy is 89%, which would be impossible to do a few years ago without LLMs, just with humans. It would be extremely expensive to create this label set. Once you create this label set, you can do all kinds of stuff. We can now train ahead in our deep learning model that just predicts these relevance labels. This tells the model that, ok, someone searched for almond milk. Don't show this very popular dairy milk at the top because it's irrelevant. The relevance head, you can tune it up and down. You can crank it up and down to say how much relevance you want versus how much engagement you want in your search results. By tuning this, we got a lot of business wins. The takeaway here is that you can do these rankings with engagement only.
What you suffer from is that semantic relevance is very low. If you did with LLM re-rankers, why don't you just go and say, I'm going to take all search results and re-rank them by LLMs. You're going to run out of money because it's extremely expensive to do this all the time with LLMs. We struck this balance by using LLMs to generate the labels offline, and then training a model that can learn from this. What happened is like relevance went up very sharply. We did an A/B test with this, and as I said, conversion rate went up by 1%, which at our scale is huge. GOV went up by 0.5%. This is real ML actually pushing business numbers, which is great to see all the time.
Then I'll go over this because it's the same concept. Once you have this three-tiered relevance understanding, we can also change how we retrieve items for search. There, essentially what we did was, as the search system is retrieving items given the query almond milk. We can actually in the retrieval state also try to separate the three classes and say, I don't want to retrieve all the irrelevant stuff. You can read this paper, which is going to come out. What we did was we used SupCon, supervised contrastive loss, and then we used this other kind of loss called the circle loss. The plain language takeaway here is like, once you have created that 100 million pairs of labeled sets, you can influence both the ranking layer and the retrieval layer to start trying to separate these things very cleanly. These kinds of losses are actually geared towards clean separation of the three levels.
Essentially, we do this thing, and the paper is called Mine and Refine, so we mine this medium and not relevant labels. Then in the second stage, we refine them. Basically, whatever is polluting my result set, I use the second stage to refine it and then train a retrieval model. By doing this, we actually got another win. This is a visual example of how this separation happens. Just to put it in perspective that you could do these things, we thought you could do this with very large LLMs. LLMs, we tried Qwen3, we tried multilingual-e5-large. Out of the box, they helped, but not as much. You can see the NDCG, which is the offline metric that we use to understand the goodness of the retrieval, it went up by 5%-ish by directly using the very large embeddings. Our method, which is using these two kinds of losses to separate out the relevance labels, ended up giving us 17%, which is very extreme. Then, you see that this is another test that we ran, and just changing the retrieval led to another 1.2% conversion rate increase. Together, just by creating these 100 million query label pairs, we were able to lift conversion by 2.2% by influencing both the retrieval and the ranking layers, which is amazing to think about.
Bringing it Together - Steerable Recommendations
We have gone through this now. We have talked about memory blocks. We have talked about how to use the memory blocks to encode behavior. We talked about semantic IDs, how we understand the items. We talked about consumer intent. We talked about improved ranking and retrieval systems. I want to talk about how all this comes together now. How all this comes together is we can now start thinking about a foundation model that we can build for DoorDash that uses all these primitives. Essentially, the consumer SIDs, you can think of also putting the consumers in semantic ID space, and then have these things, and then have a foundation model learn from the sequence of behaviors that we see from consumers, how to predict the next thing that they'll do. This can be used in our homepage, in our complementary recommendations on carts, in search, store page, and things. This is the direction we're going, which is going towards encompassing all these new techniques into a toolbox that lets us build our own internal foundation model.
Acting on it All - Building the Infra to Evaluate and Improve Agents
Last but not the least, I want to talk about how these things also come together in another way of shopping, which is agentic assistants. Here is an example. This is a work in progress prototype. It came out of a hackathon. Basically, you see we're asking the chatbot, can you help me order groceries for the week? Please add coffee beans, milk, for protein, get me chicken breast, two steaks. It's a fairly complex instruction that we're giving. First, the agent goes and says, ok, because I have the memory blocks for you, let me see where you purchase your grocery from. It figured out that for me, it's like Metropolitan or whatever for this consumer. Then it also knows when this person says chicken breast, they mean this brand or this certain kind. These chats can go multi-turn. What we're seeing, internal employee, when they're using it, typically it's 10 to 20 turns.
They will go and they'll clarify, and then they will say, ok, now also add some fruits, and then the agent has to do that mission again, and things like that. Sometimes they also put constraints like, can I have these under $25? Then the agent has to go and like, again, reason over what it's doing. The quality of these agentic systems is the whole trajectory. You need to say, all these turns that happened, the whole trace, is it a good one? It's not about if one turn inside that in isolation was good or bad. Just to connect this left-hand side with all we talked about today. When I said, do my grocery run, it understood the intent. That's the intent understanding. It's like, ok, now I have to call the grocery agent, not the restaurant agent. Memory, it knows which store I shop from. It's doing the search against that store.
When I said, add coffee beans, it understands that this person likes decaf, not caffeinated beans, just from the fact that it has that in the memory. Semantic IDs helped because their favorite coffee beans were out of stock. It found in-stock substitute. Then, again, like memory and search comes in, all this improved search that we talked about, it comes in because it can surface very clean, relevant candidates over which the LLM can reason and pick the right thing for you. Everything that I talked about comes together in this system in a very nice way. This is the architecture. It has a few things under MCP. It has all the agentic tools exposed to the agent via MCP. There is also the memory thing that I talked about, the session memory and the agentic memory, they all work harmoniously together to make this thing work.
Lastly, evaluation problems. Essentially, as I said, these shopping journeys in an agentic system are multi-turn long trajectories, and many things happen between the consumer and the agent, as we're doing these 15 turns. Quality is about the whole trajectory. It's not about a single turn. Then, as you're changing the system. If I say I want to change the system from, I want to change how the tool calls are happening, we need to be able to do some A/B testing between the new setup and the old setup. These are the two main problems we face that we're solving. The entire input is the trajectory. We have the tools. We have the prompt. We have the environment. We have the scenario. Every time you're doing something in this agent, you are changing the environment. I tell it midway that, no, I want under $25, and it has to change the whole environment and find the new things.
What we came up with is this multi-dimensional rubric. The rubric is based on communication, is the agent's communication helpful? Shopping execution, was it successful in satisfying your needs of your shopping journey. Then, personalization, which is like, ok, when I asked for coffee beans, did it understand that it's only decaf. Essentially, these three dimensions set up the rubric. Once you have the rubric, then you have to actually turn that into an evaluation criterion. Then, what we did was, instead of going and piecemeal looking at these trajectories, we take the entire transcript, which has everything from what the consumer said, what the system said, what tool was called, just take the whole log, for example, feed it through a system that can break it down into different views for another AI grader to look at, say, this was good, this was bad. One thing to take away, for when you're building these systems, binary is the best way.
You cannot do graded stuff with LLMs very well. Good or bad, that's the best way to build these systems. What we did at the end was have a coding agent look over all these raw transcripts and do optimization of the system on its own. It goes and runs GEPA style loops. It goes and fixes tool calls and everything. This is the new emergence of what we are calling agentic harness engineering. You use a coding agent as your harness, in some sense, and it goes and fixes everything that's broken in the agent. We are getting really good outcomes by doing this. The main thing to take away is that we created user simulators, we created transcript builders, and we created rubric graders, the three LLMs that sit inside a box. Then the system that sits outside is the coding agent that looks at all these transcripts that are being generated and goes back and improves those.
See more presentations with transcripts
