Transcript
Aditya Mulik: My topic today is, Batch Intelligence at Scale, which is, Cost-Efficient Multi-Agent LLM Workflows with Built-In Resilience. My name is Aditya Mulik. I work as a senior software engineer at Walmart Global Tech.
The problem I want to walk you through today is one which all retailers are wrestling with right now, which is inventory accuracy. What I want to say is the retail at massive scale, so Walmart has around 4,700 stores within the U.S. They have multiple stores in Canada, Mexico, and so on as well, but the major chunk is in the U.S. itself. There are hundreds of thousands of SKUs. There are varying departments, like varying products, any product that we can imagine of, those are all available at Walmart. The physical space is also quite large and so many items have to be tracked very carefully. The problem that each retailer is dealing with, as I mentioned about inventory accuracy, is where we have to determine what is the right inventory required for a particular occasion. If it's Thanksgiving, we have to make sure that much inventory is available at a particular point in time.
What I want to mention over here is, can we predict what inventory a department needs? I'm particularly looking for a particular department here. Can we predict what inventory a department needs for the next n number of days accurately, automatically, and at scale? Because the scale is quite large, and if you are targeting a particular department, it has to have a particular set of items available. If you are walking towards a shelf and if that shelf doesn't have a particular item available, it's not a good experience. We have to ensure that each item has to be available and in stock.
The Accuracy Imperative
The accuracy imperative. Accuracy isn't a nice-to-have, it's very mission critical, on any particular item. What we worked on was fresh food, or basically the items, you can take an example like rotisserie chicken or fresh beef, these items can expire very quickly. The store manager has to decide very accurately like, what items are there in stock and what items need to be ordered, to make sure that the inventory is very accurate for the next n number of days. The line between a stock shelf and an empty rack is what we are looking over here right now. If there are stockouts or overstock, then it could be a problem. If it's an overstock, obviously the items are going to go to waste. Either it has to be donated or it has to be thrown away. Consider this compounded across millions of square feet of retail. We have 4,700 stores, so you can imagine how critical this could be for a particular store. The next thing I'll say here is, how do we keep inventory accurate at this scale? Our first answer has to be an obvious one, we have to forecast the demand based on the history.
Backward-Looking Machine Learning Model
Now the thing is like, in Walmart, we already have machine learning models which can actually predict the amount of inventory that has to be ordered. We have backward-looking machine learning models which are trained on historical sales data to forecast future demand. We use like a year's or maybe two years of sales data, where we are trying to determine, ok, this much amount of inventory has to be ordered for the next maybe 30 days or maybe next 60 days, and so on. It's reliable for steady, recurring, and seasonal trends. Walmart is very focused on Thanksgiving. We know during Thanksgiving we have to order n number of items. That is very accurate for prediction. It's fully automated and it scales across SKUs and stores. The problem right now is sales forecasting accuracy. The pro over here is like, based on the historical trends, it can get it properly.
What if there are blind unprecedented events. Consider an example where there's a storm coming. Now the machine learning model won't be able to accurately determine if there's a storm, and if we need water or certain other items. Will that be actually available well in advance? That machine learning model might not be able to predict that. These are the things that the store manager has to deal with. They can't just rely on this particular backward trend. The question became, how do we add real-time, real-world context on top of the signals we already trust. The thing is, since we already have the machine learning models and we already know the prediction can be done in a specific way, this was an engineering challenge that came on to us, where we had to determine how should we just not rely on backward trends. Like, this is something that's going to happen, and then this number of stock has to be ordered. What if we can add some real-time context to it?
Agents, Not Monoliths: The Sequential Workflow
That's where we came up with a multi-agent — not monolith — basically a sequential workflow of agents where this agentic flow can actually determine, like get the real-time data along with some forecasting model data as well, and get the exact accurate decision that has to be given to the store manager, where these are the number of items that they need to order. How did we do that? Basically, in the sequential agentic workflow, we added multi-agent orchestration, like one agent after another, where first agent would determine, so in this case, it gathers context from various data sources for a particular department. Consider if it's a department, or like as I said about the fresh food, it's getting all the metadata from various sources like BigQuery and so on. That context is then fed to the agent, which outputs more accurate data. If it's like rotisserie chicken, it has to be ordered for the quantity that needs to be determined.
Then the second agent consumes that data, and then it synthesizes past sales data prepared by the data engineering pipeline. We have another pipeline which is set to get more context. What it does is, there's an amalgamation of two datasets which is collated, and then it is fed to another third agent data. Now this agent is quite important, which analyzes the context and output from the previous agent, and there is a grounding mechanism over here now. Consider, as I said, just a backward ML trend is not going to get us the exact prediction. There might be a weather agent, or there might be some weather API call. There could be a trending event that's happening, a sales event is happening over there. It can gather that context and it can make more accurate predictions, now that it has to order more or either less of the inventory. Then, finally, the fourth agent which produces the final natural language recommendation directive, which is like, increase the stock by this percent, and so on. That's the entire sequential workflow that we created. This was particularly targeted for a specific department.
The Output and the Human-in-the-Loop
Going to the next part, so the output, and this is the human-in-the-loop. What we were trying to do over here was, all this context generation and everything was fine, but our only goal was the store manager has to get a clear directive. The clear directive was not a dashboard, it was just a clear command. Increase apple stock 15% in produce ahead of the weekend. They are having a clear context on what they need to do. It's not just that, they are just going to increase the stock right away. There's a real-time data integration as I mentioned, which is like, MCP lets agents pull live data sources during reasoning. All of that is verified using live tools, MCPs, and so on, which is grounded properly. The critical checkpoint is, the store manager reviews each recommendation, and the store manager can either accept it or override it, because they will manually go and check and verify if there is actually a shortage of apples, if they do want to order it. There was a human in the loop in this entire pattern. This is a feature not a fallback. Human oversight in the final validation layer was catching the local context that the data cannot. That was the additional validation that the store manager was doing.
Production Rollout, and Subsequent Failures
Now this is where the important aspect comes. Then we went to production. The pilot was magic, because we tested it on maybe 10 stores or 15 stores, and then we had a clear output, coherent recommendations, and the stakeholders were quite happy about it. Then we flipped the switch to full-scale nightly runs. This was not like a real-time context where the store manager is communicating with the app, it's just a recommendation that the store manager gets during the start of the day. As Walmart stores open in the early morning, they get this recommendation and then they act upon it. Yes, we did this. We went to full chain, like all 4,700-plus stores that we rolled it out to. Then, what works at the scale of one rarely survives the scale of thousands. Here are the four failures that taught us why. Because we saw that, ok, we went to production, we made it like full chain available.
Initially for a 10-store pilot, recommendations were great, we got good feedback. There were a lot of engineering teams, data science team getting involved, and they had everything analyzed. The main thing was, me and my team's task was to get it to production, and then we started observing a lot of failures. There were four main failures which taught us what exactly happened.
I'm moving on towards the failures now. These are the common failures that we have always seen in the LLM world, and this is how we also tackled it. The very first one was hallucinations and schema violations. Here as you can see, hallucinations and schema violations were like, we had to enforce structured outputs, because the agentic workflow was getting the context, it was getting all the data, the output as well. The output was vague at times, because it was not always validated how it has to be given to the end user. Here you can see like we had to add some schema layer, which is basically to validate outputs via some JSON format of identical validation. I can give a bad example output over here. This was one of the recommendations that went to one of the store managers. As you can see over here, I think you should probably increase some stock for the weekend, maybe around 15% or so, for produce items that sell well.
It's like a coherent recommendation. It doesn't make sense, because it says that you have to increase the stock, but what stock? We don't know. Is it apples? Is it rotisserie chicken? What is it? We don't know. It's just saying that around 15%, and it should be about something that's in the produce section. Now the store manager has to go and verify which is the item that's low on stock. This didn't make sense, and it still gave a pretty high confidence. The thing is it did validate across the schema but it didn't make sense from a good output perspective. What we did is we fixed it with structured output, but also making sure that the item is also validated across a specific SKU, whether it's like which department it belongs to. If it's a produce, the item name has to be there, which is the Honeycrisp Apple right over there, and the SKU.
This confidence level made sure like this is adhering to the department that it has to be. This can be extended as possible. The stock directive, the schema which is given, and this is all validated across a common schema. As I mentioned, here's the same output. Now schema enforced, every field typed and validated: the exact SKU, item, department, and adjustment. No ambiguity left. This was the first failure.
Then, again, now the second failure. This is something that we encountered once we were full chain as well, where it was semantically wrong. What do we mean by that? You're ensuring data integrity with semantic validation. Even though we did have structured output there was a problem. The problem was the data engineering, because the workflow that was created was an overnight job, where we gathered data from various sources. Walmart has so much data for each produce, or each department. Once we pull the data, there were certain things that were getting mapped properly but certain things were not getting mapped accurately. This was a one-off situation maybe out of hundreds of items that we found out there was an issue. I can give an example over here. As you see here, everything is matching to the schema, but as we can see I have highlighted over here.
At Walmart there are multiple departments. The SKUs are unique as well, but the problem is it can map one of the items which has the same item number, but the department is different. The other department might have the same item number. The only varying part is the department over here. If there's a product that's coming as Honeycrisp Apple, but it is coming from auto_parts, so there should be some recommendation for auto_parts which was showing to the end user, but for an auto_part it was showing as a Honeycrisp Apple, which doesn't make sense. This was a failure at the data ingestion layer, where basically, wherever the data was being fetched from, it was not correctly being mapped. We had to also validate if the SKU is correct and also the item name is also coming from the same SKU. This was one of the failures. It was a dangerous kind of failure, perfectly valid JSON, but it maps Honeycrisp Apple to auto_parts because the SKUs collided.
The schema can't catch this, only schema validation can. How did we ensure to fix this? This is the part where we did the grounding mechanism. As you can see, we introduced an MCP tool call over here, basically validate coherence, where if you are getting a stock directive, we have an MCP client which is connecting to the proper data source. From that proper data source, we are looking up the proper catalog that belongs to that item. We made sure that any item that is not relevant to that particular data pipeline, it should not be processed as well. That's where we ensured, ok, this is something that every output coming from an agent has to be validated with some grounding mechanism. This was another failure that we encountered.
Then there was one more. This is the place where it's very common, where outputs were vanished. What do I mean by that? Over here you can see unreliable inference, which is failure disrupts perfect inferences, and then there is resilient inference. What I mean by that is there could be network issues, like network partition can happen as we know. Here you can see we had some logs enabled. Because in the pipeline, the way it was, there are multiple agents doing multiple tasks, and then their task was to just save the output to some persistent storage. What if there is a db_write failure, like here you can see clearly there is a timeout error which is the connection reset that has happened. Because of the connection reset the output has just vanished now. We didn't have a mechanism initially where if the output is vanished, maybe there are like 4700 stores, maybe some of the stores were not even saving the output and the store manager was just opening the app, and seeing like, ok, there's no recommendation today, that means everything is proper.
That was a failure that was observed. How did we fix that as well? This is where the mechanism of the agentic workflow agent comes over here. We had a sequential agent, and we then determined to let's switch it to a loop agent. As you can see, the main thing that I'm highlighting over here is the max iterations. These are the bounded retries that we have added. If there are any sort of failures happening, the agentic workflow will ensure to retry. Right now, you can see there are multiple sub-agents over here, like data gathering agent, synthesis agent, grounding agent. We add an iteration for the entire loop. This might be not that cost efficient, but maybe we can add this max iteration to a particular agent as well where the main work is handled. In this case, I'm showing the entire agent can make sure the workflow is computed properly, so that the output is saved and you can increase the number of iterations as much as you want. Like here, I've just given as an example of five iterations.
Next one, so we were flying blind. Observability is a must. This is a very important thing. Since we had full chain available, the very common thing that we want to ensure is we should have observability in the entire workflow. What do I mean by that? The regular way of doing the engineering is you can have maybe 5xx errors or 4xx errors. Even here it's the same case, like if we saw like there's a schema violation, or if there are retries that didn't happen, or something that went wrong, we were adding structured logging. Structured logging was ensured. Over here you can see, as an example, is how many times the retries happened, like two attempts. Or if there was a schema violation, how much latency it had. All of this was enabled just to make sure that we have clear visibility into what's happening in our entire workflow.
As you can see over here, we had proper logging if the LLM response doesn't match the schema, if there is a tool response that failed, so it was retrying again. We were trying to reach a success rate of 99%, so this is some example that I added, like if there's a 97% error rate, 2.3% of failure rate, and so on. We made sure that observability is also in place.
The Cost Factor
That being said, these were the four main failures. Now I want to get to the very crisp part of what we were building over here. With failures behind us, the next question on the table was cost. This was the biggest question that was running on to us, because we wanted to get this into production. This was a very critical thing that we were trying to get in, but we were leveraging all foundational models, and with a high scale, like 4700 stores, the amount of tokens generated is quite high. How much of this whole solution is going to cost to run, is what was the next question. The thing over here is the PTU problem. When the solution went to production, we had humongous amount of input and output tokens being used. Since it was a foundational model, we decided to secure 400 PTUs for that amount of scale that we need to run.
For 400 PTUs, we were almost being charged around 100k to 120k per month, and the solution was not that, because we are trying to also make sure that the inventory is accurate, also save cost. There should not be wastage happening in a particular department. Running at this position, it doesn't make sense, if these much PTUs are going to be used. PTUs is like reserved capacity, and we have to reserve it for the entire day. Any of these vendors who are providing the PTUs, we have to take it for the entire 24 hours, and it's a monthly or yearly subscription that we have to take in. Now the problem is, this is a batch intelligence application, it's only going to run overnight. It's not going to run the entire day. We are not serving any live traffic. For us, we were being charged 400 PTUs for 24 hours, and PTUs reserve capacity for the entire day.
Our workload was nightly batch run, which was not 24 by 7. We were paying for the round the clock capacity which was being used. This was our most critical problem that we were trying to solve in a particular defined set of time, which didn't make sense because of the cost involved. We started exploring options, what are the options? Because this is a foundational model, the best part about foundational model is they are well trained, they are good for such enterprise scale applications. The next thing that came to us was, what AI model should I choose? Here as you can see, we had cheaper models where we can go with open-source models or any other models, which might introduce some latency, or they could be cheaper compared to the one which we already had, whether it's a GPT-4.1 or Gemini, and so on. Or we can just reduce the PTUs, that was another option. Or we could go with pay-as-you-go. We were debating between quality versus cost, and both were not a good option, because we needed high quality but lower cost. That was the main problem that we had to solve.
How did we do that? The decision that we made was, we went with pay-as-you-go. Now just to go with some background, the problem with PTU versus pay-as-you-go, is pay-as-you-go will come with latency issues, as I've mentioned over here, and there will be rate limiting happening. We did observe a lot of rate limit issues, but now the thing is, as I mentioned in my failures as well, we had added a retry logic. This retry logic enabled us, even if there is a rate limit, if there are any 4xx or 5xx errors, I'm just going to retry again. If I do retry again, the output will be generated again. Now, if it's a pay-as-you-go, it's still a foundational model and it still gives very accurate responses, and there was no reserved capacity as well over here. We were not paying round the clock, we were just using as much compute as we require.
The payoff, so what happened over here is, we were supposed to pay around 100k per month, now we are paying just 10k as the final usage, because even with the retries, the cost is just very low. It's just going to happen in that specific 30 minutes, or maybe the time might increase. We don't mind because the stores are already closed, this is going to be a nightly job. Then, eventually, whatever compute is being taken is going to just happen at night, even if there are retries and everything, the cost will be minimal, output will be still matching the quality that we have set, because there is schema enforcement, and all the data engineering pipelines were fixed. This was our biggest win. As I mentioned, same tokens, same model, just a pricing tier switch from PTU to pay-as-you-go, and for a drastic drop in cost. That was the big win that I wanted to mention over here.
Learnings, When Building at Scale
Every failure and every fix left us with a lesson worth carrying forward. Here's what we learned, building at scale. These are some of the learnings that I'm going to mention once again. Basically, schema enforcement. Since we are retrying, everything is all planned out, but schema enforcement is not just the one thing that happens over here, basic JSON validation. There has to be semantic validation, because data, what is coming from various sources, like in the AI domain, what I've learned very well is context has to be very accurate. With the data pipelines that we have, whatever context we get, if there is even a slight mismatch, it can just return vague output. We have to make sure semantic validation is in place. Data usability, making data easy to understand and use. We need to have some observability as well. Whatever the data that is coming in, we need to see how the pipelines are running.
All those things have to be taken into consideration. Then comes the data integrity. Maintaining accuracy and consistency of data is very important, is what I've learned from this entire perspective. That's one of the key features. Second thing is data engineering limits AI quality. As I mentioned over here, data quality over here is reliable AI results, accurate and trustworthy outcomes, data rectification, fixing upstream data issues, data engineering, ensuring data quality, and flawed data. What do we mean by all of this? The entire context over here, since it was a nightly job that was being run, this had a huge emphasis on the data engineering aspects itself. Because we were getting data from various sources, there was a lot of focus that was added into this perspective, where data coming from various sources has to be verified. The amount of scripts that we created were validated with the business and the data team, with the engineering team.
We ensured that once the data is very accurate, then that context will be fed properly to the agents with a proper particular prompt, and that will generate the exact output that we are looking for. Because it's a multi-agent workflow, we have to ensure that the data that is coming from various sources has to be very accurate. That was our main goal that we learned from this entire workflow. Also, like flawed data, as I mentioned, unreliable input is also like, once the input is not proper, all the agents will react in a weird output, because it might either not match to the schema, or it might just match to the schema also, but just generate vague outputs, and that's how it would return. Building retry logic, this was one of the most important aspects that I went through. We have been very much emphasizing on this.
Retries can be a very bad loop as well, where if you don't plan it properly, it can just keep retrying, and that would just increase a lot of costs. Just because there are tokens involved over here, you have to plan it very accurately. As I mentioned in the initial flow, we used to retry it at the base, like root level, where all the agents are, like where the workflow runs. Ideally, the agentic retry should be in such a pattern where it is actually needed, whether we are getting the context from the proper data source, so that the further agents don't have to hallucinate, or there could be something that can go wrong. Retry over here, implemented from the outset, reduced unexpected failures. There's proactive system enhancement, cost saving and efficiency. It results into better user satisfaction.
The next learning perspective is observability. This is one of the key things from the entire workflow, that everyone is focused on. Traceability is important, structured logging is important, and enhanced system understanding. Because once you have tracing enabled, for us, initially, when we were having this entire workflow running, we were not able to determine, ok, what's going wrong, just because there are no proper traces set up. If you're not able to identify if the LLM is hallucinating, or if the model is just getting rate limited, all that has to be clearly available. There are tools which help us add those visibility, and then having that in place, make sure that your application is very robust, and can survive any failures to make it very resilient. That's a very important aspect that I'll mention over here. The next thing is matching infrastructure to requirements. This is the key thing that we realized, which is the infrastructure should be used for a nightly batch job.
Here, we can see that the real-time infrastructure, like if you have a real-time infrastructure where users are having conversations with the LLM, in that case, you can have an accurate set of PTUs set up, and so on. In cost effective infrastructures, you need to ensure that you don't need to actually have PTUs only allocated. Here, in our case, we were opting either for like pay-as-you-go, or you can even go with open-source models. As long as it's giving you the proper performance, the correct output, that's the ideal approach that you should be going with. The next one is, I've reached the end, which is, tomorrow at 6 a.m., a store manager will confidently adjust the inventory. They don't know about our schema validation, our loop retries, our nightly API optimizations. They don't have to as well. It's just the ultimate metric, which is the resilient engineering. That's what the perspective is.
Summary
Just to recap over here, schema is our table stakes, semantics is the real work. Focus on validation checks that go beyond basic JSON structure. Data engineering limits AI quality, so garbage in results into garbage out as well. Having retry logic is very important. If there are network timeouts, hallucinations, retry can ensure that your output is always accurate. Observability is mandatory. Tracing, logging is very important. Then match infrastructure to the regular requirements. If it's a nightly non-latency sensitive batch job, then you have to ensure that you don't need to actually rely on PTUs, because that would just cost you even more than what's expected. That's the main goal.
Questions and Answers
Participant 1: How did you assess the benefits or effectiveness of the new model compared to your legacy time series forecasting models? You said before you built this agentic flow, you had some forecasting models, the classical ML time series models. How did you assess whether this new agentic model was better?
Aditya Mulik: The thing is, basically, we had like time series forecast model, what we were using in this use case, because we had a lot of sales data. We were looking at the past two years of sales data for the forecasting model. Now that the forecasting model can just output in an accurate pattern for a fixed set of events, if it's a Thanksgiving event, the time series forecasting model will accurately determine what is needed. What if there is a situation where there's a flash sale happening somewhere, or there's a competitor having a flash sale, or consider if there is a certain situation, like a storm is coming. Then now the forecasting model won't be able to determine that. This is where we wanted to introduce agentic workflows. The entire goal was we have to create an output, but we wanted some real-time data mechanism as well. That's where we decided to go with the agentic workflow. That's how we decided to get the output here.
Participant 2: Did you keep the log only in testing, in the trial phase, or what about in production? Do you still keep all the logs in production?
Aditya Mulik: For production, you mentioned?
Participant 2: Did you keep all the logs in production?
Aditya Mulik: How did we determine?
Participant 2: We understand you write down all the logs, you check all the logs, during the testing it will help you to analyze what's wrong. Did you keep all the logs in the production phase?
Aditya Mulik: How did we ensure logging is maintained in production? There are multiple tools that were being used over here. Initially, we opted with cloud vendors themself, where they have their own logging tools and mechanism. We had our own custom logs added to every agentic workflow. Each agent that had custom functions, like as I mentioned in one of the MCP tools, where it was validating across one of the SKUs. In that SKU, we were trying to determine, like we were adding some logs, if there is a failure happening, or if there is any db_writes that's happening or db_reads happening. If there are any failures, we added custom logs, we had our own dashboards. Initially, we went with the cloud vendor dashboards, but we had our own custom dashboards to verify that.
Participant 3: At this conference, we had multiple sessions about evaluating the agents. I think for your case, it's especially interesting to hear, given it's deployed in production on many users, if you do changes to your agent, how do you evaluate it? How do you do regression testing and evaluations?
Aditya Mulik: This is the thing that I didn't include in my presentation as well, because this was our initial workflow where we went with an MVP. Because initially, our process was to come with an MVP, how should we determine the inventory accuracy. The things that we didn't include were evals. We did work upon in an iterative fashion where we had a certain perspective, like if there is a particular department, in this case, as I mentioned about rotisserie chicken, the shelf life is about 24 hours to 48 hours, is what I know about. There should be a certain eval set for particular departments as well. What we did initially was to go with a flow where we are getting real-time data. Once we have that data coming in, then we were able to iteratively add evals and a grounding mechanism, and even governance. Because at times, they were like, we had to make sure there are pricing information and everything that should not be coming in. All that was considered in an evaluation set. That's how we had approached it. That was not one of the perspectives that I did include, because that's something that was iteratively performed.
Participant 4: You mentioned that the store manager needs to act on the recommendation, either agree or override. Is there a possibility that the store manager does not take any action? Does the recommendation become stale because the next nightly run might have different parameters, and that might be a different recommendation.
Aditya Mulik: The thing is, the store managers didn't act upon it right away. The way it was done was initially when we were in the pilot phase, this was all being validated very thoroughly. The store manager has the ability to first go because it's kind of like their day-to-day process, they have their own applications where they're verifying the inventory that's coming in. This was just as a recommendation that was initially being shared to their store manager. They were manually validating it. As this went to full production, and over the iterative process of weeks and weeks, they were able to determine that the predictions were becoming more and more accurate, because that was actually the context that they were getting. The only thing was there was always a human in the loop perspective over here. Because even if the recommendation has come in, they won't just take the action right away, they will verify the inventory and then only they'll take a call. That's how it was done. It's not that they always acted upon it.
Participant 5: You mentioned the cost aspect. How did business decide in the end, is it worthwhile, given the cost to roll it out globally, or is there a tradeoff to take into account?
Aditya Mulik: This was an initial POC that we started off with. The way it was working out was initially recommendations were just based on machine learning models themselves. That's how it was running from a longer time. As this agentic workflow started coming through, and then we did already have something which was planned out to have this rolled out, like to certain pilot stores. The plan was initially to see how this fits in the entire ecosystem, because we didn't want to disturb the existing flow, because as the stores are operating right now, they're well efficient. This is just an additional layer that we added over here, just to make sure like, ok, this is getting more accurate recommendations. Then it went out as a POC, POC to the entire production, like 4700 stores. Then it was foolproof as a concept that got deployed. We iteratively kept adding more and more recommendations. Our initial plan was to do it for a specific department itself. Then that's how we scaled it up.
See more presentations with transcripts
