BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Presentations Reliable Retrieval for Production AI Systems

Reliable Retrieval for Production AI Systems

45:43

Summary

Lan Chu explains how to build production-grade enterprise RAG systems beyond simple prototypes. Drawing from real-world deployments at Rabobank, she discusses replacing naive OCR with agentic multimodal parsing, encoding non-textual temporal signals to prevent outdated retrieval, utilizing bootstrap statistical evals, and establishing continuous observability to survive strict banking governance.

Bio

Lan Chu is an AI Tech Lead and Senior Data Scientist with 7+ years of experience building production data and machine learning pipelines and 3+ years in building GenAI products. She specializes in designing, implementing data and AI pipelines and responsible AI practices. She has a background in Data Science, deep expertise in Natural Language Processing.

About the conference

Software is changing the world. QCon London empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Transcript

Lan Chu: How many of you trust and rely on the AI output without even fact-checking? Do you rely on the AI output? I found this shocking statistic by Columbia Journalism Review, and they tested the performance of eight different AI search tools. Here is the output. Each square is an answer, and so red means incorrect, and green means correct. As you can see here, it is almost all red. What does this mean? It means that retrieval is hard, and even top AI companies like Google or OpenAI still didn't have it all figured out. Then, what does this mean for us professionals who need to build AI search tools? From my experience, it's going to be a bit messy before it gets better. In fact, bringing our demo to production is going to be a bumpy road. I will be sharing with you my real experience building a production search system, the challenges that I faced, and how I overcome them.

My name is Lan. I work as an AI tech lead at Rabobank, one of the largest banks in the Netherlands. I came to the Netherlands in 2019, but I'm originally from Vietnam.

Building an AI Research Assistant

Imagine you are a banker, and you are going to have that meeting with your client tomorrow. You need to find that specific information from a thousand documents. What are you going to do? You have two choices. You either quit your job, don't do it, or you ask me to build an AI search tool for you. That was what I did. I developed this AI research assistant that allows my colleagues to search through thousands of documents and to find the insight that they need in a quick manner. The system has access to more than 10,000 documents. It also has access to the external API to call for the external data sources. Currently serving 300 users. This is how it looks like in action. Basically, you can ask the questions, the AI will return you the answer, and it also gives you the citation and the source documents so you can do the check if you want.

You can interact with the system, give the feedback. You could also change some certain settings through the UI. I love this. This is my baby. I hope you like it, too. Let's move to the more serious stuff on how I actually built this. This is basically a RAG application. I have these three main components here. The first component on the left-hand side is a RAG data ingestion pipeline. We start with some documents from the source. In my case, it is SharePoint. I work mostly with PDF and PowerPoint format. We ingest those documents. We parse them, chunk them, index them. Actually, before indexing, you need to turn them into a vector, which means embedding, and then index them into a vector database. In the middle, we have the inference layer where the user can interact with the system through a UI. The user input will hit the FastAPI backend, which, again, calls the vector database in the backend to retrieve the relevant documents.

With those relevant documents, we feed into the language model to generate the answer. That was simple RAG components. What I like the most is the observability and monitoring layer that allows us as developers to track every single trace, spans, and the RAG evaluation metrics of our whole system. More about that later on. If you look at this, it is quite straightforward and simple, but things get complex quite quickly.

Top Three Challenges

These are the top three challenges that I have seen in my project. The first challenge is getting the data ready for the AI. This is not a trivial task: garbage in, garbage out. The second challenge is how to get the right context to send to the AI, or how we implement the search algorithm. The third one is about how you do your evaluations and make sure that we have a good flywheel to improve our system over time.

1. Data Ingestion/Document Processing

Let's start with the first challenge about document processing. I want to zoom in a little bit about my data ingestion pipeline here. Remember I said we have these three main steps. You parse the document, so you flatten the text, turn it from a PDF or PowerPoint format into text so that you can feed into the AI. The second step is chunking, and then you embed it, and then index into the vector database. Now, let's talk about document parsing because this is where things get really interesting. Document parsing is not sexy as building an agent running on the internet, but this is where you can lose or win with your RAG application. The reason I say that is because I was able to improve my retrieval accuracy by 16% by just improving my document parsing pipeline. Many teams are still using the traditional document parsing pipeline.

You have the raw document. You flatten the document into text using an OCR model, and with that text, you parse into the AI to generate the answer. At the surface level, this seems simple and even elegant. In reality, this is how a document looks like. It is just not text. It contains so many blocks of infographics and much cells. The moment you flatten the document, you lose the structure of the document. What I meant with that is something like this. Look at this example. I was using Docling OCR to parse the information from this infographic, and there are two issues here. The first one is that EasyOCR misread some numbers in the ground truth document. The second one, more importantly, it actually loses the context and the structure. You see, it was reading the numbers from left to right instead of top to bottom, and now the numbers stop making sense.

If you parse it into the AI, you can only hope for the best that the AI can understand what is going on here. Instead of treating your document parsing pipeline as a simple text extraction, you can actually use a more complex approach using agentic document understanding. This would be the output of such an approach. You see, the context is maintained. The structure is maintained. With this output, the AI has a much better chance of giving you the right answer. How do you implement such an agentic document understanding? The idea is that you can combine traditional OCR with vision language model reasoning. One way to orchestrate that workflow is to build an agentic framework where you have an agent that has access to different tools, and then it will call, for example, a vision language model for parsing the image and infographic and use a traditional OCR for just parsing the text.

You can build this workflow yourself from scratch, but you don't really have to because there are some hosted solutions out there already. I do the benchmarking on the performance of different document processing tools on my retrieval accuracy. As you can see here, I was able to improve my retrieval accuracy from 72% until 90%, so almost more than 16% improvement by just using the agentic document extraction approach.

With that, I want to give you some takeaways that you can bring home with you. If you are evaluating the document parsing tools, here is a small little guide based on my experience, some of the tools that you can choose. If your document is full of tables and you need high accuracy on table parsing, go for Adobe PDF service, rock-solid performance every single time. Adobe invented PDF, so it is not surprising that they have such a good solution.

That was about document parsing. Now let's move to the second step about chunking. You may say that language models these days have a really large context window, like 1 million tokens. Why do I need to do chunking anyway? Why don't I just dump everything into the AI? The answer is you can, but you don't want to. Why? Because there is this concept called context rot, and you might have heard about it in the context of context engineering. It means that the moment you increase the token being sent to the AI, it impacts the performance of the LLM. You can see here, this is a global issue. It doesn't just like apply to a specific language model. It is the same issues with every single language model family. Now you can still argue, but language models are getting better. Maybe at some point, I don't have this context rot issue anymore.

Maybe you're right. Even if the curve flattens, you still want to chunk because you want to think about the cost. As you scale up, you don't want to send like 100 batches into your AI. I'm sure that your boss wouldn't be happy looking at the cloud bills. Secondly, you also want auditable and traceable citation to your users. That is almost impossible if you are returning them PDFs of 100 pages. I hope I convinced you that you need to chunk. Now, how do you do chunking? There are so many different chunking strategies out there, from simple to more advanced ones. I think we should always start with the simple techniques, and then you only add the complexity when you have to. When you do chunking, there are two things that you need to keep in mind. Your chunk needs to fit inside the context window of the LLM that is going to generate the answer.

It also needs to fit into the context window of the embedding model, which is generating the vector embedding for your document. The thing is that the LLM context window is really large, but the context window of the embedding model is much smaller. I think the largest one is around 8,000 tokens. You have to chunk anyways if your document is more than 8,000 tokens. Again, I do the benchmark on different simple chunking techniques. You can see here I have multiple configurations of fixed-size chunkings and section chunkings, which I use structured chunkings using the natural division of your document. I see the strongest performance with section chunkings, and you see a clear margin of 93% accuracy as compared to other configurations. For my case, the structured chunking works really well, but it doesn't mean that it will always work for you. The thing about chunking is that it is very easy to implement, but very difficult to get it right. Let's say your document has different sections. Each section has a very different length, so you don't really want to chunk by sections because some sections can be really big while others are super small.

I just want to give you some reminders on how you should do chunking. First, test with your data and start with simple chunking techniques. If someone is telling you that this is the best technique for all of the use cases, then maybe you should take that with a grain of salt. This might be my hot take, but please don't chunk if you don't have to. If your document is small enough, maybe a couple thousand tokens, it is still within the good performance range of the LLM, and at that level, cost is not really your concern yet, then maybe you don't need to chunk, and save yourself a headache. If you do chunk, try with structured chunking first and see how it performs. This is just not based on my experience, but also what I have learned from other big tech companies too. Of course, when you chunk, you want to think about what is the chunk size.

Imagine, in the end of your RAG application, you want to return top 5 documents and send it to the LLM. Then you have this, for example, 4,000 tokens budget, which means you need to divide that budget into each of your chunks. That will help you to determine how many tokens you want to have in one chunk. Finally, of course, you want to think about cost as well. If you use advanced techniques like semantic chunking or LLM-based chunking, then you incur the cost there.

That was the second step about chunking. Now let's move to the final step about embedding and indexing. At this point, hopefully you parse and chunk the document right. Now you are ready to embed them into vectors and push them into a vector database. This is not necessarily a challenge for me, but I think it's quite interesting to mention. When I started in 2023, I was just using dedicated vector databases like Qdrant or Pinecone, by default. Everyone was using those by default. It was because back then, the native database like SQL or NoSQL doesn't really support the vector search or K-Nearest Neighbor search. Now everyone is catching up. All of this vector search capability is built-in in the native database as well. Now there has been a shift recently. Everyone is talking about coming back to traditional native databases because maintaining a separate component for a dedicated vector database becomes less necessary. That was all I wanted to share about document parsing. Please treat your document parsing as a foundational step for your RAG pipelines, because you will be surprised on how this will impact the performance of your RAG application.

2. Search and Retrieval

Let's go to the second challenge, which is about how you find the right context to send to the AI. This is really not a trivial task. I spent months on this. This is how we all started. We have a user query. We turn it into a vector. Then, given some vector, you do hybrid search through your vector database. You get the top five documents. You put all of this into the AI and it will generate the answer. It works until it doesn't. This is an example from a real user query. The person was asking about Mexico monetary policy for 2026. I did everything by the book. I converted into vector and I searched through my vector database. This is what I received, a very relevant but outdated document. You see, I have documents that were written in 2020, 2021. You don't want that when you ask about something in 2026.

What can we do here? A simple approach that I tried was just adding metadata filtering. You get this temporal signal from the user query. Let's say people are asking about something in 2026. Let's just filter out all of the documents that were written before 2026. That sounds elegant. This adds another problem because you need to introduce a cutoff date. It could be the case that the right documents were written one day before your cutoff date and you will miss this completely. A more elegant approach is to add the recency score with your documents. Think about it. The more recent documents, the more boosting it should receive. You can see this S-curve here. If you ask a question from today, a document that was written yesterday would receive more recency score. At some point, when the document is so old, you want to give it a penalty because you don't want it to be returned.

One way you can construct that recency score is to use a temporal vector. What it means is that at the time of indexing, you do not just convert the document into the vector, you also convert the timestamp of the document into a temporal vector using sine, cosine functions. Now you have the temporal vector as well. At the time of inference, when users were asking about the questions, you do the same thing. You convert the user query into the text embedding. For the time embedding, we would use now as a time vector. You combine these two vectors and you search through your vector database. As such, you actually combine both the similarity but also recency signals into your search algorithm. You can build something like this from scratch, but you don't have to. There's also hottest solution out there. Superlinked has a very elegant implementation of sine, cosine function to turn non-textual signal into vector. Please do check it out. After using this approach, this is what I received, much more recent and relevant document. I do not say that it is a perfect solution already, but that is already much better than how it was before. Then you can tune your curve to make sure that it is reflecting what the business wants.

Now you think that, yes, we have this time decay curve. Everything works fine. Users are happy. We are done. Not quite. The thing is, people ask about different topics and different topics need a different decay curve. Or sometimes, I don't just want to search through my vector database. I want to get some live updated data. For that, I need to call an API to another database to get the live data. Because of that, I need to introduce some agentic component to my system. I use this live router, which is really a simple version of agentic RAG. You introduce a classifier that given the user query, you classify the topic and the router will decide on which pathway and which tools it should call to answer the user query. This adds a lot of power, but at the same time, whenever we add, like we have a solution to something, we introduce another problem.

The problem that was happening in my use case was that now the LLM actually needs to also decide on the arguments, on the other parameters that it need to put into the function to call that tool. Then many times we see that it was failing here. Why? Because the user query is really vague and ambiguous, and then the AI was just blindly guessing what should be the arguments that I should be putting into my API call. That is actually an example of what happened in reality. We just came up with a very simple solution when the AI is not clear about what would be the next step, we introduce this user in the loop where it asks the user to validate and confirm, what is the right arguments I should put into my function? Very simple, but really reliable approach. Makes the product so much better.

That was all I wanted to talk about search and retrieval, and a few takeaways. We all started with hybrid search, that was a good starting point. Then real user queries contain non-textual signals too. In my case, it was time, but you can think about numbers, geographical location. You want to encode all of these non-textual signals in your search algorithm. Finally, these days, adding agentic components, yes, it's really common, adds a lot of power, but it introduces more complexities that you need to think about.

3. Evals

Let's go to the last topic, which is about evals. Many of us probably come from a technical background, but then you will still be surprised to see that many people, when building product, they will still do this. I've run it a couple of times, it looks good, love the vibe, let's deploy. I've seen it a lot, and there are so many better ways to do evaluations. I usually approach evals as a two-phase process. You have pre-deployment phase and production phase. Let's zoom in a little bit to see how they complement each other. In the pre-deployment phase, you are doing evaluation in a more controlled environment, so you want to test whether your product is good enough before it interacts with the user. In this phase, you want to think about what are the evaluation metrics and how I can build a good reference dataset to test my system.

Let's talk about evaluation metrics. There are so many evals libraries out there that offer you all of these generic evaluation metrics, accuracy, truthfulness. They are good as a starting point. What happened to my use case was that when I received the feedback from users, actually, they're really insightful, and when I look at the feedback, I can get an idea of what is going wrong. Then my evaluation metrics are not really telling the story, because I'm not checking them. In fact, you want to capture the real signals. You want to discover some of the common failure patterns that are happening in your applications, and you want to check them instead of just general evaluation metrics. For example, think about routing failure, or in my case, temporal reasoning failure that I want to check, they are common failure patterns of your specific applications. You can use like Code-based or LLM judge for these custom evaluation metrics.

Now, you have the evaluation metrics, let's think about a good reference dataset. Many teams see evaluations as a big upfront investment, because you need to build thousands of samples to test your system, and then because it is expensive, let's just not do it. Then it doesn't really have to be expensive, you can really start very small, but start with real user queries and covering the key use case that you want to check for your system. Every single time, real user queries will be like 100, 5,000 synthetic samples, that might never happen in reality. Let's just start with a small sample dataset, and then you can always iterate and expand it over time. I'm telling you that let's just start small, but how much is small? How much is actually sufficient? I'm going to share with you this really cool technique to build the intuition on how you can decide how much data you need for your system.

It is an example, I have these two chunking strategies A and B, and I want to test the performance of these two chunking strategies. You can also use other examples, like if I swap a new model, anything that you want to change in your pipeline, and I'm testing the performance of these two systems using only 20 samples in my dataset. If I look at the mean retrieval accuracy score, system B is a little bit better than A. Can you confidently say that from a statistical point of view, B is really better than A? One way you can do so is to construct the confidence interval and the underlying distributions of the accuracy score, and in this case, we have these two confidence intervals of two systems, they have this strong overlap, which is not a good sign. Then you cannot really say that B is better than A, and for now, it is also even at an 80% confidence level.

Actually, if you want to increase the confidence level to 95%, which is what most people want, then you need to double your sample size, for example. It is a really cool technique. Think about bootstrapping when you need to decide how much samples do you need for your evaluation dataset. Once you have all of these evaluation metrics and the evaluation dataset, you can experiment really quickly, like you can change models, swap the prompt, and run your evaluation pipeline until when you are happy and confident enough to launch it to production.

In the production phase, you want to keep monitoring what is happening, and you want to collect more feedback from real users, and of course, you want to build guardrails. Let's talk about observability and monitoring. You can't really improve what you can't see. With the GenAI-based applications, you really need to zoom it out, because now we are not just monitoring the model, we are monitoring the whole system, and you are making so many LLM calls, tool calls, job retrieval logic, and so many things can go wrong. You need a way to track every single step in your pipeline. If your organization has such an observability platform, great. If they don't, then there are some hosted open-source solutions out there like Opik or Langfuse that you can use. It looks like this. With an observability platform, you can track every single trace that is being sent to your system.

Basically, a trace contains every single step from the moment the user asks a question to when the system generates the output. You can see here, I have like 10 different steps in my pipeline, and I can see the input, output of each step, and the execution time. It is super useful for debugging, but also if you want to reduce latency, you know where you can cut down on the execution time. Super useful. Remember in the pre-deployment phase, we have this controlled dataset. Now when you are in production, you receive thousands of requests a day, and you get more signals from the users, and you realize, actually there are many other metrics that I haven't thought of in the pre-deployment phase. You can feed it back to your product development cycle, and you might need to develop new metrics that you want to check in production instead of all the metrics that you already developed in the pre-production phase.

Finally, you want to implement guardrails. A good product is a safe product. You want to think about hallucination issue for output. You want to implement critical business behavior or compliant behaviors that are specific for your applications. There are a few ways that you can implement guardrails. You can just start simple with a Python script using regular expressions. Another time, you can also use LLM as a classifier to block user questions, depending on your business rules. Sometimes you also want more dynamic guardrails, and that is when you can use embedding-based guardrails. For example, you have a dataset of well-known attack patterns, and then you generate the embeddings of those examples, and you compare the embedding with the embedding of the user queries, and then you can block some queries if they exceed a threshold, for example. Just a few ideas on how you can implement guardrails. Just start simple with Python script. Most of the time, it can work quite well.

Some small takeaways about evaluations. Please use a two-phase approach. You want to make sure that your product is good enough before launch. At the same time, after you launch, you want to continue caring about the user feedback. Start small with a real user's dataset, and capture monitoring from day one.

Summary

Let's just take a small summary of what I talked about today. First thing, please treat document processing as a foundational step in your RAG pipeline. For search and retrieval, you want to encode the non-textual signal, too. Finally, with eval, yes, use this two-phase approach, and monitoring from day one.

Questions and Answers

Participant 1: When you're building evaluations, and your document set changes over time, how do you keep your evaluations up to date?

Lan Chu: I update my vector database every day, so, yes, my documents change over time, but that is why you also need to do the evaluation after you launch the product. The users interact with the new documents, and then you get these new signals, because of that, maybe you can implement some sort of log filtering to look at the new signals from production that you haven't really looked at in the pre-deployment phase. I think it is very common that you will realize that there are so many different use cases that you haven't thought of. For me, it was just more like continuous integration and continuous calibrations. You start testing with the new documents, on a daily basis.

Participant 1: Do you then drop the evals post-launch and focus just on the user feedback, or do you keep the original evaluation suite around?

Lan Chu: Just to make sure I understand the question well, you mean like, if you get new feedback, do I drop the original dataset?

Participant 1: No. I mean the evals that aren't based on user feedback, useful six months after you launch, or do you drop them and start only looking at the signals that come from production?

Lan Chu: Once I am in the production phase, I don't really look at the old evaluation dataset that I have in my pre-deployment phase anymore. I do this to run experiments before launching.

Participant 2: Are there ways to connect user feedback to evaluations? Like, of course, there is thumbs up, thumbs down, but that's very little knowledge. Is it a good idea to go more sophisticated into gathering user feedback, either directly or from their interaction with the agent, actually? What are your ideas about that?

Lan Chu: For now, I don't really have such things in place. I only have the thumbs up and thumbs down. Then we look at the user feedback from our log. I think what you could also do is like, you think about the user frustration metrics. For example, if they ask multiple questions in a row, that is a signal that you need to capture that. In the agentic component, the user can also give me the decisions that they make, and so I can check that to see if my systems are actually doing well with the suggestions of what are the right metrics or the parameters that users can choose. I think those are a bit like hidden user signals that you could also check as well, but I haven't really implemented those yet.

Participant 2: You update your eval dataset from the actual production? Do you go into the actual production and find certain cases, and then you update your dataset from that?

Lan Chu: Yes.

Participant 3: We're also from the finance industry. We're building a very similar use case, like research assistant for investment research, alpha research, like all that. A lot of our use cases are stuck in POC. They are not launched to production. I do want to understand what's your biggest challenge for bringing POC to production, and then what's the learning that you can share with us?

Lan Chu: We are in the banking sector and we are compliance-heavy. The road to production is really bumpy and slippery. Actually, we have this internal AI way of working that we need to follow every single step before we can go to production. For example, you need to think about your solution design. You need to think about the risk tier of your system. In my case, my system is using published data source, so the risk level is low, and so I can move a bit faster. For the use cases where you use sensitive data or customer data, then you really need to do it in a more controlled environment, or a platform that the bank provides. Then there are so many controls from the risk side. You need to have a project start document, architecture, all of these things in place, making sure that it is well tested. There are these model validation teams who would check every single step in your process, your evaluation step, your chunking strategies, and they are really strict with all of these things until when they're happy, then, yes, you can go.

Participant 4: I was wondering for the different benchmarks that you had and different techniques, what metrics did you find most helpful when comparing the chunking strategy, things like that, which metric did you use for comparison?

Lan Chu: You mean the benchmark on the chunking strategies?

Participant 4: Yes.

Lan Chu: I just started with simple chunking strategies. In this case, I'm just using fixed-size chunking and section chunkings. Very simple to implement, and there was no cost involved. That is where I started. I think the really interesting insight that I saw from my documents was that. Let's say you want to explore the distribution of the data in your whole corpus. Let's say I look at the token size in every section in my document, and then you can see, let's say 75 percentile was just 500 or 700 tokens, which means that there was an equal length in each section in my document. I can go with section chunking, but then you could also have the case where your document has completely different length for each section. Then you might not want to do that because some sections there might exist the context window of your embedding model, and then you cannot really embed them anymore. In that case, you have to think about breaking those sections into smaller chunks, for instance. Then you need to think about how you can do that in an optimal way.

Participant 4: The accuracy was against the evaluation dataset with the input queries and the output?

Lan Chu: Indeed. The idea is that you can infer the chunking quality using a QA pipeline.

Participant 5: Because currently you've got a very batch pipeline, you've got data that gets crunched, then gets turned into useful assets at the end of it, do you ever run into the issue when things are starting to change faster? Like, if your input data is viewed as bad, how do you deal with the real-time overlay of very up-to-date data, let's say, to the hour, to the second, to the minute, sort of thing?

Lan Chu: I have a pipeline, like a scheduled job to update my index on a daily basis as soon as my documents arrive. It doesn't really happen to me that I have to re-index the old documents, for instance. I would just check the document ID, and then if that document was already in my vector database, then I just embed the new documents.

Participant 5: If there's overlap between said documents, because let's say a document changes and now overlaps with a secondary document, how do you resolve this conflict if one document starts to overlap a second one?

Lan Chu: That was a big discussion for my project, too, because I have also a lot of documents on the same topics, and then we were thinking, why don't we just improve the quality from the source. We remove the old documents, and then only keep the recent ones. It doesn't really work out, because there's no efficient way you can do it. At the same time, sometimes people want to ask about the trend five years ago, or in the past five years, so we want all of these documents. What we were able to agree with the data owner was that once a document is updated, it is the final version. Yes, so that is full stop.

 

See more presentations with transcripts

 

Recorded at:

Software is changing the world. QCon London empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Dec 16, 2026

BT