Transcript
Ricardo Ferreira: My name is Ricardo Ferreira. We're going to be talking about this context engineering stuff. This presentation is actually going to be a mix of lessons learned when I've developed a specific solution that I'm going to talk about. I learned the easy and the hard way. Also, it's going to be a mix of storytelling, because these applications that I've developed actually have a story. I was setting up my Alexa device, which has to do with what I built. I work for Redis.
Initial Implementation
Let me tell the story about what I actually have built, so you can understand where I'm coming from. I work inside Redis from this part of this team called developer experience. One of the things that we started doing, probably I think about two years ago, is to build an open-source project called Agent Memory Server, or AMS, on top of Redis. The idea of this open-source project was to create a very quick, fast memory layer on top of Redis, so developers can build either short-term memory and long-term memory on top of it. Build more natural human-like conversations with it. Then our team that developed this project, we're also being asked to develop applications that would put this open-source project to use. We've released it open-source, but we've also tried to test internally. Basically, the requirement that we're asked was, build something that can put some stress on this type of open-source project.
I come from a background where I've developed some Alexa skills in the past. I know for sure there are some very unique and interesting challenges when you develop Alexa skills. One of them is that the Alexa interaction has a fixed timeout of 8 seconds. It needs to respond within this time frame. I thought, I'm going to try to use this in this open-source project. That was the first motivation. The second motivation is that probably in this slide that you're looking at right now, how many of you have interacted with Alexa devices in the past, and then most of the time is spent not knowing things than actually knowing it. I think if my Alexa is ready right now, we can test it.
Alexa, can you explain what is context engineering?
Alexa: I'm not quite sure how to help you with that.
Ricardo Ferreira: How many of you have spent like listening to the same thing? Point is, I had this idea, what if I could put some brains behind Alexa? What if I put an LLM behind Alexa, and then I create this skill that I've called My Jarvis. In a minute, you'll get the reference for where the name came from. The way it works is like this. My Jarvis is a skill. Basically, what you have here is the Alexa device, and this is going to make this interaction, this is backed by a Lambda function. This is usually how Alexa skills have the backend spot. Then on top of the Lambda function, I have two layers. One of this is the actual SDK from the Alexa skills. That's a bunch of wrappers on top of JSON payloads that comes from the Alexa interactions. Then I use this framework called LangChain4j, which is a Java version of the LangChain framework, kind of, because actually technically LangChain4j is not a part of the LangChain projects, it's more like a LangChain along with LlamaIndex, along with Spring AI, so it's a more complex framework. The point is, that's what was backing the LLM, which in this case I use in OpenAI. I was able to do this. Alexa, ask My Jarvis to explain what is context engineering.
Alexa: Context engineering is the art and science of designing systems that understand, interpret, and respond to contextual information. It involves creating algorithms and models that enable machines, like AI assistants, to process and prioritize relevant data from user interactions and environment cues. This discipline is crucial for developing intelligent systems that can provide personalized and accurate responses by understanding the user's intent and the situational context.
Ricardo Ferreira: That's a very insightful answer.
Different Types of Context Problems
The point is, that's how I felt after I started developing this. The name My Jarvis came from the famous AI character from the Marvel Universe, and I'm a huge Marvel characters fan. Then, as I got excited with this development, I started asking more questions, more elaborate questions, and tried to incorporate this skill in my daily life. What I ended up realizing is that, yes, the skill was able to retrieve answers, but it's fairly easy. Once you actually make a call to the LLMs, it's going to be virtually able to respond to any possible question. The human aspect, the mannerisms, when you ask questions, when you interact with, let's call it the model, is what takes place most of the time. What I found is that the experience of dealing with that was odd at first, and then I started realizing that, ok, I am running through a set of problems that has to do with context poisoning, which is things that didn't make any sense at first, or context distraction, too much data, too much information that was not necessarily spending more time creating elaborated answers, then creating the right answer.
Context confusion, which is a bunch of irrelevant data that was a mix and match. Also, like the famous problem of as your context grows, the context rots. Ultimately, the context clash, that happens a lot, like a mixing up of this is true, this is false, this is the versioning and timing of the answer, and all of this mutation of the data creating a very poor answer that was given.
Using Newer Models as a Solution
I'm not going to make a bet here, probably the first instinct that most software developers do in situations like this is, I'm right on the backend, and if the backend is malfunctioning and is not keeping up, what am I going to do with the backend? I'm going to improve it, or I'm going to change it. Like what we used to do in the old days of databases, relational databases on-prem. If the relational database on-prem is not keeping up with the workload, what's it going to do? More CPU, more memory, or put a cache in the side. I tried to use a newer model for ChatGPT, like 5.4, yes, I increased my cost by doing this. Also, my assumption was, if I use a newer and a larger context window, it was able to create a more elaborate answer. Silly of me thinking that, because changing the model didn't do a thing in the experience that I was having. I think that was the first lesson learning of this, that not all the time, upgrading to newer models in your AI implementation is going to solve your problems. I know for sure that in my case, it didn't solve mine.
Agents Need Context
What I actually needed, and I didn't know at the time, that's one of the fun stories about this implementation, I didn't know that I needed context engineering. That implementation started probably like one year and a half ago. The term itself was originally starting to catch up, but just like I started catching up recently to harness engineering. Then, at the time, what I actually have felt is that there's a bunch of situations that I need to fix as an engineer, as a software engineer, I need to fix each one of them pragmatically. As I was fixing them, I was using some instinct approaches, some others I actually was collaborating with the rest of the team. Then, "No, Ricardo, there's a name for this, there's this context reranking," because my background is not AI engineering. I'm a software engineer. Most of those practices that I would assume that are pretty common, like in the AI world, it wasn't for me.
The huge majority of the time was trying to fix them instinctively. If you ask me, not the industry, but if you ask me, Ricardo, what I have felt when I actually fix all the things, I would categorize it and I conceptualized context engineering exactly as this. It's something that goes beyond prompting, because that was another instinct of mine, like let me create a more elaborated system prompt that would help the agent to think a little clearer and better. Another mistake. What I ended up creating was a series of very intentional, very well-crafted approaches to pragmatically solve each one of those bad experiences. If you ask me, this is what context engineering is about. I know that there's a very more conceptual and more elaborate answer that Alexa just gave us. The point is, I think that's how I felt at the time.
Chapter 1: Pragmatically Solving Each Problem
This presentation, I'm going to break it down in two chapters. The first one, I'm going to walk you through all the single problems that I went through and how I solved them and some of the lessons learned, that I learned the easy and the hard way. Chapter two, we're going to talk about cost, which is the incidental type of cost that by using context engineering, I ended up having, which is by itself my most important heads-up for any of you that would like to start implementing context engineering as a practice in your applications. There's a cost involved.
The first problem that I went through was this, probably this is a no-brainer for everybody, but LLMs have a hard time deciding what day and time is today. How many of you have used an LLM and then all of a sudden, you rely on an answer that, yes, because we are in 2021, this has happened, this, this, and that. No, we are not in 2021, we're in 2026. The first thing that I actually had to do, and because one of the first use cases that I've implemented was a reminder system, I wanted to use my skill to remind myself about things.
Actually, let me do a test here. Alexa, ask My Jarvis to remember that I have to say hi in two minutes.
Alexa: I noted that you need to say hi in two minutes. Would you like me to set up a reminder for this?
Ricardo Ferreira: Yes, please.
What you're looking at right there, this is the actual, what we call LTM, or long-term memory, stored internally in Redis. This long-term memory is basically a combination of the text, which is the actual information or memory that you're supposed to have, and the text vector because there's a bunch of vector searches that happen behind scenes in order to do that. With that said, the first experience is that, like I mentioned before, LLM has no concept of time. How do I solve this? Using tools. I created a tool that I was able to use it in order to teach the LLM, along with the context about date and time. Hopefully, that's how it's going to remember the exact day and time. That became even more important when I had like, remind me about the appointment next week on Wednesday. Those things really matter. For those of you that have never worked with LangChain4j, this is as simple as to create like an agent in LangChain4j. Basically, you associate the models, the list of the tools, and you build it, that's your AI service, ChatAssistant, and then you can start invoking your queries. It's that simple. Then, that's what actually starts to happen, like the reminders start popping up exactly in the right time that I was supposed to do the presentation.
Then, here is my second problem that I went through. I start asking questions that would require a subsequent remembering of those questions. For example, "Yes, my name is Ricardo." "Nice to meet you, Ricardo." "You know my name?" "Apologies, I don't know you." Because at that time, I haven't introduced the context of short-term memory yet. Probably some of you didn't hear this, but that was the reminder that just popped up like two minutes later. Apologies, but I should have set up my Alexa device in advance.
Another incidental realization was LLMs are stateless. I think this is no-brainer for everybody. First of all, I used this open-source product called Redis Agent Memory that has support for short-term memory, we call Session Memory because they are session-bounded and they are supposed to be short-lived and TTL-based. LangChain4j, the way it works, you basically implement that interface called ChatMemoryStore, and then you create whatever persistent vector store you want to create. Basically, I create a wrapper to call that either via MCP or via REST APIs. I think, originally, I used the REST APIs and then I switched to MCP. The way this Session Memory works, and let me try to show here. We're all developers. I think it's good for you to have an idea about how this works. If I refresh this, those are all the interactions that I had with Alexa so far, and as you can see here, this is a JSON payload.
It's a forever-appending, forever-growing JSON payload. This is how a Session Memory looks like behind the scenes. This is the Redis Agent Memory Architecture in a nutshell. Basically, what it does is a thin wrapper on top of Redis, and also uses LLM behind the scenes to create long-term memories out of short-term memories in the background. As you keep having conversations, the Redis Agent Memory keeps using the LLM. What is important here to realize is to promote to long-term memory. It does this automatically. This is how you can associate it with LangChain4j. The interesting part here, this implementation of a ChatMemory here is how you can get a hold of always the last 10 messages that were exchanged between the user and the system. You can create an idea of what I have said before. I was able to fix this, like it started remembering my name.
One of the things that I realized with this are first and foremost at the time the building implementation of that ChatMemory was a malfunction because LLMs also make bi-directional calls between tools. That implementation was causing a forever loop of calling all my tools. As you can imagine, the timeout for Alexa is 8 seconds, and then it started like having more than 10 seconds because of the forever loop. I had to replace it with my own implementation that I've called WorkingMemoryChat that was a simple pass-through between the messages. It didn't get a hold of the last 10 like a buffer, and it was forever growing the ChatMemory. It solved two problems at a time. The first one was the forever loop, and the second one is that my conversation would be there forever between calls because session memory you associate with a TTL. My TTL was five minutes. Five minutes was an ok time for you to remember the last thing that you have talked to.
Then, if I would ask Alexa five minutes later what was my name. "I don't know that," because the TTL had expired so it had no recollections of who I was anymore. That was the trigger for me to start implementing long-term memories, LTMs. The context must use long-term memories as well. The way you implement this in LangChain4j, and arguably with LangChain itself in Python, you basically have this concept of a ContentRetriever. A ContentRetriever, like that's a direct reference for RAG. It's an abstract implementation that you've given an input, you can create a list of outputs. I create my own implementation of getLongTermMemories. I create my memory server that was able to talk with my Redis Agent Memory. This at the end of the day is a simple vector search semantic search call. Here's the prompt. Give me the top-k results that would match this on a vector search level, using HNSW index type, so highly efficient at scale, not necessarily very memory hungry.
Then you would retrieve all the recent memories so I can correlate this and try to provide a more relatable answer. I was able to start like, ok, I have an appointment next week. Then even many minutes later, if I ask the same question again, yes, you're having a dentist appointment next week. Can you see how my experiences were increasing as I was putting more information and functionalities at the device? That didn't come for free at first. Because first, my naive implementation was I would just put the LLM here and the LLM would take care of everything. I think you're getting a feeling of how when you start dealing with more human-like experiences, all of those things have to be a little bit more intentional. It doesn't come for free.
Another lesson learned about that is that, first and foremost, I started using this Alexa skill inside the house. My son and my wife started using it as well for their own memories, their own interactions. I had to figure out a way to use multi-tenancy. I ended up having to use another technique of the AI world, which we call vector search with post-filtering. What is vector search post-filtering? You basically do a vector search, you retrieve a bunch of results. Then out of those results, you're going to do a post-filtering based on metadata, which is my tenant ID and actually owner ID at the time. You know when you approach Alexa and then it detects based on your voice or your picture who you are, so that internally is an ID that I was using for doing this filtering. I had to do this, otherwise, the questions that I would ask could be answered by memories, from my wife, or my son, who is 15 years old, and quite dangerous these days. You have to separate these tenants over there. Secondly, probably this is going to be a very simple thing to understand, but like, how many should be set to the top-k? Five, 10, 15? How much is enough memory to use to formulate a right answer. This is where I ended up using it.
Another use case was not necessarily everything has to be about memories. What I mean by this is that this was an actual use case. "Dad, can I actually teach Alexa how to decorate the code that you keep changing for the garage door?" How do I do this? I could create my own memory, like, Alexa, remember this. Or I could maybe fill a PDF document with a bunch of instructions about everything that has to do with the house. I ended up creating something that I called knowledge bases, which is yet another category of long-term memories. Knowledge bases would require a little bit of more elaborate usage of how I would implement this. Now I would have two retrievers, one to retrieve the long-term memories, the user-specific memories. Another one to actually retrieve all the general knowledge base. As you can see here, it's shorter because of the slide length, but it would have a more descriptive description, that's redundant, I know, but a description of what each one of those use cases do.
As you can see here, I use in LinkedIn for the old LLM to decide based on the question, ok, should I retrieve this based on the user memories or based on the knowledge base, depending on the nature of the question? If you remember the nature of the question, what is the wireless garage door for the touchpad? You can unlock your garage door with the code 7121. That was the result of the knowledge base. I changed this code already, so this is not a security concern. This is interesting because, first of all, you see how I am using the LLM with yet another use case, not necessarily to answer questions, but as a subsidiary to actually sustain an ongoing answer. That was an inline call to the LLM just to make the actual call to the LLM, so in order to decrease the knowledge base. This is another example of you have to craftily and intentionally design your implementation.
The other use case was a very interesting one. It was, ask My Jarvis to recall, who was the John Doe I met last week? John Doe was blah, blah, blah. Then if I use a subsequent question, where he lives, because I am in an ongoing conversation. Human beings would be able to capture real quickly, but then Alexa would say, where lives who? I just told you, John Doe. That kind of confusion has no longer to do with memory, but it has to do with how the LLM is going to do some associations based on the context. In LangChain4j terms, you use a tech, which ended up, I realized that is a technique called compression of a queryTransformer. You can use the actual LLM, given the prompt, and then it can decide to do things like this. There's this replacement and associations on the fly. He, in this context, is John Doe.
This is one good example. Compression is another technique of context engineering that for a human being, it would be so natural, but you have to teach those things when you interact with your AI services. That was another practice from context engineering that I implemented, using something that was pre-built with LangChain4j. That was interesting. For example, can you confirm what is my favorite color? Then, yes, your favorite color is black, which is true. You also love writing code in Java. Why did he answer that? Can you guess why he did answer that, based on what I've explained before about when you do a vector search, it retrieves the top-k result before answering the question? Vector search is not an exact search. The name tells everything. By the time I actually asked this question about favorite, two entries had to do with being favorite, the favorite programming language and the favorite color.
Those two entries came with the results and became part of the context that the LLM uses to respond to the question. I can't blame the LLM. It gave a very complete and elaborate answer. Yes, you love writing code in Java and you love the black color. Again, for the human experience, you have to solve that problem. How did I solve this? Yes, the model is dumping everything it retrieved, it needs to be a little more selective. How did I solve this problem? I actually used another technique. This is not necessarily popular in the context of context engineering. It's more popular for the context of AI in general, AI engineering, but it's a technique called few-shots design. What is few-shots design? It's when you give a set of examples, positives and negatives, contradictory to the LLM in the form of a system prompt, so you can actually better explain the notion of intention from the user in that particular domain model that you are designing.
First and foremost, I had to use a little bit of a more structured context. LangChain4j allows you to use this with a context injector. You can create actually a structured input for this. There was a bunch of them, like I think I created like 17 in my original implementation, but of course I couldn't put the 17 here in my slides. This is one example of, ok, given the question, what programming language do I use? The context would include favorite color, birthday, October the 5th, enjoys Java, so it would select this. There would be almost like a hint to the LLM about how to behave, not just think, but how to behave. It's more like a behavioral engineering type of thing, the use of few-shots. When I implement few-shots, I start having more objective answers, which is cool, which ended up with what I was having.
Another important one, which is a variation of this confusion that LLM does every now and then, Alexa, what is my priority for the week, or priority right now? Adding support for JSON and going to the dentist are my priority. In this example here, obviously, since I mentioned the word PR, I assume as a human that you would understand that I'm talking about pull requests. If I'm talking about pull requests, it would be logical that the answer would be adding support for JSON. Not going to the dentist because this is not a pull request, I think. You don't open a pull request just because you're going to the dentist. This one, I had to create a more like, teach this semantic meaning of the context itself, which is yet another challenge that I have to undergo. In this case, what we're talking about here, like the technique for the context engineering that I ended up using was reranking.
It's a very popular technique. The reason why I had to use reranking was, remember, before I had those bunch of memories that was retrieved out of the top-k results. I have to teach the LLM without the reasoning about importance and priority. I've solved this with the few-shots technique. Here, there might be two or more memories that are correct, and they're supposed to be used in the context. Some of them would take higher priority. It was about the reranking, as we call it. Luckily, LangChain4j also has support for reranking. Basically, you instantiate something called a scoring model. In this case, I used Cohere, which, I don't know if that changed, but at the time was the only open LLM model that actually supports reranking and provides an API for that. I don't know if there's something new, but I'm still using Cohere because it works very well and it's super cheap.
Then I ended up associating this with this construct called ContentAggregator, which makes sense. You aggregate content. All of this happens in flight, if you think about it. The actual invocation to the LLM to answer the question hasn't happened yet. All of this is happening in pre-flight mode. You create a ContentAggregator that's going to pick up all the top-k results from the vector search, is going to perform a reranking, and a minimum score, which is 80%. I had to manually calibrate this. I played with different numbers at the time. Then you start seeing this, that my pipeline execution now has a context injector. I have a queryTransformer for the compressing thing. I have a queryRouter to decide for the knowledge base or the long-term memories, and now I also have a ContentAggregator to perform reranking. Again, it all started with a simple assumption, I just want the LLM to answer the questions.
That is a very good example, and what I've mentioned before, I'm going to actually stress this term throughout the presentation, how when you apply context engineering, things have to be deliberately intentional. You have to mix everything intentional as you implement. This was the example of what I ended up getting as an answer after implementing the technique of reranking, which is very popular within the scope of context engineering.
Two lessons learned about this exercise, the first one was more like an operational one. Remember I told you that the backend of Alexa, the skill is an AWS Lambda function. Lambda function has this limitation, if you upload your Lambda function directly through Terraform or through the console or whatever, the JAR file has to be top of 50 megabytes. If you upload, which I ended up using this approach, which is using S3 buckets, there's a maximum limit of 250. None of this was enough because my original intention was not to use Cohere. I ended up using Cohere because of this problem, which was, I used one of those like ms-marco-MiniLM-L-6 models, which is an ONNX format. Conceptually, you could deploy your own models just to do reranking, and that model specifically solved all my problems that I want to use. The JAR file ended up being twice what is supported by the actual AWS infrastructure, so I couldn't use it.
That's why I had to switch to, let me use a service, which is a Cohere, lightweight, I don't have any dependencies, and then my JAR file ended up being probably less than 100 megabytes, which is very lightweight. Also, something that I've noticed as well, and probably this is a heads-up for everybody doing this process. If you change your models, this has a direct impact on the scoring model or min score that you're going to use for the reranking technique. You're saying that the same 80% that you were using with like ms-marco didn't work with Cohere? Exactly. I had to calibrate those numbers. Yes, changing the models have a direct impact on the calibration process that you do for the reranking process. It's almost like, yes, on this release, I've solved all the problems. I can close this issue here on GitHub, and then the issue goes back again just because someone changed the model. That's the type of thing that will happen systematically.
Chapter 2: Dealing with the Monthly Cost
Now, let's go to Chapter 2, which is talking about cost. Before I actually move forward to this, have you noticed that as I was implementing and solving systematically all the problems with context engineering, I ended up using LLM more and more than just the actual call to answer the questions. That is a prelude of what we're going to discuss here in this section. I did an analysis at the time. This whole experiment, like I said in the beginning, was to check whether the Agent Memory Server, which is an open-source project we developed, would be suitable for use cases that would be stressed by human interaction, and it was. Virtually everything was going to have to be fixed outside the Redis Agent Memory Server. That was the conclusion at the time. One of the realizations that I had was for a minimal experimentation, I basically was using me, my wife, and my son at the house.
The average number of queries that we would have per day would be 10 per user. Cost with the LLM was a flat cost with like $4.20 per family. Cost with Cohere, Cohere is super cheap. Basically like 10 queries per day per one user, less than a dollar. Now let me remind you of a mistake that I did. You just chatted in the beginning. I planted that seed when I was explaining short-term memory. Can you remember where it was? When I was explaining that, yes, I used that built-in implementation from LangChain4j to keep the last 10 messages of the chat history. Then I kind of, it was having problems with tool calling and all of that, so I created my own that was simply a pass-through that was a forever-growing context. The experience was amazing because even like up to five minutes, if I actually ask any of those questions, I would have very elaborated answers because it would still remember the context.
The context growth exactly different from the LLM cost and the Cohere cost for reranking was not linear, was exponential, because the context kept growing for every single interaction. Believe me when I tell you, because when the LLM makes multiple invocation calls to your tools, all of those if forever append into the JSON data structure that I was using, the context and the number of tokens being consumed was very high. Obviously, again, if you think, Ricardo, you're so cheap, we're talking about like cents of a dollar for one user. Yes, for a small family of one user or three members of a family. What if it is put at scale? At scale, that would become a very serious problem. I did some extrapolations here, and like I said, it would be like an exponential growth of the cost. What we want is constant time. We want flat and constant time, fixed cost.
This type of question here, that's where it's explained before. You add a question data to the context, and then after some elaborated answer, you keep adding all those answers to the context, along with every baggage of information that comes with LLM tool calling and all of that. What I actually did to solve this -ish, I'll explain this why -ish at the time. There's a built-in implementation, a LangChain4j and LangChain as well, called TokenWindowChatMemory. How this implementation works, first of all, you have to set what is the maximum amount of tokens supported by that model. I was using GPT-3.5 at the time when I started developing this. If I remember correctly, I think the maximum number of tokens was 4,096. That was the amount of tokens supported by that model at that time. That was the number that was provided on that constant over there. Also, you also have to provide a very specific to the LLM implementation about a count estimator.
What is a count estimator? It's, teach how the specific LLM counts tokens, because believe it or not, when I switch this model from OpenAI, for example, for Anthropic, how it parses all the information about tools and human calls and AI calls is different. The data model, the actual payload is different. The strategy of counting varies for every specific model. The point I'm trying to make, given this, is that this token window basically creates a cap out of this max tokens. By the time you start getting closer to 4,096, it would cap. It would behave similarly to the very naive implementation of keeping the last 10 or last any as a buffer. Here, it would trim all the old ones and keep only the recent ones, but based on the amount of tokens, not the amount of messages, which is an improvement if you think about it. At least that way, I would solve my cost problem, which is keeping the costs flat. That would be a pragmatic way for you to actually solve the cost problem and turn the cost from exponential to flat. Good.
However, what is the problem that actually started happening with this? Can somebody guess? Losing context. Exactly, because the trimming that would occur, it would basically have this built-in rule, which is trim everything that is old. What if that single message that indeed is old, but was very relevant to the conversation? We could not simply start deleting arbitrarily like this implementation. That was my first attempt to fix this. Recently, I use another technique also from context engineering called summarization. What I did, I created my own implementation of this, which is the ChatMemory, but uses another LLM call to, "Here's the context that I have so far, extract everything that is relevant to this conversation that is either a fact, or a preference, or a strong statement and summarize it." I took a look in the code here, how this implementation does. I also use this notion of using the token estimator because I think it's pretty cool. At the end of the day, I have to maintain the cost flat, but also having a more intelligent implementation of, it cannot simply delete message arbitrarily. It has to delete message consciously at very best. This is how I fixed this problem.
The other problem that I will start having, this is one of the most interesting and worthy things that I ended up discovering. When you actually make calls to the LLM, like, who is coming for dinner tonight? Both John and Susan are. Sometimes, either you or some other member of the house would ask the same question, whereas the answer is exactly the same. Perhaps they're going to ask this differently. Perhaps they're going to ask this in another session, but the answer itself didn't change. The immutable vector entry there on that JSON payload stored on Redis is essentially the same. It hasn't changed. Why do I have to incur into another call to the same prompt? For those of you that know how Redis works and how Redis became famous throughout all the years, what do we do best at Redis? We cache data. I started using semantic caching.
This is another technique that became part of the umbrella of context engineering that we call semantic caching. What is the difference between typical caching? Typical caching is a point in time, very objective, very direct, hit or miss. Given the key, do a lookup. If it's there, go to the backend, populate the backend, and retrieve the answer. The difference with semantic caching, and that was the ugly part of the implementation, was, you go to the cache, same thing. It evaluates the semantic meaning of your query. In here, for example, can we agree that those are the same questions, just asked differently? It's the same question that would return the same answer, but they were asked differently. That's why semantically, they have the same meaning. Because semantically they have the same meaning, they have to perform a cache hit. There couldn't be a cache miss. It solved the problem.
I reduced most of the 75% of my redundant calls in more than half, which is good. That's the positive part. The negative part is that sometimes, even for a small fraction of words, or even in my case, because English is not my first language, I'm from Brazil originally, sometimes I would ask it with this terrible English accent. It would do a cache hit on the semantic cache, because of these very particular, unique words. It ended up asking from the cache instead of going to the LLM, which is annoying sometimes. Semantic caching is another area that you should pursue in the implementation of context engineering. One of the dangers of semantic caching is exactly this, just like the reranking model, you have to calibrate how this cache hit and miss is going to trigger all the time. What we actually do on this implementation called LangCache, it's called LangCache because it was originally intended to be the cache for LangChain, so that's why I originally called it LangCache.
If you go to the LangChain documentation, you're going to see the mentions of this. We have this parameter that basically calibrates how the assertion goes to the vector similarity search that's going to be performed against the cache. That's how we use it. How many of you can tell me what is the name of this pattern that I'm implementing here? Starts with cache and ends with aside? Cache aside. Basically, it's a cache-aside implementation, no big deal. You go to the cache, if it's not there, go to the LLM.
Final Note on Context Engineering
Context engineering, I would say that this is not just like a fancy thing that you would put for decoration purposes in your implementation, like a feature. Like, it would be good to have this or not. For any serious implementation that you're going to do with AI that has to provide human-like experiences for users, I think it's become part of the architecture. My main lesson learned from all those implementations that I did with this Alexa skill device is that there was a bunch of things, like you can see the amount of things that I had to change throughout the implementation. There was a bunch of things that I should have thought sooner in terms of architecture, in terms of design, in order to actually start implementing things and not keep changing on the fly. I did the reactive mode, but I could have been proactive. That's one of the reasons why I came here to share all those lessons with you, so you also can be proactive. My impression is that when you get context engineering ingrained in your architecture, everything else comes much easier. I think it's more natural.
Resources
I would like to make two invitations for you. The first one, all this implementation that I have done, you can find here on GitHub. Clone the repo, play with it. I think ultimately, at least if you want to learn how to develop Alexa skills, it's all there, in Java at least. All the context engineering patterns are also there. Obviously, this is going to be a minimalist version of what I have built at home because what I have built at home also has some very specific, like I have to make sure that I read my blood pressure every day. I created a tool that actually keeps storing all the blood pressure readings. I can say to my doctor when I have the appointment, this is the average that I have been through in the weeks. This is that.
Summarization, with Different LLMs
When I do the summarization, because in order to trim the messages, I have to summarize so I can keep the cost flat, did I ever use a different LLM to get better answers?
I didn't. I ended up using OpenAI as well. I haven't checked if they are able to provide better answers. That's a good thing to check.
See more presentations with transcripts
