Transcript
Baruch Sadogursky: I want to show you something. I had CLAUDE.md, which I fed everything there is to know about my project. It has code samples. It has all the documentation. It has best practices. It has everything there is. Now, all I want to do is add error handling to one of my endpoints. I gave it a lot of context, it should probably manage. Let's see what's happening.
Patrick Debois: I think it's easy right now.
Baruch Sadogursky: Should be easy. It's a small project, and I gave it a lot of context, so let's see what's happening. Let's see. It adds try and catch, which is good. The test failed. Look at that. Look what's going on.
Patrick Debois: How is that possible?
Baruch Sadogursky: Yes, there is an AssertionError here, Expected 404 for missing order, and got 500.
Patrick Debois: It's not working.
Baruch Sadogursky: What's going on?
Patrick Debois: Maybe you stuffed too much in there? What do you think?
Baruch Sadogursky: Who thinks that it just ran off context window, and it needs more context window?
Patrick Debois: Maybe level setting. Who has used Claude Code for coding? Who has ever run out of context window? Congratulations. You did the first step.
A lot of people say, let's try another model, see if it works, a bigger model. Obviously, a lot of people are looking for the ideal model. At a certain time, they will all have the same problem, that have a maximum context window, the more we stuff with it, like Baruch showed a big CLAUDE.md we put in, but there's even other parts that go into that. Remember, Claude as a coding agent, it has its system prompt, which already takes a bunch of that as well. There's your chat conversation, that as well. Everything you keep typing and everything it returns also goes into the context window. All that together will eventually blow up your context window.
Baruch Sadogursky: Let me get it straight. What you're saying is that the system is stateless, and every time, whatever you send it, goes back and forth forever?
Patrick Debois: Luckily, otherwise it would keep all our chats on their servers and all the stuff.
Baruch Sadogursky: I don't know what is better, that would be their problem.
Patrick Debois: Yes, depends who you talk to.
Baruch Sadogursky: If I pasted a log file into my context, like into my chat window, it will travel back and forth?
Patrick Debois: Yes, as part of your session, and it just keeps adding to that whole thing. It's one of the demystifying things. People think that it's only your chat, but it actually always sends the whole shebang over the line to the LLM. That's why this context window, even if you're typing tiny messages, just fills up.
Baruch Sadogursky: They lied to us. Who would have thought? What does it have to do with engineering?
Patrick Debois: It's more like, why do we keep stuffing everything in there? There must be a better way. What do you think?
Baruch Sadogursky: Yes. There is a lot of talk about prompt engineering. Is prompt engineering and context engineering the same thing?
Patrick Debois: Prompt is how to craft the perfect query or prompt that does exactly what you want, but it's only part of a whole conversation, which is your context. That's how to relate. Initially, everybody was obsessed with the prompt. Getting the perfect incantation to do the model on the one shot. You can send it to do it. Then, later, as we got the tricks with exclamation marks and all the funny things, and it's like treat it as your life depends on it, it became a part of giving it the right context. We started putting more and more context.
Background
Baruch Sadogursky: My name is Baruch Sadogursky. I'm a context sommelier because I love context engineering and wine at Tessl AI. This is Patrick Debois, father of DevOps. The man who actually invented the term DevOps. Now let's talk about AI.
The most important slide of all this is this. If you go to speaking.jbaru.ch, you will find there the slides with the speaker notes. The agent code and the demo recordings, and the chitchat about what we're going to talk about, and all the links, it's conveniently on the bottom of every slide.
Antipattern 1 - The Stuffed Prompt
Patrick Debois: We're explaining this whole problem set by using coding agents, because a lot of you are using coding agents. Obviously, when you're writing your own agent within your product, they have the same problems. A lot of people put too much documentation, too much stuff in there. The agent gets lost. If it becomes too long, the context, then it loses parts at the end, and it doesn't become efficient. In a coding agent, you've done this before. You might have just created a new session, new session, just to keep track of that.
Baruch Sadogursky: This was the stuffed prompt, what we saw an example. How do we fix it?
Patrick Debois: Over the time of building the coding agents, we started having CLAUDE.md. Then we put things in some files. Then, eventually, the idea of skills came about. The idea there is it is also context, but part of the skill describes when to load the context. We're not loading everything up front, but it's like trigger words that inject that context into the prompt that's being sent over. I think that would be the solution.
Baruch Sadogursky: Let's see what we can do about it. That was our entire CLAUDE.md when we stuffed everything into the prompt every time we ask anything. That's what you saw previously. This is our skill. What are we looking at here?
Patrick Debois: The first part of the skill is it has a name, but that's just for us humans to recognize it. The description is actually describing when it's supposed to trigger. On coding conventions, error handling, and so on. The agent takes that description of your skill to actually figure out, should I load this file? It's a big difference because your CLAUDE.md, when you start your coding agent, gets immediately loaded. Everything you put in there will be every time in your context. Skill will be lazy loaded. That's the difference between.
Baruch Sadogursky: The description is critical because this is where activation is decided. The agent looks at all the descriptions of all the skills it has at its disposal, and it goes like, ok, for what I was asked, this skill will do the job. Developers are lazy, if we have a field description, we will probably copy-paste the title and move on. Don't do that with skills.
Patrick Debois: Who has used skills?
Baruch Sadogursky: Who wrote a proper description? How do you know you wrote a proper description? We'll talk about that. Should we apply the skill instead of CLAUDE.md? Let's do it. What we're going to do now, we're going to delete the terrible CLAUDE.md, and actually put the skill right there. You see, this is our skill. I just printed out the titles, so you will see it, like the parts. Then we are going to ask for the exactly same prompt. It's exactly the same text. Error handling, we didn't have it. Let's add it, and let's see what's going on now. Who thinks it's going to work? Of course, it will work. I didn't do a live demo. It worked. The skills work amazingly well.
Patrick Debois: The difference is you cut up the pieces that you know, maybe your frontend advice to the agent doesn't need to be loaded when you're doing backend stuff. Maybe your directory layout doesn't care when you're doing UI stuff, so those trigger words allow you to cut up a lot of the context. This was of course one example, but this is the way to go.
Baruch Sadogursky: Before, we stuffed everything into one prompt, and we had conflicting conventions. We got the wrong answer. The 500, it wasn't purely hallucinated. If I go back to my terrible CLAUDE.md, and search for error handling, there are four hits. There are conflicting instructions there because we keep adding there. No one checks that it's up to date. We're just keep throwing stuff on it. Do you remember Back to the Future? Who remembers Mr. Fusion, the engine? Tell me like just out loud how it worked. You put in what? You put in garbage and you get out energy. This is the opposite. You put in a lot of energy and you get out garbage. This is like 700 lines of code, conflicting conventions.
Patrick Debois: One caveat, as you're chatting away and it's loading progressively, it is keeping that skill that you loaded previously also still in memory. Again, that's why it's useful to just switch to a new session because your skills are then fresh and not loaded. I haven't seen a system that unloads skills, but some day that will likely come to keep track of what it does. Another way Claude deals with it is it delegates things to sub-agents. They get a clear zero context again and then they go off, they hand off the results and come back. While we're saying conflicting conventions because we're splitting things up in different files isn't happening, now we have another problem, conflicting skills. Remember that description? If it overlaps, that's another problem, but it's probably easier to deal with than conflicting descriptions in everything.
Baruch Sadogursky: Skill is just a markdown file, and the first instinct we all have when we have something like a text file or a source file is, how do we manage it? How do we manage source file, text file? GitHub. We just shove it on GitHub.
Patrick Debois: Version control, yes.
Baruch Sadogursky: Then we go on LinkedIn, context engineering. In this day of AI, it's very important to, "Look, I wrote a skill, here is my link to a GitHub file."
Patrick Debois: It's a great way to show off.
Baruch Sadogursky: Is it the way to actually do stuff?
Patrick Debois: If you put something on LinkedIn, I can't argue over that. On GitHub, it's there, but how does somebody else use it? That's one of the problems you have. Then it also is a skill, no pun intended, to write a skill that somebody else uses within the organization. You have to think more of edge cases, maybe they're hitting some other things that do. It's a little bit like open source. You share it, it could be rough, but if you really want it to be used, you have to put some love around it. People expect things. They want you to have a test and version control so they know when it's failing. We'll get into that later, but it's a whole bunch.
Baruch Sadogursky: I spent more than a decade in JFrog convincing people that shoving jar files and other artifacts into GitHub is not proper artifact management. It was just a warmup for this. Actually, skills shoved into GitHub is not proper artifact management. Because, think about stuff that you really need once your LinkedIn post took off and everybody wants your skill. Versioning, I installed your amazing skills from your LinkedIn post, and then a month later, how do I know that's the latest version? How about distribution? Is my skill public or private, or for my team, or for my organization? How do we manage that part? There are a lot of aspects that skill, although being just a text file, actually have more in common with artifact management, Docker images, npm packages, than with source files. Treat it as an artifact.
Patrick Debois: Even within a skill, now you can have pieces of code, you can have pieces of documentation for that skill. Think of it more like a zip file or indeed like a jar file of a bunch of things, not just the markdown file. That's why version control and putting it somewhere reusable makes sense from there.
Baruch Sadogursky: Let's summarize the first part.
Patrick Debois: It's not a junk drawer.
Baruch Sadogursky: It's not.
Patrick Debois: Be selective on what you put in there, break it up, and then make them reusable so when you add to the next project, you don't have to copy your CLAUDE.md from the one project to the other. Think of this as a reusable component that you share more and more across, whether that's across your own tooling, whether that's across your team, whether that's across different agents. They're all pieces of context that you can reuse from that as well. Only one way? How many to go?
Antipattern 2 - The Wrong Tool for the Job
Baruch Sadogursky: We can try another one? You want to see another one? Let's do another one. What else can we use? Here's an example of a search. How many of you heard about RAG, Retrieval-Augmented Generation? Retrieval-augmented generation means that our generation now is not based on what the model knows. We actually augment it with more stuff that we retrieved, which means, found somewhere. The way we plug in RAG into Claude Code is by using MCP. Here we have a RAG server that is connected. What we're going to ask now is to add notifications to our amazing system that we're working on. Add the notification. pidge is a very sophisticated library that I vibe coded for notifications. Then we can run tests and see if it works. Same idea. It'll find API docs in our vector database. It found something. It found actually two hits. One with the relevance 0.46, not great.
The other 0.76, the closer to 1 the better. This is a good hit. This means that it found a very relevant piece of documentation that it will now use to do that. How many of you think it will work? Let's see what actually happens. Looks clean, straight from the docs, tests fail. Seriously, you thought that it will work? Didn't you get the pattern by now? We'll check the next one. Something went wrong. Score 0.7 relevance was very high, but it was actually wrong. There are two versions, version 2 and version 3. It chose the wrong one. How did this happen?
Patrick Debois: One of the first things I learned about RAG and thinking it was the holy grail to find everything back on my hard drive, in my email and stuff like that, I started indexing that. I must say that I was very disappointed when I was asking it a question and it returned me an email with the same question. That's because it doesn't look at whether it's relevant, it just looks whether it's similar. The same thing as they show with documentation, there could be lots of documentation, but whether that applies to the piece that you actually need is not up to the RAG to decide. This is another problem. Then RAG also suffers from the stuffed prompt a little bit in a way that if it's a big document, in order to index it, we need to chunk it up in pieces and it will only find pieces.
Then maybe the cutoff of the piece just was in the wrong place. It missed actually the relevance because it imputed parts of it as separate parts. RAG is great if you know the limitations and where you put it in. I would also say a lot of the early coding agents had RAG indexing, not just for text, but for code. What you've seen is they've moved on to using grep and search and so on. It wasn't that they believed it was going to be better, but it was empirical that they said, we get better results because the grep actually takes context and searches where it fits in. RAG's great if you know what the limitations are, what you're looking for, but it is not the ultimate grep.
Baruch Sadogursky: What else do we have? RAG, we just mentioned.
Patrick Debois: Another piece is obviously just searching the web. Why do you need to index? Indexing is slow, especially with dynamic information. You don't know it ahead of time. There's a cost effect of doing that. Why not just search and go out? Again, do you know whether that is accurate? Do you know that's relevant? You have to ask the agent. It's also a bunch slower. One of the early products we had, the company I was working for, like, do we do a web search or do we do a semantic search? Then when both give results, which is the best one? That's another challenge. Like, which is the more relevant? Because this one gives you relevant scores, but this one might give you a keyword. Then people are experimenting. If I give it a search, a lot of people are still giving it a keyword search instead of actually a question. When you give it keywords, then your embedding is not matching, but your web search is, but it might get the different things there as well. Challenges everywhere.
Baruch Sadogursky: Another tradeoff with web search is obviously will work fine for whatever current trendy thing is that has a lot of relevant results on web search. If you do something that is niche, something that is not very hype, you'll probably get wrong answers from Stack Overflow of 2009 in your answers, which is also obviously problematic. MCP, what can we do with MCP?
Patrick Debois: MCP is something you have to look at more as a connector. It's not like a specific search way of searching. You could do semantic search over MCP. You can do grep over MCP. It's not that it's doing anything specific. What it does add, like, when it gives a description of what the MCP function can do, there is another prompt that goes into your system prompt, but in your old prompt that it's being sent over. A lot of people said like, I'm going to give it all the tools. Then you started seeing that explosion of MCP tools descriptions, which eventually led to the first thing, like having a whole context full way faster than you had before. People are now mixing MCPs with skills. There's also now new RFCs about making MCP more lazy loading, discoverable, in there as well.
Baruch Sadogursky: What else do we have? This one is interesting.
Patrick Debois: The versioned documents, you can do web search, keywords, you can do semantic search, but sometimes it's very important to give it also maybe more, I was going to say static information, but let's say we are on version this, so filter out all the other stuff. Think about like when you're searching, what do you filter with the input or the context that you already know, because that will improve the searches to limit it back. If you search all documentation, we might have seven different versions of the same API being described or various states on your wiki. If we say we're on this version, it might be more selective. That is the filtering mechanism that works well.
Baruch Sadogursky: There are a bunch of companies and tools today that do that. That's another option. The last one, rules. What is this about?
Patrick Debois: The rules is something that also goes into the system prompt, most of the tools loaded there, or it's lazy loaded based on when the coding agent is doing certain files. Then it loads it up. Rules can be managed by your IDE as well as a way of loading things in the context while your coding isn't doing it. It gets appended based on the file suffixes and stuff like that. A rule could be anything. It basically is also a prompt that you add, but it's another snippet that you're not putting directly in your CLAUDE.md, but could be more lazy loaded locally as well.
Baruch Sadogursky: MCP, all of them are consumed through MCP. We saw RAG consumed through MCP. Web search, you can install MCP that will do better search. Then versioned doc artifacts, all those tools are consumed through MCP. MCP is not really a channel, it's a plumbing. We have four context channels. Search in database, search in web, rules which are, always do that. Then pieces of documentation which are correct for exact that tool. Which one do we need to use?
Patrick Debois: Use the best one for the tool.
Baruch Sadogursky: Let's use the best one for the tool. What we're going to do here is we're going to use the documentation for our pidge notification. Here is the reference of the notification library written in the way that our agent will have the best time with. You can see that it's version 3, not some unknown version. It goes into the full documentation, including examples, including stuff that it should go, integration pattern, error handling, and whatnot. We can also have here rules for this particular library. Here are the rules. Use async API. Do not use obsolete versions. Config, construct messages with this class. Very strict rules that should always apply when we use this library. We're actually going to use two. We're going to use versioned docs and we're going to use rules.
Patrick Debois: One of the things that vendors of libraries or providers of libraries are doing, they're starting to provide you not just the code of the library, but they're providing you the rules or the best practices that go in. Imagine if they provide it as an artifact that you can just download and you all of a sudden get the best practices, you don't have to write everything in your CLAUDE.md. That's what brings it up almost like a life cycle. A new version of the library, new version of all the descriptions and you're golden. The LLMs can't keep up because they are only trained on a limited set or they have a cutoff date until when they were trained on.
Baruch Sadogursky: Let's see what we can do. This is my tile. Think of tile like a plugin. This is the plugin of context artifacts, piece of context that has all the needed information for our context about my pidge library. You can see there are docs. That's exactly what we saw previously. There are rules. We can also write skills. Skills, as we spoke about, is, this is how it should be done. That would be best practices on how to use our library. It's all packaged together as an artifact, probably versioned and distributed and managed and whatnot. Once we have this, we can actually go ahead and ask Claude to use this instead of doing RAG to get the right documentation. Here we go, let's see. Exactly the same prompt, obviously, send confirmation notification. Again, I didn't manage version previously, I don't mention version now, it should figure it out.
Let's see what's going on. Who thinks it will work? Here we go, successfully loaded skill, twice. We have two skills now, the one is for our project, this is how we write project, what we solved previously. The second skill now is this is how to use our notification library. Now it knows exactly what to do, it knows how to look for the right documentation. It's obviously going to use the right version, it's going to use version 3 because, remember, in skills version 3, in docs version 3, and in rules, always use version 3. It's like belt, suspenders. Three levels of suspenders. Obviously, everything passed. Isn't it nice? This is what changed. Before we had search_docs through MCP. It was a similar topic. Version 2 was a little bit better indexed, and we got the wrong answer, and after, version 3, docs tile from the registry that it found in the registry, downloaded, exact docs, and correct answer.
Patrick Debois: For those using Claude, Anthropic has a skill that says, like, whatever I did right now, can you turn that into a skill? You can save that quite easily, and then it becomes reusable. Same with docs, I keep repeating this over and again, save this to my rules, save this to my docs, and that's like your lip that we need to know, start being reusable as well. Version control?
Baruch Sadogursky: This is our context artifact now. It has skills. It has rules. It has docs. It has a piece of content which is packed together, versioned, tested, and distributed. How do we test context artifact? We're going to talk about that. Meanwhile, let me just do something here.
Patrick Debois: I think there's a talk on evals.
Baruch Sadogursky: We're going to talk about that. What did we learn?
Patrick Debois: Similar things, but not always useful things. Context artifacts allow you to make it reusable. You know whether you're searching or accessing for just like a plain search. Or, I want exactly this, is another way of thinking about finding the right tool for the right context that you need to use.
Baruch Sadogursky: Two down. Patrick recommended clearing context. Should we clear context? Let's clear context.
Patrick Debois: This helps with bloat in the context window, so definitely can recommend it.
Baruch Sadogursky: Let's discuss. You remember our discussion, should we use version 2 or version 3? Claude learned something. Let's see if we can decide on which version is better based on our previous experience, everything we know. Go with version 3, because that's the right thing to do. Rules and whatnot, it explains why we do that. Then I'm going like, ok, version 3, sounds good. Now, what we're going to do is we maybe have a little bit of stuffed context there. Let's clear and ask what we just decided. Will it work? I don't have any context from previous conversations.
Antipattern 3 - The Goldfish Agent
Patrick Debois: This is the exact reason, you're in a coding session, it's giving you the right results, you do not want to close this. It's so good, this one. I know if I close it, I was like, it's really terrible. It's not FOMO, maybe somebody has a term for this, but definitely closing it is not. It doesn't remember anything, and that's the challenging part.
Baruch Sadogursky: Also, modern agents do have memory. You saw it all, it does something with the memory all the time. It's like, updating the memory, and then, like, looking in the memory. Do you have any idea what's in this memory? It's like when you're in therapy, and then the therapist starts to write something, and you're like, what are you writing? Is it good? They will never tell you. It's that. It's like updating the memory, and you're like, what did you write down? It will never tell you. It has memory, you just don't control it.
Patrick Debois: You've seen it a little bit, like writing in MEMORY.md, maybe. Over the last couple of months, it writes some files about your project, it saves some context that you set it. You can obviously do it yourself by saying, everything that I said, save that to a markdown file. Now the tools are doing this automatically. They're also even looking at past sessions or something, trying to figure out if they need to remember something as well.
Baruch Sadogursky: It has memory. The problem is you have no idea what it is. You know what's worse than that secret notebook that they write stuff? The compacting. This is like the worst idea ever. What does compacting do?
Patrick Debois: It saves you space in that big context window.
Baruch Sadogursky: How does it do it?
Patrick Debois: It usually does it by taking pieces and say, let me summarize some pieces, and let me remove pieces that are irrelevant anymore.
Baruch Sadogursky: Who decides that they are irrelevant?
Patrick Debois: Claude, when you give it /compact, you can actually specify the reason or the focus that it needs to keep while cleaning up your context. If you just do /context, it's just going to do clean, whatever. We can nudge it a little bit. Still, it feels like, ok, I'm not going to know what's coming.
Baruch Sadogursky: Exactly. This is a black box. Both memory and compaction are black boxes that you have zero control. What do we do instead? Who's got the reference? Amazing Christopher Nolan film called "Memento." It was about the guy that couldn't retain memories. What he did is he wrote everything down, including on his own skin, so when he wakes up in the morning, he can reread it, and then remember what it is about. This is exactly what we are going to do. Our next demo is going to do exactly that. Should we use version 2 or version 3? It will give us exactly the same answer, obviously. We should use version 3. Look what changed. Want me to document this decision? Where does this come from? You know the answer by now. Where does it come from? We just spoke about it, how to instruct the agent to do what is needed. Skills. This is something that we can easily program with skills. First of all, who thinks it will work? It's going to work. There we go. Let me check what we have on the record. It searches our memory.
Patrick Debois: .memory/decisions, Reading.
Baruch Sadogursky: You saw that line, memory/decisions. It's already collapsed. There is like a schema of how it saves stuff. This schema comes from the rules of our project. Remember, so we have here our skills, that this is how we do things in our project. All we did here is added a new set of instructions. Memory Bank. Architectural decisions are stored in .memory/decisions with a clear format of how to do things. Then in the rules of our project, we will say every time there was a decision made, go and save it in the memory bank using the rules in the skill. When you are in a new context and you have no idea what to do, use the memory bank to load all the decisions using the skill.
Patrick Debois: I just want you to take a moment and look at this in awe. Imagine the lines of code that you would have had to program to have the same thing. You just describe it in a prompt in a skill, and it works. That's the time we're living in. That is amazing. I find that amazing.
Baruch Sadogursky: Me too. I'm blown away. This only thing was invented for me, because I don't need to write code. I just can tell, do this, and then it works. Amazing. Again, before, clear blank slate, what decision? After, decision read from .memory/decisions, and it says we decide on what needs to be decided.
Patrick Debois: The built-in memory is currently a black box. I'm pretty sure that will be opened up in the future. Now we install things through a skill. I'm pretty sure there's going to be a service that under the hood saves this for you, allows this to be manageable as well. It's like one of those progression things that we just see coming in a lot of the tools, because they will use this to avoid their own mistakes, and you have a whole feedback loop of learning, where if you weren't putting in what you needed, it wasn't learning, and it was just as bad. Maybe the model was better trained, but now with the better context, it just keeps going, improving the context, rewriting the memory, much a little bit like humans do, unfortunately.
Baruch Sadogursky: Let me do one more thing here in the background, and we will continue. We got skills down, we got right retrieval down, we got the memory set up. Who thinks our agent is better now? You think it's better now?
Patrick Debois: Yes.
Baruch Sadogursky: It's definitely better now.
Patrick Debois: Probably.
Baruch Sadogursky: People agree?
Antipattern 4 - The Vibes Eval
Patrick Debois: Yes, I think what you all did is, as much as you hit it, you did a vibe check. You said like, let's get to me a little bit.
Baruch Sadogursky: Also, there is Patrick, we cannot say that it's bad, although we think it is.
Patrick Debois: Yes, I'm a Belgian, I can't take choices. It's hard to predict this upfront. As much as we want it to be, and we believe it to be, it can still get things wrong. The word vibe check actually came from people looking at when a new model was introduced, whether it was better, yes or no. They didn't have a set of tests. The only thing they could do is, we'll check. That was a vibe check. That later evolved into vibe coding as I'm checking my code. First, that was a term as a vibe check in the ML engineering.
Baruch Sadogursky: I need to do another experiment here. Who heard about evals? Who thinks it's just a fancy name for tests that people like to show off? Who thinks that it's like a super complicated data science shit that you will never be able to grasp? It's actually neither. How does it work?
Patrick Debois: Most of the evals, you have a certain state of your project, you give it a prompt, and then the whole agent does something to it. Then you want to check that. Now, how do we write the tests? You all know tests, the unit tests, end-to-end tests that you're doing. In this case, it's not always that clear. One of the things I learned with AI is if something isn't working with AI, you just use more AI. For the testing, actually, they use AI with a concept LLM-as-a-judge. You have rules that say this is how you produce your code. There are rules that can say, here's how you look at the code where it's good. Now, hopefully, there's a little bit of overlap, but it's a different focus, like this is for review, this is for code generation.
Baruch Sadogursky: You can use other models as well, that will give some kind of neutrality.
Patrick Debois: Other models, there as well. You write the same narratives as you have in your head. This is what good code looks like to me. You describe that, and then you use scenarios to see whether that is working, yes or no.
Baruch Sadogursky: Ideally, you will review those scenarios and say, yes, this is what you really need to evaluate. Maybe do, maybe don't. Then, what happens?
Patrick Debois: They check it. Now you have two problems. You have, how do you write a good prompt and the good rules, and how do you write a good evaluator prompt? The problem just multiplied. Then you iterate on one, and then see if this got improved. Then you improve this and see if your tests pass again. There is like two works to it. A lot of the unit tests mirror this, but in this case, you also have to mirror whether that is actually a good review, yes or no. There is now double the work. We save time, but you got to put some work in there.
Baruch Sadogursky: Here's an example of eval criterion, that was obviously generated by LLM. It goes into, ok, this is the problem description. This is the output specification. This is how we want to judge it. What should happen? What do we expect to see in the results? What is the weight, the max score that a certain rubric can score?
Patrick Debois: The scores are based on what the LLM gives you, based on your criteria and kind of that. Like, is it a 10? Is it a 9? At least it gives you some indication how far off you are. Again, this is never perfect. It's not like an exact test, like a unit test, but it does give you a way of better writing your prompts and getting feedback on how to do that better.
Baruch Sadogursky: Those evals can evaluate our context artifact, our piece of context as a whole. At the end of the day, LLM checks behavior. The comprehensive behavior is skills, rules, and docs working together to produce better results. We run evaluations on those pieces of context together and see the results. Who thinks that our behavior with our pidge notification library is now better when we packed it with rules, skills, and docs than it was if it just had to find information about it out in the wild? Who thinks that it's better now? Who thinks it's better by what? 30%? It's like one-third better. Who thinks it's better by 50%? It's twice as better. Who thinks it's three times better? Let's see. This is the summary of all scenarios. Baseline without context is 35%. It's terrible, because this is a library that I vibe coded. Basically, the agent has no idea what I'm talking about when I refer to pidge, probably thinks pigeons. With context, skills, docs, and rules, 98% of correctness when we use it. Impressive? I think it is. Here we go.
Patrick Debois: The concept is it even can not only grade your thing, it can also create the scenarios out of your code. Then you say, this is a good scenario. Then you do the eval of that. There's a lot of AI help to deal with this whole flow to make it less tedious of typing and testing and doing that as well. It's also important that you review every time your evals, whether they make sense to you. The LLM can get it wrong, much as it can get the coding wrong. You're the domain expert. That's the work that you need to do. While you're not coding, this is the work that you can put the effort in to get better results.
Conclusion
Baruch Sadogursky: Here we go. Fifty minutes ago, you watched the agent fail, and now you know exactly why. Twenty years ago, you looked at dev and ops and said, this is the same problem, we need to solve it together. Today, you look at AI agents and developers and do the same?
Patrick Debois: Yes.
Baruch Sadogursky: Four antipatterns, four fixes. Stuffed prompt, we do skills. Wrong tool, we use the right channel. Goldfish agent, you don't trust building memory or compaction, write it down yourself. Context engineering is an architecture problem. Context artifacts are actually building blocks and not stupid .md files, so treat them accordingly.
See more presentations with transcripts
