Transcript
Cassie Shum: Today I'll be talking about knowledge graphs and retrieval to reasoning, and some of my experiences, and what I know and what I've been working on for the past few years. Let's go ahead and just start talking about me just so you can get some context about my background and actually why I care about all of this. Today, I actually spend a lot of time building and running agentic systems. I just can't get enough of it. It's like the gambler thing. Every time I hear that ding, I'm like, ok, what's next? What's next? That's actually how I started off 20-plus years ago. I started off, as many of you have, as an engineer, as a coder, and I got really excited about solving problems, creating applications, getting customers to see what you've actually built. Then, as I went through my career at Thoughtworks and a few other places, working with customers as a consultant, I turned into this hand wavy person, talking about microservices and architecture and all of these things.
I accidentally, but probably somewhat intentionally, got away from the codebases and started managing all these teams and organizations on how to create good software and things like that. Then, as we all know, what we're talking about is how AI and this agentic world has now come and now the agents have showed up, and they're actually taking a lot of this work in a good way, I think. One of the things I wanted to do throughout my journey here is I wanted to go back into the codebases and I wanted to see, what are these agents actually doing? How are they doing it? How can I apply the knowledge that I have around basic software engineering practices, the work that I've been doing around knowledge graphs at my current company, RelationalAI? What does this mean for agentic systems? That's actually what this talk is about. I pivoted from the last time I talked about this to make it slightly more personal on the journey that I went on and what I did, creating even a development harness for myself and for my team based off knowledge graphs and what I've learned through that.
Framing
What this talk is, is really about my lessons learned. This is not a dogmatic, everybody needs to use a knowledge graph. My friends always joke with me because I see knowledge graphs everywhere. I see everything in graphs because that's what I've been doing for the past three years. The reason I got into graphs is because I do think modeling data and modeling your domain is very graphical in format. It's about relationships. It's about tying this thing to another thing. That's why I got really excited about it. It is not a, this is how you should do it, this is how you should work with agents. This is just Cassie's experience and how I work with knowledge graphs and the agents and everything that we're doing now. It is also not like a GraphRAG tutorial. I'm going to say this because my last presentation at QCon in New York, I went much deeper in how you create a knowledge graph using GraphRAG.
I'm not doing that this time because I had a lot of lessons learned. Just for context, that presentation I did was in November of last year, and between then and now, everything has changed again. The worst part about giving presentations is you have to change it all the time now because it's rapidly moving. I'm pivoting away from that a little bit and talking about some of these lessons learned even in these last six months. This is also not a framework prediction. I don't know what's going to happen. We are just evolving. I'm hearing a lot of talks about, I don't know what tomorrow's going to look like, but we're sitting in a place where we're trying to put our foundations back with the agents in order to evolve, to pivot as quickly as possible. That's where this talk is surrounding as well.
Outline
This talk is in two parts. The first one is why I believe knowledge graph is the foundation of agentic systems and why this matters. Then the second part is around the lessons learned. That second part, I'm actually going to try to demo the harness that I built for my team on top of a knowledge graph and some of the things that we've learned around using that knowledge graph. I'm the customer of a knowledge graph in this particular demo, and the lessons learned there.
What is a Relational Knowledge Graph?
I wanted just to level set with everyone on what a knowledge graph is. We've heard words around graph databases. I just saw a talk around decision models, which actually look like a graph when I saw it on the presentation. It was great. What I want to do is just explain to you what I believe are critical in a knowledge graph. I would say a graph database, like Neo and all of these other types of databases, are the underpinning of it. A knowledge graph in itself is really the relationships and the entities and the logic that you're embedding between these relationships. That's where the rich semantic data and this rich semantic layer lies, is within this knowledge graph. The way I think about it is how do we actually position all of our different datasets, our different domains, and how do we connect them together? Why am I related to this person over there?
We're both from Denver, Colorado. I'm at QCon, and then I'm going to fly to New York tomorrow, where I'm going to meet a friend. We are all interconnected by all of these different relationships, and how do we actually store them in such a way that we can actually retrieve them? Retrieve that information, that rich semantic knowledge. That's what I mean by knowledge graph. I know a lot of definitions around graph and graph technologies. This is really around the rich semantics around it.
Is GraphRAG Dead?
This is one of the things when I said earlier in November, I was talking about GraphRAG. How do you retrieve a lot of this very rich domain type of information from a graph, and how is that better than getting it from just your basic databases and microservices and application layers and things like that? I was in a conversation the other day, and I added this slide, because they were like, isn't GraphRAG dead? Aren't the LLMs so good now, and the agents are so great, that you just can send anything in there and retrieve whatever you want, and that's the context. Why do you need a graph in order to retrieve that? I'm like, you're right on the retrieval part, for sure. It's just gotten a lot smarter. I would say, maybe even six months ago, I saw the differentiator in actually retrieving from a graph.
I'm not seeing that differentiator as much anymore, so clearly my talk had to change very quickly. I don't think it's dead, but not in the way you think. I think what we're trying to pivot from is just using graphs as this retrieval system. It's not just about optimizing your question and answer, and getting a more fuller answer. I actually believe that the knowledge graph can serve many purposes, not just grabbing the information, but following and understanding some of our software development practices, thinking about visualization, thinking about tracking things through a system. There's a lot more that we're seeing with the agents, and how much more powerful it can be, where the knowledge graph can be the substrate and the foundation of your system, as opposed to just optimizing for better results when I'm trying to gain information.
Why the Knowledge Graph (KG) is the Foundation
As I said earlier, why a knowledge graph? Why not just a long context? Because right now, agents and LLMs can do amazing things. In one prompt, in one window, when we dump everything in there, they can actually come back with a lot of very good logic. They can actually come back with a lot of good information. I think there's a theme that I'm hearing throughout all the talks around how is that repeatable, or how is that consistent, or how is that accountable? How do you actually know, and like, my context or my prompt that I'm doing may be very different than your prompt and your prompt and your prompt. How do you actually do this in an organization, in a team? How do you have that consistency? For me, one of the biggest things of a knowledge graph is it can be a source of context that you're sharing within your organization and within your teams themselves, which can be composable in all these different terms.
You can have a lot more reliable source in the knowledge graph. You can actually go back in time and query that knowledge graph and understand what you did. How did those relationships form? How did new relationships form as you're learning on top of that knowledge graph as well? I think auditability is a huge thing. How do you audit decision-making? How do you audit when a decision was made, and then code was created from that, and then, how did it change? You can actually audit that through a knowledge graph because you can actually feed that feedback mechanism back into that graph. Then, of course, the main thing here, too, which is how I started in knowledge graphs is, how do you pull in information from multiple different domains? When I'm working on different applications, 5, 10 years ago, we would do a lot of microservices. My team would be in one domain, and this is what we cared about.
What happens if you have dependencies on all of these different domains, and now you have a distributed monolith where you don't know where all these dependencies are? How do they all track with each other? A knowledge graph is a very good way to think about how to bind some of these different domains and create relationships between them so you can have a fuller picture, not just your own specific domain.
One of the things that I think about is really about the moat isn't really the models. The models are changing constantly. You can fine-tune your models. There are different domain models. Everything is just getting better and better and better. I don't think that's the moat. Again, this is just Cassie's opinion on things and why I care about graphs. I believe it's actually the domain knowledge and the graph that you put together from your domain. Here are some questions that were asked in one of these financial institutions. This is one of my customers. If you ask just an agent right now, you open ChatGPT or you open Claude Code and you say, which compliance regime do I actually need to follow? Or, what does Tier 1 mean here? Or, what was decided last quarter? The agents are going to turn around and be like, "I don't know.
Let me just go look up what everybody has said, what a compliance regime actually looks like, or what Tier 1 means in a general sense." If you think about the organization itself, this financial institution had very specific answers to these questions because that's not only your domain context, but your organizational context as well. How do you actually turn your agents to looking at your own knowledge graph, your own organizational knowledge graph, and answering these questions with way more fidelity and way more accuracy?
Here's an example of that financial institution. When I first thought about knowledge graphs, I thought mostly about it is the domain knowledge, it's just how this financial institution works, how this domain maps with this domain, and this and that. It's actually so much more. This is why I believe the knowledge graph is a substrate, not just knowledge retrieval. If you think about an agent who's assisting some sort of business analyst or an analyst in these particular financial institutions, they are asking about what is that compliance regime and which one applies. They're figuring out who is the policy owner, how is that mapped to the different policies. I'm sure different companies have different policies, different teams have different policies. These are all specific to what you are pointing to in your knowledge graph. Then, actually the thing that I learned, and I don't want to give too much away in my second part of the conversation, but it's really about how domain and context changes over time.
My decision that I made today based off the technology, based off like the domain rules, based off how my organization rolls, may be very different than six months ago to five years ago. There's a lot of institutional knowledge that is kept within organizations, and I'm sure everybody has seen this. There's an old wiki page that had a rule that was this like five years ago, and clearly no one knows what happened to that rule. That became like some dead code somewhere, or maybe that was a rule that was re-implemented in many different ways in different places. Thinking about that lineage of decisions and context is really important, and how do you track that in a knowledge graph and be able to figure out, why did we get to this decision, and where did it come from, and what is not a decision anymore? It's really important as well.
Then, I also say things around like veterans, tickets, internal wikis, tribal knowledge is a real thing. Agents don't know what to do with tribal knowledge. Half the tribal knowledge is in all of our engineers and business analysts' heads, and the other half of it is in old Confluence pages or wiki pages, and there's really no rhyme or reason why they are, because they're just stale data. If you say, agents, go to my organizational wikis, go find out what this answer is, are they going to get confused very quickly? How do you actually think through all of that?
If the KG is the Foundation, What Does Building on It Look Like?
If my hypothesis, or at least my opinion is, if the foundation is a knowledge graph, then what does it look like building on top of it? Just context as well, for about a few years now at Relational, I've been working with many different customers on building their knowledge graphs. As I said before, I really focus on building their knowledge graphs based off of their own business rules of their applications, their domains, and whatnot. What I turned around doing in the last six months now is, I said, why don't I build my own internal developer tools on a knowledge graph as well? Because as I said before, here, there's a lot of internal wikis, there's technical design docs, all of these things that were held within my organization that, after many years, I couldn't keep track of it all. I was like, how do I actually start developing our application or improving our feature sets based off of our own organizational information?
That's what we did. I found four patterns that I'm going to share with you. There are many other patterns, I'm sure, but these are the four that are going to stick out with me as well, and I wanted to share with you all. Before we get to the four patterns, I'm going to, of course, look at this very blank slide around this is how things just happen in the engineering world. We go from an idea to building to validating to learning. This is a very typical SDLC. We care about the foundational pieces of that fast feedback loop. I don't think anybody would disagree that this is a normal pattern that we actually follow. As I think about how we build our systems, this is something that's core. This is our core entities. These are the core things that we need to think about. I want to touch upon this of like the bigger lesson learned.
This is not a pattern, just more of a big lesson learned, is that when we first started out working with all of these agentic technologies, there were many agents. We went crazy. My team was spinning up agents left and right, and we were actually pushing features very quickly. However, the thing that I realized is that we were pushing a lot of things, but I'm not sure like how useful these things were. One of the things that I do know is that agents accelerate that feedback loop. Something very traditional like back five years ago when I have a scrum team and I was saying, ok, we're going to go story grooming, we're going to pick up five features or whatnot, or five tickets to pick up, let's go ahead and do that. If I had a very bad spec or I had a very empty spec about what that feature is, the business requirements were ill-defined, all of these types of things.
It would take maybe about two to four weeks to find out that that thing was wrong, or we're actually building something that we didn't want to build. That feedback loop was a longer feedback loop. I'm not going to even talk about 10 years ago when that feedback loop was six months and realized we built something wrong six months later. Now we're in this world where agents are actually accelerating this like crazy. If I wrote a bad spec or the agent wrote a bad spec for me, I'm going to find out within 40 minutes probably that that had not actually built the right thing and that feedback loop is super quick now. We should take advantage of that. We are taking advantage of that right now, that's why we're all here talking about what are the best practices, what we're actually trying to do here.
Really, what I've found out is agents aren't really magic, they are just the fast, honest mirror of your engineering culture. If you don't have some of these fundamental practices in place, those agents are going to shore them up. If you do have some of these fundamental engineering practices in place, then you can see that, you can see some productivity gains on what that looks like. Just as a quick continuation on the fast, honest mirror, these good engineering cultures, like a task that's actually created here can be shipped correctly very quickly, but these bad engineering cultures like with these undefined rules, these brittle handoffs, these inconsistent data, you're going to find very quickly that these things are going to produce very poor code. Just quick story on this one, just because I was baffled. I think one of the things that we're seeing in the rise of the agentic world is, a lot of people, not just engineers, are like, I can build software so much more quickly now, and it's going to be fabulous.
I'm going to just build this app and it's going to work and whatnot. What we're finding out is just the AI slop that's coming from this codebase, because if you don't have testing practices, CI/CD, all of the things that we've been talking about, then you're not going to be able to maintain a sustainable application like that. My partner just joined a startup, and it was the co-founder and the founder who vibe coded an entire application in three months. Great. Awesome. They even put it in front of a customer. Even better. That customer was super excited about that demo. However, once they got to going into production and thinking about scalability and all of these things, everything fell apart because you didn't have that engineering practice in place, therefore being able to do the things that you need to do in production just fell apart. This is the kind of thing that we very much learned in the very beginning of actually introducing agents into our world. What did I say? I'm like, where's the knowledge graph? Didn't you guys have a knowledge graph as part of this engineering type of thing?
The Four Patterns
Let me just explain the four patterns that I found actually in the last six months, but I'm sure many more patterns have come out. After listening to all the talks, I'm starting to think about some patterns as well. Here are the four. One is around context as a bundle. I'll talk about all of these in great depth. The second is provenance for decision making. The third is code as truth, not spec as truth. I'll talk about what that means. Then the third one is about visibility to drive this team effectiveness. These are some of the patterns I saw and wanted to use a knowledge graph in order to help bring these patterns to light and actually fix some of the things that we are seeing here.
Pattern 1 - Context Assembly
The first pattern. As I said earlier, we really worked on retrieval as a query. How do you have all this domain knowledge and all of this logic that are all bound together in this knowledge graph, and how do you query it? One of the things that we thought about was the RAG part, the GraphRAG. I had a question, we have all these chunks within the graph, and then we had a really high-fidelity answer. Great. I'm not saying walk away from that. That's still great. That's what you need in your organization. What we found out is a knowledge graph can actually serve much more than that. You can actually assemble your context for the agents with a lot more than just the domain knowledge. You can actually create that context bundle using the spec itself, the standards that you're placing within your engineering team or your organization.
As I said, prior work. What work preceded this that you can actually use? Then, of course, the relevant documentation, whether that be the domain knowledge, that be the decisions that were made previously, but some of the documentation that you usually get in wikis. How do you pull all of these in into what we call a context bundle? I will now show you, just switching over, the engineering harness that my team and I had built to help us. We built this on top of a knowledge graph in itself. Just to focus a little bit, just switching context a little bit. The way we built this was actually, how do you create a new idea? How do you create a new task? Think about a task as being a story, an idea being an epic. This is a very just minimal way that we've built out our own personal Jira, because we didn't want to use big, heavy Jira.
For the purposes of this talk, this is a quick demo that I have here. What we actually did was think about how to build some of the contextual information on top of a knowledge graph in itself. If I click here, we have a couple. Here is an example of being able to actually have an idea, this idea 47, is, how do you implement Auth, Auth? Very hard task, but very basic task for an engineering group to actually pick up. If you sent this to an agent and said, I want to implement Auth, Auth in my application, how is it going to go about it? It's probably going to go to the internet, see the best way, see how Stripe did it, see how Facebook did it, and say, yes, you use OAuth, you use Auth0, and just come up with some sort of implementation. If you're in an organization that already has some patterns or some ways that they've already done Auth, Auth, or an architect has already created a spec that's more current here, then you can actually bind this into your own internal knowledge graph as your context that you can use in order to create this idea or create the spec.
For example, I have this idea, but I also have some code and a codebase that I've attached this to. I have this authentication doc, so you can see here, the Google Doc that actually has an architecture that an architect has already put together. GitHub is pulled into this as a source that you can pull into your knowledge graph that says, this is how someone has actually created this, and this is how your codebase looks like, thinking about your Auth, Auth. Then you have the idea and the spec itself. Instead of just saying, agent, go tell me how to do Auth, Auth, what you can actually do is say, take my current GitHub repo or repos that have different implementations, or hopefully the same standardized implementation, take the architecture diagram that is now more current, and then take my own spec of the requirements that you need and pull that in in order to create this thing. This is what I would call a context bundle. You're bringing in many different contexts, and you're putting it into your own graph in order to drive implementation of this task or this story or whatnot. This is an example of that.
I love experiments. I used to be in the sciences before, so I love running experiments and actually having data, because this could all be theoretical, where I could say, if you have a knowledge graph, this is going to be so much better, and it's going to be so much accurate. Your implementation is going to be a lot better. One could guess that that could be the case. I did run a little bit of a small experiment based off of my own experience with my customers and things like this, but this is one that I could show you guys. Let me talk about the use case here. The use case is we have two different teams with this on-call policy. In these two teams, my on-call policy in my team is very different than an on-call policy somewhere else. We're all still part of the same organization, and if I said, agent, please go and give me what the on-call policy should be and let's go implement that.
It's going to probably come up with many different documentations and different policies that are being used in the organization. How do I know which one to use? I wouldn't know which one to use, so Lord knows how the agent is going to know which one to use. One of the things that I did was, in this experiment, I said, within all of these documents that we've curated, and let me just show you even like this is what we did here, if I go back to resources, you can actually see many different compliance and on-policy documentation for different teams. We have one for Meridian, and then some for Squad Cypress, and then some for Atlas. Many different on-policy type of documentation. What do we do? We put this all into a knowledge graph, and we can actually say, which ones matter to me? Which node is in my neighborhood?
My team is called Atlas, I believe I called it. Then, what are the documents that actually matter to me, and what is that neighborhood of documents? This is why we put this in the graph, so we can quickly see what is our nearest neighbors versus what are the documents that are much further away, and maybe not as applicable.
Essentially, what we ran as an experiment was, how do we actually run a task that says, give me my on-policy documents, and then, how do I run an implementation of what that actually looks like? In this experiment, I ran the same task twice, once with the knowledge graph, and once with just feeding it all the documents. What do you think actually happened when I did that? It was better? It was more accurate? It actually surprised me a little bit. What happened was nothing. The correctness of actually the implementation I was trying to do was actually right in both instances. Guess what? Models are really good. They can actually scour through many documents and try to figure out what your policy is. It's like, that's maybe not going to work for me anymore. However, one of the things that did come about was the tokens that were used.
The number of turns. How often did that agent wander before getting to the right answer? I think that's incredibly important now. My initial hypothesis of, knowledge graph is going to be more accurate for you, it's not about that even. It's about many different facets. It's about how quickly you can actually turn around with the answer, even if the answer is correct on both instances. Take this implementation and take it times 10,000 engineers. Those tokens are going to add up if you just keep feeding it all the documentation in the world and say, go figure it out. If you can be way more specific using a knowledge graph on what you're actually trying to do, you can actually optimize on how many tokens you're using. That goes against that token thing of like how many tokens are a developer using. It should be, how efficient are you using the agents, really?
How many turns is this agent taking in order to get to the point that you're trying to get to? We did learn this as a pattern. Instead of saying, it's going to be the most right, it's actually what's going to be the most efficient. Now I'm thinking about that developer productivity, that agent productivity, and how quickly are these things actually going to be going. That's the first pattern that we found out. Not what I expected, but still very interesting information. With that back there was 30 docs. Imagine if I did that over thousands of docs? The next step for that one is to actually test it in one of our customers to see if accuracy is still there, but also to see at what scale you're saving those turnarounds.
Pattern 2 - Provenance for Decision Making
The next pattern that we found out was really around the knowledge graph helping us with provenance for decision making. As an engineer, the minute I start the implementation and you're in Claude Code and the agent says, "Picked it up, going to do all these things," you have some traceability of what the agent has done. How did you come to that decision and how many times does a human have to intervene with the agent, and are you documenting that intervention? Are you documenting any of that human feedback loop that you are doing with the agent itself? One of the things that we've built in our harness was collecting even the human interaction with the agent so you can have some provenance all the way from the idea that you actually created the spec, to where the task got picked up, and then how it got implemented, how many human interventions happened with that implementation.
Then, once it gets to production, you have from idea to production between the human and the agent, all the decisions that were made between that, and all being mapped in a knowledge graph. This is what we did. I know you can do this many other ways, but this is actually how we did this around the audit trail, is the graph artifact. Every time something happened on a status timeline, it becomes, the action was taken, it was written to the knowledge graph, and then it was summarized. You go into this flywheel effect where you're actually feeding back into the knowledge graph because that itself becomes the context as I showed before, but you're feeding back into it with every decision that you're making as well.
Going back to the harness, this is something that we did. We tried to figure out how to trace a merged PR back through the graph. Again, that's why I had that first slide around idea to the hypothesis standard, the prior context, and the merged PR. I can show how we did that. This is an example of a task that I had in that harness, and you can actually see all the way back to when the idea was created, not when the agent picked it up, but when humans said, I have an idea, I want to change the box colors of my app. With that idea, I went back and forth with the agent, and then all the way to the time that that task got picked up by the agent. That provenance starts much earlier on, not right when you actually picked up the task.
Thinking about how do you actually follow the decision making and how that happens is very important in terms of provenance. This is something that we stored on our knowledge graph so you can actually weave through like how things happen because not everything is as simple, from A to Z. It will take a hop here. It will take a hop here. Those are the kinds of things that you want to see from a provenance point of view. That was pattern two.
Pattern 3 - Reliability Under Exploration
Pattern three is really around code is truth and the spec is a kernel. I was recorded on a podcast saying, "Code is cheap, spec is the real truth." I said that six months ago. Because I was like, could you imagine, all you need are specs now because an agent can just create that code as quickly as possible. Great. I love this new world. This is how it is. I very naively said that. One of the patterns that we showed was thinking about how that spec is not the source of truth anymore. Like, everyone's thinking that. Like, once the agents came around, all these people said, ok, I'm just going to create a bunch of specs and that's going to be the truth of our applications. As we all know, as an engineer, how many times have a business analyst or product manager come to you with a spec and it was exactly implemented the way it was and it was exactly the business criteria?
No, zero. Never. The implementation and the truth of the application came through the implementation itself. The spec was the starting point, and the more you learn through that implementation, the more you found out that is actually what evolved that spec. How do you actually bring these things together and actually follow that provenance? How did you start with that idea and where did you end up as well? Very important here, the spec will always capture the intent. Why are we trying to do that? The code is still what's real. It's what is actually being implemented and driven to production.
One of the things that we think about with the knowledge graph is, how do you model both the intent and the reality? Essentially, the intent is a spec and the reality is your code. If these are all together in your knowledge graph bound by that relationship, then you actually have that lineage of how things actually happen and what that reality actually looks like. Reconciling the two in its own loop. This is actually one of the things that we did, and very tactically how we did it was when we had an idea, that spec, we actually said, ok, agent, let's work with the spec. Agent, let's pick up the task. Agent code it, I'm not coding it. I haven't coded in a long time. Then, what happened at the very end of that? What was that back and forth that happened? You can actually see here that we can actually have a very nice contextual summary of what happened with that and then put that back into the knowledge graph, so you can actually query what is this feature actually doing?
Because if someone goes back to the spec, chances are it's not actually what the actual application is doing. This is the kind of thing that you need to take that same feedback loop and bake it back in. That's actually one of the biggest things that we got wrong when we were thinking about the spec. Again, I'm the first person to tell you I was wrong about a lot of things. One of the things that we got wrong was regenerating the spec with each pass. One of the things that we actually had to do, which is why you saw that last picture, is we had to create a summary of what the actual implementation was. What happened was the agents erased what the builder discovered. Really, it's just as important as a spec as what the agent or the humans discovered together as they were implementing it.
The implementation definitely knew way more than the plan did. The other thing that we found out is treating the agent as a fire and forget. Here's spec, agent, go do it. It doesn't work that way because, in the end, it's actually not what we wanted. There's no flywheel there. Then the other thing was we were assuming that the engines were interchangeable. Mostly it was. How do you actually keep some of that interesting context that I showed earlier and feed that for the next feature? If you actually did something wrong or implemented it in a wrong pattern, how do you bring that pattern forward, the right pattern, and take that forward?
Pattern 4 - Visibility
Pattern four, a little bit different than provenance. Provenance is being able to capture all of the data so you know how something started and how something ended. Really, the last pattern that I realized was a dumb moment for me is visibility. What on God's green earth are our agents doing? How do I make that super visible to our engineering teams? Because our teams, agents and humans alike, we still want to know how things are happening. What are the agents doing? How many tokens are they using? What are the context bundles? Where did I start off from? These are some of the things that you can actually query the graph in order to see. This is like the end all be all. It's like, that's why we held everything in the graph because now, weeks later, I can actually go back in time and figure out what was the cost per task that I actually had, or, what was my audit? Can I actually see that audit? Can I actually show that in a report to someone? What was the context bundle size? All of these different things are things that you can query the graph later to show visibility of what your team had actually done.
Very quickly, going back to the harness that we built, here's a task. We can actually look at the flow. This is a query that we did on the graph to understand how many tasks were shipped in a certain time. How much would a feature actually cost based off of these tokens and things like that? Being able to make things super visible, this was super easy to bring up because we already had this all in the graph. Being able to bring information back was incredibly important, so visibility of any of these things that we could do. In the last slide that I had, those are queries that we made on the graph in order to pull in this information because everything got pulled into that. Essentially, the last part of that demo was you can actually see the agents running. What we can do in this particular task here is, if I'm actually running a managed Claude instance, then how do you actually see the agents running from a UI like this?
It's starting here. Essentially, if I look at these different tasks, I'm the only one on this one because it's my demo. Imagine that I can see all the different agents running from my team and what tasks they're actually working on. If you click into any of these, you can see the agents and that visibility of that. Really important to bring all of this to light and bring all of this to front.
Recap - The Four Patterns
Just wrapping up, four patterns recap. Context is a bundle. It was incredibly important, the pattern that we found out. As I said earlier, provenance is the decision making. You can actually see how did ideas start, all the way to got to production. Also, not just production, how do you actually see the customer feedback and bringing that back? That's part of the next steps that we're going to try to figure out. Then, code is the truth. We talked about specs being the owner of all the information, it's not just specs, but it's the code as well, that implementation, that knowledge with that. Then, of course, the visibility. In this visibility, you can actually drive a lot of team effectiveness based off of a lot of things that you see in that knowledge graph.
Open Questions: What's Still Open?
What is still open? Things that we're thinking about and things I heard throughout the talks is around, manage and supervise agents, and how do we actually go about that? All the different models. I heard a really great talk about adaptive reasoning. How do you actually adapt and use a graph when you can? You don't have to always use a graph. Like, when do you use what? How do you actually make those decisions? Then the next step is really thinking about, within this graph context, how do we have agentic memory architecture? We have a very light architecture right now, but how do we do this at scale, given all of those four patterns that I just talked about? Hopefully more to come, more learnings.
Questions and Answers
Participant 1: This whole topic that you have seems to me, belongs to the context engineering piece, where you would have the agent, and then you have the context engineer sitting below that. Then, do you think your knowledge graph sits within the context layer, where it's between the agent and the context engineering layer?
Cassie Shum: My opinion is, is that the knowledge graph should be the context layer. That would be my opinion, because I'm very biased on that end. More of a pragmatic sense, I would say that the knowledge graph should be definitely part of that context layer, along with any other documents context that you guys have. I do see that the knowledge graph helps with like a source of truth in your context. There's so much context sprawl, like what team is doing what. Knowledge graph gives that organization.
Participant 1: I really like the live demo you have, and I was just wondering what framework you use to show like, there's open-source framework, or you can say how the agent runs. What framework did you choose to use?
Cassie Shum: We built that framework, actually. It was actually just my team building this framework. That's not actually the product. That was just us trying to help our own development team make things visible and things like that. We have thought about maybe making it a more formal thing, but that's just a bunch of devs clamoring something together.
Participant 2: Knowledge graphs are highly effective. We've used that, but there's an overhead with knowledge graphs. When would I shift to knowledge graphs versus having localized agents which have specific links to the necessary context?
Cassie Shum: The knowledge graph definitely has a lot more overhead. My advice is only use it when you see that benefit. That first pattern that I showed you, we used knowledge graphs because it was way more accurate six months ago. The models have gotten so much better that accuracy is not the problem anymore. Tokenization is now the problem in how many turns we're taking. I would actually be very careful, too, because knowledge graph and the overhead to building it is actually still high. We do have a lot of ways in our company and skills that we're building in order to make that 0 to 100 much easier to build that knowledge graph. Really, you need the domain expertise, like what your team has, the context, and how to build that together. We're using agents right now to figure out how to build that knowledge graph way more quickly. Yes, good call.
Participant 3: How would you characterize the performance penalty of knowledge graphs? Is it driven by that first embedding call to try to figure out what's close, or is it driven by the size of the graph that you're searching? Or, does it matter depending on how you store it? Not to ask another $64,000 fire all question.
Cassie Shum: It depends. That's where my consultancy background shows. It definitely depends. If you have large amounts of data within your knowledge graph, then, yes, performance might be an issue if you're actually trying to traverse certain things. However, if you're trying to do something small, like I was doing there, performance might not be an issue, but it's the way you actually create that.
See more presentations with transcripts
