Transcript
Brendan O'Leary: My name is Brendan O'Leary. I'm a founding developer relations engineer at Kilo Code. How many people have heard of Kilo Code? That's key, because I'm about to present you a whole bunch of our data. If you don't know who we are, not really helpful. How many have heard of Codex or Claude Code, or OpenCode maybe? What Kilo Code is, is an open-source agentic engineering tool. You can think of it as like a Claude Code or a Cursor, but it's open source. We have surfaces in the CLI, like a Claude Code or a Codex. We have surfaces inside of VS Code, so like a VS Code extension. You can think of like a Cline, or maybe like a Cursor, if Cursor is a fork of VS Code, in some ways you can think of it the same way. Then we also have cloud agents. The only point in telling you all that is like, where does all this data come from?
It's all real-world data from our gateway. We'll talk a little bit more about the specifics of the data. Being that I'm at Kilo, and we're this open-source competitor in this highly competitive space, I'm sitting at an uncomfortable intersection of like, what is the data that we can see about how developers actually use AI tools? Then the story that industry is telling us, especially the larger players in the industry are telling us about how AI adoption works. I think the problem is that not always do those stories match up really well. That's why I wanted to bring some real-world data to this talk to try and talk about like what we're seeing, and how that matches or doesn't match with the industry narrative. The industry narrative is a lot about capability, a capability narrative. This model X scored better than this model on benchmark Y, and model Z has this size of a context window.
This new release beats the old release on human eval by 0.02 points. It's not that that information is useless, but it has very little to almost nothing to do with whether developers can actually adopt those models to use it in an agentic tool like ours. I think thus also has very little impact on how you might choose what model you're going to use as you're building an agentic workflow. I've watched teams get access to the best models, the best capabilities, best benchmarks, and not use them at all. Or other teams build real dependency on tools that actually benchmark worse than other models maybe, but really work for them in their use case for certain reasons. I don't think that gap is random. I think it really is a lot about trust. I want to talk about that specifically in the context of developers using, and pull from this dataset that gives us a pretty clear view into what's actually happening on the ground. We're talking about from the title, 25 trillion tokens. We'll see in a minute that's not quite right.
Outline
What are we going to learn from that? I want to leave you with three things today. One is developers don't adopt the AI tools and the models in the way that I think we tend to assume, and not in the way that we think about adopting other tools. I think that's partially because the nature of AI is very different as a tool, being non-deterministic. I think that creates what we call this trust ladder. We'll talk about it in specifics, but we had auto-complete, and then we had chat with some agents in code. Then now we have this idea of like let's orchestrate many agents to do all kinds of things at once. Unlike tools in the past, you really have to have a solid foundation. If the lower rungs of this ladder that we'll show are shaky, suggestions are slow, or I don't get the right feeling, or it finds the wrong paths or the wrong file and edits the wrong thing, then the upper rungs of this AI adoption really don't get a chance because the trust is broken and I can't climb the rest of the ladder.
Second is I think where adoption can predictably break. There's three different technical points. One is context construction, like how we construct the context. Two is model routing, how we choose what model to do what. The third is what feedback loops are available. We're going to talk about that. I think there's a consistent pattern that we can see in the data. I think each of them has a concrete fix. As you're thinking about, of course, we're thinking about it from the perspective of how can we help developers use more AI better. I think it also hopefully applies to what you all are trying to build with AI.
The third is this trust thing. This trust thing has a lot of vibe feeling to it. I think there is a lot of that. I also think there are measurable ways or ways to measure this trust. It looks different depending on how high or low up the ladder you've moved. I think the instrumentation is possible beyond the philosophy of like how do I feel about a given model or what I'm getting. The higher up you go on that rung, I think the more sparse and delayed the signal becomes. That has design implications, I think, on anyone that's building or deploying a system about how we design for that signal and those feedback loops. I think at this conference, there's been a lot of meta about how we should think about implementing AI agents as actors, or across certain domains or with the right security parameters.
That's all very important. There's been some great talks about that. Probably most of you aren't thinking about building coding agents. You probably would have heard of Kilo Code and OpenCode if you had. That's ok, though, because I think that the reality is that for some time now, like this concept of agentic engineering, using AI in coding has been a premier use case for the models that a lot of the labs have been focused on. We've seen that in public usage patterns. OpenRouter is a router that publishes rankings about token usage. Kilo ourselves were actually at the top of that for months and months. One of the most prevalent use cases, I think, for token consumption today is developer productivity in coding. I'm hoping that the data that I have about that can help all of you when you're thinking about implementing AI systems in the other domains that you're all really interested in. That's especially domains where there's going to be other knowledge workers, like software engineers. Knowledge workers trying to leverage AI to increase their productivity. I'm hoping that a lot of this data will inform you about how you can do a good job of both implementing that and measuring for its success.
The Dataset
I want to talk in a little bit more detail about the dataset. As you saw, the title of the talk was what 25 trillion tokens taught us about adoption of AI. The reality is that it's changed a lot since we submitted that. In fact, we've now processed over 40 trillion tokens. I think this is just indicative of the asymptote of AI adoption that we've seen across the industry, and of course, especially in coding. I want to talk about the methodology of this data a little bit. It comes from our infrastructure. Kilo Code, as I mentioned, has this number of open-source surfaces. There's our CLI, our VS Code extension, our JetBrains extension. Then we also offer a gateway where folks can access inference. Because it's open source, you can bring your own inference. We actually don't care what you do for inference that much. We don't make any money off of inference.
To make it easy, we allow developers to just use our gateway right away out of the box to access over 500 different models. Access the large frontier lab models of Opus and Codex, and access all of the open weight models that are available through both OpenRouter and all of our other partnerships with inference companies. This data comes from that. We exclude anonymous users, our internal users, who are pretty heavy users, obviously. Then also, we have an autocomplete system that I've excluded from most of the data that we'll see today, because that's a little different. I think those patterns don't quite tell you the same story I'm trying to tell. Anything that looks like automated traffic. Obviously, you offer inference, you get a lot of special friends with automated traffic. Then specifically, we're going to look at what models developers chose, looking at what they requested, not necessarily where it was routed on the backend.
It's not reflective of our infrastructure's behavior, although we do have some auto-selecting models. When I'm talking about developer selecting models, that's actually clicking in the interface to select the right model. I think that just makes it clearer about where this data is. The other thing to keep in mind is, I think that the Kilo's user base probably is not a great random sample set. It skews heavily towards early adopters. It's not perfectly random datasets, but hopefully, it still gives us some insight. Most of our users are either developers or teams that have actively sought out open-source configurable tooling, which tells you a little bit already, I think, about their relationship with experimentation. If I say things like 68% of developers, you can asterisk that in your head from now on. It is our cohort. Again, I think it is also an aspect of the data that makes it really interesting and useful.
I think what we're looking at here is a preview of early adopters, if anyone's ever read the Adoption Curve, "Crossing the Chasm" book. The teams that come to us are ones that are moving fast, wanting to experiment the most aggressive that they can, trying to discover through production usage what actually is going to work for them. Obviously, the portrait of what I'm going to show today is not necessarily reflective of the average adoption today. I think this is where adoption is going, and where we're going to see adoption go in the next 12, 18 months, if it's a leading indicator of what teams are going to be using.
The Adoption Ladder
Let's go back and talk about that adoption ladder I was talking about, because I think that's critical. Like how are these teams that I'm talking about experimenting so much? What has given them this confidence? I think, if you do ask the average engineering team today what their relationship with AI is, might be like, we've got Copilot everywhere, or we bought everybody a Claude subscription, and we're using Claude Code here and there, and we're trying to experiment with maybe building some of our own agents. If you look at the data in industry, outside of ours, you see the trend is most teams are somewhere between 1 and 2, or 2 and 3 here on this four-rung ladder. The gap, I think, is larger as you go up this ladder. The ladder has four rungs and is a nice analogy, but I think each rung is just a little bit higher.
The first is what we've been used to for a long time, autocomplete, ghost text. I'm hitting tab to accept. There's a very low threshold here to trust it, because it's synchronous. I'm seeing what's happening exactly in it. It's in line with how I'm thinking about writing the code. I can accept or reject it really quickly just with a keystroke. The cost of failure is basically zero. You hit a tab, it's wrong. You can keep it or delete it. Maybe it's annoying. As we'll see, much lower cost than getting it wrong up the ladder. Rung 2, I want to be able to chat. This is the early days of Copilot, 2022, 2023. I want to ask a question or request a block of code. Maybe I'm just asking ChatGPT for code directly. I want to get explanations as I'm reviewing the code and decide what I want to use.
The trust threshold there gets a little bit higher, but you're still manually reviewing it. You might even be copying and pasting from ChatGPT. You're actually physically moving code around, like you used to copy and paste from Stack Overflow. That feedback loop is still really tight, measurable in seconds because I'm right there, or minutes. Then, once you get used to that, you can move up to having an agent do things for you. This has been the revolution we've seen the last couple of years. The AI can do things. We give it sets of tools where it can read and write files. It can run commands. It takes actions actually in your environment, like maybe authenticated as you if you're authenticated in your environment. That trust is a huge jump. That trust threshold is a huge jump and is significantly higher. The feedback loop takes longer. It might go off and do things, and try to figure stuff out while you're grabbing coffee or something. That cost of failure, thus, has gone up quite a bit. If the wrong file path happens, it's not just bad text, it might break your build or it might do something worse.
Then the last rung, the highest rung that I think we're seeing a lot of early adopters at now is, I want to get a design together and then let a bunch of agents go orchestrate how to do it. Let it break it into subtasks, delegate across multiple agents, specialized capabilities for different parts, and then just come back to me with something coherent that works. The trust threshold here is obviously the highest because there's a very long feedback loop, probably at least minutes to hours. If something goes wrong and you've given too much autonomy to the set of agents, you might not find out for a while or until something really bad happens. I think a really interesting thing to look at here is if we're thinking about these jumps in trust up the ladder, where do teams fail along that? We looked at some data from Gartner about massive amounts of AI projects failing before they reach production.
I think the reason for that is you didn't do this crawl, walk, run up the ladder of trust. What does that look like in developer tools where they still want autocomplete and be writing code themselves? They didn't get past that kind of I want to chat phase. Then they jump to, wait, I just saw this talk at this QCon where I can go and have swarms of agents go do things. I jump straight to that. I haven't built the muscles about how do I understand that these things are going to have hallucinations and these things are going to have interesting failure paths. If it didn't establish that trust at that foundational layer, then I wouldn't be able to orchestrate them. In fact, Gartner talks about governance gaps cited in this study. I think that's just like a polite way of saying we didn't design any trust into the system.
Because I do think the capability for trust exists, but the infrastructure for earning trust is something we haven't put enough time into. I also think, again, there's this asymmetry in the ladder that's underappreciated. The rungs have this feedback loop and the speed, and the higher the trust is, the longer the feedback loops, those are actually moving at opposite ends of each other. You have to have more trust at the rungs where you have less immediate evidence about what's going right. That's an interesting dichotomy.
Three Failure Points
I think there's three failure points that I've tried to bring out from what we've seen in terms of teams trying to adopt more and more of this trust. I think each one manifests at a different point in that. Then, again, it affects everything above that if you're not able to accomplish it. Let's go through those one by one. The first failure point is what I call context construction. I think this is where the most friction happens, because it's all the things that happen before you send the request to the model. Like which files are included? Like, should we index the codebase or not? Should we do semantic reasoning against the codebase? Does the model understand the relationships between things when it's looking at a problem? Have I stated the problem well to the model? It's like a model that I've given one file, and a model that can reason across hundreds of thousands of tokens and actually understand a representation of the project.
Again, when autocomplete's bad, if it suggests something that doesn't make sense in the structure, that's not a bad model. It's typically bad context. Like have we given the model enough relevant signal, or did it have the wrong signal? Or did we let it have the wrong signal for a long period of time, and now we've got this polluted context that we're not taking care of. If I say, where does the auth logic in this project live? The agent starts to walk down the wrong path, then that retrieval failure can have impacts for the rest of that chat. Because the context doesn't have the right information, it's going to start making guesses. Then by the time you get to, I want to have an agent go run by itself, or I want to have orchestrators run through the code, if that construction is broken, you're going to just multiply that failure.
If it's working from this incomplete set of context, then it's going to be much harder to walk it back and get it to deliver you meaningfully working code at the end. I think this shows up in the usage data indirectly. Developers are stuck with that heavy chat, but haven't started to use agents, consume a lot higher token counts per request. We see the tokens per request growing. They're doing this manual thing of pasting in a bunch of context themselves, or walking down the road when the context isn't right. I think that signal, for us, means that if context construction doesn't work right, and the developer doesn't know how to solve for that, they try to start patching it manually, which could work, but if the task is big enough, you can't solve that problem. Again, if you give it the wrong context, you can't get out of that.
I think there's three things to get right here. One, context windows have to scale with the task complexity. Here, people talk about models and the size of their context window. I think what we put into that context matters a lot. What is a good interaction versus a bad one can often lead to, if the tooling treated all the same context the same, you don't have that trust built. I also think that stale context is a real thing. We've seen folks that build semantic models on top of their codebases, and then leave it, and then it's almost worse than just letting the model explore on its own. Because even if it's just a little bit old, the agents having a map that doesn't go in the right place is going to eat up a lot of time and trust. Then I think the other key is relevance versus recency.
There's often like a recency bias towards, this is what changed most recently, but that doesn't necessarily mean it's most relevant for the task that we're on. The retrieval strategy that you have in those environments is very important and can produce different things. You have to have this distinction between like, what's the right bet to make for the agent to give? I think teams that have solved for this have really understood, and again, gotten that feel and trust, and they're using rung two of chat, or starting to do agents, for how can they understand to build that context correctly and give the agents the right context?
I think the second failure point is model routing. I started to talk a little bit about that with the context size of the model. I think the mental model that a lot of organizations are using today is, let's just pick the best model and deploy that, and then we're done, and we're just going to use that model for everything. If there's a new release, maybe we'll revisit it, but that's all. Or maybe there's like a little step forward where we have one frontier model for heavy work and something cheaper for lighter tasks, and we just have the one provider, it's a simple deployment. Again, I think the observed behavior that we've seen from early adopters is very different behavior. In April, 68% of folks that were manually selecting a model used models from two or more different AI labs in the same month. That was actually an increase over March.
Forty-five percent of those developers switched models in the same hour. That is like inside of active working sessions, not just experimentation, they're actually making a conscious choice to change models. Then among enterprise developers, people in like our structured managed AI development environments, 71% of them ran models from two or more different labs. Those folks that have come to us and want the structured environment are actually doing it so that they can access more than one lab. That number was up significantly from the first quarter of this year. I don't think it's like a small percentage of folks that are on our platform that are experimenting. I think it's the dominant behavior pattern that we're seeing from those folks who, again, may be an experimental group selected for, but are doing those experiments very intentionally. This multi-model development is what we're seeing these developers actually do once you give them access to multiple models.
I think there's some reasonable questions to double click in on here, like, are they exploring because they can or because they should? What we brought here is the usage intensity versus model depth and lab depth. If you segment by the intensity of usage for users here where we define them as light, regular, heavy, and power. The light users that did 100 requests a month, average 1.5 labs, 34% of them are multi-lab. Regular users, it's 100 to 1000 requests a month, is how we define that, 76% of them use models from multiple different labs, an average of 2.9 labs. Heavy users, which we defined as 1000 to 10,000 requests a month, 89% of them were multi-lab, averaging using models from 5.3 different labs. Power users, which was over 10,000 requests a month, 92% of them were multi-lab, averaging 7.6 labs. I think that pattern is like a learning curve, more than random experimentation.
I think the more intensely someone uses AI and understands how to use it for development, the more multi-model they become. It's because they've discovered that the real task completion for different models give them generally different, better behavior, whether it's designing frontend, or coming up with an architecture plan, or doing things with a database. These different models have a different feel and they have a different trust barrier for that. We see the same curve. This is for our enterprise organizations, so organizations that are using our tools in the enterprise, it's a very similar or basically the same curve among enterprise developers versus just our population of all developers. I think that's why we've seen the market diversifying to match this. We haven't seen consolidation. The top 10 models in our dataset by request volume came from six different AI labs. On some days we would see MiniMax-M2 Pro be the most requested model, not one of the premier labs.
In fact, we've seen huge uptake in use of MiniMax and StepFun models and xAI models. No single model on any of these days ever exceeded 37% of total requests. I think it's really interesting because I don't think this tells a story of a single winner in the model market. It tells us like there's an ever-expanding ability to use more in different models.
Then there's one other breakdown here that I wanted to show about like, how do folks think about the cost of the model? Thirty-eight percent of those that do switch models often are switching between free and paid models. We offer a set of free models that typically are ones that are just released and the lab is incentivized to grow the user base. Ones that are in stealth, where the lab is collecting data on usage for building the model. Then also some that we just have the ability through our scale to either subsidize ourselves or work with model providers to share the subsidy of. It's interesting here to see that 33% of that time folks are actually doing a paid comparison. There's two different paid models between two different maybe frontier labs. This is theorizing, like checking between which model maybe is more expensive or less expensive to do a certain type of work.
Then, again, we have a huge early adopter benefit, where folks are trying these different free models to see which ones are the best, but at the lowest possible cost. I also think there's a business continuity dimension that I think comes in here and is critical for some of the enterprise customers we're talking from. As the competition between labs is heating up, I think the policy changes, the infighting and acquisition and all those kinds of talks about consolidation are implications that enterprises are worried about. Over this time, we've seen this big decrease in the share of requests from all the frontier labs. Even as their usage grows still, Anthropic's share of our gateway requests dropped from 11.2% to 6.5% in the eight weeks leading up through April. The total users and request volume grew 3x. It's not necessarily that someone's losing. Again, I don't think there's one loser or one winner here.
I think it's showing us this maturing across using multiple models. Decreasing the concentration risk and doing that. The teams that already were doing this experimentation didn't have to worry about this, but the single provider teams were thinking about it more. I think that design implication for all of us is straightforward there. If you're building an AI tooling for an engineering team or another knowledge work team, I don't think you can assume a single model. You have to use some observed behavior of users, understand how they can either work with or around your tooling, and graduate up that ladder.
Then the third failure point, I think is maybe the hardest to fix, and again I've broken it down here based on our rungs to help talk about it. I think this also doesn't really show up until you've solved for the other one, which is, what are the feedback loops? We talked about that feedback speed. Again, I think it applies differently depending on the agents you're building, but for us autocomplete's immediate. Chat is seconds to minutes. Single agents is probably minutes. The loop's getting a little bit longer. Then when we're trying to orchestrate, we're trying to let agents do things on their own, that feedback loop gets a lot longer. I think that creates a pretty key problem for trust building. The whole model of like, if you just use it enough, you'll build trust with it, which is probably pretty critical. It breaks down if the feedback loop's that long because you're not using it that much.
It's that much longer to get that kind of usage experience with it and you can't iterate on the trust as quickly. In the data that we've seen, it creates a specific problem where there's a meaningful percentage of developers that we see who are comfortably sitting in the chat rung and don't really want to move forward. I don't think it's because they failed when they tried to adopt agents. I think it's just when something went wrong, they didn't have enough signal to understand why, and so they just went back to what they were comfortable with. I think the distinction matters because most of our discussions around a failed agentic project deployment talk is on, what did the agent do wrong? How did we design the agent wrong? I think the more useful question in most cases, and I could be wrong, but is like, did we design the right feedback loop?
Was it appropriate for building the trust as we started to give the agents more and more autonomy? The problem often isn't the agent itself, it's like folks had no way to evaluate if it's doing what we want. We don't have intermediate signals to take something that turned into a problem but make it recoverable earlier on. In practice I just think these feedback loops are pretty critical. Your tooling has to surface what agents decided not just what they did. What was the thinking that the agent did? Not just the output. Of course, the output's important and we want to have the right output, but what context they had, what decisions were being made along that feedback loop are really critical. I also think the architecture has to match the rung of our ladder where, what is the amount of autonomy we're giving and how do we match that?
Because again I think it's a trust building experience that has to match this like how much trust are we requiring to give to the agent in order to get what it wants to get done. I think we don't want folks micromanaging the agents, but if we don't have the right signals, what I see is like the developer watches the agent work for a little bit, decides that they don't know what's going on and so they just stop it and restart or take over and do it themselves. Because they don't have the right signals to know, did I give it the right context? Is it walking down the right road? Those intermediate signals are really critical and replace that anxiety with you being able to build the trust.
Trust Is Measurable
I've been using that word trust, and I think it's a mushy word to use in an engineering talk. I do think it's measurable. I promise that I would talk a little bit about how I think we can measure it. I think the measurements then matter for how you design these feedback loops we were talking about. There's a couple of things we use that I'll give you as proxies, again that are specific to the AI agent use case, but hopefully gives you some idea. One is, at autocomplete rung, we look at the override rate. What percentage of suggestions get rejected? Then, can we tell if that's model related? Is it context related? Is it folks that just haven't empirically learned that that judgment is better than the suggestion? Is it bad pattern matching? That maybe is recoverable, but if you don't measure this then we're not going to be able to trace it back.
At the chat level we look at session depth and follow-up rate. Developers stop asking for follow-ups and will reject it or treat their first response as final. That can be a trust deficit sign for us. They have decided that's not worth iterating. This is not right. I'm not getting that value. For agents we look at session completion rate. Did the developer let the task run to completion? Did they interrupt it? What is the interruption rate for an agent session? That's like a proxy for us for that trust then. It's hard to distinguish like the developer correctly interrupted, it's like a bad run where it was walking down from the developer just got anxious and took over. It gives us signal about that feedback loop design. Have we exposed enough of the feedback loop to the developer to tell them when they need to intervene and when they don't?
Again, it's very asymmetric, this trust. Many positive experiences are required to build a meaningful amount of trust. Only one bad experience can really destroy that. We talked about that in the last slide. I think this also has a social dimension that is also underweighted in most deployment strategies for agents. What I think that looks like is, yes, it's often a social failure rather than a technical failure. One highly visible failure of an agent in front of the wrong audience, in our example would be a senior engineer who's already very skeptical of AI, or in a high stakes moment with an audience like if I was live demoing on stage, which is why I'm not, can create this narrative inside of an org that is going to take a very long time to reverse. If we read the story about AI wrecking prod or overwriting a database, that's going to spread a lot faster than like, yes, actually I feel like I've saved probably three weeks' worth of work over the last week with AI.
I think the counterintuitive implication there is you have to have the right rollout strategy. The right rollout strategy is probably not just give everyone access and let them figure it out on their own. I think the optimal approach optimizes for this coverage to build trust and have a trust building time. Start with developers who are the most patient with ambiguity. Moving to a more, they're motivated to iterate. I talked early on about our dataset being focused on those early adopters. I think that's how organizations have to think about adoption too. It's like you're going to have a set of folks who can build that trust and iterate and figure out the right way to build social proof for the adoption from skeptics. Again, depending on where you are in that rung of the ladder, the stakes are different. Unlike most technology, AI failures are very visible. It's like if you think about Waymo or other self-driving, it's like you hear about every self-driving accident that exists versus you don't hear about every single human driving accident that exists. Those visible agent failures make it easier for a team to remember that rather than the things that are actually working.
Then, I also think as you're building that like, do developers actually, for us, build trust over time? This is when we're looking at retention. Do developers move up that ladder? We ask questions all the time in our product teams about like, if we're looking at developers who used our autocomplete in the last 30 days, what have moved to chat? How many of those have opened our agent manager to orchestrate multiple agents? Those kinds of questions help us see it more as a continuum of adoption rather than just, here's a token leaderboard. Who's winning the token leaderboard? Then, lastly, this data that we have, how does it help us build? Again, hopefully that can help you think about how you're building your systems. That's treating context as infrastructure, not as a feature. We spend a lot of time thinking about the context that gets constructed. I think the well-designed retrieval system for getting the right context is the number one thing to get right as you're building up.
Two is we've designed for provider independence from the beginning. Maybe it's pretty clear to you hopefully that I think we're going to be in a multi-modal environment by default. That pattern is accelerating with the usage intensity. I think if your infrastructure assumes like a single provider or a single model, you're going to constrain your most sophisticated users from really using it, and going to create incentives for them to route around your tooling instead of using it. That's definitely what we've seen. The third is again, matching the right feedback loop to the right level of adoption. That's the design intervention that I think reliably gets you to understand what does this transition between adoption look like. I think those are the really critical things to consider.
Three Questions to Take Back
Then, lastly, I want to come back to where we started, which is, most of what gets written about AI coding tools today is about those capabilities, the benchmarking of what it can do, a great demo video. I think that has its place. Of course, that's critical for adoption. What we've learned over these 40 trillion tokens we've seen, is that adoption really depends on if the system can earn trust. That trust is a problem that we can engineer against. We walk through those failure points. I think they show up in usage data before they show up in user complaints. That makes them more predictable and solvable. If we look back at the developers that are real power users, it's not simply because they have access to better AI. It's because they've become better at choosing the right model for the right problem. They've solved these failure points.
They've built the trust at each rung, and really climbed that ladder of trust. That I think is the signal to really look for as you're building agentic toolings. I think that's all the kinds of things that evals and benchmarks, as much as I love them, won't tell you. It really takes a focus on those levels of adoptions to actually understand if you're being successful in what you're trying to do.
Questions and Answers
Participant 1: You mentioned the importance of working with multiple models from multiple labs. What about the different agent harnesses essentially, and what different agent providers have built in them and the different capabilities in each of them? Should we be using multiple?
Brendan O'Leary: It's funny, somebody asked me that, obviously I should just answer, no, you just have to use Kilo. I said, I wish that the harness was the answer, because then I could just sell everyone in the world my harness. I do think that the model matters a lot more than you think. I would encourage you to try multiple harnesses. Of course, this is my bias, I think the harness needs to give you freedom to try multiple models too. It's that combination of harness and model that matters the most more than one harness or one model. I think that the harness mattered a lot more even six, nine months ago, because the agentic nature of the model was a little still being proven. Early on or at the end of last year and early this year we're talking about how good is this model at tool calling.
I think that's now table stakes, that most of the models that come out of either a frontier lab or the large open weight model labs can all handle tool calling relatively well, which gives the harness less ability to differentiate on that specifically. The harness differentiation now is where it meets you. What's the surface that you want the harness to live in? You should choose the one that you're most comfortable with. Hopefully one that lets you experiment with models.
Participant 2: What area of a system do you find the most success with people adopting agentic vibe coding? Let's say that like, go build me Google versus go build me a web page versus maybe even something simpler of building an API route. I'm curious if you're seeing signal on certain teams or departments or areas in engineering that are getting more success than others.
Brendan O'Leary: I don't have a lot of data about this in this presentation but I've written some on our blog about it with some of our data. I found it interesting that for us internally and then for our customers it's some of the most senior engineers that have gotten the most value out of it. I have another talk that I call, From Vibe Coding to Agentic Engineering. I think that our senior engineers who really understand systems thinking and have a deep understanding of the systems we've built are able to do more with agents because they're able to know that I'm giving them the right context, that I can trust them to go do something versus the go build me Google, make no mistakes, is obviously not going to be successful.
Participant 2: A hundred percent I see that too. Do you even push that further? You're not seeing the signal of certain areas of systems that are more vibe coding friendly versus others, or anything like that?
Brendan O'Leary: I think there's been a huge focus on frontend from the model providers because frontend was not a great place. I think they've done a great job there. Dex Horthy has this great statement of like, you still can't outsource the thinking. I think if you have a well-established system, you can vibe against it really well if you already have the right guardrails in place. If I said APIs or REST APIs, I don't think that's a great answer because if you just give it whatever you want it's going to be really messy, versus if you have a really well-established pattern of APIs, it's going to be great at like, I need this next thing. It's going to be like this but a little bit different like this. It's going to be really good at following those patterns.
Participant 2: Piggybacking off of that, are you seeing any patterns when it comes to popular sub-agents, so like UX review, or QA? There are so many ways to slice and dice your sub-agents with coding. I'm curious if you're seeing just even one or two that stand out, like you are gun to head, I only got two agents that ride or die with me. Who are you bringing with you?
Brendan O'Leary: I think the models have gotten a lot better at reviewing even their own code. Of course, they don't have any real bias towards their code, so we added a local review agent at the end of a turn. I used to not use that at all and now I use it almost every time. I think that reviewing the code for a given goal is something that the models are getting really good at. The thousands of other skills and agents that exist, I don't know. I don't know that there's a standout there. I think reviewing code for intent is something that they're getting really good at doing.
Participant 3: What I had observed while also using OpenRouter is that depending on what provider you choose for a specific model, the performance changed, and that, of course, how their inference settings were set up for their vLLMs and all of that stuff. Is that showing up somehow and is that an angle that gets lost in some of the metrics that you are looking at? How are you compensating for that?
Brendan O'Leary: OpenRouter which is one of the backends that we use, it also has multiple providers. If you're talking about especially an open weight model, you can end up with very different implementations of the same model. Normally it's not something I think most people care about, but you care about it, so it's good. It's almost all we think about. What we've found is that people just expect it to work. What we end up doing is building a lot of filtering there in order to make it work the way you expect. Because there's a lot of things that once you start getting in an LLM, like how well are you doing caching, like all of that stuff then starts really mattering for the frontend performance to the developer and of course to anybody using that. That's then when we end up spending a lot of our time in thinking.
See more presentations with transcripts
