Transcript
Aleksandar Mitic: I'm Alex, this is Jo. This is our talk, "Rewriting All of Spotify's Codebase All the Time". We're here to tell you about our background coding agent that helps us do exactly this. Everyone meet Honk. Honk helps us rewrite all of our code. Honk is already much more famous than I'll ever be. It's been mentioned in the billboard, papers. We'll tell you the story about how this started, what we learned developing it, and where we are now.
Solving The Maintenance Problem
We'll rewind time back slightly more than a year ago, beginning of last year. This journey started with us trying to solve what we call the maintenance problem. I'll explain in more detail what this is, but before that, a statement. Developers code for more than one hour per day. Do you think this is true? Developers on average code for more than one hour per day. I'm sorry to say you're all wrong. We see in data from the industry, but also internally, that on average, developers spend less than one hour writing code. This number might be slightly different now. With all the tools we have, it's gotten even more fun for people to code. The point still stands, you don't spend your full workday writing code. You go to meetings, and you do all the other things you have to do. If you've done software for a while, you know you can't just make features.
You have to maintain your codebase. In practice, that one hour is full of bumping dependencies, migrating to the latest Java version. Your platform org is pushing a new logging framework that you need to migrate to, and this is basically an endless list of tasks. This never stops. You have to do this. Otherwise, your software won't work after some time. This is what we call the maintenance problem. This is what we try to solve. This is nothing new. This has been here since the dawn of software. At Spotify, we've been thinking about this for a very long time. We have this thing called fleet management which is many things, but, mostly, it's a way of thinking. It's about us bringing our fleet with us as we make changes. If I am the owner of a library, it is my responsibility to make sure that all of Spotify gets to use the latest and greatest version of this library.
What does this mean in practice? Let's say I'm the owner of this chat library, and I've released a version 3. It's my job to make sure everyone is on version 3. How do I do that? I write a little script that, given a codebase, checks if it's using the chat library and bumps the version. What our fleet management system allows us to do is to specify this migration first by saying what we are targeting. This is a Java library. We're targeting all Java components at Spotify, which is thousands of codebases. Then we specify the transformation we want to run. In our case, this is our script. What our fleet management system does is it picks up this configuration. For each target, it starts a Kubernetes job. Each job clones the repository, runs the transformation, and opens a pull request. This pull request is then sent to the code owners for review and to merge.
In some cases, where we can fully reason about the change, or there's some way to test it, we can even auto-merge, so no one has to see this happen. This works quite well. This graph shows you how many days it takes for 70% of our fleet to adopt the latest version of our internal service framework, and you see how this changes over time. Before we had fleet management, it would take us almost a year. If we release version 3, we need almost a year for 70% of Spotify to use it. With fleet management in place, we're down to just under a week. This is all before LLMs. This is amazing and super helpful. When I look at this graph, I wonder, why not 100%? Isn't fleet management about all of Spotify's codebase? Yes, of course it is. In practice, that last 30% is quite hard.
That's because the long tail is complex. When we run our script on that 30%, what happens is we see an error. It turns out, in version 3, we removed one method that we didn't know was used by some parts of our codebase, and our script, it can't handle this. This is fixable. We add some code to replace post with send message, that new function. In practice, it's not this simple. We don't want to replace every post, just the ones from our library. Our script starts looking like this. We're parsing the abstract syntax tree, and it's getting more complicated. At this point, there's only one person that truly understands how this works, but it works, so we run it, and then this happens. Turns out, we made some performance optimizations in version 3, and those don't click well with some part of that 30%. What do we do now? As platform teams, we say, this is fine. Let's keep both methods. We migrated most of it, we proclaim success. Migration done.
The Birth of Honk (AI Migration Tool)
Now we have a problem. Now we have two methods, and we have a diverse codebase which brings a diverse set of problems. Every time we interact with the chat library, either with a new migration or building features on top of it, we need to keep both methods in mind. It's at this time where we see LLMs start getting good at writing code. They've been writing code for a while, but at this point, we're getting that aha moment. We start thinking, what if we replace our script with an LLM? They're more versatile. They can handle these edge cases. At this point, Honk is born. That's a simple idea, let's just replace the script with an LLM. I wish it was that simple. In practice, I want you to remember this is more than a year ago. This is before Claude Code was a thing. This was when we were doing tab completion.
A lot of writing code with LLMs was using a chat UI, prompting something, getting code back, running that. Maybe you get an error, shove the error back to the chat UI, and you go back and forth. This works, and it's cool, but there's no way we migrate the hundreds of chat library use cases doing this. Thinking about this, what is it we're actually doing here? This is the normal software development process. Given a set of requirements, we write some code. We build. We test. We iterate. It just happens to be so that the code is generated by an LLM, but this process needs to happen. We realize that to get Honk to rewrite all of our codebase, we somehow need to package this nicely.
Jo Kelly Fenton: This is where Honk starts to come into existence. Sure, we have the LLMs that can write the code, but the building and testing is where it gets particularly tricky. Claude is pretty good at verifying your code now, but the tools that we had at the time did not do this. When you think about running Honk on a single repository, it's pretty easy to build and test. You know exactly what to run. When you start running it on an entire fleet, that becomes infinitely more complex, because we use different build systems for different repositories, so just the tools that we use are different, the commands that we run are different. How do we generalize this so that we can run it on everything? Our first thing we have to do is to create this verify tool, that the LLM can call on any codebase and it will verify the code by building and testing it.
This is a single entry point that fans out to many verifiers under the hood. We have a Maven verifier, a Yarn verifier, a Bazel verifier. We even allow people to write their own verification scripts. Like, maybe you're running a migration and you have some particular logic that you want to verify, doesn't or does happen, so you can write your script and you can plug it into the verify tool. Then you can put it in a loop with the LLM and it starts getting this good feedback. This is the very basic genesis of Honk. We make it a CLI. We start to run our little chat library migration, maybe on one or two repos, and then we start to unveil more and more of the problems. The first problem is this. This is the most disgusting thing you're going to see all day. This is a Maven build output and this is what we see all the time as developers.
It's a lot of text and a lot of it is very unimportant. If we fed this failure back into the LLM directly, it would just fall over. Really, all that we care about is this tiny little thing over here. The chat library was not correctly migrated. Now we have to worry about actually extracting the exceptions from the build output.
We try being smart. We wrote a lot of scripts to try and do this. For Maven, it's simple because there's some standardization in the build outputs, but a lot of build systems have very unstandard outputs, so it gets very complicated very quickly. At some point we just went back to basics. We just put it in an LLM. We asked the LLM to summarize it for us. This actually worked really well, because this is fundamentally what LLMs are really good at. They're very good at summarizing text. We haven't really looked back from this. At this point, we can run Honk. It can summarize the failures, pass it back to the agent, operate like this in a circle, and produce some code, which is good. This is looking pretty good. We start to see this. We realize that the thing that Honk is getting good at is not necessarily running a migration, but it's getting the build to pass.
This is the easiest thing to get the build to pass. Agents these days, they're getting smarter. You see less and less of this, but we saw a lot of this to start with. It's a little cheeky. We also see things like this. As the agents got smarter, it would still do things like this. This is it downgrading a Java version in order to get something to work, because that was just the easiest way to get the build to pass. What we did to try to tackle this problem is we introduced an LLM-as-a-judge. It was pretty simple. It would take the initial prompt, the generated code from the LLM, and it would put it into another LLM to then evaluate whether the code addresses the prompt. It would come back with a verdict, a pass or fail. If it failed, it would block the migration from completing until the agent fixed the issue.
For example, the judge would say removing a test was not part of the original requirements, and then the agent would be forced to then uncomment the test and actually fix the underlying issue, which is pretty good. Then we started to see some issues with our judge, which is that, take this example, not all repos that we're migrating used the post method. Sometimes the generated code does not include a migration from post to send message, but the judge is like, the post method wasn't migrated to send message, and so it blocks the migration from continuing, but the LLM did the right thing. Our judge was simple, but in a bad way. As we found the models getting better and better, we saw less and less need for an LLM-as-a-judge. If you just put verification steps within your prompt, the LLMs are getting really good at doing that themselves. We ended up actually removing the LLM-as-a-judge eventually, and we haven't really looked back. It's not that they're inherently bad. There's probably a place for them. We haven't really found the need for them. The code and the tests are really good at evaluating the correctness.
Fleet Management Tooling
At this point though, we've migrated our tests. This is our generated code. We test our Honk prompt on a couple of repos locally, and then we decide, we're going to roll this out with our fleet management tooling. This is where we hit arguably the biggest problem, which is that when we roll it out across hundreds of repositories, we start seeing hundreds of failures. We start seeing missing permissions, failed to load Docker. Then we start to dig into these failures, and we realize that these are actually nothing to do with the generated code, but they're very much to do with the infrastructure around the code. The integration tests of all of these repos, they require specific permissions to run. They require Docker to be loaded. We realized that previously we were running Honk on our laptop, and now we're running it on a Linux VM, which is inherently very different.
Our laptop has all of our dev permissions set up. It runs a CLI directly. Now we're running in a container that has a service account. Sure, we could give our service account permissions to run the integration tests, but that is a lot of permissions that we would have to grant in order to get the integration tests to run for all of our services. We could look into supporting Docker in Docker, but our infra is not set up to do this very nicely. What do we do? We skip the integration tests, and now Honk can continue with its migration, because it's not blocked on these infra errors. That's not great, obviously. The integration tests are very critical in validating the correctness of your code, but we can move forward with our migration, so it's good enough for now.
Then we hit an even bigger snag, which is that we can run iOS builds on our laptop, but we cannot run them on a Linux machine. We wanted to start making migrations in our app. It's a very important codebase. We basically couldn't validate any of our changes that we made to the app, and then this list kept going with these differences between these pods and the local developer experience. We had to take stock as a team. We couldn't keep skipping tests. We realized like, what is the thing that makes Honk really good? It's this automated verification. The more dimensions of correctness that we can automatically verify, the more correct the code will be. Instead of minimizing it, we need to focus on ways that we can maximize it. There's actually a system that's already purpose-built to run your build and to run your tests, and it's your CI system.
We know that works for all of your repos. We had this valuable lesson, which was like, we're trying to recreate our CI system on our Linux VM, which is our agent runtime, but actually, if we could separate out our verification runtime from our agent runtime, then we could leverage the pre-existing CI system that's already set up. I'll take you through a little bit of our architecture of how we do this. We still have our pod, and we have our agent harness running inside the pod. We still have our verify tool, so the LLM calls out to that. We still try to do what we can locally, because that's quick, it's cheap, it's efficient. At some point before we create a PR, we want to do a full end-to-end check that the code is actually correct. How we do that is we push a branch to GitHub, and then we kick off a build.
We created a service called a verification service, which abstracts several of our CI systems. We've got a few at Spotify. It starts the build and waits for it to be finished. Then when it's finished, it'll summarize any failures and return it back to the verify tool. If it's incorrect, the agent goes again, but if it's correct, it'll create a PR. What's really nice about this is that when you have hundreds of PRs being generated in parallel with a migration, you know that the PRs are going to have a correct CI build, because you've already validated it before you'd even created the PR. We don't want to create garbage PRs, and this is kind of this system that allows us to assert that they're going to be correct. This is kind of like, it's simple, but it's effective. At this point, Honk is a really good AI migrations tool.
Honk's Integration with Slack
Aleksandar Mitic: Now we have a great migration tool, and we see this in our metrics. This shows you the merged PRs. As platform people, we start using this immediately. All of a sudden, we're running migrations we just couldn't do before. Super exciting. What happens then is we have a Hack Week at Spotify. This is a week where everyone is allowed to explore and build whatever they want. Many of the features you use in the Spotify app came from a process like this. One engineer, he exposes Honk over Slack. Instead of using our fleet management migration tooling, you just talk to it on Slack and have it make an ad hoc change. As platform people, we're like, this is cool, back to migrations. Time goes on. We keep using Honk, but we start being curious about the Slack integration. Not just us, but basically every developer at Spotify.
Then this happens, we see the line sharply going up. At this point, there are more PRs happening with Honk, and we start thinking, what is it that people actually want? There is a need for migration tooling, but there is something more here. We realize that what people don't want to do is to leave the surface where work was planned in order to act on it. Let me give you an example. Let's say I'm debugging my Honk service which has a problem. What do I do? I ping my colleague on Slack. You go like, hey, Jo, did you see that Honk service has high error rates? Jo links our dashboard, and she sees that error started 20 minutes ago. I look at the logs. I find a suspicious stack trace. I post it here. This jogs Jo's memory, and she recognizes that a few months ago, we saw a similar error.
She had created a Jira ticket with a bunch more context on how to fix this error. At this point, in this Slack thread, we have everything we need to fix this problem. Why should I have to go somewhere else to fix this? If we have LLMs that can write good code, given all the context being there, I don't have to. I just ping Honk, fix this. I chat with Honk. It gives me a plan. We go back and forth. The plan looks good. I let it go ahead, and, after a moment, it gives me back a PR. At this point, we realize that what people need here is a background coding agent that allows us to go from prompt to PR from any surface.
If you look at this architecture Jo showed us, this is tailored for our migration's use case. What needs to change to enable work from any surface? We touched on a key point during that Slack thread where some of the context was written messages, but some was in a Jira ticket. Some of that is in a monitoring dashboard. The agent needs to be able to access all of this. The agent, when it works, needs to have access to all the sources where we have context. This might be logs, monitoring, Jira, whatever. Also, any surface truly means any surface, not just Slack. Maybe you, in a PR comment, want to trigger a task, or from Jira. We expose this architecture with an API. We build the integrations we think are most useful, but we also let all developers at Spotify integrate with it and build the workflows they think are necessary.
This is where we are now. The result of this is that when Jo and I presented a version of this talk, this was still a migration tool. We were so happy to announce that we had 1,000 merged PRs in 3 months. We were over the top. Today, we do this in 10 days, and this number keeps going up. What is the result of this? It's a bunch of PRs to review. This is becoming the new bottleneck.
The New Bottleneck - PR Review
I'm going to take you on a slight detour here. This is Lisanne Bainbridge, she's a cognitive psychologist. In the '80s, she wrote this paper titled, "The Ironies of Automation". I highly recommend you read it. It's four pages. It touches on what happens when you try to automate an industrial process like power plants or aviation. There's a bunch of good lessons here. One that sticks out to me, and this is slightly paraphrased, is that, when we automate the process, we tend to leave the hardest tasks for ourselves. This is because the hardest tasks are the hardest to automate. If we think about aviation as an example, airplanes can basically fly themselves, not fully, but a lot of it, yet we still need pilots. If we didn't have pilots, many of us wouldn't be here today. What do pilots do? They plan the route. They communicate. They might navigate smaller emergencies.
They fly some parts of the flight. At the end here, interestingly, they monitor the flight systems, and they say, this is safe, or this is not safe. In my view, that is the hardest job. Looking at all these measurements, all the observations, and being sure that this is going the right way. This is where we're getting now. We have all of this code to do, and the PR is only valuable once it gets merged and into production. We have a bunch of this work to do now, the hardest part, looking at a PR and saying this is safe to merge or not. This is the new bottleneck. I would be lying if I said we've solved all of this. We haven't. This is a tough problem. We're definitely thinking about it, and there are some things we're doing to make this better. One, this is quite simple, but it is important.
It is a culture shift. We've always had a review process, everyone has, but it is becoming increasingly more important. If I make a PR, I might ping Jo on Slack, and Jo reviews it, and that's it. Now we need to have some expectations on simple things like review timeframes. When can I expect my PR to get reviewed? If this is the new bottleneck, we also need to be structured about it. If there's a bunch of PRs that need to be reviewed, we need to understand which ones shouldn't be reviewed. Stale PRs get closed. We will have to try again. Interestingly, in some cases, a driver of migration can approve the PR themselves sometimes. Why is that? If I am the expert of the chat library, I know best if it's correct or not. Maybe I am the best person suited to review and say this is fine.
Of course, the receiver of the PR needs to see it and be aware that is happening, but I should decide if this is ok or not. Second, we need better tooling. We will all have to review a lot, so let's make sure we have good tooling. Again, some of this is simple and obvious. We need a PR inbox. Before this, I might get some PRs on Slack and some on mail, and some in GHE itself, but I need one place I can go that tells me which PRs I should prioritize, the one Jo sent me today, or the one PR from last week that I forgot to review. Maybe there's a PR I don't even know about because I haven't been assigned it, that I am the best suited to review. If someone is migrating to the chat library in a part of the company that I'm not, maybe I should review that.
Some tooling is more complex. We started this talk by saying that we auto-merge some PRs, even before LLMs. Now, with LLMs, we can write even more complex code and better code. Naturally, we should be able to auto-merge more PRs. The question is which? This is a hard question, and it's not fully answered. We're starting to think about this. Maybe there are some heuristics. Maybe documentation changes can be auto-merged. Maybe internal systems that, if they go down, we just revert, not the end of the world. There's something we need to figure out here, because, surely, there's more things we can offer.
The Biggest Tool - Standardization
Jo Kelly Fenton: This is arguably the biggest tool that we are leaning on. It sounds simple, standardization. If we think about why reviewing all of these PRs is actually hard, there's a lot of PRs, and that takes time. It's the complexity of each PR. Where does the complexity come from? We're back to this old chestnut. A diverse codebase brings a diverse set of problems. What was hurting us before LLMs is still hurting us today. Remember all of those migrations that we only completed to 70%? We introduced two ways to do the same thing. The one that's on the 70% and the one that's on the 30%. Our prompts don't actually look like this. They look like this. This is a lot of complexity here. Now when I review, there's a lot of edge cases, like, if this, do this. We have to think about a lot of different things when we craft a prompt that will run a migration across our services.
If we think about what is complicated about reviewing a PR with this much of a prompt is that we have to then consider all of the context around the generated code when we actually come to review it. We have to understand whether the agent actually parsed this correctly and executed it, which is really hard. If we go back to our chat library, we can take a look at what this looks like in practice and why our prompts are so long. Remember, we have to migrate from post to send message. Then send message had some performance changes. There was something that I didn't bring up, which was that version 3 of the chat library has a minimum Java 25 version requirement. This is where we start to come into contact with one of our unfinished migrations, the Java 25 migration. We did it to 80% of the fleet, but God knows what the other 20% is on.
They were really hard to migrate. We just had to move on to the next migration. Now when we migrate the chat library, we have to think about like, is this repository using Java 25 or not? If it's not, if it's using 11, we have to do this and that. If it's using 17, we have to do this and that, 21. Your prompt is basically filled with these if x, do this, if y, do that. That's how it ends up like this.
What do we do? We tackle this very explicitly. We want to get rid of this long tail. We want to get rid of the 20% of our codebase that looks and behaves differently to the rest of our codebase. If we do that, our code becomes extremely predictable. It becomes much easier to write our prompts, and also to review our code. This is very hard. It's so easy to say standardized. Spotify is about to be 20 years old. We got a lot of services. We got a lot of engineers. How on earth do we standardize, and why haven't we done it already if it's so easy? The thing is, is we finally believe that we're at this inflection point with our AI tooling that we actually have the tools to execute this now. What are the steps that we're taking? The first one, this is the hardest.
This is defining your standards, because you've never met a more passionate group of people than a bunch of engineers discussing the pros and cons of dependency injection frameworks. I'm pro. Alex is against. It's a massive conversation contention. If I post this on Slack, I come back after lunch and I have like 342 replies. Everyone at the company has an opinion. They got to voice their opinion. If you leave it up to the engineers to determine what the golden technology is, you will literally never get it done. It's just impossible. We take it out of the control of the engineers and we put it into these advisory boards that we set up. The advisory boards exist in all of our major disciplines, and they're comprised of five or six of the most senior engineers at the company. Hopefully, they can come up with some consensus. They choose the golden technology and then they publicize that to the company.
Next, we have to standardize our codebase based on these standards. This is where Honk comes back. We have a lot of new golden tech. We have a lot of codebase to migrate onto that golden tech. We can use tools like Honk to actually drive our migrations all the way to completion now, which is great. How do we pick which migrations to start with? Because there's a lot of them now. We pick migrations that remove this long tail. We pick migrations that make other migrations simpler and easier. For example, we could tackle the Java migration. Once we do that, once everyone's on 25, then migrating the chat library is now a lot easier. Once we standardize our entire fleet, we have to keep enforcing these standards. How do we make sure that new code that enters our codebase does not have these kinds of quirks that doesn't use the golden technology?
We do this quite explicitly through the use of monorepos now. Spotify had a massive migration to get all of our codebase into monorepos. When your code is centralized, it becomes a lot easier to observe. It's much easier to have stricter enforcements of standards. For example, in a monorepo, it's very easy to enforce one version of every dependency. Now your code is a lot more predictable. You can make a lot of assumptions about that because everyone is using the same dependency version. When you upgrade, you upgrade everyone at once. You have to do the migration 100%. Then, we actually introduced a lot of linting. The linting asserts that you're using the golden tech. If you try to use the dependency injection framework that's not allowed, you'll get a linting failure. You won't even be able to push that code into production. We keep our code very nice.
Then you also get the benefit of having a centralized set of skills and tools. Everyone using the monorepo now can use the same skills for writing their code, and that means that all the new code that our agents create is effectively going to look and behave quite similarly. Using this strategy, we are reducing the long tail. It's happening.
Is this how we're rewriting Spotify's codebase all the time? We think so because there's a powerful thing that starts to happen when you standardize. When you focus on aggressively standardizing your codebase, we know that our agents and our Honk can generate more correct code. Because it's easier for agents to generate the correct code and it's also easier to maximize our automated verification when the code is more predictable. When Honk generates more correct code and our codebase is more predictable from the standardization, our reviewing and planning gets much easier. This is our bottleneck. When we can review and plan more code, we can write more code. When we have more capacity to write more code, we can then further standardize our codebase. As this loop starts to build momentum, we become faster and faster at generating new code into production. We really believe that we are now rewriting Spotify's codebase all the time using this.
History of Honk
Luu: All I heard is about Honk. You guys created Honk. What's the history of Honk? You guys still able to talk about that or not yet? I was just curious about the naming of Honk. It's a very catchy name.
Aleksandar Mitic: Yes, there's a big bird thing going on right now. A lot of things are getting bird names.
Questions and Answers
Participant 1: I was wondering, in a culture where you enforce aggressive standardization, how do we deal with innovation? Because that would by definition not follow the standards.
Aleksandar Mitic: I understand your point, but I think there's a middle ground there where if something goes into critical systems, critical Spotify features, it's good that it follows the standards. That doesn't exclude innovation. You are free to try new things and you can have experimental components and writing new code. If you're able to show that that works, it's a very open culture in that sense. It's not about hindering innovation. It's about making sure the hot path looks and behaves the same way.
Jo Kelly Fenton: In a way, I think that it actually enables innovation because you can write code so much quicker now. You can leverage agents to now build services and whole features super quickly. Your innovation, maybe it doesn't look like trying a particular framework, but it looks like creating different features and services and flows, and things like that. I think personally it's enabled my innovation at least.
Participant 2: The thing for me that was quite interesting though, is you guys went away from a deterministic way of doing migrations. It feels like if you've got rid of the long tail, you could just go back to that. Presumably, Honk would be best pointed at the bits that fail that deterministic migration. Is that something that you guys do or is that on the roadmap, I suppose, in the future?
Jo Kelly Fenton: I think that definitely like if everything looks the same, it would be a lot easier to have the deterministic scripts. You could even use an LLM to create a fairly complicated deterministic script. I think it's possible. We might try it out at some point. We've talked about it. I think agents are just getting so good at writing code though, aren't they? Deterministic is much cheaper. That's a benefit.
Participant 2: One of the things we've done for migrations is write a test to ensure the migration has happened. Then like that TDD loop and agents are great at picking up all the issues for that. Is that something you guys do as well, or it is just, here's the prompt? Because it just felt like there was a lot of if statements in the prompt that could just be deterministic.
Jo Kelly Fenton: A part of our verification system initially was, you could plug in your own deterministic verification script that would allow you to have very migration specific checks. Like, don't update this file, or maybe you run some tests or something like that. I think that that's a great approach. We've used something similar previously.
Aleksandar Mitic: I think what has happened as well is like, what we consider a migration, we are happy to take on, has broadened much more. I think many companies have no breaking APIs policies, and so on. This is starting to get pushed now, where we're like, maybe we can do this. We try even harder things that we wouldn't even think of deterministically migrating before. I think it's a combination of both things.
Participant 3: You said you moved to monorepo and you glossed over that. You've talked about thousands of projects and lots of code, that must have been a massive task in itself. Obviously, that has its own cons in terms of build time and cognitive load and merging and pull request issues, all of that stuff. How did you mitigate any of that?
Aleksandar Mitic: An important piece of context there is that since we have these disciplines, like data and client and backend, it was not as if we went everyone into monorepos already. Some disciplines already had monorepos. For example, a client discipline had been running with a big monorepo for a long time. We had a lot of learnings that we could take from those past monorepo migrations. This is mostly now bringing the other disciplines that have lagged along. There is definitely learnings. I'm sure Spotify will share more learnings from those journeys, definitely.
Weakly: You talked about a lot on the quality frameworks and the ways you figured out how to get the AI from being cleverly stupid. Were there any things that stood out to you as really surprising how effective it was or really you didn't expect it, but you implemented this one little quality thing and this one little check thing, and somehow it was really impactful. Does anything come to mind?
Aleksandar Mitic: As you said, one thing that comes to mind is the examples Jo was going through, are cases where the agent did the wrong thing. What is interesting to me is in some cases, it may look like the agent is doing the wrong thing. Then once I dive into the codebase, I realized that my understanding of the problem was lacking. It's like this was actually impossible to do given the constraints we had. It's like there was some library missing or something, and the agent was basically hallucinating things. A small thing that has helped us there is gathering these insights quicker. Vibe coding simple tooling that allows us to understand, what did the model try to do here? Why did they do these things? This has helped us understand how these migrations are supposed to get done quicker. It's a half answer, but I think that is definitely one thing that surprised me how in many of these migrations, our understanding of the migration has been incomplete as well.
Jo Kelly Fenton: I feel like previously, like all of my software engineering experience before working with agents was like, to monitor a system, you'd have very deterministic checks, you'd have metrics and things like that. Now as we monitor Honk and to see how effective Honk is, you have to pause these huge LLM conversations that Honk has. It becomes quite complex to monitor. I think what you said, like having another agent pause the logs and be like, how could we improve this prompt so it gets to the solution much quicker, is actually really effective and it's super easy.
Participant 4: Have you tried some modern instruments like Claude Code or Cursor? I understand that you started this Honk a long time ago, as you said before there, but have you tried now, nowadays, some new instruments? How do you mitigate the problem that when developers start being lazy and just giving the tasks through the Slack channel to the AI, they essentially lost the understanding of the codebase. Imagine in a couple of years, nobody will actually know what's happening there. How do you mitigate that?
Aleksandar Mitic: On the first question, as you pointed out, one downside of us starting early was that none of the good tools we have now existed. The upside is that we were able to build in a way that allows us to plug and play. Where we are today, we're using all the things you can imagine under the hood. It's just that when we started, these didn't exist.
On the second question, hard to answer and hard to say.
Jo Kelly Fenton: We're still reviewing the PRs there. It's not like we're pushing straight to production. Like if you call it on Slack, you are reviewing that PR. It's going through multiple reviews. You're still aware of the changes that it's making.
Aleksandar Mitic: I think there's also like a time and place for both. If I'm building an internal tool for myself to test something for my team, maybe I don't need to know every single line of the code. There are certain places where I need to. I think people are generally good at making this distinction, when to apply different skills.
Participant 5: When you mentioned insights and tracking things, I was curious about the cost aspect of it. You mentioned thousands of PRs, agents, monitoring agents. For me, that sounded like a lot of tokens. Did you have to do some sort of analysis to, ok, if we're doing it with maybe static analysis, and not using LLMs, this is how much it costs versus now this is how much it costs. It justifies with the benefit that it has.
Aleksandar Mitic: I understand this varies between companies and how willing you are to do these things. We definitely monitor the situation, how things are going. In our particular scenario, the costs are definitely manageable. It's not something we are bothered by right now. We're seeing these tools. Sometimes it's hard to make that comparison you're suggesting. The chat library is an example here. We're definitely doing these migrations, which we just could not do before. There is no equivalent comparison. There was no AST tool we could use to do them. The value of completing these things is often so much larger than the token cost that you spend doing it.
Participant 6: How do you deal with the traffic control of if someone in Slack can put in a symptom by ask that can trigger Honk, how can they see what the blast radius of, it's gone away and start to do this work and start to affect thousands of parts of your system, and they're disconnected from seeing what's happening?
Jo Kelly Fenton: Definitely, like I mentioned before, they will review the code. You still have to have a code owner of the codebase that the changes in review that. They are aware of the kind of impact that they're having. I do understand like in Slack now, one of the cool things about Honk is that it's democratized coding in a lot of ways. Like you are seeing some maybe non-technical people start to make code changes. Like I said, we're still having that final quality assurance gate at the end that our engineers do to understand the blast radius. We leverage skills and things like that to try to guide non-technical users to make appropriate changes. We try to make it hard to make bad code changes through the use of things like skills and context.
Aleksandar Mitic: In both our fleet management tooling and on Slack, you are always prompted before, and it asks you like, I'll do this on a thousand repos? You have that feedback before. In our fleet management tooling, this is not driven by an LLM. You target and say, all Java components, and then it shows you that this is 2,000. There's always someone making that conscious decision. The same on Slack where you get prompted before saying, this will impact this many things.
See more presentations with transcripts
