Transcript
Andrew Swerdlow: I'm Andrew Swerdlow. I currently work at Roblox. I support the engineering acceleration team as well as the core services and platforms group. I'm a manager. Before this I was at Instagram for a couple of years and I was supporting safety, AI fairness, team mental well-being and a few other contentious people areas. Then before that I was at Google for my entire adult life, for almost 16 years. I worked on a lot of different products like Google Assistant, YouTube, Android ecosystem, mostly around developer tooling as well as privacy and security. I've worked at some of these larger companies and I've seen how software is developed at scale. I've tried to make this presentation fit some of my experiences around transitioning to AI for Roblox. Roblox itself is actually a 20-year-old company, and so we've been around for a long time, and we're doing things in a very classic way. More recently, we decided we needed to accelerate with the rest of the industry, and this presentation is about that.
Shifting Up the Maturity Ladder
To start off with, who in the last week or two has shipped AI-generated code? How many of you people trust that code? Do you totally trust that code 100%? That gap is actually the main gist of this presentation here, is that we've gotten really good at generating code, but we haven't gotten great at digesting the code and trusting the code. This is the paradox. Essentially, we've solved typing but we haven't solved trust. This creates a lot of backpressure on engineers. It creates a potential for a lot of SEVs, people that are having to wake up at 3:00 in the morning to debug an incident but don't understand how the code works is a real problem right now. This is the problem that we tried to solve at Roblox, and we affectionately called our effort, Prompt to Prod, because we wanted to get to a point where you can go purely from a prompt all the way into production without any human intervention.
A lot of the techniques and tools that we built were to support that effort. I think it's probably obvious, we've made this strong transition over the last six months from really sophisticated autocomplete, which I'd say Copilot was at the beginning, to trying to get to more autonomous software development. That's with agents that are running, not just to generate a line of code or a file, but potentially re-architect your entire codebase in a short period of time and build features end-to-end for you. That's a fundamental change in how we need to think about many different aspects of the software development life cycle. Part of it is not just a change from how we're thinking about what are the best models, what are the best harnesses, but more fundamentally, it's about the infrastructure that's used to build your applications. Where are people at in the journey? Is anyone here allowing agents to ship directly into production right now?
When you do that, do you do any code review? Are you just YOLOing it? You have a set of gates that you're using to try to make sure that the code change doesn't actually cause a major SEV. I'd like to point out that there is a pretty big opportunity here, because if all we're doing is generating code, but we're not actually doing the full end-to-end product lifecycle, it means that we're never really going to get the full promise of AI. It means that we're not going to be able to really increase the productivity in the way that the industry hopes that we are in the valuations of the companies. What we think at Roblox is that if you just increase the speed of autonomy without focusing on safety, you're actually just creating technical debt. What I mean by technical debt is I mean that there will be a lack of people that understand the code, how it works in production. There will be security incidents, and potentially people will have to wake up at 3:00 in the morning to answer a pager.
Three Takeaways
The rest of this talk is going to focus around these three areas here. The first is around alignment and guardrails. This is how we encapsulate expert judgment and build operational systems. The second is around security and access. This is how we let agents run real workflows without compromising the company. Then the third area is the one that's most nascent, it's about how do we rethink measuring productivity in the world of agentic software development. I'd love to have a conversation with you all about that one.
1. Alignment and Guardrails
Let's start with alignment and guardrails. Just as a data point here, most of the LLMs, including state-of-the-art frontier models, were trained on petabytes of data, which is a lot of data. Actually, if you look at all the data that's locked inside businesses and corporations right now, you're talking exabytes of data. Does anyone know what the factor difference is between a petabyte and an exabyte? It's a million. It's like a million times more data that's locked within companies than what was trained for the model. It's not unsurprising that when you try to leverage these state-of-the-art models within your company, that they don't always get it right, and they make a lot of mistakes because they don't have access to all that data for their training. Another issue that we ran into when we first started our journey here was that we wanted to start allowing the agents to be more autonomous, and the first reaction from folks in security and throughout the company was, no, that's too dangerous.
We don't want to have a data leak. We don't want to have a SEV. No. Just don't do it. At the same time, we're also spending a ton of money on tokens and agents and not seeing the true value of them. We're seeing 10%, 20%, 30% productivity improvement, but not to the extent that we're spending without the autonomy piece. Most of what we had to build was the trust infrastructure needed to allow these agents to run more freely. Security was probably the largest bit of work that we had to do. There were lots of fears around prompt injection, around agents running with inherited permissions that were acting as you. There was an interesting case from one of my friends who works at a frontier lab where he asked an agent to work on a Jira for him overnight, and it ended up Slacking other folks on the team to ask them to merge the PR that the AI generated as my friend, and saying that it should just skip the checks, it's ok.
Because it was trying to be helpful and solve the problem. We've seen similar issues as well at Roblox, where an agent trying to be helpful does something that's actually not great for security. Lots of very interesting problems to solve. Our approach to trying to solve these things were these three main key areas. One of them is around sandboxing. Who here runs their agents in a sandbox? Does that mean that everyone else is not running their agents in a sandbox? I think you all are taking a lot of risk if you're not running in a sandbox of some sort. A sandbox is pretty configurable. Most of the major frontier labs offer sandboxes that you can install in your enterprise and put in place controls. We built our own sandboxes because we wanted an even higher level of security and safety. Your sandbox is there to protect not only the underlying host and all the files that are on the host, but also the network access that the agents can do.
We spent a lot of time thinking about policy gateways, making sure that the agents only have access to what they need when they need it. Just-in-time permissions, least privileged access. We also tried to make sure that we didn't have any long-running secrets that could accidentally be leaked by the agents. Then, finally, making sure that the agent identities have a separate auditable annotation from a real human. For example, that Slack message, it should be obvious in an audit that that Slack was sent by an agent and not by a human in that case.
The next pieces once we solved the fundamental security pieces, it was reliability. We felt like reliability was secondary because security is something where you lose trust of your customers, whereas reliability, we have reliability issues all the time. We know how to deal with those. We have a lot of operational runbooks and processes for those. Still important to the business because we lose revenue when we have a SEV. A model can be really great at solving industry-level problems, but it might not know your organization very well. We had this idea that AI could operate best in our environment if it was able to act like our best engineers. We tried a bunch of different things at first. We tried to fine-tune a bunch of open-source models. We tried to do special system prompts that tried to teach the models to act like our engineers. None of it really worked in the way that we wanted it to for many different reasons.
Our expert information at Roblox was actually already stored in our codebase. It wasn't at the top of tree in the head of the codebase. It was actually in things like code reviews. We had this idea that if we were able to look at every single code review, and we're able to extract insights from all the different code reviews, and then cluster them by similar feedback, that we'd be able to automatically pull out special rules that we call exemplars that are fundamental pieces of institutional knowledge that are encapsulated in a YAML file piece of code, and that are testable, and that are extensible and used by AI to improve future operations. We call those exemplars. Then we have a system that we use to make those exemplars useful in agentic interactions called the alignment engine. It's actually like a React loop. This is the system diagram here, where you can see that.
We take a look at the historical data from our repo, and it actually was about 1.75 review comments, which is about 700,000 PRs over 3 years. We extracted all of the useful pieces of feedback from those code reviews, and then we did a clustering, and then we extracted the exemplars. Then we actually at first weren't sure whether or not these would be high quality. We built a system, a UI for people to see the exemplars that were extracted from their repo. What they saw was they saw a UI where you would have a list of specific rules that we thought might be important to your repo. Then the interesting thing about that is that we found that the ones that people ended up saying were the most useful were the ones that were annotated by the authors that originally provided that piece of feedback. We found that if you showed our employees the best engineers and their exemplars, that they'd be most likely to adopt these within their repo. We wanted to make it opt-in at first, so that way we weren't inadvertently causing friction. Being able to understand who an expert is, annotate an exemplar with the expert, show it to the repo owners, and allow them to adopt it for their repo was really transformative for us.
This is one of the UIs for how if you wanted to hand author, not automatically extract an exemplar, you could create one. You essentially have the rule description, you have some patterns. Then you have the performance of that exemplar on the codebase, mostly coming from code review stats. Because we inject the exemplars in alignment engine during code review. There's a quick test feature here, which is a playground. You can actually give it a PR and say, run this exemplar on this PR, and see if it finds the issue that you hope it did. If it doesn't, you can tweak or improve your exemplar to make it more effective. Exemplars in action, like when we first started building our automated code review agent, this is an autonomous, standalone, single-purpose agent for code review. The accept rates were really low in terms of the suggestions that the code review agent made.
Over time, what we saw is that adding exemplars ended up getting us to, it's like almost 68%, 70% acceptance rates of all the suggestions that were given in code review. Mostly because of the exemplar addition. That's pretty great, since I don't know if you've seen this or not, but even only 55% of human code reviews are accepted. Our AI agents are actually producing better results than our human code reviewers are at this point.
2. Orchestration and Access
We'll move on to access. I told you before that the name of this presentation was called Prompt to Prod. The inception of the project was pretty simple. Roblox is a pretty popular gaming platform. We have 150 million monthly active users that come to the platform. It's really popular with kids. You've probably seen Roblox on your credit card bill. It's maybe the first time you've ever seen the words Roblox. Our homepage is really the main place where everyone starts on the platform. We had this idea that you should be able to go from a prompt all the way to a production change with no human intervention. It sounds pretty simple, but then when we started to unpack that, it got pretty complex. We decided that we should be able to author at least an experiment. The blast radius is pretty small. It's an experiment. You can have a small population.
That would be at least the entry point for this project. As we started to do that, we unpacked what it takes to actually create an experiment on the homepage. It actually takes multiple weeks. Some of it is because you have to let the experiment bake. A lot of it is because there's 18 different human touch points along that journey. A bunch of these things were inaccessible to agents. None of them had APIs. There was no MCP integration. All these tools were missing the fundamental plumbing to do agentic software development. A big part of the work at the beginning, it wasn't AI, it was essentially plumbing and infrastructure. It was to try to make sure that we could get to a point where all of these different systems had CLIs and APIs and MCP access. We used Playwright, which is a pretty cool tool, to help convert your UIs into CLIs and make them agentic-friendly.
We did a four-week sprint, and we were able to get this full stack of things plumbed. We were able to get to the point where an agent could actually author an experiment and make a change on the homepage. That part was great. Now the plumbing worked. The next piece was around safety. Making sure that you could actually trust the fact that you weren't going to break production. What we found is we were missing some of the basic fundamentals. We didn't have great unit test coverage. We didn't have integration testing. We were still not doing some staging deployments for some of these changes. We didn't have an auto-rollback and an auto-revert for these particular changes. Once again, the hard work actually wasn't AI work. It was good infrastructure work. It was about making sure that those things that we've been saying for 20 years in the industry, like testing and canary and telemetry and all that stuff was there. We spent a lot of time working through those particular problems. You can see here, these are some of the different systems that we had to instrument with CLIs and make sure that we had the right telemetry for.
The other piece here was around policy. We ran into some pretty serious friction around what can be allowed to be done. There was only one person that said that they allow PRs to be automatically merged into production. We ran into a similar problem as well when we talked to our policy and compliance folks, and said, we would like to still do a code review, but it's only going to be an AI code review. Are you ok with that? Do you know what their answer was at first? It was no. Actually, one of the reasons why we have a lot of the data around the human effectiveness of code review suggestions and AI's effectiveness of code review suggestions was because we wanted to build some confidence making sure that the AI code review, that we would replace a human code review, was actually as good or better.
It was a policy change for us. We had to rethink some of our underlying assumptions for software engineering at the company that every code change requires a human code review, and essentially say that's not going to be true anymore. Is anyone else here thinking about removing human code review? I'd love to know why the people that didn't put their hands up are not thinking that, because I see it as so fundamental. Imagine you're producing 10 times more code. How are you going to review that with the same amount of people? There is no way. Not only that, but you will have not gotten the gains that you need to from AI. I really encourage you all to definitely think about the policy pieces as well. We also had other rules around deployments and when deployments could happen. We would do a production deployment moratorium between certain hours.
We had to remove that. We had to get to the point where we trust AI enough that it can build the code, it can review the code, it can ship the code, and it can revert the code. This is essentially where we wanted to get to. Which is like write, review, fix, approve, deploy, and iterate. We tried to make this as transparent as possible as well. We didn't want this to be un-understandable by humans. We tried hard to make sure that any time that an agent would author code, would modify code, would do a code review, you could see that action. You can also provide negative and positive feedback signals to the agents to be able to say, this was a good change, this was a bad change. Then we take those negative and positive feedback pieces and put it into a cycle for self-improvement. The way that we do self-improvement is we make sure that any of the negative feedback is added to our eval set for those agents. As well as we use it for extra context for some of our RAG operations, as well as use it to distill it down to exemplars.
3. Measuring What Matters
We built all these things. We built sandboxes. We've changed policies. We've fixed a bunch of security things. We're spending a whole bunch of money on tokens. Does it actually matter? Is it actually making us more productive? I'd be curious to know here, who here thinks AI is making you more productive? How do you know it's making you more productive? Do you talk to the engineers? You're an engineer. You feel it right now that it used to take you three days, and now you can ship it in a shorter period of time? How do you measure that, besides the feeling of it? Do you track your feature velocity in some way? How do you do that? Ticket tracking. Through Jira style ticket tracking. Great features. Anyone else have an interesting metric for how they measure productivity? PRs. I think like in the past, a lot of us would probably look at PRs and cycle time and all that stuff.
The point is, is that I think most of those metrics are failing us at this point. Because generating code itself and reviewing code and all those things, like I said before, if they're all automated, like what are we measuring? We're measuring maybe the productivity of the AI, but certainly not the engineers at that point. I think most of the metrics that we relied on was SPACE and DORA and stuff like that, need to be rethought. This is the beginning of how we're thinking about doing that transition ourselves. As you can see here, I'm trying to be a little bit contentious by saying things like lines of code and PRs and things like that actually should just go away in terms of productivity measurement. I think we need to reprogram our language to stop talking about those things. One for us where we invest a lot of time and energy in is around agent quality.
When I say agent quality, I'm talking about two different types of agents. Either the focused single-purpose autonomous agents like a code review agent that you build and it runs indefinitely. It's not inherited permissions or anything like that. Then, also, when we bring in a new harness and a new model, we want to know whether or not that's better than the other one. Or, is this vendor better than the other? For us to get confidence in any of those decisions, we rely very heavily on evals. Maybe 30% or 40% of all the work that we do is around evals and data. It's about making sure that we know how to measure the quality of agents that we're building, the single-purpose agents, as well as the harnesses and models that we want to spend money on. The example here is that we found a whole bunch of code in one of our repos that didn't have any unit test coverage at all.
Then we take a state-of-the-art foundation model with a new harness and we run it, and say like, how much unit test coverage can you get without human intervention? If it can get 60% unit test coverage and the previous model was 50% and those tests look good, then we actually promote it, and say, we're going to adopt the new thing. If it's a regression, then we don't adopt it. Evals are super important for all the work that we do. Does everyone here use evals? Does everyone here know what evals are? Who doesn't know what an eval is? An eval is a way for you to be able to look at the quality of something that was generated by AI, essentially. Because generative AI can be hard, it's not binary. If you say, generate a unit test for this file, you want to know whether or not it's a good unit test.
It might be a really crappy unit test that's testing something that's not that important. An eval gives you the ability to start with a set of data to run it through AI and look at the output and gauge whether or not the output is good or bad, and then compare that to a previous version of the AI. If you're building a single-purpose agent, let's say a code review agent, you saw that graph that I had there. All of the ways that we knew that we were getting better and better, like all the things that we were doing that were making an improvement, were all measured through eval. Definitely if you're not thinking about eval, you should definitely get that on your radar. This is really important. It's a fundamental way for you to be able to make sure that the agents are actually high quality.
The other one here is feature velocity. I was really happy that you said features, because as much as I would love to be able to tie our AI efforts to top-line metrics like R&D, or revenue, or those sorts of things, it's really difficult. The best proxy that I could think of right now is feature velocity. The idea is that if you're shipping more features as a company, you're probably going to be shipping a lot more innovation. If you're shipping a lot more innovation, then you're hopefully going to be a more successful company. What is a feature? I think that's the big question is like, in the past, we always really struggled with how do you define a feature? Is adding a new capability to a backend service a feature? Probably it is. It was very hard to annotate the code and know whether or not that was a feature, and if the features are increasing over time.
Because we're living in an agentic world, it's actually gotten a lot easier because now you just ask an LLM to analyze the PRs that are being submitted, and say, what kind of PR is this? Is it a feature PR? Is it a configuration change? Is it a bug fix? Is it refactoring? They're pretty good at that. They're actually quite good at being able to determine the intent of a PR and classifying into a category. We actually do a primary and a secondary, but this chart here is just for primary categories. Now you have an idea of the different types of PRs. You know how many features are coming out of your system. The question is, how do you actually use that and turn it into something useful? For us, we look at median number of features per engineer. We normalize it, because actually all the buckets go up with AI, because people are able to do refactoring, able to do more configuration changes, able to do all of their work faster.
The thing that we care most about is actually the number of features per engineer going up. We mostly look at like a p50 here, and we can see like over time, 22% in feature velocity in the last six months or so. This is a way for us to say more concretely that investments in AI are actually translating into innovation for the company and hopefully making our product better.
This metric here is the one where I'm not as confident is the right thing. When I started this talk, I talked about how we want to do Prompt to Prod, but ultimately our goal is to get to something like 24 by 7 AI. We have AI agents that are continuously working on your behalf, even when you've left the office, shipping into production safely, scaling it out by 10 parallel instances, doing all that kind of stuff. For that level of autonomy, you have to give a task to an agent and it has to be able to go and do something big. What we're seeing, you can actually see in this data snippet here, is actually that the p50 turn time, this is like, you talk to your agent, you tell it to do something, is 40 seconds. That's not very autonomous. That's 40 seconds of work.
That means that a human needs to attend to that agent, has to sit there, every 40 seconds, tell it what to do. You can't get to 24 by 7 AI at that rate. The idea here is we want to look at what I like to call long turn rate. Looking at how if you give a spec to an agent, will it go and actually go and converge on the right output of the spec? Will it problem solve by itself? Does it have access to the right tools? Will it converge on something that's good? Because you can also have a long-running turn that isn't good. You can see here, we also try to filter out turns that work out in weird ways: missing a tool, some network interruption, all these kinds of things. Our goal is to be able to orchestrate the work in a way that allows people to do these long-running turns and go for 8 hours and get a giant chunk of work done on their behalf.
Right now, you can see even where we're currently at there, the p99.9 is only 2.1 hours. That's barely my commute home. Not much work is happening while I'm out of the office. This is our main success metric right now for 24 by 7 AI. We want to look at this and see whether or not we can get to an 8-hour window. There are some circuit breakers in the agentic tools themselves that will actually just stop after 4 hours too. You have to not just solve the coming up with a good spec, giving it the right tools, putting it in things like Wiggum's loops or those sorts of things, but you have to also make sure that the tools are configured in a way that allow you to burn those tokens overnight.
Summary
I think if you combine all those things together, this is what we're trying to do to get to a 24 by 7 AI. It's about the alignment guardrails. Making sure that you have institutional knowledge that's accessible by your agents. It's about making sure that you put in place the right security foundations, sandboxes, access to all the tools, and that you're also measuring the things that matter the most. I would definitely keep away from just measuring lines of code in PR because it is trivial or even less than trivial to game that at this point. Please rethink some of those metrics.
Questions and Answers
Participant 1: You got me thinking some stuff, especially about the autonomy. I was thinking differently in terms of human intervention rate, instead of agentic autonomy and stuff like that. Because the number of interruptions that you have, it will get the people thinking between tasks. It's context shifting. That takes like 14 minutes for the human to shift context.
Andrew Swerdlow: I think there's a term for this that's emerging, it's the orchestration tax. If every 15 minutes you're getting interrupted by an agentic workflow because you have to re-steer it in a different direction or answer a clarifying question or deal with an error, you're paying a tax. I don't think that's where we want to be. Our goal is to try to avoid those interruptions.
Participant 1: How are you measuring the interruptions?
Andrew Swerdlow: In that long-running turn time, those are single turns, no interruptions.
Participant 1: Autonomy-wise, you have to measure how autonomous it is, what is the actual metric and how you're capturing?
Andrew Swerdlow: I was thinking about it in terms of the length of the turn itself. A turn is like, every prompt that you do to the agent, how long will it go before it successfully completes the task that you assigned to it? If you're doing spec-driven development, if you give it the spec, you put it in a Wiggum's loop, then how long will it go before it actually stops either because of an error or anything like that? Our goal is to get it to take a long period of time, but also successfully complete.
Participant 2: I know you mentioned in order to get the agent truly automated, you basically have to do a lot of plumbing work for the existing codebase, setting up MCP and stuff. What kinds of tools did you use and how long did it take for your company to do that?
Andrew Swerdlow: It is interesting. One of the pods that I support is called the AI Pod. Actually, almost none of the work that they're doing is AI. It's actually all infrastructure work. It's adding the CLIs. It's increasing test coverage. It's extracting and providing context to the agents. I wouldn't call any of that the traditional context engineering AI work that we've seen in other previous presentations. We were given a very ambitious target by our CEO. This is about three months ago. He said to us, I want you to be able to unlock full AI for everyone at the company. Autonomous workflows in six weeks. We did this company-wide code ride. We could bring in anyone that we wanted to work on this exercise. Within six weeks, we went from more casual use of AI to every single person in the company using AI tools and shipping into production.
The precursor work for that was to make sure that their tools all worked with their agents. About six weeks. A wide variety of different types of technologies. There's not one specific type of technology. Playwright, for example, was something that we leaned on to help get us to CLI capabilities faster. We try almost all vendors. We have a rapid eval process, which is a way for us to quickly onboard and try new state-of-the-art tools and see whether or not we should add them to our tool chest or build our own or use a combination. That's also another fundamental problem that I've seen in the industry, which is, it takes too long for people to get onboarded into new tools. I know some folks that still don't use any of the frontier lab technologies because they haven't been able to get security approval, budget approval, or any of these things. I believe there's going to be a Blockbuster and Netflix moment in the industry right now. The companies that fully adopt AI are going to be like Netflix and really excel. The ones that don't will become like Blockbuster and become out of date pretty quickly.
Participant 2: You mentioned the eval framework. What's your recommendation for coming up with an eval framework?
Andrew Swerdlow: There are many vendors out there that you can look at. We actually built our own eval framework and harness to start with as well because there wasn't anything that was great with coding a year and a half ago. There are some better things out there now. You can look at Weights & Biases and things like that. We don't use them, but they're pretty popular with people. They have some sophisticated tools. I'd also recommend investing in observability for your agents as well. Using OTel to collect tracing data, to be able to debug all your agents like that is one of the most important things that we invested in as well.
Participant 3: I'm still very skeptical about shipping everything all the way to prod with AI. I trust you with all the beautiful graphs. However, I think a related problem happens. You still can have issues. Having a good remediation plan is very important. I want to understand, did you ever have any problem with shipping all the way to prod causing problems, troubles, losing data? How do you remediate it? How do you also make sure the same thing won't happen in the future? Or if things happen again, how do you minimize impact in your system?
Andrew Swerdlow: I think maybe an even higher-level question is like, what is the role of humans in this new world? I try to write some code myself still. When I see the volume of changes that are happening, even within the repos that I'm working in, it's more than any human can really grok and review. I believe that most engineers will not actually understand most of the code in their codebase moving forward, that it will be abstracted away from them and that they will live above this AI abstraction layer. To make that actually work and deal with problems and SEVs, we need to also have AI abstractions within production. If you're not investing in agents that can help you debug production incidents, then you should probably be working on that, because it's like the order of operations is coding agents, code review agents, production agents. If you bring all those things together, then that's when you can start to really unlock the full agentic lifecycle.
I agree, it is scary to say that we will allow AI to do these acts on our behalf. We don't trust AI for everything. For us it was a bit of a maturity process. It was saying like, do we have all the right plumbing in place? Do the agents have access to the right tools? Do we have the right policies? Do we have the right graduated promotion technologies? Does it go through canary? Does it go through staging? Does it have integration tests? Can an agent do a rollback if it detects a problem and a revert? If the answer is yes to all those things, then you might as well try experimenting on some of the low-risk scenarios. We also started with trying to use AI to identify low risk PRs to auto-merge. Because you can actually say from an LLM, does the blast radius of this look like it could be something that could take down production? They actually do a pretty good risk assessment. Not always right, but sometimes.
Participant 4: If you've managed automation from coding through review to production, it feels like the bottlenecks are going to move forward then to product. Are you thinking about how to validate product requirements?
Andrew Swerdlow: We haven't hit that bottleneck yet. I think that human imagination is unbounded. With AI writing tools, PMs can still write PRDs, but even super-fast now. I do think certainly if you compress five years' worth of roadmap into one year, you need to write five years' worth of roadmap. That does take time. My hypothesis is this, is that it used to be expensive to do stuff, and yet you wanted to get it right. You would do a lot of user research and customer interviews, like all these kinds of things to try to make sure that you don't ask your engineers to do something that's going to be throwaway. Now, the building part is becoming a lot faster and easier. The deployment part is becoming a lot faster and easier. Maybe the barrier for coming up with ideas and making sure they're right ideas is not as hard anymore.
That you can actually just pump out more things. Your agents will also tell you through your experiments, which things are the most effective. Your product will self-iterate that way. I think that's where you'll see a lot of value is like, if you can take the loop from generating code, testing the code, building the code, deploying the code, but actually like full-on experimentation and iteration on experiment variants, then you'll see a lot of people moving faster. There might be a new way of thinking about product management and coming up with the right features and that kind of stuff. No answer for that one quite yet.
Participant 5: How are you solving engineers continuing to build a theory around the problem and the solution of the problem? Are you reaching to pair programming? Are you reaching to other things that are happening before PR review?
Andrew Swerdlow: Tell me more what you're thinking?
Participant 5: The theory building, the mental model we keep in our head of the system, the domain of the system, and the how that this problem is solved and continue to keep that mental model as we develop it over time.
Andrew Swerdlow: I spent a lot of time working at Google. Google love design docs. That was the thing when I first started working there. We all love design docs. We spend a lot of time on design docs. The idea was like you would create this design doc, you get a lot of feedback on it. It was a way to build that mental model, to get feedback, all that sort of thing. I'm going to argue and say design docs are dead, and that you can build a prototype as fast as you can write a design doc now. What's better than a prototype? You can touch it. You can feel it. You can see whether that's the right thing. You can inspect the code. You can look at the architecture. You can do all that kind of stuff. I think that ability is unlocking a higher fidelity shared understanding of the things that you want to build, because you're making it very concrete.
You're showing with an example. I see a lot more of that happening right now. That instead of like even a junior engineer coming with a design doc and iterating for a long time and trying to get that shared consensus, they just come and show us what they think they should build. Then we're like, yes, that looks good. Ship it. A little bit of a contentious take there.
Participant 6: My experience has been when I'm, say, vibe coding for a while. It's working great. Code's going nuts. I'm adding features left and right. Then after a while, it's like, you have to fix that bug. Wait, that created another bug. That created another bug. Then I go and I look at the code and I'm like, no wonder, this is a complete utter mess. Everything is repeated everywhere. No wonder it can't stay in a steady state. Then I go rewrite it or re-architect it a little bit. Then I'm back again for a while until, again, it's messed up all the state. I'm wondering if you've experienced anything like that. If you have any mechanisms to try to detect and pay down and refactor tech debt and so on.
Andrew Swerdlow: I think it's interesting. I feel like I see more of the whack-a-mole style. Like, you try to fix a bug and you create a new one with more of like the vibe coding kind of like prompt this, do that style of AI software development. If you're doing more like spec-driven development where you're talking about APIs and contract boundaries and convergence criteria and all these sorts of things, then I find that the code is not quite as messy and that you have some semblance of order rather than things changing all the time and bolting on new features and that kind of stuff.
Participant 6: What format are you using for your specs? What does spec-driven development mean to you?
Andrew Swerdlow: It changes on a week-to-week basis, and team-to-team it's very personal. It's the difference between specifying like I want to build a specific feature, to talking more about the properties of the systems that you're trying to build and the constraints around it. More of like the requirements and the boundaries and the architecture of the system. These can be pretty large documents. We have giant-like specs that are multi-pages and have to be chunked up sometimes to process. I don't have a perfect answer for that right now. What I've seen is that the people that are just saying like fix this bug, or add this feature, like that's where we see more of the whack-a-mole and the less sophisticated agentic software development.
Participant 7: I'm from a medical device company. I have to admit that we deliver zero code using AI for half a year, because we have to go through FDA.
Andrew Swerdlow: Just brain-coded.
Participant 7: Yes. One thing is you're talking about the code review that accept rate for humans is 55%, and the machine and AI.
Andrew Swerdlow: Turns out that humans aren't always that great either.
Participant 7: How do you determine this metric? Because when we do the code review, it's not necessarily rejected or accepted. Sometimes it's just a suggestion or it's good and then follow-up discussion.
Andrew Swerdlow: We look at just the basic accept rates within GitHub. Whether or not we see if someone made a suggestion and then whether or not the diff against the merge code was that the suggestion was actually accepted. We just do it programmatically by looking at the comments in the GitHub diff to what gets merged into production.
Participant 7: How do you train your team to get on a similar level using the AI? Because different people have different acceptance for the AI accept rate.
Andrew Swerdlow: We set up this other metric, it's like AI intensity. It's about like how much you use AI and what you use AI for. We do look at that amongst engineers. We do see like a broad spectrum for some people, although not many people not adopting at all. A large cohort in the middle that are what we call casual AI users. They use it on a day-to-day basis, but not to do all of their work. Then we have our high-intensity AI users. The high-intensity ones are just self-motivated and have learned to do this by themselves. Then the middle cohort has adopted because they see everyone else doing it, but they haven't quite learned how to be highly effective 10x engineers like the top cohort. Then we have other folks that are just like, I don't trust it. I don't want to see it. I don't want to touch it.
We're not really sure like how that's going to play out in the industry. As a manager, when I'm doing calibrations, you're starting to see the difference between people that accept and use these technologies versus people that don't. Just based off of like the output, like this person was able to get through this many roadmap items and some of the same level only got through a fraction. What's the difference there? How do I as a manager start to calibrate and promote those people? That's still an open question right now. I don't have an answer for that. We do try to do a lot of info sharing sessions. There are ways for people to share their prompt history and show how they've been building things. We do lots of lunch and learns, all that kind of stuff to get people excited. I think right now people are more excited than not excited. I actually get hate mail from people at the company because they're like, why haven't you brought in this tool yet? I'm like, "It just came out two weeks ago. Give me some time."
Participant 8: You guys do no human looks at the code prompt straight to production right now at Roblox?
Andrew Swerdlow: Not for everything. This is a small subset of our production changes.
Participant 8: Some of it?
Andrew Swerdlow: Yes. The homepage changes and experimentation, that's like been the main focus area. We have eight or nine other development workflows that we've been automating.
Participant 8: Code review is initially based on what is being accepted in GitHub. At the moment, a lot of our code review is done by AI. Eventually, all the code review that the AI is looking at is going to be done by AI. How do you retain confidence in the code review once the human is out of the loop?
Andrew Swerdlow: I'm not sure the code review matters, in the way that it did before. It's still an important validation checkpoint to make sure that nothing breaks going into production. The point of it, I think before there was like two main reasons why you do a code review. One is to make sure that you don't ship a bug. One is to make sure that the institutional patterns are upheld within the company. The third one is probably just for education and knowledge sharing. Those are the main reasons why you might want to do a code review. Certainly, from the quality standpoint, and the adherence to a standard, I don't think those things are going to matter as much, and that an AI can do as good or better than a human for those. Making sure that everyone understands the code, and that we have institutional knowledge and share that knowledge, I think that's a different problem that needs to be solved. I think it is going to be a problem.
See more presentations with transcripts
