Transcript
Sarah Deitke: I'm a software engineer at Duolingo. I've worked there for about two years. Most of that time has been dedicated to the team that I'm on, called DevEx AI. You might be familiar with the traditional DevEx team, working on things like merge queues, CI/CD, the works like that. My team takes a different angle at that. We're focused on how do we enable our engineers to use AI effectively in their workflows. That's what my talk today is mostly about. What I'll cover in this talk, kind of where we see the industry going in terms of culture change with AI adoption. How our internal AI literacy programming through education helps our engineers. How that can enable autonomous systems and AI agents in your company with the case study of redesigning our code review processes. Then, how you can scale that in your organization.
AI Adoption is a Culture Change
Let's talk about what we're seeing in the industry today. Most organizations do have pretty good tooling access. How many of your organizations have Cursor or Claude Code in your organization? How many of your organizations have a dedicated team towards helping you use those effectively? Maybe by the end of this talk, you'll advocate in your organization that you might want a team like ours in your org. Furthermore, we see AI as becoming more prevalent in workplaces. Maybe last year we saw copilots taking off, but now agents are the big thing. With that, AI has a lot more responsibility. Where does AI adoption in your org get hard? I think the first one is that engineers can be skeptical. It's a big change to what people are used to, what they studied going into school. That can be a blocker for teams like mine. Core systems, like code review, are viewed as sacred.
They're best practices that have been honed for years that are maybe now getting challenged by the bottlenecks that come with AI being able to write code faster. AI also challenges and changes accountability. This is a conversation my team has a lot. With something like code review, maybe we saw both the engineer and the reviewer are responsible. When one of those is AI, who's responsible? Then, finally, trust is different, same as accountability. It's slow to build, fast to lose. With AI, I think a lot of people are afraid and skeptical when it comes to trusting more and more of their processes to AI.
Building AI Literacy Through Education
I'm going to switch gears now and talk a little bit about, in depth, some of the things that my team does to make our engineering organization feel more comfortable using AI in our workflows. The first thing that we do is structured, lab-style workshops. These are things that are pretty hands-on, that are outlines. People can work through them at their own pace. The topics that I have in this screenshot include learning how to use MCP servers, learning how to use Cursor rules, learning how to do batching requests to LLMs, and do evaluations. One thing that we really have found with doing lab-style workshops is that engineers trust the pre-vetted content from their engineering orgs a lot more than, say, bringing in a vendor and doing a more generic solution. This is a screenshot that we ran at one of our company all-hands last year, and 95% of our engineering org reported learning something new and that they would recommend this style training.
I would definitely recommend giving this a shot if you guys are focusing on some new features coming out that you want to train your organizations to be more effective in. One thing that can really help you with figuring out what to put in your workshops is actually building AI observability dashboards. This was a small initiative that my team ran with last year that we've seen a lot of success with in return on investment for, and continue to build out today. We started with just doing a daily active user chart for one or two vendors and in-house tools. Then, over time, we scaled that up to include looking at what functions, like engineering, design, finance, are using different AI tools. We've added more AI tools. We have charts around what developer community, for example, in our organization, that's iOS, Android, and backend, but it could be definitely different in yours.
Then we also look more at what the breakdown of AI consumption is through things like what IDEs and programming languages people are using, token usage, cost, model family, lots of different groupings. Then, finally, just anything else that may be interesting or useful to your organization. One thing I do want to call out is we did start with this as a vibe coded project, and it has continued to be vibe coded. We could just go to our AI tools dashboard and say, ok, add me a chart that looks at this now. That's been really effective for communicating well with our leadership around what people are doing in the organization and really helping us grow from there.
Another thing my team does is live office hours for AI support. This is just a 15-minute time slot that different people in our organization can book. It gives them a lot of support in terms of like, I might want to use AI in my workflow, but I don't know what to do or what are the best practices. One thing I've found really interesting with the office hours is it's not necessarily used by engineers, but it is used by these fringe parts of people that might touch engineering a little bit that now need more support. I think people like our learning designers that might need to know now how to use GitHub because they're vibe coding more projects, or people like our QA support team like using AI more and more in their workflows and figuring out how to actually scale what they've done off of their computer.
That has been really cool in terms of the impact can be broad-reaching beyond your engineering organization. It's just really awesome to see like those people have those light bulb moments and take them back to their teams and get them to use AI more. Another thing that we do is encourage shared learnings. We have a Slack channel for engineering building with AI, and then in parallel we have a biweekly engineering building with AI meetup. In those we nudge people in our organization to share what they're doing with AI in their daily work, and keep it low stakes. The culture of learning, even sharing failures or successes is really shared and encouraged. That makes people a lot more comfortable using AI in their work.
Finally, I've put this slide up to talk about investing in your AI vendor relationships because that's been a really important part of enabling AI for our organization. The first point is around, it accelerates your access to new AI tooling through beta programs. One example that I have for this is, we have a relationship with Cursor that nudges us on new things before they're available. One of their new features last year was around adding team rules, and that was a pain point we had had in our org before they had introduced this feature. Things like letting the AI agents know like what CI/CD platform we're using, what are our best practices when it comes to writing Python code? What libraries do we have in-house and how do you use those? When this AI vendor came to us and said, we have this new feature, are you guys interested in trying it?
We said, yes, let's see if it can solve this pain point for us. That had been really successful. We do this with a lot of different companies. We're very fortunate to have a very active partnership with our legal and security teams where we can say, "We're trying to get these new betas turned on. What do you guys think of this? Does this fit within our security and legal boundaries?" Quite often the answer is yes, and we can test out these tools, provide feedback to vendors and advocate for our organization through that. That's kind of the third bullet point here also is our vendor partnerships help us advocate for the organization. When we have these longstanding conversations with a lot of the different AI vendors, we can tell them, we're seeing pain points here. Vibe coding for design is a great example of one of them where we've communicated that a couple times.
We can really nudge them to develop products that help our organization have that be a two-way communication. Finally, a great part of our AI vendor relationships also is it can help us identify where we are maybe lagging behind the best practices of industry. One example that I have for that is around evals. As we were bringing online more AI agents, we realized that a lot of these vendors are actually using evals in their organizations and seeing success with that. Maybe we can talk to them about how they're doing that and then copy that and do the same thing. That has been a really successful partnership also is just helping us figure out where the gaps are that we can learn from them.
What are the results from Duolingo investing in so much AI education and literacy for their organization? Over the last year, we've hit the tail end of getting to 100% AI tool usage adoption for the engineers in our organization. Before that, it had been 80%, which is still quite high. I think we can definitely say now that the whole organization is far more literate than they were a year ago on this topic. Engineers understand the strengths and weaknesses of using AI in their engineering systems, which I'm going to touch on a lot more in our case study. It reduces fear, uncertainty, and doubt in using AI in their day-to-day job. Then, finally, engineers are more ready to adopt AI agents into our existing engineering workflows like code review, which I'm going to talk about in the next section.
Case Study: Redesigning Code Review for Autonomous Systems
This is a case study of the work that I did over the last year around how can we improve the bottlenecks that we see in terms of code review slowing down shipping code from AI being able to generate PRs a lot faster. This is a little bit of a look at what the user experience now is like for us in terms of code review at Duolingo. We have a system called the PR Risk Assessment. It essentially buckets code changes by low, medium, and high risk, or if your code review is far too big, then undetermined because it exceeds the token limit. Then if it falls into the low-risk bucket and it meets a bunch of other criteria, then it will get auto-approved by our friendly auto approve bot here. What this means is that we allow you to merge your code without a human reviewer in these cases.
Why target code review as an effective AI agent to run in your organization? In my opinion, there's a lot of pros and cons when it comes to code review. The pros obviously being catching bugs and getting feedback from your peers on best practices. Another pro being mentorship, especially for junior engineers. Then, finally, going back to earlier, shared accountability between the author and the reviewer. In terms of cons, there's quite a few cons of having all of your code changes reviewed by humans as well. It's inefficient for a lot of portions of code changes. We switch teams a lot at Duolingo, and so moving around what team owns a codebase, it's pretty predictable, but it doesn't necessarily need to have an actual code sign-off. It frequently interrupts developers. You might be aware of Duolingo's famous notification system. We try to replicate some of our gamification in our company's internal practices.
Then you do get lots of notifications. We see that a lot. It's also an inconsistent developer experience. How many of you have sent your code at one point to maybe a person that really likes to nitpick, and you've had multiple cycles where you've needed to turn that around? Versus, sometimes you just really want to get that code change in, so you send it to your work friend, and they sign it off without looking at it. It's not consistent. Then, finally, as I've mentioned already, it's easier to make PRs now, and so we see the bottleneck shifting to code review for shipping quickly.
How does our AI auto-approval bot work? It's pretty similar to other CI checks you might be familiar with. On a pull request push, it triggers our risk-scoring pipeline, and then that updates the pull request CI check and bot approval status. How does our PR risk scoring work? It takes in a bunch of parameters, the PR title, the verification steps, what the actual diff is, a user prompt mainly consisting of, you are grading this code change on its risk levels, and then an actual risk prompt, which buckets what kind of code changes are low, medium, and high risk. Sends that to an LLM, and then we get a risk tier and a risk explanation from that. Then I do want to call out, too, that Meta pioneered this work with their diff risk score. Our system is a lot more scaled back because our engineering org is only 300 people and not 300,000 people.
The core principles are still the same. I want to talk about what is covered in our risk assessment. At a high level, low risk changes look like changes to Markdown files, straightforward and small code changes, so these are actual feature differences in the repos. Then things like approved metadata and configuration changes that engineers have signed off on to say, this is a routine update I make, I just want this auto-approved. In terms of medium risk, we see things like updating the API versions for our learning app. While this can be a straightforward change, the feedback has been, if the API version mapping is wrong for some reason, it's just too high risk for us to ship that as an auto-approved change. We would love to have two people be accountable for that. Updates to foundational components like auth, logging, things that have a larger blast radius even if the code change is small.
Finally, large feature code changes, so adding a new feature, but it's straightforward still. Then, finally, high risk changes. There's a lot that goes into high-risk code changes. The three I've boiled down to here are changes to user permissions. We always want to get that reviewed. Changes to large scale infrastructure, and then changes to specific and core features of the Duolingo app. There's also a lot I have not included in here that is very Duolingo specific. If you are going to want to do something like this for your organization, you're going to see a lot of things come up and learn a lot from the organization around what things are wide-reaching. For example, one thing in the Duolingo world that I didn't include in these slides because it's not relevant to everybody, but you'll see things like this, we have just these asset animations, and changing those can have pretty wide-ranging impacts. Think of Duo at the end of a lesson popping up and waving at you. Things like that we have bucketed in here, but it'll change from organization to organization.
You might be thinking at this point, Sarah, it is completely crazy to not have people doing human review for all code changes, and to that I would say we have a ton of guardrails in place to mitigate risk. The first chunk of guardrails that is included is providing config options for the auto approve bot in your repos. The two that really come to mind are, one, locking down the auto approve bot to only code owner changes. For our repos, we typically have 5 or 6 people that own one codebase, and maybe 20 or 30 people contribute to that codebase overall, but only those 5 or 6 people would have the option to turn this on, only have auto-approve from them. Eighty percent of the code changes in that repo are already from the small group of people, so it is effectively the same, but it's just like a nice sense of security that that group of people has.
Another restriction option we provide for configs is including and excluding directories in the repo. You might have some parts of the repo that are very safe for an auto-approve system, and some that are maybe not safe. I think of something like our CI/CD repos, where we have like the finance CI/CD, definitely no auto-approve, but we might have the scratch CI/CD, totally fine. A wide range of things can live in repos from time to time. Another thing we have this completely disabled for, and we debate this from time to time, but it's how it is today, is disabled for changing any AWS resources. That's just a preference from most of our engineering org. It's disabled legally for repos that have higher auditing requirements, like SOX or ISO for our Duolingo English test, so no auto-approve running in that area and domain. Then, we also have restricted it to new engineers during their onboarding period.
We've shifted that around a little bit, but we've found that for people's onboarding periods, they don't know the best practices for the organization, and so it is just a little bit safer to have that blocked time window before enabling auto-approve for those people. Then a final guardrail is just around opt-in notifications per day. Some parts of the industry have advocated towards moving towards a ship, show, ask framework, where you ship most of the code changes, you show people before shipping, and then you ask for permission in terms of code review before showing or asking. Opt-in notifications are a really nice way to have the auto approve bot, but still have that visibility in terms of moving fast and seeing what's been shipped once a day instead of seeing what's been shipped every time you get notified that somebody wants you to review a code change.
The feedback loop portion of the PR risk scoring I think is really interesting and ties in to where we see the benefits of our AI literacy programming really take hold. I have a Slack channel, proj-pr-risk-score, and in it, people report inaccuracies that they see with the bucketing of our risk scoring so far. The top one we have is, this PR probably should be marked as high risk because it changes some services, or this PR should be auto-approved and it's not. Then, I'm changing how we connect to every endpoint, this should probably be medium risk instead of high risk or low risk. Things like that, I probably get a couple messages a week in the Slack channel. I think what's really powerful about this is that since we've really invested in AI education, we see that people are pretty indifferent and willing to accept this system, whereas before people had been like, I might not want to turn on the auto approve bot at all.
Now we see people are like, I really like having auto-approve on in my systems, but I just think there's an inaccuracy here that I want you to fix. We take these reports and essentially put them in an eval list to say, this PR should be low risk, medium risk, high risk, and then pick out keywords in the explanation that we just want to evaluate and validate the risk scoring on. Then, from there, once we have this eval dataset, we can fine-tune the risk prompt as new developer feedback comes in. The risk prompt grows and changes over time, but is backwards compatible with all these different reports that I've gotten from people in our engineering org around what things they want to be like low, medium, and high risk. It's a really cool flywheel and feedback loop system. I think that this is really what is enabled by having excellent AI training and support in your organization.
I also wanted to talk about the results of our auto approve bot system because they have been pretty dramatic and pretty awesome. This top line we have here is our median time to merge of hours for all pull requests that are in our organization. This is over pretty much the last year, and we see it steady at about 18 hours per pull request. Then this bottom line here is the percentage of all pull requests that are being merged through our auto approve bot. It climbs from about 0% to about 10% over like the last 6-ish months, and we see that with that climb is the correlation of our median time to merge dropping from about 18 hours per pull request to about 12 hours per pull request. Some really exceptional things about this, this 10% of PRs being merged through our auto approve bot, that includes all of the code changes that we can't do an auto approve bot for from compliance or resource change reasons. This is like everything. It's not just what is eligible for auto approve. Then, also, this is not just the low-risk changes. It shows that when you accelerate the low-risk changes, there is that cascading effect that the median risk and the high-risk code changes also helps them, it gets everything merged in faster.
From Education to Safe Autonomy
To reiterate what actually changed in the culture to enable an AI agent in our code review system, I would mainly argue that the educational training for our engineers helps us build an understanding for our AI systems that enables AI agents to work more for your organization. This foundation, it lowers the skepticism, it lowers the uncertainty and the fear around adopting AI in your organization, and helps people to use AI more effectively. Like I was saying, pull request auto-approvals are possible because our organization was ready for AI workflows through our education program. I do want to emphasize that our auto approve bot is not special and our engineers are creating and expanding their own AI workflows, mainly through our support for AI literacy and training to enable them to scale. Some other examples that immediately come to mind where I see people making AI agents in our org, we have a support bot for our organization just to do question and answers.
That has been really effective in helping get people unblocked more quickly. We have an AI agent working to fix some of the bugs for our app, especially things that are simple and straightforward. We are using AI a lot in terms of generating some of our content for the app. These are not all just me, these are all different parts of the organization using AI effectively in their work. Branching out in terms of like how to scale AI in your org, it starts with access. Like we've seen here, around 50% of you are using some agentic coding in their organizations. Then we move towards, if you want to scale that more, looking at enabling experimentation through things like hackathons and learning sharings, doing literacy through things like workshops, creating observability through things like your tool dashboards to be able to understand where is AI being used in your organization.
Sometimes the tooling actually can help you say, "I noticed somebody in our Android org is using AI way more this week than last week. Did they learn something that we can have them do sharing on and expand their knowledge?" I've done that a lot, actually. I see people using in new ways, spikes in our observability dashboards, and nudge them to share their learnings. That's been really positive for our organization too. Some people just need that nudge to share what they've been working on. Additionally, guardrails. I did the case study and talked about how we're guardrailing the auto approve bot. My team works a lot with security, audit, legal, to identify what guardrails we want to have in our organization to help us use AI more effectively. I think that that's a story that every organization is definitely working through now. Then, finally, once you have all those building blocks into place, you can look at where autonomy and expansion are occurring in your organization. That's pretty cool.
Before you add AI agents and automate, I want you to ask yourself a couple of questions. The first is, have we built AI literacy in our organization and not just AI access? Do engineers feel empowered over the AI systems that they have in place in their organization? Is learning encouraged and visible? Would AI systems feel empowering and not just imposed in your organization? If AI is going to participate in your workflows, you need to empower your organization first.
Questions and Answers
Participant 1: I'm part of a team very similar in my company. I was wondering if you were looping the actual result of code review, meaning like when they do manual validation, they choose to manually go through code review. Are you looping on which of the manual code review are indeed going to do a change of the pull request? Which manual code review are really involving a change and getting a feedback loop on the teams to say maybe this kind of code review, you should make it automatic.
Sarah Deitke: Your question is about, for code review changes, are we verifying that we're getting the right set in that low-risk bucket to make the code reviews automatic?
Participant 1: It's more like when they choose to make it manual, does this code review lead to code change? Because sometimes they review manually, but in the end, you see like a dozen of exchanges that don't lead to any change of the pull request at all, and it gets validated anyway, which is costing a lot of time to our developer, to be fair.
Sarah Deitke: Are you asking around like how often when people do manual reviews that they are actually leaving feedback that is resulting in another change?
I think my answer is twofold there. One part of the research that I did to get the risk bucketing accurate was actually looking at what code changes were people essentially just rubber stamping already? I found that we actually could bucket pretty well like what code changes were low risk and correlate that with what code changes were already just getting a sign-off with no comments. That was one thing that made me feel really comfortable to say, we should actually scale this in our organization. We have like 30% of our code changes with no feedback, so what's the point of pulling in people if they don't need to give feedback, it's like a one-line change. That's part of it. Then the other part of it is our auto-approval bot is optional. Sometimes people still want to get a manual review just in case there is, like they do want that feedback or that mentorship on a code change.
I would say maybe like 20% to 30% is just ballpark numbers of the auto-approval bot still people just reject it and say, I want a code review anyways. That's fine. We make it optional for that reason so people can have that mentorship. We don't see too many complaints around the auto approve bot being too aggressive to where we would be missing that kind of feedback that does cause churn. One thing I do want to call out is, this, to my knowledge, has never been involved in like a side incident. It does have that level of conservatism where it's not really causing the churn and outages that you might be afraid it would to turn this on for your org.
Participant 2: I think you mentioned that you're right now around 100% of developers already using AI and efficiently using AI. How do you measure the actual efficiency or, in other words, what's the real value for your organization? Of course, you have this speed-up, for example, in a number of reviews or time for reviews. Obviously, this is because of your AI literacy, but not necessary because of the developers using AI. What's your gains, the value from this AI adoption?
Sarah Deitke: I think that is the golden question in industry right now overall is around, we now see 100% of our developers using AI in their workflows, but what impact does that have to the organization? I would probably say it's twofold for us. It might be different for your organization. The first is like people do feel more empowered to use AI to build, like to integrate in their systems. Even in our product, we have a feature called Video Call with Lily, where you can video call in your other language. Having our engineers be familiar with AI makes adding cool AI features to our product valuable. Then in addition to that, it also empowers our engineers to be a little bit more ambitious in their work, maybe to take on projects that are slightly larger than they would maybe even a year ago. That's something that my team talks quite a bit about.
Like, we face all these new issues now of like, I'm trying to work on this new project that I'm not a subject matter expert in. I would have never ever been assigned work like this if I didn't really have the support of using AI and helping me to speed myself up and answer questions. We see that throughout our organization too.
Participant 3: You've showed very nicely the effect on reducing mean time to pull request approval. Have you noticed or have you been able to discern any change in defect rate with automatic approval versus manual approval? Has there been any signal at all in the data there?
Sarah Deitke: For the most part, no. We have about 200 pull requests that go through a day, if my math is correct, I would say maybe one or two of those are reverts of the auto approve bot requests. It's also hard to determine like how many of those reverts are because like we needed a human reviewer versus just like a human reviewer would have. I think engineers do a lot of things of like, I'm going to test the CPU going up and see what happens. "That was bad, let's revert it." They know what they're doing. How many of it is like those cases versus like, we probably should have used a reviewer in the first case. I think this project probably would have been shut down almost immediately if we had seen a significant defect rate like that.
Participant 4: When you actually went through this journey, what were the objectives that you set yourself as your success metrics? Where do you think you are in terms of your objectives overall?
Sarah Deitke: I think that when I first set out to do this project, so Meta, they call it their diff risk score, but it's essentially the same concept. They actually use this primarily in a completely different way, which is gatekeeping high risk code changes to go through at like safer times. I think that's a phenomenal idea too that has a lot of merit. When I first started to introduce that idea, Duolingo is a company that likes to experiment and likes to ship fast, so adding friction was just not acceptable. From there, we could pivot and say like, actually this is still a great system for the low-risk changes, and how can we enable our engineers to ship faster? It's something we've seen in our developer surveys and in the metrics, is like, people really do love to have something like this in their organizations. One of the highest pieces of feedback in our most recent engineering survey was like, "Love the auto approve bot, you guys should keep it up." Kind of a pivot, not really what the original intention was, but having an organization that supports pivots too is always great.
See more presentations with transcripts
