Transcript
Suhail Patel: This is going to be a talk about how we shape our software engineering organization within Monzo and how we think technically, how we think about our technical stack to allow us to ship continuously with humans and beyond, with autonomous agents, with LLMs, whatever may come next. Many organizations that I speak to, to this day, their development and deployment environments and flows look something like this. If you're able to start your own organization, be completely AI native, you may be able to bypass all of this. Most of us in regulated industries and places that have been around for a fair amount of time, I've been on panels and numerous talks where engineering teams in other organizations have just accepted that this is part of their process. You have to form some form of assessment before you want to start anything new. Then you have to do a whole bunch of steps before any code gets even thought about or written.
Before you know it, it takes days, weeks, months to get anything released. A large chunk of that time was the implementation phase. Systems do take time to build. No matter how much planning you do up front, getting the code down in your editor takes time. It takes people. It takes testing. Now, that equation has completely changed. Anyone who thinks otherwise, I think might need to see some of the talks on how quickly it can be to generate code. Yes, you might spend a couple cents, a couple hundred dollars generating that code. Code can be generated really quickly. One of the key capabilities that a lot of people are missing in this equation is that you can iterate on multiple variations really quickly, too, in parallel. What used to take days, weeks, and months and have a high cost of iteration, usually you'd only have time to do one or two iterations of your particular implementation. Now you could have 10, 20 iterations of your particular implementation and find which one is best. That can be achieved in minutes and hours.
Adapting to the Rapid Rate of Change
There's a ton on the internet about measuring developer productivity and being really objective about it. There's all sorts of developer experience surveys. You can have DORA frameworks. You can put in significant amounts of telemetry and be really numbers-driven on where you put that investment. Ultimately, this is the kind of testimonial that proves the value in investing in technology behind the scenes to automate some of the boring processes. When your engineers use and rely on these tools and are able to leverage them for great benefit, that is the best testimonial for your organization, because then other leaders, folks who are in non-tech, notice. Engineers only see what they are missing when they move organization and no longer have access to that same cohesive ecosystem of tools, and vice versa. We have engineers join all the time that are like, we have this other system in my other organization, we're able to move significantly faster.
That's a really good opportunity to learn. When your developer experience has a lot of friction, this is a reason that engineers might choose to leave the company. We shouldn't be under any illusion. The rate of change is also significantly increasing. Many companies are investing heavily into AI to augment their existing engineers. Ultimately, there is a business goal. We want to increase the rate of productivity per engineer. Engineers don't come cheap. There is a strong desire from a financial perspective to increase the rate of productivity per engineer. As engineers, we are willing to be complacent in this equation. We are not freeing up more time in our calendar. We are taking on more work. Therefore, there's higher pressure to go and ship these changes. Part of this enablement also wants to be extended outside of engineering. Tools like Claude and Cursor give access and give license to engineering-adjacent disciplines like designers and product managers to go and prototype with code and make changes directly in your primary codebases.
I was thinking about the first time I downloaded Photoshop and thought that I could be a designer. What an illusion that was. Or the first time I opened FrontPage and thought I could design something for the web, seeing all of that code being generated. Nowadays, if a product manager or a designer has got a small bug or a technical implementation question or question about data, for example, what are our current business KPIs and where are they trending? Why spend time creating a ticket to go ask a data person or to go ask an engineering person to go and fix that particular thing or investigate that particular issue when you can take a shot through an LLM to go and solve it for you?
On the flip side, even some of the largest companies with tens of thousands of engineers and numerous teams dedicated to change management and software testing and software quality aren't immune to the rate of change and the impact on reliability and availability. There was a Financial Times article recently. It went extremely viral across like all the social media and Hacker News and all the various places that AI was to blame for a bunch of reliability outages within Amazon. Amazon had to publish a retort on their blog, because the article said that senior engineers had to sign off on AI-assisted changes. Amazon did correct the record that these weren't AI-specific changes. It's very clear how the industry is viewing the role of AI in software development. They had to justify, they had to put out a public retort on why this was not the case. Putting aside whether the retort here was due to change that was human or AI-driven, there was a shared goal in not wanting to break things.
Nobody wakes up in the morning and instructs their agent to go and write bad code, just like a human doesn't wake up one day to write bad code. Nobody wants to be in the news because of outages. On the flip side, I find it fascinating to see what processes companies put into place after an incident occurs or a deployment goes wrong. This is how rules in our industry, like we don't deploy on Fridays, come about. A really common practice is the introduction of change approval boards, especially in large financial institutions. A group of individuals, usually looking quite miserable, like they are in the photo, typically across a bunch of areas, like risk and compliance, and platform, and security, and a bunch of other areas, go out and have the ability to go and stop and delay a release if necessary. Then you end up in really miserable and unproductive meetings like this to go and ship every small change.
In practice, these forums become necessary because the scope and nature of these changes have a large blast radius, or changing things that need a lot of confidence building. You can't just vibe code your confidence out there. You get to a point where certain software, you lose the confidence to make change. Then you have a lot of inertia from your engineers to go and change even a simple configuration value. That risk profile even mentally becomes exponentially higher. Even technically, that risk exposure might be quite low.
Background
My name is Suhail. I am one of the principal engineers at a company called Monzo, based in London. I work in the platform group. I help lead the platform group, where we provide all of Monzo's underlying infrastructure, libraries, and tooling, so that engineers can focus on building a bank and not have to worry about infrastructure. I spend most of my waking hours thinking about developer experience. If I'm being quite cynical, what can developers do wrong? How can they ship bugs to production? How can I avoid them shipping those bugs to production? I know firsthand how a fractured experience leads to frustration and then leads to more bugs and more incidents. Monzo is a consumer-facing retail bank in the UK. We've also got a small presence in the U.S. We are also expanding into Europe geographies this year. We have 14 million customers and growing rapidly across our personal account offering and business banking as well.
We don't have any physical branches. We power all of our banking features through an app. There's no opening hours to worry about. You handle all of your banking needs at a time and place that is convenient for you. What that means on the other side is that we don't have downtime. People expect our systems to work 24/7. I still get notifications from banks, especially a lot of those based in the U.S. If any folks are in U.S. fintech, I'd love to speak to you about how they're going to be taking down their systems for a couple hours to handle daylight savings. Customers nowadays expect a flawless online experience, especially if you can't service them in a physical bank branch. That means we need to have a diverse offering that is reliable and accessible. Having downtime and having bugs means that customers don't have access to their money.
Anatomy of a Microservice
This is quite an old picture, but it's a connected call graph of many of our microservices that power our payments infrastructure and also our product offerings. Every dot represents a microservice that we build and operate and write on our platform. The lines connecting them represent RPC flows between those services. Imagine every time we wanted to go and make a change or add a new service or build a new system, add a new dot or add a new edge, we needed to go through a complex change governance process. We'd basically get nothing shipped. This talk is on some of the key investments that we've made to facilitate that ever-increasing rate of change. To start with, let's start with the anatomy of what a microservice looks like. This is our developer experience contract to engineers that work at Monzo. A key principle that we've had ever since we've gotten started is uniformity in how we write our services.
For the vast majority of microservices, we build and follow the same template and use the same underlying abstractions. Our platform and security teams deal with the infrastructure and then also embed the best practices into our service layer within these libraries. Within a microservice itself, the engineers filling the top part, inserting their code in that little thin envelope, this is where they add their business logic. Engineers are not rewriting core abstractions like HTTP handlers and consuming from a Kafka queue, or metrics, or marshalling of data. They can rely on a well-defined, well-typed, well-tested set of libraries and tooling. All of these shared library layers provide batteries-included telemetry by default. Especially in a microservices architecture, one of the key questions is figuring out the murder mystery. If you've got communication flows happening between services, how do you figure out what went wrong and where the data flows happened? In order to do that, you need to have really good telemetry. We embed that right at our library layer to have that across all of our services, whether your service is serving one request or 1 million requests per second.
We've been quite fortunate. Our platform stack is relatively modern. We spent some really early innovation tokens in 2016 betting on technologies like Go and Kubernetes and microservices as part of our foundations. Since then, we've been very deliberate in adding to that stack and introducing new technologies and integrations. A key principle for us is that we don't expose these platform-level components to engineers directly. Instead, we build Monzo-specific library abstractions on top, which provide an easy-to-use abstraction that is well-tested and well-maintained and especially has that telemetry built in. These foundation bets have massively paid off. We have built and scaled to over 3,000 microservices running in production, and are shipping changes hundreds of times per day. We've only got a couple hundred engineers, and we're shipping many hundreds of times a day. Engineers don't need to understand the semantics of Kubernetes or YAML or complex Kafka library logic in order to write services and get stuff shipped.
They want to have key primitives and key abstractions that work really reliably and durably. We maintain a monorepo of all of our services, and in that monorepo is a bootstrap generator which outputs a consistent folder structure for every service. As I mentioned, the vast majority of our microservices are written in Go. This reduces the cognitive overhead of engineers. Whether you've been around for a long time or just joined, you learn the muscle memory of navigating a service. All services look broadly the same. They follow the same folder structure, and you expect your database logic to be in the dao folder. You expect your queue consumer logic to be in the consumer folder. You expect your RPC layer to be in the handler folder. You know where these bits of logic are. We follow the exact same software stack across the board.
A key goal for us ever since the early days is to keep that iteration time really rapid. When you generate a microservice, the first thing you do is fill in an API spec in a proto file. We've written generators on top of those proto files to compile the code into RPCs behind the scenes that embed with our service HTTP libraries and provide that batteries-included telemetry. This definition can include things like validation, which is auto-generated beyond the standard data types that you get in protobuf. This generates a whole bunch of Golang code behind the scenes so that engineers are not handcrafting HTTP requests and unmarshalling and stubbing of data into Go types. Testing and mocking becomes far easier because you're working with static types. We're using proto files to generate much more than RPC code. For example, we also generate queue handling logic. Queue handling, same basic principle.
You're ferrying data over some transport bus, so you can define that as a type. We will generate all of the communication primitives behind the scenes. Engineers can specify an event payload definition as well as all of the validation criteria similar to RPCs. Again, we have a proto plugin that will go and generate the Kafka consumer and producer logic calling into our libraries. Engineers can work with these strongly typed objects and auto-generated function signatures and not have to worry about that underlying transport. One big benefit that we get as an infrastructure team and for our security teams is that if you have any bug fixes or any new features or performance improvements, these can be applied across the board. We just regenerate your service stub and you get those benefits for free.
The Paved Path
You're going to see this as a bit of a theme. We've spent a lot of time building and operating our own tools from scratch and leveraging open-source tools as well with a deep integration in mind to fit within our ecosystems. Many of these tools are built with modularity in mind. We want to expose the features that these tools and packages have to offer, but in quite an opinionated and batteries-included manner, rather than just throwing tools across the wall and seeing what sticks. Our goal is to spend our time and energy getting engineers and making it easier for engineers to do the right thing, to get them onto the paved path. Make it so compelling, make it easy and fast and secure by default, that they choose to build for software on the paved road. Then we make it really easy to release that software with confidence.
This is how I believe you build a really good developer experience, when you hyper-focus on helping engineers build the right thing well, and then getting that code to production and release quickly and safely and iteratively. When an engineer joins, and in order to get permission to work on our backend and go and deploy code to production, we have a very structured backend 101 guide, which takes you through all of the tools that we have and all of these primitives. Once you've learned it once, it is something that will stick with you throughout your entire career at Monzo. We take you through all the tools that we have, how we write systems, our opinions on writing Go, our logic, how we structure code, our microservices architecture, and much more. One of the big benefits is that we're able to take this documentation, all the standardization that we have in mind, and encode it as skills to put into LLM tools.
These LLM tools are not hallucinating service packages and service structure. They're able to go and navigate code similar to how an engineer would go and navigate code. We could just surface this documentation directly because we've spent the time and energy standardizing all of our tools and all of our processes. Documenting it for humans, and now we could put it into augmenting engineers as well.
Coming back to that service anatomy, automated tools like Claude and Cursor learn our processes for writing code just like humans do and conform to the service structure. Another big benefit is that we have 3,000-plus examples in a monorepo that you can go and reference. These AI tools are constantly learning about our consistent patterns and are able to go and look at implemented examples behind the scenes. They're not just hallucinating random stuff out of thin air. In the end, each service is a couple thousand lines of actual business logic. That size is really understandable for a group of engineers. You can put out a PR, and it's something that is very easy to understand. It's also really easily referenceable within a context window. Quite a few distinct services get involved in real time when you make a transaction on our platform. For example, when you tap your Monzo card or any payment card, many services get involved in that flow in order to make a decision.
We have a very small time budget in order to respond to whether we accept or deny that particular payment. Things like checking if you have the right amount of money in your account and whether this is fraud, making sure that we're protecting against financial crime and a bunch of other areas that we need to categorize for a particular payment to be accepted. A key challenge that we have is that engineers are constantly making changes to the system to adapt to the rapidly changing environment. Engineers want to be able to test these flows end-to-end. We have a strong incentive to encourage them to do so. Fundamentally, we are a bank. A key principle of a bank is accepting payments and handling payments. That is something that we want to continuously test end-to-end. We want engineers to have confidence in the changes that they are shipping. Historically, we've not had a great local development experience on getting the full stack up and running.
Imagine, in order to change something in this flow, you had to spin up 250 microservices, each with their own dependencies and configuration and a bunch of other things. Then, on top of that, you've got machine learning models that are running as part of this pipeline as well. Machine learning models, then you're setting up a Python environment, and you're praying to the Python gods, let this be the pyenv that works for me and this version of pip and getting all of your dependencies in order. It's a bit of a nightmare. Having that happen across a broad spectrum of engineers is something that is really complex to manage. Imagine we want to scale that to a group of non-engineers as well. Imagine asking a PM to go and install Python, and go and pray to the Python gods and make a sacrifice and hopefully they can get the machine learning model up and running. They're going to have severe friction and that's something that we want to be able to go and solve for.
Development, Iteration, and Staging
There was a really good presentation a couple years back from Soam, where he was representing Stripe, and they talked about how they built around this problem. Stripe would provision remote instances that would get allocated per engineer. An engineer could have one or more instances. They built tooling to automatically sync code from their local machine into this dev box. Effectively, this was an instance that would run in the cloud and it'd check out a bunch of the code, and you can go and spin up the entire Stripe stack, or a subset of the Stripe stack onto this machine. They significantly worked on the user experience to make it feel like you were developing on your local machine. It felt like you were running and testing the change locally. We wanted to mimic this experience. We wanted to allow engineers to go and develop and iterate locally really quickly, but then execute the actual logic when it came to doing the full end-to-end test in a remote environment.
We started with a very core premise. If you want to go and make a change in a subset of services, you shouldn't need to run the entire chain of services to test the change end-to-end. Many of these services I mentioned will have their own data model and third-party dependencies and many other things. We didn't want to put all of that friction on top of our engineers. We had a few options on the table. If you're looking at it from a first principles point of view, one idea is to lean really heavily into mocking and test contracts, and recorded outputs and mocking across service boundaries. That emulation only gets you so far. We found the safest approach is to actually go run the code itself, rather than trying to emulate or mask its side effects. We have a perfectly functional staging environment that is running a copy of all of our services and all the integration is taken care of for you.
We run our entire platform stack on top of this staging environment. It's fully configured and fully operational. We've integrated it into, for example, a test rack that speaks directly to MasterCard and a bunch of other areas. Naturally, what we had is we had engineers converge on mostly testing their changes onto the staging environment and to test their changes end-to-end. Engineers would effectively go and deploy from a branch, and go and deploy that onto staging. As we added more engineers, we encountered numerous services under contention. For example, if you're changing something within the account service, we only have one account service running. Engineers would be contending to go and make changes into that account service. Multiple engineers wanted to go and run their code. We needed to come up with a way so that numerous engineers could run the same service without having to run the entire platform and not disrupting the entire shared global namespace.
Just a quick primer on how our staging environment works. When a request comes into our staging environment, we route the request just over standard HTTP to the service running within the same staging namespace. For example, let's say that you've got the pink service on the left. It wants to go and talk to the Aqua service at the bottom. It may have dependencies within that call path on a bunch of other services. It'll go ping-pong some requests and send the response right back. What we did is we built a layer where engineers can spin up their own individual namespace within their staging environment and deploy only the services that they have changed. We call this system tenancies. It means that an engineer doesn't need to run all of their services that are part of the end-to-end request chain that they are testing. They don't need to know about all the 250 services and get them running in a completely separate environment.
That means they're able to test their scope changes end-to-end. For example, if they have a service that is being depended on as part of that path, it goes and calls into the shared namespace. We also balance not spinning up dedicated infrastructure per engineer. An engineer can have one or more tenancies running at a time. These tenancies are namespaces in our staging environment, so they are really quick to provision. For example, if you've worked with Kubernetes, we've got a Kubernetes namespace. We also expand onto that with other infrastructure abstractions as well. A Kubernetes namespace is very easy to create and very easy to destroy. These are namespaces in Kubernetes and in Cassandra and Kafka. We've got intelligent handling of this at our library layer. When a request comes in and is routed via a tenancy, we intelligently route these services that are not running in the tenancy back to the same shared staging namespace.
This is bidirectional. A service in the shared namespace can call into the tenancy if it needs to route outwards. The method to enable this isn't very sophisticated. What we do effectively is add a header. We add a header to every request, similar to how you'd add a header for OpenTelemetry, or any sort of thing that is passing across request boundaries. We add a header to say, this is running as part of a tenancy, so you may need to do something a bit funky with your request routing. We interject all of these requests routing config at our service mesh, and we do this intelligent routing.
This works really well for stateless requests. We also want to avoid cross-contaminating of data. Example, let's say that you've got a service deployed in this shared namespace, but you generate some garbage data. It might look valid for the services that are accepting it and pass all the validation checks. For example, it's still garbage when you think about it from a logical point of view. That should not permanently contaminate the queue and datastores of the shared service. What we did is we implemented handling of this at our shared library layer. Each of our shared components supports tenancies. They understand what it means to handle a request that comes from a tenancy. For example, at our Cassandra layer, that might mean segmenting that data away into its own container. For our Kafka layer, that means routing the Kafka message into its own topic and queue, and not polluting the main topic.
That means that you don't end up with a queue message generated in a tenancy, which might be valid but illogical that blocks up a queue because the code to handle it isn't in the global namespace. As I mentioned, since these tenancies are really lightweight and can be provisioned in minutes, we can have hundreds and thousands of them running simultaneously. When a tenancy is no longer needed, we can spin it down equally as quickly. This paradigm has worked extremely well to solve for this engineer contention problem because we can now provision all of these tenancies for all of our engineers, and engineers can have numerous tenancies on the go. We've been able to extend on this paradigm to also enable this for acceptance tests. Historically, we had acceptance tests running on our staging environment. These acceptance tests would need to be scheduled sequentially and would be flaky if an engineer deployed a change midway to a service that was part of the acceptance test, maybe with some bad code. We increased confidence in these tests, and now we run these acceptance tests as part of a pull request, because we're able to deploy this acceptance test logic into a tenancy itself.
Code Quality, Maintainability, and Security
Once you've got all of your changes lined up, you can go and submit a pull request and go seek a review from code owners of that particular service. Behind the scenes, while you're going and seeking a human review, we're already running a bunch of CI checks and validating the structure and the logic of your code. Before we make these checks required, we're continuously adding checks. We're continuously inventing new checks to add based on learnings that we've had. We make sure to clean up our existing codebase so that engineers are not tripped up by failing checks that may result as a result of code that is irrelevant to the change that they were making. That means that these checks are really high-quality signal for our engineers, and engineers don't default to ignoring these checks. What we've been able to do is add significant friction in bypassing these checks.
For example, you have to escalate to numerous people and effectively have a break glass procedure and justify why you are bypassing these checks. We want to make the paved road the correct behavior, the path of least resistance by mandating all of these checks and having really high signal. I firmly believe that making your checks robust and fast and easy to understand is the biggest areas of technical leverage. It's not something that you could just buy an off-the-shelf product for. There are some vendors out there that will go in and say, you can make your CI this much better. I think fundamentally you need to embed this as a technical discipline within your organization. It takes a lot of effort to make CI checks high signal, and it's not something that I think is optional. When an incident occurs because of a code bug, a concrete question that you should be asking is, how can I turn that into a check so that that bug doesn't occur in the future?
If your CI checks are slow or error prone, this is a really fantastic area to work alongside LLM tools to find the bottlenecks and speed them up. Every PR that goes through production requires a human review, so all services have a human code owner, and it's a mandatory reviewer to make sure that a change is well communicated to the owners of the services. We have services changing all of the time. We have engineers contributing across the board. It's not just that engineers only contribute to services within their team. Engineers are contributing to the entire microservice ecosystem within Monzo, because it's all in one monorepo, and all the service structure looks the same, so you have that element of familiarity. CI gives you confidence that you have built the thing correctly. We're able to check the logic of the code and make sure you're doing all of your error handling and things like that, which frees up time and energy for our human reviewers and teams to go in and assert whether we've built the correct thing.
These are the two fundamental things. You want to make sure that you've built the correct thing and you're solving for the product requirement that you're trying to solve for, and then you have built the thing correctly. Make sure that all your logic is the same. We spend a significant amount of our time and energy having deterministic tests for whether we have built something correctly so that we can spend our human tokens making sure that we have built the correct thing.
An area that is continuously under development, and an area that gets a lot of attention, is the ability to go and reason and review code. It can definitely be a leg up, but implemented incorrectly, you just get a whole bunch of nits, just stuff that is completely irrelevant, not finding big logic flaws, but just pointing out typos. From my point of view, you don't need a fancy AI code reviewer to go and solve all of your problems. Instead, spend the time and energy writing really high-quality CI checks that will pay for itself. It is deterministic. When you are using AI-based reviews, make sure they are high signal in actually checking for business logic issues, making sure that you've built for the right product rather than trivial semantic nits. For us, these CI checks that we have built deeply introspect code to flag incorrect logic behavior or behavior that we want to discourage.
We lean very heavily into the Go AST parser and new packages like analysis to go and build up these checks. What we have is a series of small and independent checks that run really quickly. You can run them locally on your machine to test for patterns that we want to strongly discourage. You can run them as part of a Git push, as a pre-push hook. For example, we have a check to see whether you're importing cross-service packages that we want to discourage. For example, we want to make sure that you're only communicating and importing code from another package when you're communicating over service boundaries rather than just directly importing its code. We want to make sure that services are decoupled and don't rely on internal implementation details. This can also apply to building secure services as well. For each of our services, we want to make sure that we have egress and ingress firewall capabilities.
For example, we have an API service. It's a relatively trivial API service that is printing all of our bank branches. I think it's part of some UK regulation that you need to have an API that, in programmatic form, gives you all of your physical bank branches. For us, a very stable API, it gives you the empty list. This service has no need to go and talk to any other service whatsoever to serve an empty list. It should have no need to go and speak to the ledger or any other third-party service to go and serve this particular API request.
Again, remember our microservices graph. We have a lot of services running, and we don't want engineers managing all of these connections by hand. It's very easy to get wrong and open up a vulnerability. In that branches API, imagine that there was a third-party dependency that was introduced as part of the software stack that communicated to a command and control center. By virtue of having all of this static analysis, we're able to build these firewall capabilities in a really nice manner that is easy for engineers to reason about, that you are explicitly adding a rule to communicate with a third party and immediately block everything else. We generate a series of these rule files that sit alongside your code, enumerating all of the valid bits of egress that you are able to do from your service and check that in alongside your service coders, alongside your pull requests.
These are peer-reviewed and sit alongside that service code. It means that we can add a check in CI to make sure this data also remains consistent on every change, and our security posture remains tight and everyone is happy. For writing these checks, I've usually been a huge proponent of things like Semgrep, because it's really easy to write these static analysis checks. Semgrep also works across a wide variety of programming languages. You can write checks for SQL, you can write checks for Python, and a bunch of other programming languages. You don't need to get into the detail or abstract syntax trees for all of these different languages.
What I found relatively recently is working through these with Claude has been a real enabler to write more native syntax checking tools. It's been an absolute game changer for complex logic testing. For example, let's say that you want to track the lifetime of a variable across functions to make sure that it's not being mutated concurrently. I highly recommend you give this a go. Especially if you generate a corpus of test cases and examples, you can get an analyzer generated by Claude without hand-rolling a bunch of AST parsing code. It can give you really strong inspiration. Static analysis is also a really good way to bring a codebase towards a path of consistency, a path of convention. Introducing a new static analysis rule stops further additions of bad code being merged and promoting areas that you want to deprecate from. Then you can track and focus on removing existing use cases.
Again, this is a really high-leverage thing that you can do with LLM tools. If they're able to run these checks across your entire codebase, they're able to identify all of the use cases and then effectively do some of the grunt work to go and raise PRs and fix some of these issues behind the scenes, and make sure that these issues are fixed because you've spent time writing a deterministic check. Rather than nowadays spending a bunch of time hard-coding exceptions, one of the key things, if you want to make sure that you have a green build, is you're spending a bunch of time evaluating all of the exceptions and saying, won't fix, or, go ignore whatever for that particular check. Instead, you can just one-shot it into an LLM and ask it, can you go and fix this? If not, add an ignore tag.
Telemetry in Prod
Once a service is out there, you need to gain insight into how it's performing. Especially in the era of LLMs and autonomous systems churning out code, it's very easy to deploy buggy or bad-performing code. We use a lot of open-source tools like Prometheus and Grafana and OpenTelemetry to run our observability stack. We invest very heavily into scraping and collecting all of this telemetry data from our services and from our infrastructure. It's not cheap. We pay a lot of money for all of this. We are thankful every single time we have an incident, and we can identify the contributive factors because we have all of this telemetry. For every service that comes online using our batteries-included libraries and tooling, we get thousands and tens of thousands of data points, which we have plotted onto a generic dashboard. Again, we can give access to this to LLM tools to go and help debug our systems during incidents.
This also feeds into alerts. We have automated alerts for all of our services based on these common metrics, which we can track back to a change being deployed. Alerts are automatically routed to the right team that owns that particular service. For most backend systems, performance profiling is usually locked away for various reasons. Special privileges, you need root access onto the machine and things like that. Very few organizations are comfortable with giving engineers the ability to go and run profilers in production. This attitude really needs to change. You need to have capabilities to go and introspect your systems. Many languages now provide out-of-the-box profilers that are really performant that basically add zero or very negligible overhead. We make it really easy for engineers to go and invoke the Go profiler into our backend systems and provide a nice web UI to go and request these profiles. Java has really good tools using Flight Recorder. If you're using Python, py-spy is also really good. Really low overhead.
I've talked about making developer tools really fast. I've been doing a lot of work in the data engineering space lately. Most of these tools are written in Python. It's really easy to write inefficient Python code. Copying lots of objects, casting something between a map and a list. You get this illusion in front of you that by having this one function call, it will be relatively cheap. Then you have a million operations that are being done in a DataFrame. This very cheap logic has become very time consuming. A simple native implementation would have much less overhead. There are plenty of opportunities to find improvement and optimization. Shaving a couple seconds or a couple tens of seconds or a couple minutes out of a developer tool has a meaningful improvement. There are also tools out there like Pyroscope, which allow you to run continuous profiling. Software is being continuously profiled so you can spot if there's a change that has made a regression in performance.
Putting it in a language that business folks will understand, the faster your code runs, the less servers you need, the less your cloud compute bill is or your infrastructure bill is. To incentivize this with engineers, we have a Slack channel called Graph Trending Downwards. Engineers can shout about their optimizations. They're able to broadcast to the world. We have all of our engineering leaders and our CTO and engineering directors routinely lurk in the channel and celebrate these optimizations, something that you can brag about when it comes to review season. This is a net benefit for the entire engineering discipline. It's an activity that is rewarded in personal reviews and feedback. Again, this is a really fantastic way to leverage autonomous coding. Rather than giving it a large program and saying, find all the ways that you can go and optimize, blindly, you can give it a CPU or a memory heap profile and access to the code.
It can target optimizations to where it's found hotspots in your code. It can, for example, reduce the amount of calls to an expensive function, or go and optimize the expensive function right from the get-go. I know there's teams that have adopted LLMs for bug tracking. For example, you might have just hooked in LLMs in Sentry or some other tools that you might be using. For example, where you can have a button that says, you've had an exception, go and try and fix it by giving it access to your code. You could do the same with performance profiling, and your engineers will be much happier and your financial folks will also be much happier.
Debugging Systems in Prod
Lastly, let's talk about debugging systems in production. It's important for engineers to be able to go and debug the business logic. You're able to debug everything else. Why are you not able to debug your business logic? Sometimes that means querying the service endpoints and seeing what state the data is in, even in production. You can serve this with privacy and sanitization in mind. I see two extremes with this in many organizations. Either engineers are forced to beg and plead to get access to the production database or to be able to go and call endpoints directly in production under some illusion of security. Or, on the flip side, they have to add lots of loglines and do lots of whack-a-mole with telemetry and deployments to go and see, like, "It was definitely not this. Definitely this. I definitely hit this function over here," to go and debug their systems.
What we've done is we've built a CLI that allows you to speak directly to production. This securely authorizes into our platform and allows engineers to go and directly interact with our backend services, manage service configuration, schedule operations, and interact with our infrastructure and much more. It's our opinionated way of exposing bits of functionality that might constitute a chain of requests. For example, configuration management. This is all handled via the CLI. We've very recently added really strong validation into our config management to make sure that engineers are not putting illogical changes into our config system. We've also embedded things like peer reviews. For example, we built our own multi-party authorization system that allows you to have a second pair of eyes to review changes. Or, for example, if you're calling a sensitive RPC, you need to get approval from the code-owning team to make sure that you're not making bad changes in production.
I see many power users proud of their command line history, adaptable shell commands. You might have this really funky Postman script or this fantastic curl command. However, when you build this as a tool, everyone can contribute and everyone benefits. One of the biggest net benefits that we see is that we have non-engineering folks who use the Pound CLI on a day-to-day basis as well. These are folks that are not going to be comfortable with having just a big IPython Notebook or a Postman script.
CLIs Are Making a Comeback
A lot of the tools that I've shown are things for backend and frontend engineers. You can also automate things in spaces like machine learning and mobile testing as well. For example, we have a whole heap of tooling that generates ML service code and gets it ready for training and testing on top of Vertex AI, and then takes a model and then builds it on top of our platform and deploys it. For example, what I'm showing on the screen, we have a CLI tool that interacts with the Android emulator to help automate tasks that an Android engineer would be doing on a day-to-day basis. Especially if you're an Android engineer and you're doing some form of release testing or you're responsible for doing the QA for that week's release, this stuff is a godsend to go and take automated screenshots and do comparing and run automated tests and spin up the emulator, and things like that.
This is part of the day-to-day toolchain for these engineers. One of the biggest net benefits that I see with all of these LLM tools into the foray is that CLIs are making a comeback. A lot of teams, a lot of companies are investing in having really good CLI interfaces. Having these really good CLI interfaces saves you a whole bunch of tokens. Rather than spending a significant amount of time going back and forth, scraping a web page or having a really long instruction set that is non-deterministic, a CLI allows you to do the same thing in a deterministic manner. Take, for example, reviewing a pull request. Are you going to go and parse the 30 megabytes of the GitHub JavaScript to go and render the GitHub UI in a headless browser, to go and look at a pull request? Or can you just use the GitHub CLI, pull the diff, and pull all the metadata in a structured format that is computer-generated and built for computers in JSON?
We have solved for this across the industry and it's very great to see that CLIs are making a comeback. Hopefully this leads to more investment in CLIs. This analogy also extends into enabling changes across a broader group of people beyond engineers. We want engineers and non-engineering folks to use our paved road. Our tools need to be fast and reliable and well understood. Even for LLMs, calling out to deterministic tools leads to a much more reliable result. Think about all the provisioning steps an engineer might have to do and compiling when working on a change. If engineers need to do a whole bunch of gymnastics to unblock their developer environment and pray to the Python or Ruby or Node gods, install lots of random packages and wait for minutes and hours for their software to compile, people will give up and they will try and circumvent your paved road. If it's far easier to go sign up to a third-party platform and ship to Vercel or some other system, then they will go and do that instead. They will find a way. Then, effectively, you have just enabled more and more vibe coding.
Synthesis
To summarize and package this all up, I've shown by example some of the tools that we have built, and these have been investments over the past decade to encode our organizational processes. Take inspiration from what we've built and find places where your organizational processes are falling down, and find impact in automating them and making them easier. A lot of companies are making significant investments in broader accessibility of LLMs, and effectively, you need to have the foundation in order to be able to go and enable that. The barrier to investing in these tools have become significantly reduced and continues to become easier. Investment in those key engineering capabilities, like having deterministic code generators, standardized package management, consistent test tooling, reliable telemetry, really reliable staging environments, and profiling have a huge payoff in the world where we are enabling humans and LLMs and beyond, where the cost of generating code has become significantly cheaper.
We want to facilitate that rapid iteration, and LLMs can augment engineers if all of these other things are taken care of to validate the hypothesis quicker and safer rather than flying blind. Standardizing on a small set of technology choices and continuously improving on these abstractions, we enable engineers and LLMs to go and focus on the business problem at hand rather than recreating underlying infrastructure every single time we use them. Of course, we don't build everything from scratch. We are relying on the foundation of the entire engineering ecosystem that we have around us with open source. We rely on a lot of open source and a lot of third-party services too, but we don't directly project the open-source technology into our microservices and system. We have a very opinionated approach. We have been very intentional. Be intentional in the experience that you want engineers to have and then project the interface to match that experience.
Questions and Answers
Participant 1: Question number one, with having such an opinionated stack and all the libraries built, doesn't that kill developer engagement because they're just solving the business logic, which is not always the most interesting part for a developer. Then related to that, when you come to more low-level high performance, high throughput problems that you need to solve, if they don't understand the libraries that are underneath them, how can they debug and investigate and tune those sorts of things?
Suhail Patel: Let's talk about the motivation for most engineers. I think this is an illusion that we have built up across the industry. Engineers want to go and solve complex engineering problems, and there is plenty of complex engineering problems within the domains that they focus on. For example, if you're in the domain of payments or financial crime, there's enough interesting work in tuning your ML models and coming up with more and more sophisticated rules, or making sure that we are processing payments more reliably to keep those engineers engaged and to keep their mind engaged. It doesn't necessarily need to be at the infrastructure level.
The second question around high throughput and high understanding. We spend a significant amount of time documenting and explaining the constraints within our systems. Our systems are not perfect. For example, we communicate our RPC stack over HTTP. HTTP has its limitations, of course. You might be relying on a particular implementation of HTTP. You might ask, why not gRPC, or insert some protocol over here? What we've relied on is a very strong set of well-tested and well-opinionated frameworks that we have proven to be able to deploy and scale significantly to a large amount of numbers. We have found very few use cases where engineers need to go and breach those throughput limits. Fundamentally, the question that we ask engineers is that, why do you believe that you need to go and go outside of that paved road? Because we've been able to prove it at this element of scale, which usually is far beyond the scale that they are trying to achieve. It becomes from just a thought hypothetical exercise to actual reality exercise, which is, on the ground, we have systems that are doing millions of requests per second with the constraints that we have built. Why are you not able to build your system within that container?
See more presentations with transcripts
