Transcript
Renato Losio: Today we'll discuss how AI is changing software delivery, so from code generation to production, how AI-assisted software delivery is changing in our crazy times.
In this roundtable, we are going to discuss how AI is changing the way organizations are going to build, test, secure, deploy, operate software. What is the real challenge today? We are all familiar with AI in general now and how it's changing every aspect of our developer life, but one of the challenges we discuss maybe less is how is integrating AI into the broader engineering lifecycle while maintaining quality, security, and reliability.
My name is Renato Losio. I'm an editor at InfoQ. I'm joined today by four experts coming from different companies, different sectors, different countries. I'd like to give a chance to each one of them to introduce themselves and share their journey, and why they are part of this panel today.
Patrick Debois: My name is Patrick Debois. I have some history in DevOps and CI/CD. I currently help Tessl with building a software factory for your coding agents.
Martin Reynolds: I'm Field CTO at Harness. I spend most of my time working with our customers on everything that we're talking about today. Everything that's after code, build, test, secure, deploy, operate. Also, I say recently, a few months back, I also now co-host a podcast called ShipTalk. We talk about everything that's going on with AI that's in the news.
Garima Bajpai: I'm Garima Bajpai. I bring 25 years of leadership experience leading software development organizations at an enterprise level. Lately, I'm being more known for my work with AI and how AI would scale when it comes to software development. I'm more interested and intrigued by community-driven initiatives, community-driven InnerSourcing work, as well as hackathons, micro and mega hackathons, which we have been doing.
Gunnar Grosch: I'm Gunnar. I'm a developer advocate at Sinch. I have also spent about 25 years or something in the industry building software, working with developer communities. These days, I build and talk a lot about AI agents in production, and working with Sinch, obviously that means a lot of conversational AI in different ways. I also have a podcast with some friends, called, Yells at Cloud.
Software Delivery in the AI Era
Renato Losio: How is AI already changing software delivery in your organization? We are all familiar with developers using coding assistants. We are familiar with maybe agents in pure development. We hope we are not all just vibe coding, but how is it really already affecting software delivery?
Patrick Debois: We're probably going through these phases that everybody went through. Like you described, first it was your completion, then it was your coding agent. A lot of people still do that with their Claude and their OpenAI working together on their laptop. Then we started seeing, there's a lot of context. We're putting knowledge back somewhere in skills and more information, so the agents become better. We improve the harness. That's all good. I would call that our solo level, team shared level working together. About six or seven months ago, we switched to using a central pipeline, what we call the factory in general, where we're all working more through filing tickets, the agents take care of it, so it's less running on our laptop. That created a different dynamic in the company from the solo dev to actually product, engineering, sales, support, all working together. That's the transformation that we went through. I think that's not unique.
Renato Losio: You mentioned before the entire conversational chat and whatever. What's your take on this?
Gunnar Grosch: I think we're seeing a similar thing that Patrick just described. There are different parts of the development lifecycle that are changing. I think part of it is about trusting it to be able to hand off things. I think we're seeing it in stages. One part that is easy to handle is the research part. Before you start writing code, previously you had to usually do a lot of research, figure out how a codebase worked or how an API worked and so on. Those are things that are easy to hand off, let agents do, and then you get the result back and you can take part of it. Same with the other end of the lifecycle, where you deploy, you test, you iterate, and you have that loop over and over again. Also, fairly easy to hand off to agents and let them take part of it.
What I'm still seeing a lot is that the step in between, that's still where developers or engineers are engaged. They're still doing a lot of the work in that in-between, so handing off the initial work, handing off that last leg of the work, but still doing a lot of the work in between. I think a lot of it comes down to trust, being able to actually hand off that entire work. It's a bit scary.
Agentic Software Delivery, in Prod
Renato Losio: I don't know, really on a personal experience, I can say that if you talk to many practitioners it's, yes, we are experimenting with it. We are doing that. We are doing delivery using agents. When you ask, so are you doing that in production completely and trust, or you did this step? Not yet. Or we have a proof of concept or we have in a small project, but that's a big leap. I think we'll come back often to that.
Martin Reynolds: I think that's something that comes up quite often, especially work with quite a lot of regulated environments, and the trust is a big thing. I think if you had good governance on your delivery process before and you can make sure you can apply it in the same way to agents, then they'd fail the same way a person would fail. The mechanism might not be exactly the same, but I think you need the governance set up the same way, so that they should have things like scoped credentials, policies, approvals, audit trail, so they're not free-roaming agents. Coming back to that gap that was mentioned, it's interesting, because I think you're right. There are things like code review that seems to be an ever-increasing challenge, like test generation, even auto-fix of failed builds and failed deployments, or the ever eternally-growing number of CVEs that seem to exist in the world thanks to AI.
It's like, here's a load more CVEs in your build. All of those things you can use AI to do. You can use it to help you with code review. You can use it to help with test generation. I think it is evolving. One of the fun things about working where I work is that we build a lot of software, and we've been tackling a lot of these problems as we go along.
Garima Bajpai: I actually wanted to add a few things here, because, again, as Patrick and Gunnar alluded to the fact that with AI, the individual productivity has improved. That's a no question that code is no longer the scarce capability. That's a given thing in an enterprise ecosystem. What we see, again, different companies, different communities, different experiences. There are three things which I think are standing out for us. I also come from a critical infrastructure background as well, very controlled and regulated environment. Our testing capabilities, our testing matrices, we see that they are broken now, because we are leading to more next-generation test matrices where the validity for implementing it on a probabilistic LLM application, testing must shift from a statistical threshold to golden data points. LLM-as-judge rubrics comes into play. The second aspect is security. Security must transition to runtime. As DevOps advocates, we have advocated for shift left capabilities, but I think it's also more shift right at this point in time.
The third part, which also from an enterprise perspective, we have been seeing and are struggling with, FinOps, how to adopt token economics at scale. When you work with a 20,000-developer ecosystem, it becomes a very substantial amount of your capability, like how your token economics would shift and how FinOps becomes a very integral part of your ecosystem. These three things I think we have seen that now people are being more conscious about, and we are also seeing governance control levers, guardrails, around our delivery agents to make it more holistic.
Patrick Debois: I love that you're bringing up the compliance, the cost. The way that we started thinking about this is you're building code. Like you said, it's easier. That's fine. Like Martin says, you need to have tests. That is still important. Good engineering practices don't go away. Observabilities that you need, you need the scaling. You need all those pieces in there as well, because what they do is actually they improve the code and they reduce the likelihood of having issues. That's the first piece of you give better context, you write tests, you get less issues. That's one way of dealing with it. The second piece that you see now more and more in people's workflows is a risk analysis. The agent has done all the steps. They added tests. There was an adversarial review. They got a better result. Now it's the fact like, ok, can I actually push this change through?
What is the impact? A simple documentation change is completely different than you touching the authentication piece in your code. People are using ways to look at, ok, should this go directly or not? That is similar to what we did before. A user says this is important, this is not important. This is a risk as well. Then the third piece, very DevOps, like you said, Garima, is when it moves to production, what is the impact? We had the likelihood, the risk, and then the impact. Many people tell me, we don't look at the code anymore. I ask them, ok, what's your application? It's some blog site. It's some random small thing, or it's experimental. Of course, you don't look at it. It has a different risk profile of you pushing something out. That's fine in that. Now imagine your risk is higher. What we did in the past is we architecture around the failure.
What if the machine goes down, can it quickly get back up? Can we restore the data fast? What is the impact? You mitigate in your architecture not just if the risk is there, but it happened, and it was the same with the humans. This just now applies more and more. A lot of people just focus on optimizing the results, but they don't think about risk. It's really hard to get that decision in an organization. What is the risk that we're willing to take? What is the impact of this? I want people to think more about those pieces as well than just saying like, everything will go automatically. No, there's a gradation of what you do. Some things are manual review. Some things are automatic through. That's the balancing act that a lot of teams are now struggling with.
Gunnar Grosch: I think it's funny because these are the same discussions that we've had throughout the years with different types of change with DevOps. Those similar type discussions, how much to automate and should we be able to push things into production and so on. I've spent a lot of time talking about resilience through the years and chaos engineering. I think Netflix was the common example there. Just like we said then, not everyone is Netflix. You don't have to do chaos engineering tests in production just like now. Not everyone is Anthropic and has to have fully automated agents that are pushing 30,000 PRs per day or whatever that number is. Everyone has to start somewhere. You don't have to start with everything just being fully automated. You probably shouldn't even be close to it.
Human Readable/Understandable Code
Renato Losio: Until now, we had some good best practices. We wanted to make sure that the code was understandable for the next developer joining the line, usually, with a few exceptions. That has always been the approach in software, enterprise, open source, and whatever. What's going to be the future there? We say for the pipeline, we're going to deploy it. We're going to have tests. We're going to have anything automated. Do you think readable and human understandable code is still something we should expect, or do we just rethink that already and move the problem to just deploy?
Martin Reynolds: I've had a lot of discussions about this, and I do think it's a super interesting topic because we've also seen that the most effective users of AI are the more senior developers, generally, which is almost counterintuitive. It feels like you still need to understand how it all works. I think ultimately, we will get to the bit where we're not reading any code. I don't think that's going to happen for a while, but I think there is. I would lean into what Patrick was saying about the levels of risk. If you've written a bunch of code for a blog site, do I care to read all of that? No. If it's my banking application that handles all my transactions, I want to still be able to understand that, and especially when it goes wrong. Sometimes you just need to be able to still read it when it goes wrong.
I don't think we're at a point yet where we can 100% trust what's generated. We know it generates more issues. We know that there's been an increase in live issues for code. We know that there's more CVEs in it. We know that we're finding CVEs faster. All of those things tell me that we should still be able to read it at the moment. Future, maybe 10 years down the line, current rate, 2, who knows? We might get to that. I think it will come in stages, and risk is a big factor for that.
Renato Losio: Do you still read all the code?
Patrick Debois: Again, it depends on the feature, what I do. I've set this up to understand better. That debate about like, should we just let the agent loose? I do TDD, and we could say, don't do it. It helps the agent because the agent can run tests and can verify what it built, that it actually worked. This is just an example. Should it write docs? Of course, because if the docs are up to date, it actually helps the context of the agent. It feels like in a way that what was good for human engineering practices actually helps the AI do a better job. If you have multiple ways of standards in your code, the AI is getting confused. If you clean that up, it will do a better job. The code is one thing, we're getting to in some tooling about reviewing the intent while not looking at the code.
Formal verification is making a comeback in a way that we specify what we want, and then it can actually verify that with the tests. That is like a new layer of how do we look at that. We also make the AI agents and the coding agents prove us that it works. They create videos. They show it. They have to really convince us. Does that mean that we're looking at all the code? Again, it depends if I really care about one specific part. If it passes all the tests and it passes all the things, it might be as good as a human would do before.
Garima Bajpai: What my experience goes is, when you take code at an enterprise scale level, of course, we'll have to shift the needle in how we are thinking about the syntax of the code and the two capabilities which we have started to take very seriously is semantic layer. How do we build code through AI and AI agents and how the semantics of that has been built around it. We have invested time in that. The second aspect is forensic, because we don't want code to have vulnerabilities and we don't want to shift the problem from software development to software maintenance. These two components are very critical for our harness layer. Then we also have some circuit breakers. When AI agent starts to write enormous amount of code, so what we have found is AI has been notoriously poor in handling those kinds of large context windows. We ensure that when there is a lot of code being written by agents, we have circuit breakers and we have to have the human reviewers in place.
Testing
Renato Losio: We love automatic tests. We love the thing that now it's possible to create them, and we can delegate testing and PR approval to AI. That's great. I have the feeling on a personal level that is, I like to go back to the standard usual example of the photography that in the past used to be you couldn't do many pictures. You had 24, 36 maximum with a camera, so you were quite careful in designing them properly. Maybe they were not the best one, but you were putting some effort. Now, I saw an explosion, at least in the project I follow, of tests. Now the coverage of tests is great and that's really a great improvement. It's like digital photography, you can create so many. The problem is that there's really a lot of noise. Of course, the big question is, do you use a very specialized AI tool on the market or you build your own, or you just use some regular testing framework? Gunnar, you mentioned before AI testing. I don't know what you use or what you do in that space.
Gunnar Grosch: I'm not using anything that's specific or newer frameworks. It's following the same practices as before, both when it comes to the regular frameworks that we use and then for the more resilience testing and so on as well. No changes in it, but just like you said, with agents, the test coverage looks very different these days. In apps or services where we would create a smaller number of tests before, now all of them has hundreds of tests. The test coverage looks very different with the help of agents.
Patrick Debois: I do two very small things. One is I say files can't be bigger than 200 lines, which restricts the pieces of the change, like Garima was saying. It's not like the big chunks. Then the other one is I actually say you need to have 99% test coverage. In my hook, I actually verify that with my test coverage tool. If it's not complying, either I need to split up the tests due to the smaller files, or creates the test. That's like two nudges that I use very practically to keep that more under control. Indeed, like Gunnar says, it's the same tooling, but we're nudging AI in a way that it uses those tools more efficiently.
Martin Reynolds: I think that's really interesting. I don't know if anybody remembers this. I was talking to somebody recently, and they were talking about the table game where you have your path to production, and you're moving things through, and every so often you roll the dice, and it says that step in the process failed. I'm picking up really on your small changes thing here. Funnily enough, whether you're using AI for all the steps or not, a big change that fails still has the same blast radius and the impact on the down the line and how much gets shipped and the shipping, lots of small steps on that production line when something fails, the impact and the blast radius is still way smaller, whether you're using AI or not. Almost going back to what you were saying at the start of, the old principles still apply. They still work. There's a reason that we developed them over the years. Just because we have AI doesn't mean that we should just throw them out the window.
Garima Bajpai: One thing which I think is changing from our perspective is definition of done, because what we have done is we have added more checkpoints, which are centric to correctness, quality, semantics. Also ensuring that the definition of done, not only the code output is correct, but also how the code would live or be maintained in the long term. These factors are very important. I think in the ongoing basis, we will see the shift happening in that area as well.
The Human in the Loop Bottleneck
Renato Losio: AI can deliver code at a pace that cannot be matched by humans, so humans reviewing the code is becoming a bottleneck. What would be your insights how to increase throughput? I would say don't fire people. That's on a very personal level. I don't know if anyone has really faced already that problem of like the human side being the bottleneck in deployment.
Martin Reynolds: That's pretty much the status quo right now. It doesn't really matter where in that chain you look, because we've talked about quite a lot of things already, but for some organizations, the human bottleneck is at the PR stage, but it could be in writing tests and making sure that they're safe. It could be in security triage and remediation. There's just a whole bunch of areas that rely on humans right now. There is this constant layering of, can we use AI to help accelerate this? Most of these systems were designed for changes that happened at human writing code speeds. We've accelerated everything at the front, and we need to make sure that we can safely accelerate those things to the right. I also agree. I do think it's a layered thing and it is by risk.
Patrick Debois: That's the clue. You can't approve all the things. You can ask for proof. You can ask to do it better. Those are things that will help the reviewer move it on the verification more on the intent layer because you don't need the code. You can verify on those pieces. Those are all accelerators. At a certain moment you say, ok, these things can go up, and that's a big chunk done. Now, unfortunately, like you said, more CVEs. Now it's like you have to maintain your context. That's work again. There's the factory PRs. That builds up. The clue is getting that risk analysis, and not just risk analysis, but the architecture that it can mitigate whatever risk that you're doing. I think that's probably the next phase in dealing with that.
Gunnar Grosch: It is simple math if the number of PRs are doubled, if you don't double the number of reviewers, you need to do it in a different way. You will have to then either do a shallower review, which probably isn't the right way to do it. It's about then, like you said, look at the risk, what are the things you need to actually dive deeper into when it comes to reviews? The solution isn't to automate it with AI either, because reviewing AI, is not the answer to the question.
Martin Reynolds: I know internally, and for some of our customers, we take a risk-based approach. We don't use the same AI for coding that we use for reviewing, just like you wouldn't use the same person to review.
Renato Losio: That's a very important point. Sometimes we take for granted that people don't do that.
Martin Reynolds: You don't want to mark your own homework.
Gunnar Grosch: You want to, but you shouldn't.
Martin Reynolds: You want to, but that's not effective. We take a risk-based approach. We review the PR if we think that is a low-risk PR. Then we might allow AI to accept that PR because the risk is low, based on a whole bunch of metrics, not just, I think this is low, but what's the stability of this? How much has this code been reworked? What happened in the last 10 deployments? Is it a stable part of the system that's changed? Taking all of those things into account, we can then say, this is low-risk, medium, or high-risk. Actually, low risks don't need a person. Then you can also use AI to help you solve that problem, and say, there's a thousand lines of code changed here. It's mostly updated versions of dependencies, but here's the bit of the code that you actually need to look at in the diff, not all the other bits, because they're files that were touched because we updated the version of a dependency.
Here's the bit that actually changed in the functional. You can help accelerate that and take away some of the stress, because there's nothing worse than a big diff, which is why I'm still a massive advocate of small code changes. When you get a diff with a thousand lines, humans are rubbish at reading a thousand-line changes. They are good at saying, I know this is the important bit, and now I can focus on it. It's the same for everything else, but I just want to give this as an example.
Gunnar Grosch: There are things that AI is great at when it comes to reviews. It can do style reviews, for instance, look at patterns. It can see conventions and make sure that the code is then readable for the next person or the next agent and so on. Because, like you said before, Patrick, if all the code looks different, it makes it harder for agents as well. You can use agents to then review for following your style and your conventions.
Patrick Debois: I want to give one example, which I think is a cool addition to this thing. Imagine you have two code changes and two different people have asked it. Imagine this is your billing thing that calculates the total of your invoice. There's a total function that you say, don't do it in dollars, but do it in dollar cents. It creates like a PR, everything passes, it's logical, that code change is happening. At the same time, there is a person asking for, I want a new invoice report, and that will use the total function, but they don't want to display it in euro cents or dollar cents. They want to still have it in dollars in general. If both PRs land after each other, they are mergeable, they are correct, but they're both semantically incompatible. Because the one change started with the concept of still understanding that it was calculated in dollars, and the other one is changing this concept while the other was landing. Now we have to look at PRs, if they're ok, not just on code merging and being correct as code, but being compatible on intent.
Renato Losio: It's a requirement. If the two requirements together still make sense.
Patrick Debois: Which I don't think we're really checking right now in our reviews.
Garima Bajpai: I think this is an excellent point. We started to also measure intent to production metrics. This is my personal belief that in no time, intent will become the backlog and bottleneck. Because there will be a lot of code into production, but if you are trying to measure mean time from intent to production, the intent becomes the bottleneck.
AI Tooling in Highly Regulated Industries
Renato Losio: How do you introduce AI tools into a highly regulated space with rigid IT security configurations, such as financial institutions and the public sector?
Martin Reynolds: Introducing AI is no different to introducing any other tool. You need to make sure that you have the right controls in place, the right governance in place. All those checks and balances need to be there. It is not different to any other tool. You need to follow those same processes. I would say now, certainly most regulated organizations that we work with, they have a very defined process for introducing AI into their environments. They know what the checks and balances are that they need to add in to do that. In general, governance is the core. I think this might sound weird, but it's more important on the delivery side than it is on the coding side. Because on the coding side, the agent is living on your machine and in the code repository and doesn't have access to production environments, doesn't have access to things that can break, probably doesn't have access to a whole bunch of sensitive information.
When you put that and agents are running in your actual production environments, deploying software, creating infrastructure, whatever it is that they're doing, running tests against that infrastructure, suddenly there's a whole bunch of things that you need in a regulated environment that don't exist. Executing agents in production brings a different level to what is required of an agent that is writing code against a codebase on a developer's machine, and that you have to build in the guardrails to be able to get the trust.
Garima Bajpai: I'll add to some of the points which, Martin, you have mentioned, because you're spot on. There are guardrails, governance, there are teams which are specialized in a regulated ecosystem which works on a central level, how this whole ecosystem should evolve, and there has to be guardrails, for the risk which we carry and the blast radius which we have. There's another risk which has been introduced when we talk about AI as a technology, because AI is evolving very fast. What I have seen in my experience is that with regulated sector, there is a high risk of shadow AI being adopted because there is a central governance which says, apply the brakes. Then people are seeing that the democratization of the technology is happening. I think it's a very interesting challenge when it comes to regulated sectors. Now, what we have done, and I have talked to several communities about this problem, the velocity trap is real.
Everybody is creating agents on their machines, and the experimentation has come to production level. What we have done is we have community-led InnerSourcing frameworks where a particular persona which works as an agent and which fulfills 80% of the jobs, then these agents can live in a centralized ecosystem and everybody can spin off those agents. That way the risk of having shadow AI on your laptop, on your machines experimenting with different tools, it's getting limited because it's a community-led InnerSourcing framework which we have experimented. I don't say that it works for every enterprise and every segment, but again, this is a very interesting problem to solve.
Gunnar Grosch: Before joining Sinch, I worked for AWS for many years, and part of that time was when all of this started, with the AI boom we're in right now. Regulated industries, that was definitely one of the most common questions at that time. The way that question was framed back then was usually that we can't use OpenAI or Anthropic or even AWS, so how can we run all of this ourselves? Luckily, we've been able to move past that as an industry, I think. These days, most of these providers have made it possible for regulated industries to run these AI tools. It is about governance, just like Martin said. Once again, I want to emphasize what he said as well, that there are two different parts. One is the AI tools we as engineers or developers or users are using. That's one part of it, and that's just pure governance.
I think that these types of organizations, they have to do it if they haven't already to avoid the shadow AI that Garima talked about as well. That's where you'll end up otherwise with people running Gemini on their personal phones or whatever it might be. The other part is using AI in their production workloads. I think we're past the point where that was a problem as well. There are so many ways of still being compliant and doing that these days as well. It's just that they need to be able to actually be able to audit things. They need to be able to see whatever they are building is doing. That's where a lot of organizations across the board are failing today. Many AI agents that I see in production today, they are black boxes. People don't know what their agent is doing.
Garima Bajpai: One more thing which is very interesting is it's still a heterogeneous technology. People are looking at their own agents, their own integration points. There is some level of standardization which has happened with MCP, for example. Agents, by and large, are heterogeneous. I think industry experts right in this panel and beyond this panel would work towards standardization because that is the need of the hour. How to make this ecosystem homogeneous, interoperable, because that is something I would look forward to for another one or two years from now.
Patrick Debois: The reason, Martin, why I was shaking my head when you said it's predominantly there, is that when we were developing traditional software, there was also the piece of what pieces of third-party software we could pull in into our codebases, what licenses, and things like that. There is a piece that ends up in that world as well of vetting and making sure things are ok and secure there as well. Now, there's the layer of context that has similar characteristics. You can install skills from a registry. You can download skills, but they can contain scripts and so on. There's a new piece that a lot of enterprises might not be thinking about as this new piece that they need to control as well. It's not just a coding agent. It's whatever you pull in there. Is it PII data? Like, whatever pieces? That is also contributing to whatever upstream your production is doing.
It could be equally harmful if you do it in the bad way, what ends up where to what systems? It might not be the shadow IT, but it's like even in their existing workflows, it might not be mitigated yet on that level because there are new pieces, new threats, the prompt injection, all those pieces, which still feels as an industry we haven't really gotten around as dealing with that governance in a good way.
Human Review in an Agentic Ecosystem
Renato Losio: I like the idea of asking the agent to prove that its work is correct and convince us of it. Besides tests, what other thing can we ask the agents to do to make human review easier, faster, and more complete, as opposed to just keeping when risk is low?
Patrick Debois: One simple thing we do is we have it record the workflows, so the end-to-end workflows, and we say, ok, make sure that it's viewable. I just walk it through it. I'm not just looking at the code, but I actually see it in action functional-wise, which is probably a lot easier. If it is a bigger chunk of work, we ask it to split up the review into multiple pieces. It's not just showing us the div. We're saying, ok, guide us through the div, almost like you would do a slideshow of what has changed. What did you do? The understanding that it helps us do that in our language. It's not just that it understands that it did a good job, but we have a narrative, a set of questions like, what did you do for this? What did you do for that? It could still lie, of course, but it helps us understand better what the change was doing.
Garima Bajpai: One more thing, I'll tell you, start from the foundational piece, the foundation layer, where is your knowledge base? You create or construct a knowledge base. Based on that, you have a harness which talks about what semantics would be there, what forensic would be there, what is your definition of done. Pieces of the puzzle which are changing. It makes the human reviewer's job easier because there is a harness layer, which is working already on these agents, which is making sure that the foundational pieces of the puzzle are correct. It's quality assured. There are security elements embedded. There are guardrails. Then the human reviewers comes in play with a long-term hat on top of their head saying that, ok, how these 20 microservices would exist or be maintainable in the longer run. That is what the role of a human reviewer would be in some time.
Deterministic Controls/Metrics, for AI, in a Zero-Trust Environment
Renato Losio: In a zero-trust environment, what deterministic controls, metrics should be added to ensure AI tools only do what they should, and is measurable?
Martin Reynolds: For me, in zero-trust, it's purely about scoped credentials, though. If you're giving an agent very specific scoped credentials, probably time-limited as well, then it's the same constraints you would give a person. "I need to perform this task. This is the very explicit permissions I need to perform that task. Here is your temporary credentials for performing that task. I know that they're assigned to this agent, which is running as part of this pipeline". Then all of that is audited. In that zero-trust environment, you're doing exactly the same thing that you would do for a human. You're making sure that it's tagged, that it's auditable, and you can pull that information out anytime you need to, to be able to show that it only had those limited permissions, and it only did what you asked it to do, and that the constraints were there to stop it doing anything else.
Gunnar Grosch: It's about scoped credentials. Least privilege is really what it's all about here, and then having an audit log to be able to follow up what actually happened. Then, of course, in certain use cases, running things in sandboxes and so on is definitely an option as well, depending on what that work is.
Garima Bajpai: Quality of your knowledge base.
Renato Losio: I was thinking as well that probably the zero knowledge and as well scoping the credential for what they just need to do, as in the past with just humans, sometimes you were really trusting a bit too much. It's like trying to relax and giving a bit more, like, why should I extend them later? Let's give a bit more permission of what they really need because I trust the person. We should basically really now apply real least privilege.
Gunnar Grosch: I think that's the thing. Previously, we usually talked about the same things. We talked about least privilege and scoped credentials and so on, but we also trusted humans to do the right thing. Good intentions. I don't think we can do that with agents. They will do what they need to do and they will do what they're allowed to do.
Martin Reynolds: We've seen that again and again in the press.
Patrick Debois: I think this is also where you really need an AI gateway to control stuff. Because the other day I was using Claude, and Claude has this thing like OAuth, it could do your subscription. I found a command, set up token, and it gave me a token. Then before I realized it, this was a token that was a year valid. I tried to go to the console, there was no way of revoking this token. It's like, really? I could have revoked this through an AI gateway, but right now it was impossible. There's not even an option to do this. This is where we're lacking. The fact that we're exposing GitHub repos, personal access tokens, it's a hassle. Apps, yes, but then it stays valid for an hour. I think we're all catching up. Like Renato said, we trusted people a little bit, and it was ok. Now it's not ok. They can do things so fast. The tooling just needs to improve that as well.
Gunnar Grosch: I know Anthropic put out the paper around Claude Code and how that works. In that, they're talking about this approval thing you get in Claude Code, where it's asking, am I allowed to do this now or for the entire session and so on? How they've been thinking about how often you should get prompted about it, because I think all of us, after a while, you just start allowing things. It's so easy, because if you get prompted constantly, you're not going to read the prompt in the end.
Martin Reynolds: There was another piece to that thing, I think it was part of the same paper where agents were running like six hours of agent time for every one hour of human time, essentially, because they were running so many agents. Then the time that those agents were idle, which was about 80%, was them waiting for somebody to say, approve, exactly to your point. If you've got six agents waiting for your approval, and they're doing stuff all the time, that suddenly gets to a lot of approvals that you're doing. You're right, humans are very good at saying yes. We've seen it historically with PRs. I've scanned that for three seconds, yes. Engagement in the PR process versus just clicking the button and letting it go, are very different things.
Patrick Debois: There was somewhere an experiment, and they saw that the acceptance rate for AI written code went up in the weekend. It's just because you don't know how to spend your time. It's just, ok. It was very weird.
Martin Reynolds: It's some Saturday afternoon.
Improving and Sustaining Productivity, with AI
Renato Losio: Talking about that, actually, there's one interesting point that is mentioning a related study. We could talk about open-source projects and AI, because it's a big topic, we could spend hours on that. Basically, he's mentioning a paper that looked at 800-plus open-source projects and found that the adoption of AI resulted in a brief increase in velocity to repair, but then reverted to the median after a few months. Have you noticed this? How can we keep productivity up and growing? I don't know if that's just related to debt, or if it's related to the maintenance of the project not being able to keep up anymore with any PR.
Martin Reynolds: I did read about this, and there was quite a lot of response around it. A lot of it was around these open-source maintainers don't have access necessarily to a lot of the enterprise tools that people might use for helping with the reviewing and everything. The velocity basically slowed down because they still needed to review them. They still needed to get the peer review. There was still the same number of people working on them just because the code was generated using AI and they're not really set up for doing all the other stuff downstream. I think also there has been an increase in caution in those open-source projects because so many have been hijacked with some innocuous thing in them that AI is great at finding, but also great at generating. Humans are great at generating them too. It's just if you've got a thousand changes, the chance of vulnerabilities being injected into those changes has certainly increased.
Patrick Debois: The trend there is that open-source maintainers, when they're set up, they don't actually accept code changes anymore. They ask people for their intent change and they have their agent code that intent. They're dealing with it in that way that they source IDs, basically, not code anymore, in a good way. Now, the tooling is also a little bit brittle, like setting up an agent flow in your GitHub, somebody creates like a ticket on your issue board. Is that safe? You don't know. Now we have to taint this and say, this came from an external person, this came from an internal person. It's the same thing in an enterprise. If somebody else puts something up, you don't know if you don't know them, which, in bigger enterprises, it's the same thing, like who did what. It can steer agents. That's another place where people are experimenting with like, how will this end up? How can we keep that spirit of open source and keep it open for people to contribute, but not have to bear the risks of doing this or the burden of a million PRs, as you say.
Garima Bajpai: I say fast is the new slow.
AI Demand - Use Case Driven vs. FOMO
Renato Losio: I'm curious to know how much of the AI demand you are seeing is use case driven and how much is fear of missing out?
Martin Reynolds: I think early on, it was a lot of FOMO, we need to be doing this. I think the landscape has changed. The huge investment that people have put in has made a genuine difference. The challenge has been that they put this huge investment in and there is no discernible outcome. A huge percentage of the projects that were started were abandoned. Essentially, I think where people are now is, this is where we're struggling and this is the outcome that we require. I think it's working from the outcome back, rather than I just want a shiny AI tool that says it might do A, B, or C. I think it's the same also for what organizations are building. I think early on, they just wanted to get agents out there, so they had an AI tool that was part of their suite because that was the cool thing to do.
I think now they're being much more deliberate about it. They're being much more focused about it. They're saying, what is the value this is going to add to my customers? What's the outcome that it's really going to drive? Is it worth the investment? I think that that changed because generally business owners, organizations, enterprises, they've invested a lot, but now they actually want an outcome. That is where I think it is moving. Now very much it is really, what are the measurable outcomes that I can get from applying this tool?
Garima Bajpai: I want to add here that, for us, it's a reality check. I think it's an acceptance that our jobs will be changing. There will be new roles in the enterprise. The question has moved from the fear of missing out to building a competitive advantage. What are the strategic opportunities for a big enterprise? Because it's a reality that a startup with 200 people can replace a big enterprise with 20,000 people, because of the sheer velocity and what AI could do. Now what these big enterprises are starting to see is the strategic opportunities, the time to market compression ratio. Which features would be more interesting for the users? Hyper-personalization, for example, democratizing innovation. These are real things which we probably are targeting as return of investment from this AI boom. Now, again, with this, the jobs, the roles, the enterprise, the organization itself will change. This is an accepted reality. Again, how soon it can change, which industry segment you are in, it all depends on a number of dimensions.
The Rate of Change
Renato Losio: There was one message earlier on that said everything has a threshold, nothing is linear in real life, where do you think this log graph is going to plateau? I'm pretty sure if we come back in one year, not even two years, because if we look at where we were two years ago or one year ago, things have changed. We don't have a crystal ball, but they are interesting times.
See more presentations with transcripts