Future Cybersecurity: Hardware Memory Safety, Automated Governance and Post-Quantum Cryptography

Transcript

Olimpiu Pop: Hello, everybody. I'm Olimpiu Pop, an InfoQ editor, and I have in front of me today Chris Swan. I'll let him explain what he does for a living, but I know him as a track host for the InfoQ event, QCon, mainly in London, and he's been part of the program committee for a long period of time. This year particularly, he had the security track on his hands, and given that we had so many things to learn from, we said that we'll just look into what happened and what he predicts for the future. Chris, can you please introduce yourself?

Chris Swan: Sure thing. So hi, I'm Chris Swan. My main job is as an engineer at Atsign, where we're building a platform that makes it easy for people to build post-quantum security into their agentic applications. So I've been doing that for a little over five years now. It involves me in open source, and I also like to spend time in the broader community with things like QCon. It was my real pleasure, sort of at the end of last year to be invited back in order to host the security track. I think at the time I didn't have a particular overarching theme or thread running right the way through it, but on reflection, I think you could say it became about understanding the foundations that we're standing on and the fact that a lot of our vulnerability comes from the leaky abstractions beneath us, whether that's dependencies or underlying platforms and things like that.

And that we can't rely upon human vigilance to keep track of all of that stuff. We need to systematize those things. We need to automate them in order to have the machines constantly pay attention to what's happening in those layers and where the vulnerabilities might be emerging. And I think that on reflection stitches together all of the different talks that were in the track.

The Multifaceted Security Landscape [02:26]

Olimpiu Pop: Yes, it was nice to see all those topics coming together because I looked into this in the last couple of years and one topic that people usually left behind, well, we all know that security in terms of software development is always left behind for somebody else or for a later stage for the next version. But what happened is that you had governance as points of the discussion. I think Sara Wells was talking about it. And then you had Viktor Anderson talking about SBOMs. And especially in Europe, that's or should be an important topic given the CRA that is coming into force and putting the limelight on the way how open source is built and how it's maintained. But what was nice is to see also points about hardware security, how the lower levels are being treated.

And funny enough, in the last couple of months, so in the six months since QCon became history, it just came out that these things are even more important, especially with AI being introduced into the AI space. Any thoughts on how things evolved? If you would've had any predictions in April, which of them came to fruition and which of them are still a question mark?

Chris Swan: So part of putting CHERI onto the track, which was David's talk, focusing on how we can make security better in the hardware was I think we're still at a point where there's an opportunity to make safety better in hardware, which will benefit the entire software ecosystem. And that opportunity comes about because the profile for RISC-V on Android hasn't yet been fully baked. And so it's still possible that CHERI could become part of that. And if we then had hardware memory safety baked into the system on chips that were in popular handsets and the stacks using that, I think that would be a massive boon for our security. Now, we've already seen a glimpse of that. If you look at the latest iPhone system on chip and the latest kind of flagship Android system on chip for the Google Pixel and some of the Samsung Galaxy devices, they've been what I could kind of call cherry-picking from CHERI.

So they've got some hardware memory safety features in them, which are sort of activated with their particular versions of iOS and Android. And it would just be great for that to become a more ubiquitous thing. And so part of my reasoning behind picking David's talk was to raise awareness of the possibility that's offered by CHERI and things like that. And so in terms of looking back and how we could have been looking forward at the time, getting better hardware protection in place seemed like something worth advocating for. It kind of runs through Alex's talk about the layers that we stand on. Alex didn't talk a great deal about their day job at Edera in terms of what they're doing there to protect containers, but I think being able to anchor what we're doing in software down to the hardware, which is what they do at Edera, is going to become all the more important.

And what we've seen since with Mythos and Glasswing and now what we could almost refer to as the AI lab leaks, these supposedly agents run amok starting to break into services like Hugging Face is really an escalation in the exploitation landscape. But I think there's also potentially a very helpful side of that because there's ultimately a finite number of vulnerabilities. And so we have the opportunity now to use these things to help us find them all and rid ourselves of them all. So if I can look back 20 years, there was a brilliant paper presented by Microsoft Research, USENIX Security, about does software age like milk or wine? And it was building statistical models of the latent vulnerabilities in software and how these got discovered and potentially exploited. And from that, you could actually build a model of how many vulnerabilities do I think would be in a large code base that I've probably not found yet?

And I think we're actually potentially on the cusp of being able to take that kind of knowledge and say, "Okay, let's grind my way through". And if I look at real world projects like Curl, it feels like they've been on this journey for a while and we've seen that journey accelerate for a bit in terms of the AI tools have helped them find a lot more vulnerabilities a lot faster than they might have otherwise done. But this can't keep going forever. And so at some point we'll find ourselves in a situation where there'll just be fewer vulnerabilities left to discover, and that should be a happier place for all of us to exist in because we won't be worrying as much about zero days and their exploitation.

Olimpiu Pop: Okay. Well, you mentioned CHERI and the fact that both Android and iPhones are looking into adopting this model. Maybe we can allocate a couple of minutes to get more details to our listeners and understand what CHERI actually is. Maybe we inspire other folks to adopt it.

Threats and Opportunities in Security [07:55]

Chris Swan: So CHERI's a project that's kind of emerged out of research at Cambridge on providing hardware memory safety. And so if we look at a whole kind of class of memory safety bugs, there's been a big pushover in recent years to start using more memory safe languages. And so, one thing we hear is, "Just rewrite it all in Rust".

But if you look at the size of the problem, there's something like five and a half billion lines of open source C, C++ and something like 70 million lines of Rust at the moment. And so if we were to plot our course of how long it would take humanity to rewrite all of that C and C++ into Rust, even with AI assistance, it's almost an impossibly long duration. And we've also seen projects where rewrites in Rust have introduced new logical flaws because okay, you get memory safety, but it doesn't mean that everything's invulnerable. You still have to think through the other ways that systems can be attacked.

The rewrite of some of the coreutils into Rust has unfortunately thrown away decades of hard won experience about what can go wrong with applications. You end up with functionally equivalent replacements, but that have some new bugs in them and might not be memory safety bugs, but they're still potentially security problems. What CHERI does is say, "What if I could catch out of bounds errors, the kind of things that cause buffer overflows, et cetera, in my hardware and raise an exception out of the hardware?"

And then I can take all of my existing C and C++, do a recompile to make it CHERI aware, and that should be pretty much all I need to do. Some cases there's a small amount of rework involved, but the research so far suggests that that's a fairly light load. And so we can now have applications that are unsafe in terms of the language that they're using, but the hardware is ensuring that the safety's there. So we don't have to do this potentially huge scale rewrite into Rust. And so getting memory safety from the hardware is ultimately going to be a lot cheaper than redoing all of the software.

Olimpiu Pop: Given the points that you had, having it in Android and iOS will have a very wide span, given the OUB code is the nature of the mobile phone. If it would be to have a magic wand and you should pick another platform that would have it tomorrow, which one would that be?

Chris Swan: So it would be RISC-V. And I say that because RISC-V is still nascent as a instruction set architecture in terms of its adoption. It's still standards-wise a little bit more malleable than the x86 and the Arm stuff that we've all become accustomed to. And I think having the CHERI instructions in the Android profile for RISC-V would be something where we would see the emergence of an entire new class of devices, and it wouldn't be the high end ones like the flagships that presently have these memory safety features. It would be entry level and mid-level phones and tablets and stuff like that, that would come with memory safety. And I think that would then drive memory safety ubiquity. CHERI kind of just missed the boat to be able to do that with Arm on Android first time around.

And there was potentially an opportunity in the past to push it in, but that opportunity wasn't taken. And we've kind of got another swing at it now with RISC-V. But I think if we did that with RISC-V, that approach would very quickly find its way into the mainstream for everything.

Olimpiu Pop: Yes, probably given the open source nature of RISC-V, we'll have hopefully a better blast radius, but fingers crossed that we get closer to having a more memory safe ecosystem. Based on the things that happened in the last time and the points that we heard during QCon, what do you think would be the minimal requirements or the recommendation for people responsible for security in companies these days? Because things change a lot.

Chris Swan: This touches very well upon Viktor's talk and Sarah's talk. Viktor was talking about software bill of materials. As you mentioned, that's due to become a part of the European Union Cyber Resilience Act. That act, the CRA, is now in force in terms of breach reporting requirements. So that took effect just a few days ago as we're recording this. The requirement for software bill of materials takes effect next year, next December. And what that means is anybody selling software or things that contain software in the European Union is going to have to have a software bill of materials that goes along with that. And that really, I think, achieves two things. One, it provides evidence that an organization making stuff with software is paying attention to their security. They have to pay a certain amount of attention just to generate the software bill of materials, and the software bill of materials becomes in evidence for that.

The other part of it is that once you've got that software bill of materials in hand, you can use that with other tools to determine your exposure to vulnerability. It was really Josh Corman who started this off a few years back where his idea was to make software bill of materials mandatory for the federal government in the US. And the notion was nobody would want to sell known vulnerable software to Uncle Sam because obviously if you're saying, "Hey, I've got my thing and here's the software bill of materials".

But you can see it's got a whole bunch of vulnerable dependencies in it, then the buyer of that's going to say, "I don't think I'm happy with that. So either you're going to fix that or I'm going to pay you a lot less money".

And if that applied to federal government, federal government's a large enough buyer of things that it would become a kind of community benefit that we would all benefit from that. And that sort of happened with the executive order around software bill of materials in the US and some of the work that was done by CISA after that. But I feel like the EU has really picked this up and run with it now. And CRA is the thing that's actually going to make it happen on a wide scale.

That sort of dovetails into how Sarah was talking about how we do governance and how we can build into our delivery process, the mechanisms by which we're evaluating security and producing evidence of security. If we look at a continuous delivery pipeline, one of the things I can do in a continuous delivery pipeline is I can create my SBOM, I can have a bunch of SLSA attestations about how the software has been built, and I can do automated processes like open source security foundation scorecards. And if I use a kitchen analogy of that, I can show the customer of that software that we've used the right ingredients, that we've prepared the meal properly, and that the kitchen has been hygienic throughout the process.

And so providing that evidence of security is I think a good thing in the overall software supply chain. But the point that Sarah put across there is so much easier if you build this stuff in. So you talked about security being an afterthought, and I think it often has been, but if we do the whole shift left thing and build security into how we make our applications, it's not actually a massive amount of effort. And once it's built in there, then it's a continuous thing that we can get evidence out of.

Olimpiu Pop: That sounds quite right. And looking at it's just a matter of choice these days because things got a lot cheaper with AI, but do you see LLMs as a opportunity or a threat in this ecosystem?

LLMs and Post-Quantum Cryptography: Opportunities in Disguise? [16:09]

Chris Swan: It has to be both. A bad guy with an LLM is a threat because they can do damage quicker and at a larger scale than they would've done without. But LLMs used defensively is also an enormous opportunity. And so what we're seeing with things like Project Glasswing is the opportunity for defenders to be able to use the models to evaluate their software and discover vulnerabilities hopefully ahead of the attackers. And I think more broadly, we're seeing LLMs being deployed to do white box testing, so internal source code analysis, where in the past a project might have waited until it's just about to deliver and then done a pen test and maybe somebody would've done some source code analysis at that point. And now people are just asking their programming assistant, "Can you run a security evaluation of this thing?"

And I'm seeing it becoming more common for that to be a standard part of people's process and delivery practice. And I think that's going to be ultimately very helpful.

Olimpiu Pop: One of the points that just thinking about how the track looked like, one thing that missed that it's still seen as futuristic is post-quantum cryptography. That's more or less your day-to-day business. What is that in terms of evolution at this point?

Chris Swan: I didn't pick a talk on post-quantum cryptography for QCon this year, and I've been asked to do the track again for next year. Almost certainly there will be at least one talk on post-quantum cryptography. And I think it's sort of like a timing thing. SBOM was clearly something that we needed to start talking about this year because of the CRA mandate becoming so close. PQ is something that we need to talk about next year because Q day is getting close. And actually, if we look at the forecasts on Q day, which is this notion of the date by which we might have quantum compute capability emerge that would be able to implement Shor's algorithm and hence break the asymmetric crypto that we've been using with classical algorithms such as RSA and elliptic curves, this stuff's stepping up closer and closer. I did a talk myself about this just a few weeks back and planning another one.

And the title for that other one is sort of we're trying to build the house at speed on wobbly foundations. And so the thing that we've been finding with post-quantum implementations is the standards for the cryptography itself, the algorithms are settled. So there was a NIST process, winners emerged and all of that's been standardized, but the implementation landscape is very immature. And so things like library availability, OpenSSL now has the implementations for ML-DSA and ML-KEM, but you will only have that OpenSSL if you're on the bleeding edge distributions. And so if we look at an enterprise landscape, they're probably not on those distributions yet. And so you're talking about having to migrate stuff to the latest distros just in order to implement your post-quantum initiative or potentially back porting stuff, which again, kind of then gets a little bit messy. But I think one of the things people are grappling with at the moment is simply knowing where all of the cryptography in their organization is deployed and all of the things that they have to touch.

So I can see this being a bit like a rerun of Y2K where first of all, we need to track down all of the places where we've got these problems and then we need to grind through the process of updating our software to be crypto agile and putting the new algorithms in. And our focus has been very much on crypto agility because there's a lot of distrust associated with the new post-quantum algorithms. And I think that comes about by the fact that so many of them failed during the evaluation process. And then there's one of the lattices has succumbed to a sort of brute force attack using Anthropic's AI tools. And so everybody's kind of side-eyeing the algorithms that have emerged from this process and going, "Well, are they going to hold up in the long term?"

And I think we have to imagine, no, they probably won't. And so we could be doing all of this again in a few years. And so we don't want to be baking the new algorithms too tightly into anything that we do now because we're probably going to have to replace them again.

Olimpiu Pop: Yes. You touched on a very sensible point from my perspective, running multiple distros or multiple versions in parallel. And one of the things that I wrote, I think in the last week, a researcher found out that the way playing out with GPT-6 Cyber allowed him to just circumvent his whole virtual machines infrastructure and then use zero days or the fact that some of the known bugs were not backported to LTSs, older LTSs, even though they were supported. What would be, I don't know, short list of not the regular Joe, but people that are not on the bleeding edge of technology? What should be the things that we should care about in terms of the Linux distributions, patching or using the latest security releases?

Roots of Trust, Infrastructure Patching, and Automation [21:41]

Chris Swan: I think most fundamentally we need to think about how we establish trust and where our trust anchors are coming from. One of the doomsday scenarios that's been sketched out for this is the private keys for the certificate authorities that we use as kind of the basis of much of the trust, especially in the public internet sphere, could be reverse engineered. And with those private keys in hand, then I could start pretending to be Let's Encrypt or Google or whoever else and issuing certificates that nobody would be able to tell were the wrong certificates. Now, actually people would be able to tell because we've got a certain amount of certificate observatory stuff going on and Moxie Marlinspike's done some amazing work in that area. But one of the things we're waiting for here is for the certificate authorities and those roots of trust to actually move to post quantum so that they're ready.

I think this has been part of the move that's going on at the moment to reduce certificate lifespans. So we've seen certificate lifespans kind of going down from a year and heading towards 90 days and then 45 days. In terms of organizations thinking about this, some of this is we're being dragged along anyway. Whether you think about it or not, your certificates are going to need to go down to these new issue deadlines and you probably need to have automation in place to cope with that. How that then relates to things like being on more recent distributions and achieving patching and stuff like that, it's all important. And I think this comes back to Sarah's talk and the governance around it. It's so much easier to do this if you've got straight through processes where you can achieve continuous delivery versus the more kind of hand cranked ticket driven processes that many organizations unfortunately are still stuck with.

And so achieving that shift is a benefit that's going to keep on paying back. This also relates to some of the AI stuff we've been talking about in terms of we're seeing massive patch bundles at the moment. So the last Microsoft Patch Tuesday I think was somewhere around a thousand patches in a single bundle. If organizations haven't already been able to catch up with simply being able to get the Microsoft Patch Tuesday out as soon as possible once it hits, then they're potentially in trouble because as soon as that thousand bugs is out there, then people are using AI, et cetera, to be able to come up with exploits to go after those. And so, one of the things we're dealing with is the shortening of that window between knowledge or even rumor of vulnerability and the exploitation of that vulnerability.

Olimpiu Pop: Yes, probably it's worth mentioning another point. We looked into AI being a threat. We looked into AI that can be a help into patching quicker, but also obviously as you mentioned, this acceleration means a lot more bugs out there that can be used for other stuff. But the other day, security researcher, I miss his name now, he found a zero day in Muse, the Meta application. And what was interesting about it's again, the human in the loop is the problem in terms of security because what happened is that you have MacOS that built a quite decent way of putting privacy in place in the Mac applications, in Apple applications. And now we are just giving them access. We give agents access to our file system, to our privacy settings to just be able to help us in terms of doing chores for us. And that's a whole different perspective into we have bulletproof security, but we are just giving the key away for our comfort into, I don't know, small stuff.

Chris Swan: So this has been a festering problem long before AI, but AI is kind of rubbing our faces in it. As I prepare for next year's security track, I reached out to all of this year's speakers and said, "Who would you love to see talking on next year's security track?"

Viktor got back to me almost straight away and he was like, "I want to see a talk about AI identity".

And it's something that he's actually started writing a little open source project about. But we've always talked about a notion of least privilege that we should give people just the privileges that they need to do their job. And actually this notion of non-human identity has existed since way before AI. And if you look in most organizations, they've probably got something like 10X more non-human identities than they have people working in the organization. And of course that's exploding now because of agents.

It's not a new problem, but it's something that I think is often being overlooked and a little bit brushed under the carpet. And so absolutely, we should not be giving our AI agents our keys to the kingdom and saying, "Off you go, do some stuff on my behalf".

We should be very carefully controlling task-based fine-grained permissions to do the thing that needs to be done and no more with all of the appropriate auditing and surveillance that should go along with that as well. And I think what's actually happening here is stuff that has been, in air quotes, "enterprise problem" and solved to a certain extent by enterprise identity management and the associated platforms that go with that is now becoming an individual problem. It's a hairy one to manage, and we don't want to be kind of manually thinking about this stuff. And so we now kind of get into the sort of Ouroboros thing of we need an AI to help us manage the permissions that we're giving to our agents because otherwise it just becomes unmanageable.

Olimpiu Pop: I don't know if it's fair to ask you to provide more teasers about next year, but for me, two points. Well, you mentioned one and that's post-quantum computing or post-quantum cryptography. That's something that you have on your mind. You did mention some kind of security net for agents. Is there anything else that you would like to tease?

The Future of Cybersecurity: Systematizing Security Across the Full Tech Stack [28:18]

Chris Swan: I think there will be some more low level talks. So there was a talk that we had lined up where the speaker couldn't actually make it this year. So I'm hoping that they're going to be able to return and we will get the newer, better version of their talk, which will be examining some of the lower level things that we can do to build security in. And I think that applies in terms of humans and agents. The working title at the moment is about how do we defend systems from humans and agents? And I think actually there's a common convergence there that the attack surface area is the same whether it's a human or an agent. The nature of the attack and the scale of the attack is probably very different between human and agent. If you look at the postmortem of the Hugging Face incident, it was described by CISO, I think, like having 10,000 drunk house robbers show up all at once.

And that sounds like a very chaotic scene, but I think that might kind of perfectly encapsulate where we are at the moment. And even if each individual attacker in that case isn't particularly capable, just the sheer scale brings a whole bunch of problems on its own. So I think I'd like to get a talk that sort of looks at that aspect of what we're confronting, but I want to kind of ground it in the reality of if you're doing a good job of defense, then that's going to defend against the human attacker, the AI agent attacker and the sort of human AI hybrid. And so getting that stuff right is always going to be the helping yourself out way to go.

Olimpiu Pop: Okay. And just to summarize the points that I think I heard, going to lower level might help us as an industry given that that will provide a better foundation. And hence the ripple effect will be a lot broader probably and help different points regardless if we are discussing about agents or humans. And best practices to hold. If we are doing our homework, most often than not, we are closer to safety than not.

Chris Swan: Yes.

Olimpiu Pop: Okay. Is there anything else?

Chris Swan: I feel like we didn't talk much about Andy's talk. Andy's talk kind of sat in the middle between something like Sarah's talk about the high level governance or something like David's talk about the low level hardware implementation. And yet Andy's talk was emblematic of we need to pay attention to how we're building things and the abstraction layers that sit beneath us. And so I'll just kind of return to it because that was the overriding lesson that we have here with security, that it's not just the software that we write, it's the entire stack that we have to concern ourselves with. There is absolutely stuff that you can do to make sure that you're on a good stack. Some of that is about making wise choices in terms of languages and frameworks. Some of that actually can be by using modern hardware assistance. And a lot of that comes to ultimately, how do I pay attention to this?

There's not enough human bandwidth to pay attention to it all the time. And so I have to systematize that attention and that kind of takes us back to what Sarah and Viktor were talking about. But Andy's lesson was don't be just blinkered about the software that you are writing. Look more broadly at the ecosystem that you're writing in and make sure that everything has been properly considered.

Olimpiu Pop: So governance, we shouldn't run away from it. The SBOMs, we need to take care of maintenance and who's responsible for that. And at the point we need to take care of that, if not for our sake, because we are legally obliged, especially if you're doing business in the European Union nowadays. And then it's a lot of things happening behind the scenes on the hardware level, in the abstraction levels. So we have to pay a lot of attention these days on security because it's getting closer and closer to us. And as you said, even if the agents are not particularly skillful, there are a lot of them and that might create new breaches that we weren't aware of before this.

Chris Swan: Yes. And of course today's agents are the worst they'll ever be. The next iterations of those attacks will be smarter, less drunk agents and potentially even at larger scale as well. So we need to prepare ourselves for that.

Olimpiu Pop: Thank you, Chris. Good luck with preparations for the next QCon. You did mention, and I have the same feeling that this year's QCon London was one of the best. Good luck in making it even better for next year.

Chris Swan: We're always trying to make it better than it was last time around. I think '26 was one of my favorite QCons ever. Some amazing speakers who absolutely smashed it. It was just wonderful to be able to spend some time with them. But of course we're always trying to raise the bar, so hopefully '27 is even better.

Olimpiu Pop: No pressure. Good luck with that. Thank you.

Chris Swan: Thank you.

Mentioned:

• CHERI (Capability Hardware Enhanced RISC Instructions)
• RISC-V
• Supply-chain Levels for Software Artifacts (SLSA)
• Shor's algorithm
• NIST Post-Quantum Cryptography Standardization (ML-KEM and ML-DSA)
• Moxie Marlinspike
• EU Cyber Resilience Act (CRA)
• Edera
• Does Software Age Like Milk or Wine? (USENIX Security paper)
• Project Glasswing
• Post Mortem OpenAI Hacks Hugging Face Incident
• Chris Swan Blog Post QCon London