Transcript
Lynch: My name is Billy Lynch. I'm a staff software engineer over at Chainguard.
Newman: My name is Zack. I am a research scientist at Chainguard.
We got to start with the obligatory motivation slide. You're all here at the software supply chain security track/language platforms. You probably have made your peace with the fact that this is a thing we need to be concerned about. Attacks are on the rise, basically, across every stage of the software development lifecycle. It's not just that vulnerabilities are getting introduced by your developers, and then attackers are finding those in production. It's vulnerabilities that are coming in from vendors, from open source dependencies, on purpose, on accident, they're happening at the build phase, they're happening at the source control phase. There's not any one fix because of the number of places these vulnerabilities are sneaking in. You can't just have one magic silver bullet that takes care of everything. Meaningful improvements here are going to require changes to every step of the development process. You see really a dramatic increase in the number of attacks that are coming in from the software supply chain as opposed to "Front door attacks." That's borne out by the data. It's also borne out by the news, and by the policy response to the news. We see numbers of academic efforts, nonprofits being founded, the U.S. Federal Government, both the executive and legislative branches are very invested in this. The media is invested in this, I think with good cause. I think this is an area that's been really under-addressed for a long time. We're starting to finally make our peace with the fact that this is something we're really going to need to think about all through the SDLC.
Why Software Signing?
One weapon that we have, a defensive weapon, one defense that we have is software signing. As you might be able to infer from the question mark after my subtitle on the slide, it is not the silver bullet that you might think it is, but it is a really useful defense, and a really useful part of a secure software supply chain. Why would we want to sign software? We're going to focus on the distribution phase of software development. You've written your source. You've built your source, and now you're trying to get your source from the build machine into production. This can happen in a number of settings. It can happen on a container registry. In the open source world, it often happens on open source package repositories. Software signing is going to help with compromise of account credentials on a package repository. If I'm a developer, and I have an npm account, and I have a terrible password because I made that account 15 years ago, and there's no second factor auth on that account, someone might be able to get at my password or cross-reference it from some other data dump, and basically be able to publish packages on me. Maybe my package isn't such a huge deal, but it is depended upon by a popular package, and now there's malware in the dependency chain of really thousands of applications.
Another thing you really have to worry about is compromise of package repositories themselves. It should be pretty obvious why as an attacker, if you can get into npm, that's an extremely high point of leverage for you as an attacker to distribute malware or whatever you're trying to get out to as many people as possible. Software signing helps clients, end users check that a software package is signed by the "owner," and who the owner is, is a really tricky and interesting question. We'll dance around it a little bit in this talk. Software signing is going to help a lot with that. Once you know who the owner is, double checking and really making sure that the software is in fact coming from that owner. It's not going to help with all attacks. Signing does absolutely nothing for accidental vulnerabilities that just get introduced as part of the normal development process. Someone making a mistake, human error, forgetting to free some memory or what have you. It's not going to help with build system compromises. If the build system itself is compromised in building bad software, it will still go ahead and sign that, and now you have a signature on the bad software. It's not going to help with so-called hypocrite commits. This is a recent phenomenon we've identified, which are commits that look good, but were actually done with bad intent. I'm a helpful contributor to an open source project, I'm going to add my patch. It turns out this patch has a dependency on a library that is secretly a Bitcoin miner, or is secretly scanning machines for credit card numbers in memory. If the project itself accepts this hypocrite commit, signing is not going to do anything with that. Finally, signing is not going to help with an attacker who is able to threaten an author of a library. If you have an open source library, and you go to him with a big wrench and you say, upload this malicious package to the repo you control or else, they're still going to be able to sign for you. Signing is not going to help with that. It is going to help with, again, these really common scenarios, account credential compromise, package repository compromise.
Software Signing Today
What's the status of software signing today? In summary, it's widely supported in open source software, but it's not widely used. Most tools aren't checking signatures by default, and they can't because most packages aren't signed by default. You have this chicken and egg problem. You can't turn on this feature and require it until there's very full adoption. There's a bunch of reasons for this. There's a bunch of usability issues, especially surrounding key management. I sometimes do an exercise where I ask people to raise their hands if they have a PGP key living on some hard drive in a basement or a storage unit or somewhere that they've lost track of years ago. It's just a fact of life, managing cryptographic keys that are really long-lived, is challenging. People lose them. People leak them. You need to reset flow, and so on. That brings me to my second point, if you lose the key, developers today expect to be able to recover access to publish their packages. If I'm an open source maintainer, I'm publishing a package, I lose my GPG key that I was using to sign it, and you tell me, I'm out of luck, there's never ever going to be another release of package foo again. That's not really an outcome we're super ok with. What if the package has a vulnerability? We will need to patch that and to fix that. Long-lived key pairs aren't super compatible with that. This is something I think that has been recognized increasingly by real world package repositories. On the right, you see an article, a recent blog post from the Python Package Index blog, talking about removing support for PGP signatures from PyPI. You might think that this is kind of a loss, it's a blow against package signing. I actually think it's a victory for more considered approaches to package signing.
Challenges with Traditional Signing
There's a bunch of challenges with so-called traditional signing, if you have key pairs that we expect developers to manage themselves. There's key management, and that includes hooking up the key to the package repository, distributing that key and associating it with the packages that it's in charge of, rotating keys. If I lose the key, or I make a new one on a new laptop. Compromise detection. If someone gets my key, I might not know until it's way too late until they've already published malware in my name and gotten that out to a large number of people. Revocation is difficult as well. There are, again, provisions in PGP for it, but they rely on infrastructure that's decreasingly supporting these key servers and so on. Finally, identity, which gets back to this question of, how do you know who's supposed to be signing the software? If all you know about a key pair is the fingerprint, this very opaque hexadecimal identifier, that's not good for usability. It's not easy to understand what it means when a package is signed by a key with that identifier.
Sigstore (Goals)
Lynch: This is really where we see Sigstore coming into play. Sigstore is an open-source project, it's under the OpenSSF, which is part of the Linux Foundation. Its whole goal is to really make software signing easy and more readily available for people to use. Some of the key ideas here. Sigstore can integrate with your existing key management solution, whether that's bring your own keys, KMS systems, stuff like that. One of the really novel ideas with Sigstore is the concept of keyless signing. Really, instead of worrying about individual keys, really start thinking about identities. I don't necessarily care about the random fingerprint for the public key, I really care about, did it come from Zack? Did it come from myself? Did it come from my CI/CD platform? How Sigstore approaches this is really relying on existing identity management mechanisms, so OAuth and other identity providers in order to leverage the existing security practices that users are already used to dealing with. Two-factor auth, account recovery, dealing with emails as an identifier, things like that. There are a lot of challenges with this. We'll dive into some of those. Really, we're trying to meet developers where they are, so they can integrate these into their existing systems.
Sigstore - Keyless Signing
We've been seeing a ton of adoption for Sigstore just in general, nice up into the right graph. We had a bunch of projects add Sigstore either for their own releases or developer tools that are adding Sigstore as part of releasing your own software. Some notable people using Sigstore today, Kubernetes. They sign all of their recent releases with Sigstore. npm, there's a public beta feature right now for attaching signed provenance with npm packages. That uses Sigstore behind the scenes for npm. Then also CPython, which is one of the primary Python interpreters also signs their releases with Sigstore as well. We're going to step through step by step how the keyless process works, to get a sense of what's going on behind the scenes. We talk about keyless, there are still keys. The analogy we make is there's still servers with serverless. The idea here is we don't want users to have to worry about keys and think about them. There are three main components that we're going to work through, one of them is the clients all the way over here. This could be your developer laptop, your CI/CD platform. Then there's also two or three server-side components, depending on how you look at it, that Sigstore provides. Fulcio, which is the certificate authority, as well as Rekor, which is a transparency log for everything that's going on as part of the signing process. First, it's going to start at the clients all the way at the left, like I said before, laptop, CI/CD. These are going to be some client tools that are aware of Sigstore, know how to go through the keyless process. How this works is the client is going to generate, basically, an ephemeral key pair. The idea here is we only use this key pair once. We use it for the signing operation, and then once we're done with it, we throw it away, never to be used again. Then, the next steps there are like, how do we tie that to the actual identities? What we leverage here is OIDC.
Some examples of clients, there's Cosign, which is for container signing. This is a very popular tool that most people, if you're familiar with Sigstore, are aware of. This can sign any OCI objects, and normally Docker images, but can also sign arbitrary blobs because OCI is just an arbitrary blob store. Cosign does this really nifty trick where it stores these signatures alongside the images themselves, which means it's actually compatible with any OCI registry. If you can push a Docker image, you can also sign it with Cosign. Gitsign is another example of a client. This is Git commit signing with Sigstore, so being able to just use your existing identity without having to worry about generating a GPG key. A similar example to Zack is whenever I give talks about Gitsign or Cosign, I always ask like, how many people have generated a key and just hit enter, no password, no lifetime, no subkey, something like that. Those are the types of key management problems that we're thinking of. These tools are really trying to address those so that users don't need to worry about those implementation details of the keys.
Now to OIDC. OIDC stands for OpenID Connect. It's basically a layer on top of OAuth. Users are already used to going through an OAuth process, you can see on the right here. You're used to the log in with Facebook, log in with GitHub, log in with Google. OIDC provides a layer on top of that, to basically standardize some of the user identification claims, so we can better interop with different identity providers. This could be GitHub. This could be Google. For CI/CD platforms, there's a lot of runtime environments, cloud providers that will provide OIDC tokens for you. This could be your company's SSO, whatever, as long as it supports OIDC, which is a common standard, we can support it. These tokens basically just include metadata about who is authenticated in the session. This is an example web token that I pulled from a real token for my own email, so issued by oauth2.sigstore.dev. For Sigstore, this is the issued at and expiration time, for billy@chainguard.dev, and the initial issuer, so this is the button I clicked was accounts.google.com. That's good for human users, and having an email identifier is pretty ubiquitous for identifying humans. For machine identities, it's a little bit harder. OIDC actually gives us a mechanism to do that as well. There's only a set of claims that are required, but identity providers can actually include additional claims as well. This is an example OIDC token from GitHub Actions that you can include within your workflow, you don't need to do any key creation or anything like that. GitHub will just provision this for your run. Again, you can do this with Kubernetes. GCP has a similar feature for running on GCE Cloud Run. AWS has a similar feature, as well as Azure. Here you can see a lot more richer details about the actual execution that's running, not just some email or some repo. What SHA did it run at? Is it a push or a pull request? Is it a manual run? What run attempt is it? This gives us a lot richer metadata that we can bind against and write policy against more than just, what key are we using? We're challenging the idea of a key doesn't always represent an identity, because you don't know who has access to the key. If it's a developer key, or it's a CI/CD key, like your developer team might have access to it. Really moving away from key means person or workflow, and more towards having these ephemeral tokens that are provisioned on-demand without users needed to take additional actions.
The client handles both the key generation and getting the OIDC token. This is on the left here. Then it needs to actually bind those two things together. This is where the first server-side Sigstore component comes in, which is Fulcio, which is a certificate authority. What happens here is the client goes through the first steps. It sends the OIDC token and the public key over to Fulcio, as well as some signed data, usually, you sign the OIDC token itself with the private key to prove that you actually have the private side of the key pair. Fulcio will check all of that data and make sure it checks out, make sure the OIDC token is valid, hasn't expired. Make sure that the signature of the data checks out. It will issue you a short-lived certificate, an X.509 certificate that you can then use to sign whatever you want. X.509 is a pretty standard format for representing certificate and public key metadata, that tons of signing tools already integrate with. That becomes our leverage point to really hook into the rest of the signing ecosystem as is today. Examples of what these certificates look like, these are a bit abridged. You can see a lot of the same similar information here. You can see issued by sigstore.dev, valid for only 10 minutes, only to be used for code signing, and includes the information, here's the email, here's who it was issued by. Then same for GitHub Actions, again, a lot of the same metadata. Fulcio will do a bit of massaging of the data. There's a bunch of different CI providers that do provide support. There's GitHub, GitLab. I think we're working with Circle right now. There are many others. Fulcio will try to extract some of the claims out and put them in common fields so that you don't need to worry about these precise differences between the different CI/CD runners.
From there, you have the certificate, you can then sign whatever you want, sign your Docker containers, sign your Git commits. Then we have a problem of, you have a certificate, but it's only good for 10 minutes. Normally with certificates, once you're past the expiration, you're no longer supposed to trust that key pair or what was used to sign it. That creates a bit of a challenge because we're constantly getting new keys, new certificates, and so we need to be able to verify this data after the fact. That's where Rekor really comes in. Rekor is an append only transparency log that we publish every signing operation that happens. The idea here is if the signing operation doesn't appear in the transparency log, it's as if it never happened. What we can do with this because it's append only and because it can't be tampered with once it's been included, we can use the presence of the signature being in Rekor as basically an equivalent check of, was this signed at the time it said it was, within that expiration window? Because Rekor will actually return back when it was signed as part of the response. To see what this looks like, so for example, for a Docker container, you would give it the, here's the digest of what was signed. The signature as well as the certificate, which includes the public key. Rekor will verify all that data to make sure that's not uploading trash into the log. Then what you get back is basically an inclusion proof of the timestamp that was entered into the log as well as basically recognition that it was seen by Rekor, and there's additional signatures in there as well. That is the overall flow. Ideally, if everything goes correctly, users don't actually need to think about a lot of what's going on behind the scenes. It's just the normal login flow that they're used to.
Demo
Quick demo. I have a sample repo here. It's just a very simple Go binary that we can build. ko just builds Go binaries without needing to use the Docker socket. Very nifty tool, definitely check it out, if you're using Go and building containers. Then from here, we can just take this digest that was produced at the end. We could just say, cosign sign the image. You can see at the top here, it generated the ephemeral key. It was trying to get the certificate. What it's trying to do here is, we actually need an identity to sign in and bind this to. This is why we're getting the warning, it's basically saying, some of your personal information, the email, stuff like that is going to be uploaded to the log, are you ok with that? Yes, we are. It's going to open up a link, if it used other tools like the local host OAuth flow. If we go back over here, we have to log into Sigstore, that opened up in our browser, we can log in with Google. Authentication successful. We can pop back over here. You can see that the T log, the transparency log entry was created with an index number that we can go and look up. Then it also pushed the signature up to the registry. It really is that simple. Sigstore also provides some tools for interacting with the Rekor log. We can actually go take that index and look up live, like, we just published this a few seconds ago, issued by sigstore.dev, Not Before, this is basically when it was created. It's still valid for 10 more minutes, but the key that was created never hit disk. It's not stored anywhere. As far as I'm concerned, it is completely deleted never to be recovered ever again. Then if we scroll down, we can see the same claims that we saw before in the certificate, so down to my email, stuff like that. Then we can also verify. When we do verification, instead of doing, this key or including in my key ring, what we're doing is, what identity signed it? In this case, it's my email, and issued by Google. We can go ahead and verify this. It's going to spit out some ugly looking data. It's basically the transparency log that was included in Rekor, which you can poke around with, if you want. If we scroll up here, we can see the things that it checked. It checked that the claims were validated. Did the identity match? Was it present in the transparency log, and was the certificate that backed it also valid as well?
One of the nice things about OIDC is the fact that we can integrate into CI/CD workflows. I have a simple GitHub Action here that basically does the same thing that I just did. We can rerun this bit of a live demo. I can pull up the one I ran before. I'm using a temporary registry, so it just means that it cleared up the cache since I last ran this. This was something I ran this morning. This is the GitHub Actions workflow that I ran. Pretty simple. The only thing we really needed to do is the ID token, which instructs GitHub to actually include the OIDC token in the workflow. Then all we did was set up Cosign, build it, sign it, good to go. The example workflow from before, this was the certificate that was issued. We can see, this was triggered by a push for the SHA, for this build, for this repo. I can actually impersonate this as my own user. I need to go through my CI process in order to generate a valid certificate, which puts a lot of controls into place if you're taking good best practices for locking down your CI/CD platform, stuff like that. You can write policies that say, I only want to trust images that come from my CI/CD system, merge to main, only coming from a push, so no pull requests, and stuff like that. This just ran, so we should be able to rerun this. This is checking something similar mentioned before. We don't actually have an email address when we're verifying CI/CD workflows. What this is using is basically an identifier for the GitHub Actions workflow, and then also issued by the GitHub Actions OIDC token issuer. A lot of other mess, just because there's more metadata in there. You can see the same claims were verified there as well. There are some more flags you need to do because there's the extra text, but yes, you can pipe it into jq, and then manipulate it however you want.
Kubernetes Admission Controllers
We showed off some verification here. Verification is very important, as Zack mentioned before. There has been signing support in general, but signatures don't really mean anything unless you're actually verifying them. There's a bunch of tools, especially for the Kubernetes ecosystem, if you're using container-based workflows, allow you to write policy for Sigstore based signatures. Sigstore has a product called policy-controller that tries to do this, to write fine-grained policies, like we did with the CLI, but even to a greater extent. You can say like, I only want push, or I only want pull request, or I only want things coming from this issuer. Then Kyverno and OPA, other super popular admission controller projects also have support for Sigstore as well. Policy-controller is not necessarily the only solution you have here.
Challenges with Traditional Signing (Recap)
To revisit some of the challenges from before, just a recap. Key management, rotation, by using keyless workflows, by making everything per signing event, per artifact, you're dramatically reducing that scope and blast radius so that even if keys were leaked, you're severely limiting the window when they can be used. You don't have to worry about years down the line, like, can anyone access this key? If you ever detect a compromise or anything like that, you know exactly what window things could have been signed in. You can write policies to basically counteract that. That by itself is hugely powerful, so you don't have dangling keys existing out in the ether that you don't know about and don't know who's using. Compromise detection. The other part of Rekor, because it is a transparency log, and you can query it, and you can see, here are all the identities that are being used and what they signed, that can become a mechanism that you can monitor and basically say, where is my production CI key being used, or production CI identity being used? What is it signing? If there's anything that pops up there that you don't know, or recognize, or trust, that can be a signal for you that, something might be going on here that's unexpected, and you should be able to tie back to the event that actually triggered it, especially for CI/CD. Revocation is still a little bit tricky, though it is possible. This is part of where policy really comes into play. Because again, if you know that you were compromised, if this account was compromised during this period of time, you can now write policy that says, don't trust anything during this window, or don't trust this service account, or don't trust whatever. Having that policy enforcement really helps with the revocation, not necessarily of the keys, because everything's ephemeral, but being able to tie policy decision about what is and isn't allowed within your workflows or artifacts that you choose to trust. Then, finally, with identity, going back to before, challenging the concept of a key doesn't necessarily mean a person or a workflow or something like that. We tend to think about things that way. Let's actually use the identities themselves and provide the metadata to make that possible.
Everything I demoed here used the Sigstore public instance. Sigstore is an open source project, you can run your own instance. If you're not comfortable with the metadata, so like the email addresses and the repo information that gets included in the transparency log, you can run your own instance. The Sigstore project, since we want to encourage adoption, especially across open source projects, runs its own public infrastructure that's backed by 24/7 on-call rotation, and is free for anyone to use. There are rate limits in place, so you can't just hammer it, but this is what Kubernetes is using, the public npm instance, even at Chainguard, an artifact that we publish for public consumption, we also put on the public transparency log as well. This is from the OpenSSF site. This is just a landscape of a bunch of people that we know are using Sigstore, either for their own release processes, or have integrated with Sigstore in some way, whether that be generating signatures, verifying signatures, things like that.
Why Do We Trust Sigstore?
Newman: Something you might have noticed, that would make maybe anyone who was around for the 1990s like Cypherpunk dream of decentralization, which has seen a little bit of a revival in the cryptocurrency world where people talk about control your keys, control your destiny, is that we've basically added a trusted third party with a whole lot of power to this ecosystem. Sigstore now is responsible for saying the signature associated with this key belongs to this identity. What's stopping Sigstore now from going around and saying, I saw a signature from the President of the United States of America. That's a really good question. We really try to mitigate this centralization with accountability. I'll show you what I mean by that. The mantra might be, trust but verify. We have a centralized service. We trust it to do the right thing, but we can go back and double check its work. This is this concept of transparency that Billy kept mentioning. To me what transparency is, is we should make this service who we want to trust, but we want to verify what they're doing. We should make all of its activities public and tamper proof so that we can go back after the fact and double check on everything that it did.
Sigstore does this in a way that's really similar to the way this happens in Web PKI now. If you go to example.com, and they present you a TLS certificate, that TLS certificate also winds up in a certificate transparency log. The idea is that certificate authorities have a similar degree of power. I'm VeriSign, I'm DigiCert, whatever, I can issue certificates for any website on the internet. How do we deal with that power? We just introduce accountability via transparency. Now, if DigiCert tries to issue certificates for google.com, Google can quickly detect that and respond, and then DigiCert will lose its authority as a CA. They will get booted out of the CA bundle that's on your desktop, and we can catch it after the fact, and we can respond to it. The way this is done is this public log has a couple of neat features. One is you can query it, you can search it. Another is that it's tamper proof. What do I mean by that? I mean we use cryptographic techniques to ensure that once you've seen a certificate inside of this log, that no one's ever going to be able to take it out and pretend it wasn't in there. Even if these logs themselves are compromised, which we have to worry about, because they're often operated by the same parties who are the certificate authorities themselves. Even if the logs were compromised, we would detect right away if they tried to pull any funny business and take things out of the log.
The Role of Identity Providers
This means that identity providers in this ecosystem also have a whole lot of power. The identity provider is the one saying the person holding this token is billy@chainguard.dev. This is good and bad. These identity providers have a lot of power, they're really a high-value target. They also already have really great security mechanisms in place. Sigstore can actually piggyback on things like the two-factor auth, that your GitHub account has enabled. It can piggyback on the account recovery mechanisms that Google has available, or they have teams of people who look at your driver's license, or whatever. That's something that we're not going to be able to ask these open source package repositories to do. They just don't have the resources to do the thorough and strict account recovery procedures that you would want for this. We're not making the maintainers responsible for identity, which is how I saw the status quo with long-lived cryptographic keys. It's these organizations that have far more resources and can do the job really well. We can put the burden of fraud, abuse, compromise detection on the identity providers, because all of this metadata is public, it's really easy then for the identity providers to say, I'm noticing Sigstore is issuing certificates to identities associated with me that I didn't ask for. That can be detected, and so we'll notice problems with Sigstore. Sigstore supports pluggable identity providers. Especially if you're operating your own instance, but even the public instance, we're open to taking new ones, you can have your own private instance of SSO that you're using for your company. You can just hook that right up, and what you're already using for identities internally, you can continue to use for code signing, as opposed to again, having to introduce this abstraction of keys and trying to manage the mapping between keys and identities.
All this boils down to like, you choose who to trust. When it comes to open source, we're hoping that a lot of that policy gets put on the package repositories. Package repositories themselves, are in the best position to know who's in charge of what packages. If you're talking about your own deployment pipeline, those are decisions you're going to have to make. You can say things like, I'm only going to want to trust an artifact that's rolling out to my Kubernetes cluster, if it was built by my CI/CD system and it's got a certificate of that. It was built from source that my development lead signed off on. You can create these policies that express exactly what it means for a software artifact to be good. I think a lot of previous attempts at software signing were extremely binary. An artifact was signed and therefore good, or it was wasn't signed, and therefore bad. This means that things like expiration dates get tricky. What does it mean for me to sign a binary? I probably didn't hex dump it and look at every single byte in it. There's some degree of trust I have in whatever procedure in whatever computer I used to do the signing, to do the building. I think the more we can tease that out, and I think providing really expressive primitives of identity-based signing enables that, the better shape we are in to make intelligent policy decisions. This is what I was getting at on the previous slide. There is no one-size-fits-all answer. The needs of an open source package repository are going to be different from the needs of an enormous Fortune 500 company, are going to be different from the needs of a 2-person development shop. Sigstore is not attempting to be prescriptive on that front, it's just providing the building blocks that you can use to make good policy decisions.
Case Study - Verifying Images
Here's a quick case study. Here's part of the configuration for that Sigstore policy-controller project Billy talked about earlier. This is basically saying, any images that are coming from registry.k8s.io are going to need to be signed by this krel-trust service account on Google Cloud Platform. That's actually the service account that gets used to build all the Kubernetes images. This is something that Kubernetes Release Team publicizes, and it's well known. That's one example of a policy. This is another example of a policy for if you're consuming content from Chainguard. As Billy said, we'd sign all of our stuff. We put it all on the public transparency log, and we can say, any images that are coming from cgr.dev/chainguard, must be signed by Chainguard on GitHub, or at least workflow. Again, that's another policy. I'll refer you to Marina's talk into this higher-level meta policy of, ok, for an image with a particular name, how do I figure out who's supposed to be signing it? That's a tricky and interesting question. The idea is that like, in different cases, you want different policy. I want to enforce signatures on open source software differently from the way I'm enforcing signatures on things that are coming from within my organization.
Case Study - npm
Quick case study on npm. npm is using Sigstore to do build provenance. A really common attack that we see in the wild in open source repositories are, there'll be a popular open source package, it's developed in the open, good development practices. The attacker will clone the repository and replace it with malware, and then run npm-publish. If you go to github.com/leftpad, like that repository looks great, but what actually you're downloading from npm is totally unrelated to that. npm is rolling out build attestations powered by Sigstore, where, basically, you need to have a publicly known trusted builder, do that build step for you. Then GitHub Actions will sign off and say, I built this software package from this publicly available source. That link between what lines up on npm and what's on GitHub or GitLab or whatever your CI/CD provider and VCS provider are, is verifiable now. This is something that's getting used now. It's in public beta. The hope is that it rolls out to really a large number of packages on npm.
What You Should Do
The moral here is that signing is really critical, I think for any DevSecOps strategy. Again, it's not going to solve all your problems. It doesn't replace a good vulnerability management program. It doesn't replace not checking dependencies on things that are going to have bugs in them in the first place. It is going to be really helpful for knowing that what you think things are is what they actually are, and they're coming from where you think they're actually coming from. You should start to the extent possible verifying signatures, in many open source projects that you might be depending on, especially in the Kubernetes OCI world but also throughout, are signing their binaries using Sigstore or are signing their images using Sigstore. For your own software, Billy showed how easy it is. It'd be a great experiment, just start signing and then seeing, can we verify our own signatures when things are coming into prod? If your tooling and dependencies aren't signing, you can say, is this something you'd be willing to do in the future? It's actually pretty easy to do here. We can show you how to set it up.
Questions and Answers
Participant 1: There seems to be a divergence of how people use Sigstore right now. [inaudible 00:41:35] and Luke Hinds' Stacklok, so like what you build. There's a couple of startups now that are building how you build. You had some examples, but the question would be, I think there has to be a convergence in this discussion of like, what people are doing, so webhook, and then chaining together the attestations. There's a bank that's doing that, there's a startup, and another couple of startups that have done this. Do you guys have a suggestion about that? Do you even care about that model?
Lynch: You're right, that is still very much like an active area for development. I think seeing the engagement across the ecosystem is evidence of that. I think, for Sigstore, providing the building blocks. If you don't have that metadata, and you don't have that verifiable proof to go from your artifact here, build system, to your source code to make that chain, you can't really do all that higher level things for like, how do we actually build things in a trusted way? I see that as more like a fundamental foundational building block. Then as relating to the next steps of, how do we go about building things securely, and responding to that? I think that's very much active development, and where a lot of efforts are.
Participant 1: It'd be nice for you to develop patterns for that. All these companies are using Sigstore, the ones that are building chains of attestations. It'd just be nice to see Sigstore promote a pattern.
Lynch: I think we've seen some amount of that. Part of the challenge is like how people build software is very different. There's tons of different build tools. I think there's been some good steps, I think, like buildpacks is a step in the direction for like, how do we standardize? Like, how do we build npm in a secure way? How do we build Go in a secure way? Personally, I would love to lead by example here, like here are the best ways that we see it. It's always a balance and a tradeoff of like, going back to before, like meeting people where they are, asking people to completely uproot their build systems and what they're doing today. It's going to take time, but yes, it is something that is on our minds.
Participant 2: I'm going to precede my question by saying, I know enough about computer security to be somewhat dangerous. It's been good to learn about these kinds of tools that are available. Going back to Radia's keynote, I'm just curious, she mentioned something about concerns with the X.509 certificate, and you guys issue that. I'm just wondering if you can speak to those concerns.
Newman: I don't think anyone in the industry today is super happy about where we're at with X.509. The concerns are basically twofold. One is that the format itself is relatively complicated. X.509 is built on ASN.1. ASN.1 parsers are where a lot of vulnerabilities are sneaking into things like OpenSSL, you see vulnerabilities basically every year in the parsing, because untrusted input fed to a parser. Parser is written in C. This is how we get buffer overflows, all these big bad vulnerabilities. X.509 itself is a little bit scary for that reason, it's the format itself. Then the other real trick with X.509 is that checking validity is really complicated. That's another thing. What we use in Web PKI and Sigstore's validation is very similar. There's an RFC 5280. It's just so long and so complex. It's because, it's got a kitchen sink of features. You can delegate to another certificate with these properties, and now it's valid only on alternate Tuesdays, or whatever. That complexity has snuck in over the years and has stuck around.
That said, we do have battle tested libraries. Every time you're accessing the internet, and you're on a secure connection, that's happening via X.509 certs in the Web PKI. That software is quite mature. I think there's some interesting academic efforts to put this on firmer footing. You'll hear terms like formal verification. Machine checkable parsers that we know are provably correct in doing the right thing. Can we express these things more declaratively and have a much more obvious mapping from the natural language specification to the code that's written on the computer, because if I look at the way this validation works in Chrome right now, and I look at the RFC, you need to be an expert to be able to match those up. Whereas if the way the validation looked was much more of a transcription of the method that was prescribed for doing the validation, we'd be a lot happier.
The other alternative is, blow the whole thing up and pick something new, which has benefits. We can get rid of a lot of the complexity. It has risks as well. Anything new, how do you know the complexity didn't evolve over decades, because one by one by one. It's the classic trap of rewriting things and starting from scratch. We are happy to lean on standards that have wide adoption right now. To say, there's enough libraries that are depending on this, that we're happy with those. I can see the argument the other way, and we're certainly not married to the format. If a promising alternative comes along, we'd be happy to use those as well.
Participant 3: This isn't completely in your purview. When I think about wanting to verify signatures with components that we're potentially bringing in, we're pretty heavily invested in software composition analysis tools, that feels like intuitively where that would be happening. Just doing some googling right now, I can't really find confirmation that it's actually doing it even though it's making sure those components don't have known vulnerabilities that are public knowledge. Do you have any thought on whether tools like that are adopting signature verification, particularly one Sigstore is creating? Do you have any just thoughts on that?
Lynch: I think it would make a ton of sense to do verification. I think where we see a lot more integration on the signing front is attestations, so being able to attach the results of those to the artifacts themselves in a verifiable chain. You can say like, my CI ran this scan for vulnerabilities, and we are going to sign and attest to the fact that this scan ran and here are the results. That tends to be the more common way, and you can use Sigstore to back that signing and things like that. From there, you can go down the stack and be like, can we look for SCA results for our dependencies, and what the SCA tools find and pull from there, so we don't have to continuously try to recompute things. I wouldn't be surprised if people do it, and I think it would make sense.
See more presentations with transcripts
