To improve how they ship software in a scalable and effective way, GitHub has adopted GitHub Actions for a part of their continuous integration system. In particular, they leveraged the new Actions larger runners to get to run 15,000 CI jobs across 150,000 cores. In the process they also extended larger runners capabilities for all their users.
This gave us an opportunity not only to transition to a fully featured CI system, but also to develop, experience, and utilize the systems we are creating for our customers and to drive feedback to help build the product.
GitHub engineers used a few key features provided by GitHub actions, such as reusable workflows, and created a new feature that enables workflows to reuse previous outcomes.
Reusable workflows are meant to provide a mechanism to manage a collection of workflows in a single repository so they can be reused in other repositories. This was key to make the process of adding CI to a repositories seamless, which would have otherwise required to create new workflows for each repository.
Reusing previous workflow outcomes is instead an optimization mechanism that prevents running a workflow when the file contents of a repo are exactly the same as they were in a previous run of the workflow. This happens when the Git tree ID for the current commit is the same as the previous commit. In such cases, running the workflow would produce exactly the same outcome, so instead of running it again, the results of the previous run are reused.
A final requirement GitHub engineers had to solve was enabling larger runners to access some of their private infrastructure, which could include artifact storage, application metadata services, and other services that enable invocation of test harness. This led to creating a gateway to grant runners access to private resources while preserving security.
This remote access solution works on the principle of minting an OIDC token in GitHub Actions, passing the OIDC token to a remote access gateway that authorizes the request by validating the OIDC token, and then proxying the request to the private service residing in a private network.
If you are interested in using the same approach for your runners, GitHub has provided a basic scaffolding of a gateway implementing this approach.