The eBay Application Platform Team has started using Kubernetes Operators, Helm Charts and Jenkins to ensure software quality at the organization. In order to perform safe changes within the associated containers and environments, the team has created a self-service "certification" solution.
A certification is required for changes such as framework releases, runtime updates, operating system updates, etc. To prevent any bad release, the certification solution provides an automated template to handle complex certification requests. Zhong Shen, Rocky Sheng, and Ramit Bedi, the members of the application platform team at eBay, provided a detailed walkthrough of the solution architecture.
The solution consists of standalone "Certification Unit(s)," consisting of application test cases, performance tests, test apps, pipelines, and traffic mirrors. For a given certification category, there are Helm Charts containing templates for certification units and certification instances.
Helm Chart and user input parameters are used to create Certification instances and Chart Custom Resource Definition (CRD) instances. Certification instances specify Jenkins pipeline git repository and pipeline parameters. Chart CRD instances define standard parameters, certification unit groups and respective dependencies.
Using Certification Service, the certification controller manages the Jenkins pipeline jobs. Based on the requirement, Jenkins jobs are created and deleted. The application platform team reuses the modularized Jenkins pipeline scripts as per individual certification type.
The application platform team seems to follow building, maintaining, and providing a curated platform experience for all communities using it at eBay. This team has achieved full automation of software quality certification and automated test comparison with the certification solution. The team members can trigger certification jobs, and configurable notification mechanisms enable quick troubleshooting. As a result, certification time is reduced from days to hours.
Platform teams provide an easy-to-use "paved-road" by reducing the cognitive load on the development team and enabling the common platform as the path of least resistance. Aligned with this mindset, the application platform team at eBay is working on future enhancements for software quality certification. The team plans to use Tekton, an open-source framework for creating CI/CD systems, and leverage machine learning to categorize failures, root cause analysis, and predicting future failures.