The Apache Aurora community have released version 0.7.0-incubating of the Apache Aurora framework for Mesos, which provides a platform for long-running services and cron jobs. This release includes beta integration with Docker, official support for the Aurora command-line client v2, and performance improvements for running Aurora at scale.
Apache Aurora is a service scheduler framework that runs on top of the Mesos cluster manager, and enables long running-services such as those used in a microservice architecture to take advantage of Mesos’ scalability, fault-tolerance and resource isolation. Aurora provides features such as rolling updates with automated rollback, resource quotas and service registration. Aurora is currently an Apache Incubator project.
The Apache Aurora blog states that the Docker container runtime has quickly become a popular technology for packaging and managing applications, making it simple to create a snapshot of an application and ship it to different machines within a datacenter. The Aurora 0.7.0 release includes the addition of a container abstraction that supports Docker and potentially future container technologies that may emerge. This allows a Docker image to be scheduled to run within an Aurora cluster.
The Docker integration feature is marked as experimental in its first release, and is officially released as beta. The Apache Aurora blog requests that feedback be provided by companies running it in production via IRC or the appropriate mailing lists. Native Docker support within the Mesos core was added in the Mesos 0.20.0 release.
Over several previous releases Aurora has included a new v2 command-line client, rewritten from the ground-up to be simpler and easier to use. The Aurora 0.7.0 release provides official support for v2 of the client, and has fully removed the original version. The Apache Aurora blog states that previous client version was deprecated in the 0.6.0-incubating release:
We’ve also made the client even-easier to use, by improving the help output that you receive. You can simply type aurora -h for a full list of available commands and helpful guidance. We’ve also documented usage of the client commands to provide helpful information about managing jobs.
The Apache Aurora blog states that performance is continually an area of focus and improvement for the Aurora team, with production Aurora clusters measuring tens of thousands of machines and tens-to-hundreds of thousands of tasks. Two specific improvements were made in the v0.7.0 release that make significant improvements on large clusters, including AURORA-121 which made task pre-emption more efficient, and AURORA-930 which fixed GC performance issues with snapshot deduplication on large clusters.
Aurora is used in production by several companies including Twitter, where development of Aurora began in 2010 by Bill Farner, an engineer on Twitter’s research team. The Twitter engineering blog states that platforms such as Twitter operate across tens of thousands of machines, with hundreds of engineers deploying software daily:
In this type of environment, automation is critical. Aurora is software that keeps services running in the face of many types of failure, and provides engineers a convenient, automated way to create and update these services.
Over time, a growing team of Twitter engineers began to contribute directly to Aurora, and the framework began to host key Twitter services such as Twitter’s ad-serving platform. The Twitter engineering blog states that today all new stateless services, and most existing services, run on Aurora.
Not only does Aurora assist with keeping the platform running, Aurora has helped push utilization of our cluster. This has allowed us to run more applications on fewer servers and substantially save costs while increasing efficiency.
The Apache Aurora blog states that as the Aurora project continues to grow in adopters and contributors, conversations have begun regarding the next-steps toward graduating from the Apache Incubator. A graduation resolution has been presented to the Aurora community on the developer mailing list and is currently being voted on.
The v0.7.0 release of Apache Aurora can be downloaded via the project’s web page, and full release notes are available in the release CHANGELOG. The Apache Aurora community holds weekly IRC meetings at 11am PST every Monday, and users are being encouraged to provide feedback by IRC or the appropriate project mailing lists.