Capgemini are currently working on Apollo, an open source application platform built on top of the Apache Mesos cluster manager, which is designed to power next generation web services, microservices and big data platforms running at scale.
The Capgemini engineering blog states that Apollo has been built to power internal microservice and big data platforms for Capgemini clients, and is now being open sourced to allow external developers to experiment with the platform. Apollo targets the following use cases: building an internal Platform as a Service (PaaS) that leverages Docker container runtimes; creating large scale continuous integration pipelines using the Jenkins Mesos framework; and managing and distributing big data workloads.
Apollo is built on top of the following open source components:
- Packer for automating the build of the base images
- Terraform for provisioning the infrastructure
- Apache Mesos for cluster management, scheduling and resource isolation
- Consul for service discovery via DNS
- Docker for application container runtimes
- Weave for networking of Docker containers
- HAProxy for application container load balancing
Apollo currently supports Mesosphere’s Marathon framework for long-running (micro)services, with support for Apache Aurora and Google’s Kubernetes included as goals on the platform roadmap. Apache Mesos supports several big data frameworks natively, including Spark and Storm, although the documentation states that these are not yet provided as native Apollo plugins. The current release of Apollo includes bootstrap scripts for running the platform on Vagrant, AWS and DigitalOcean.
InfoQ caught up with Graham Taylor, a senior engineer at Capgemini UK, and asked several questions about the Apollo project.
InfoQ: It is great to see this kind of IaaS/PaaS application platform being built primarily out in the open. Can we ask what are the motivations for open sourcing the project?
Taylor: Initially, we started out building the platform as an internal project to support one of our clients needs. We quickly realised that what we were building was fairly generic and could be really useful to other people and organisations. At that point we decided to open-source the code and switch to working in a completely open way.
As engineers, we understand that other developers don’t particularly like proprietary / closed solutions or those that have complete lock in to a particular cloud provider. Since we were already leveraging other open-source components, this decision made complete sense.
The response has been positive so far and we’ve received a few external contributions. We’d be really interested to hear if anyone has successfully deployed the platform in their organisation, or indeed if they need help or advice. Please get in touch with us and share your story!
InfoQ: Apollo is based on the Apache Mesos cluster manager. Can we ask why you chose to build on top of this platform?
Taylor: At the time we evaluated a few other open source technologies in the space. We really liked the fact that Mesos does not make too many assumptions about what you want to run in your cluster, and does a few things really well. This meant we were able to easily mix running Docker containers, alongside orchestrating basic tasks in the cluster using frameworks like Jenkins and Chronos, which gives you real flexibility.
We also found the barrier to entry a bit lower compared to some other PAAS out there. This was important as for the client we were working with at the time, we had to deploy alongside legacy applications and support stateful services in a private infrastructure.
On top of that Mesos seemed really solid - having already been employed in production at large scale companies such as Apple, Twitter and AirBnB.
InfoQ: We see that the listed Apollo use cases are not just microservice-based application workloads, and also include data science scenarios. Are you trying to make Apollo a 'one-stop-shop' for an organisations infrastructure needs?
Taylor: There is potential for that, though that was not our initial motivation. We are strong advocates of selecting the best tool for the job at hand, whatever that may be.
When we were building the platform, our initial focus was to support a microservices infrastructure. At the same time, we also realised that we needed to build and package these microservices as well, in a typical CI/CD workflow. Thankfully, the guys at Ebay had already done some work on integrating Jenkins as a Mesos framework. This allowed us to have a more scalable CI infrastructure. We saw this as an ideal opportunity to kill two birds with one stone and use the platform not only for running these services, but for building and testing them as well.
When you start to look into some of the other Mesos frameworks, that opens up even more possibilities, especially in the big data space. We’ve not done much with Apollo there yet but we’re really interested in taking the project in that direction.
InfoQ: We notice that three Hashicorp open source products, Terraform, Packer and Consul, are currently being used within the project. Do you have any particular relationship with Hashicorp, or is this the best technology available?
Taylor: We don’t have any particular relationship with Hashicorp, no. However, we are big fans of their tools and were already using them separately in a number of other projects internally. Each one of the open-source tools they release just seems to become an invaluable part of our daily workflow, they are that good.
We saw this project as an ideal opportunity to bring all those tools together. We’re really excited to see what’s coming next from Hashicorp and the potential of Atlas and it’s integration with the Hashicorp suite of tools. Last week they announced the Atlas Terraform Github integration and that is something that we will be looking to integrate into our CI/CD pipeline as soon as possible.
InfoQ: Several other organisations are releasing PaaS-type infrastructure platforms, such as Cisco's microservice-infrastructure and Mesosphere's DCOS. Do you have any comments about this, and will you be looking to share ideas and collaborate?
Taylor: Yes, these are both fantastic projects! They are both reshaping how we think and operate within our datacenters and these are exciting times. It just shows there is a real demand for this type of platform built on solid open-source foundations.
I had a call lined up to talk with Keith Chambers, but he left Cisco to take a job at Mesosphere (and I was on leave at the time) and we haven’t had a chance to catch up since.
However, we are already in touch with a couple of the other Cisco guys and have plans to collaborate on a some things that would benefit both of our projects.
InfoQ: The Apollo Github README.md states that the project is in alpha. Can you provide any more details of the upcoming goals or roadmap, and do you know when we can expect a 1.0 release?
Taylor: At the moment we have no immediate plans for a 1.0 release. A lot of our recent changes since the first 0.1 release have been focussed on stability and adding a CI/CD pipeline, primarily for testing. Now we’ve got that in place, we plan to start ramping up adding more features and examples/demonstrators on how to use the platform.
We will definitely be expanding cloud support to Openstack, GCE and hopefully Azure, as well as adding support for more Mesos frameworks. One of our aims is for Apollo to be composable so that anyone using the project can pick and choose which components or frameworks they want for their use-case and launch a cluster with those plugged-in.
We’d gladly welcome any help or ideas from the community to help shape our priorities and roadmap.
InfoQ: What is the best way for InfoQ readers to get involved with the Apollo project?
Taylor: The best way to get involved is to head over to our GitHub page at https://github.com/Capgemini/Apollo .
We’ve labelled issues specifically for new contributors if they want to dive in and get familiar with the codebase https://github.com/Capgemini/Apollo/labels/new%20contributor
Also the project README has a link to our Gitter chat room and we encourage people to open a Github issue if you need help or run in to any issues. We’re usually quick to respond, and we don’t bite!
Further information about Capgemini Apollo can be found on the project's Github page. Developers wanting to contribute to the project can also find 'new contributor' issues within the Github repository, which have been designed to provide a good introduction to the platform's codebase.