Bio Andy Piper is widely known as a Social Bridgebuilder and speaker, and is a Developer Advocate at Twitter. His passions span a wide variety of areas: cloud, devops, mobile devices, the Internet of Things, Arduino and similar technologies, social computing, education, LEGO, and photography. He is project lead and developer on the Eclipse Paho lightweight messaging project.
Software is Changing the World. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.
1. I'm with Andy Piper, Developer Advocate at Pivotal, at London QCon 2014 and I'd like to start, if we may, just by explaining a little bit where Cloud Foundry came from and how you came to be involved with them.
Sure. So Cloud Foundry is a Platform as a Service, and what that means is that it is essentially an elastic scalable surface for you to deploy apps onto. So, it's the next layer up from Infrastructure as a Service. Things like the VMware vSphere, Amazon EC2, those kind of things, are really just looking at provisioning virtual machines, it's more broad than that, it's cloud scale. The Platform as a Service is really the next layer up, trying to provide a set of middle ware services so that you just deploy an app rather than have to set up the VM. So, Cloud Foundry actually came from VMware. Originally, they were started there as a project in 2011, open source and unusually for VMware, it was both open source and it actually targeted non-VMware platforms.
So, the idea of Cloud Foundry is to really be an abstraction layer above the infrastructure service, whether that is vSphere or EC2, or OpenStack or CloudStack or Google Compute Engine – one of those folks. So the idea is that we are sitting above that, VMware kind of recognized that vSphere wasn't the only possible target, so they decided that they were going to start up this new project. I actually got involved because I was invited to join the developer relations team at VMware. At the time, at VMware we had Spring; we had Rabbit MQ; we had Redis; Cloud Foundry – a bunch of open source developer-focused technologies which was again, somewhat unusual at VMware which is traditionally known as a more of an infrastructure organization. Yes, I got involved. It is an open source project. I love open source.
Yes. The foundation is an interesting area – just got announced last week, as we are speaking. About the middle of last year, we started to get involved at Pivotal, we started to get some other big organizations involved in the Cloud Foundry project. It has always been an open source project, it has always been on GitHub, but it's primarily been driven by VMware and then by Pivotal, a company that has come out of VMware. We started to talk to IBM, NTT, Verizon, SAP – lots of bigger organizations who were really interested in the direction we were going. So, around the middle of last year, third quarter of last year, we had a conference in California – PlatformCF – and that was co-hosted by IBM, GE Software and Pivotal. From that we formed something called Customer Advisory Board.
So really, the intent was to enable those organizations to start to review what was going on in the Cloud Foundry project and actually have some influence in the direction and help to drive the project develop in the right directions. More recently, we've announced the foundation. The foundation will be set up some time during 2014. We have a few different levels of membership. It is actually based on, or will be modeled on, the Eclipse Foundation in terms of governance and structure. In terms of what that will actually formally look like, about the middle of this year, end of second half, we'll put out a draft set of guidelines to say: “This is what we think the foundation should look like. What do you think, community?” So I can't tell you 100%, but I know that we've got HP, SAP, IBM, EMC, VMware, Pivotal and Rackspace as platinum members.
Then we have gold members who are currently ActiveState and CenturyLink and we'll have other members joining as we go forward. Similarly to the Eclipse Foundation those members contribute funds to ensure that this foundation is able to run as an independent organization. Then, rather than having Contributor License Agreements assigning rights to Pivotal as happens today, because of where we've come from, that will actually be assigned across to the independent foundation and so everyone will have an equal voice. That's the goal of the foundation – really to set this thing up in an open way that is not unfamiliar to other open source communities like OpenStack, Eclipse, Apache and so on, and really collaborate in a much more open and transparent way.
Yes. If you think about writing an application, you might chose any language. I am a Java developer, I am a Node developer, I am a Scala developer, I am a – I don't know, Haskell or Cobol potentially. Maybe let’s put Cobol to one side for the moment. So, you want to chose the right language either through your own choice or the right tool for the job, those kinds of things. When you deploy an application to Cloud Foundry it actually gets started inside a container on top of the virtual machine and that container looks like a very minimal Linux system. Actually, we run our hosted Cloud Foundry – Pivotal Web Services – on Ubuntu. That's our chosen operating system. As you deploy the application, there is no runtime for it. So you can't just run the app – it's a minimal Linux system. There is no JVM, for example, there is no Ruby interpreter, whatever you might need.
So, what a BuildPack does is simply a couple of scripts – three scripts in fact – and it says, as you deploy the app, let’s detect what kind of application this is – Java, Ruby, etcetera. Does it have any dependencies? If it is a Java app you're going to need a JVM, you may need a container or an app server and you may need some frameworks or libraries or Spring or something and you need to pull some other stuff in. If you're Ruby, you probably have some gems that you need to download or compile natively. So the BuildPack really just prepares the container for the app. So what we do is we actually go through a process called staging. As you push or deploy your app, we combine the BuildPack, the runtime with the app and you get what we call a Droplet, which is like a fully executable little blob of code.
That's what gets deployed. That's a really flexible system because it essentially means that we can support anything. I like to joke that Google seems to invent a new language every six months or so – we've had Go, we've had Dart, we've got Red Hat with Ceylon. There are new languages coming along so it is actually kind of a way of future-proofing the system because we can continue to bring in new runtimes and frameworks as they develop. We're not limited to one or two frameworks and languages. There is an exception, and the exception is .Net and Windows.
Yes. So, we've got a process to bring projects into Cloud Foundry. If you look on GitHub, we got three organizations: we have Cloud Foundry Community; Cloud Foundry Incubator and Cloud Foundry. Cloud Foundry is the core one where everything that's going to be part of the Cloud Foundry release sits. Then we have the Incubator. The Incubator is kind of a half-way house. It is kind of a way to being in the core, it is being curated and approved and seen as mature by the Customer Advisory Board today, the Foundation in the future. Then you have the Community where anyone can have an account and play and commit. In February 2014 we actually brought the Iron Foundry project; and Iron Foundry is a .Net implementation of Cloud Foundry. So in order to support .Net applications, you could use Mono to run them on Linux but, whilst Mono is an excellent project, it wouldn't necessarily be first class support. So what Iron Foundry lets you do, is actually use Windows hosts as Cloud Foundry VMs and therefore you can actually run full .Net stack applications. So that support has now come into the Incubator and that means that it's on its way into the core. So we are going to have mixed environments so both Linux and Windows machines with runtime stacks.
5. You talked about the dependencies being downloaded. Do they come from the central places like Maven Central or Ruby Gems or do you cache them? And how do you deal with moving versions of these dependencies in the BuildPacks?
Great question. By default, yes, we typically go off to an S3 bucket or as you say, Ruby Gems or Maven Central somewhere to grab those dependencies. If you are inside an enterprise, that may present some issues, some security issues. We actually have a couple of things that we can actually disable: the ability to use externally community contributed BuildPacks, so that you can curate the BuildPacks and runtimes available to your developers and also, we can limit where those BuildPacks go to get their sources. The other part of the question you asked was about cacheing and yes, as we run the BuildPacks, as we run staging we do do cacheing. So we don't constantly go off to the internet and pull down these massive blobs of libraries.
In that case it is beholden to the developer to actually go ahead and say “Well, Java update; Java BuildPack's been updated. Re-push the application”. You won't need to make changes to your app. You just need to essentially restart the app which will cause the latest Java version to be installed underneath it.
Well, again. Somebody has to look after those BuildPacks: whether that's Pivotal or whether that's another company; IBM has a WebSphere BuildPack. I am sure that over time there are new releases of WebSphere and security patches for WebSphere so you would expect those organizations also to update those BuildPacks. BuildPacks as a technology came from Heroku which is actually a PaaS competitor, but they open-sourced the BuildPacks, the language runtime layer, so that we share that community of BuildPacks. The same thing applies there: when there is an update to something like that, you need to restart or re-push your app to cause that runtime to be refreshed.
Really, a huge list: Go; you can get a traditional LAMP stack, PHP with either Apache or with nginx for example; you can literally get one for Haskell; Erlang; Python; Ruby; in the Java space, specifically, there are a number of customizations of the Pivotal’s Java BuildPack to add in different containers. So whether that's JBoss or Jonas or WebSphere, there are variations available. Sometimes they're community contributed, other times they come from the vendors, so IBM, Zend is working with us on PHP support, those kind of things.
I think that the Platform as a Service layer is tremendously powerful because it really means that as a developer, I no longer have to worry about setting up my own VM, installing all my own dependencies, worrying about database management and so on. So I think the PaaS space, although there has been a lot of debate about whether it's a viable layer in the Cloud stack: SaaS, PaaS, IaaS? Are the infrastructures innovating up the stack to nibble away at the PaaS layer? Those are good questions but I think they miss the point because what we are trying to do with Cloud Foundry in particular, is make that PaaS layer the operating system for the Cloud. What I mean by that, is that if you look at the infrastructures today, they are the new hardware.
What I mean by that, is that in the 80s and 90s we used to go to the store and buy a beige box, it gave you a CPU, some network and some storage. Inevitably, it came with Windows installed – let’s set that to one side for a moment. You did need an operating system to make use of those things, but if you look at the infrastructure Clouds today, you are getting virtual CPU, virtual network, virtual disk and of course, you install an operating system into that virtual machine, but they're all incompatible. EC2 is not compatible with OpenStack; they may have compatibility layers, but they're different.
As a developer I don't want to learn managing EC2 and managing OpenStack and managing Google Compute Engine and managing vSphere. I just want to make my apps run somewhere. So actually having an operating system layer, which when we were at VMware, one of our former executives used to refer to this as the Linux of the Cloud. I think that is some really good analogy because actually, if you look at Linux in particular, it's been ported from x86 to Arm to all kinds of other processor architectures and it has become this operating system that runs on all kinds of platforms. That's what we try to do with Cloud Foundry. We want to abstract the infrastructure. So, in terms of what's coming next, I think more of that – today we run in a supported way on OpenStack, VMware and EC2.
We've got alpha support for Google Compute Engine and CloudStack – I would love to see for example Azure support – and the ability to run on different infrastructures – IBM runs on Softlayer, their own Infrastructure as a Service. I think that's important and I think more innovation around the platform itself, adding value added services, making it easier for developers to get started and I think actually, as a development community, we’ll start to see the Cloud Foundry community become a lot more – I won't say friendly, but I mean – consumer friendly: as a newcomer, where do I start contributing? What are good projects to come and look at and I think that is an exciting space right now and something that Cloud Foundry community is starting to do – document the projects more clearly and help people come in and help. I think those are spaces that we'll see more development.
Alex: Andy Piper, thank you very much.