Engine Yard Releases Cloud Management Framework Vertebra

Posted by Mirko Stocker on Jan 21, 2009

Sections

Enterprise Architecture,

Operations & Infrastructure,

Process & Practices,

Architecture & Design,

Development

Topics

Ruby ,

Release ,

Choreography ,

Cloud Computing

Tags

XMPP ,

Management ,

Frameworks

Vertebra is Engine Yard's "platform for developing and managing secure cloud applications", and was announced in June 2008. Now, it has finally been released—under the LGPL3.

So, what exactly can Vertebra do for you (from EngineYard's press release)?

Vertebra can be used for automating the cloud as well as for writing distributed, real-time applications. The platform stands out because of its ability to embrace the differences of many clouds and to automate processes and application management.

Vertebra's features include:

• A powerful, standard XMPP (Extensible Messaging and Presence Protocol) infrastructure
• A security and discovery agent to manage security policy
• A process automation agent to orchestrate operational tasks involving both machines and people
• A system provisioning registry to allow your applications to become self-organizing
• A federated design that allows applications to operate seamlessly and securely in a manner not unlike Internet e-mail
• Distributed auditing/logging capabilities
• Distributed job control for operational awareness

Coming from Engine Yard, we suspected that Vertebra is also used to manage their own infrastructure, so we talked to Engine Yard founder and architect Jayson Vantuyl.

We aren't using Vertebra extensively. Much like Rails was extracted from a BaseCamp, Vertebra has been extracted from various techniques we've used to manage our internal cloud. So, while it has grown out of them, we haven't yet completely integrated it back in. That said, we have rudimentary infrastructure for Virtual Machine management and status querying in place, with a few tools that use them.

We feel pretty good about this situation, as Vertebra has needed to develop faster than we could do if we had to keep the deployment completely synchronized with the development. We've released Vertebra at a very nascent stage in order to provide its value to the community as soon as possible. As it progresses beyond it's current stage (as of now, version 0.3), we expect to integrate it more completely into our systems. This is largely a balance between perfecting the technology that we need and respecting that our customers are not guinea pigs.

Vertebra's website mentions its security features and that "similar to more venerable systems such as Domain Name Service and the e-mail system, Vertebra is designed to automatically federate". Jayson elaborates on how this:

Like most networked systems, Vertebra concentrates on security at two levels: the "transport-level" and the "application-level".

Transport Level Security between servers and clients is provided by XMPP and is the TLS/SSL with which most people are familiar. Client connections are also verified by means of a login system. Additional security in federated server connections is provided in the form of "server dialback".

To put this in perspective, consider e-mail, which federates in a very similar way. Though e-mail federates automatically, it does it rather promiscuously. There is very little in place to verify the source of the e-mail (although SPF and DomainKeys are working on that). When an XMPP server receives a connection that purports to be from another server, it contacts that server. It then gives that server a piece of information that it must hand back over the original connection. As long as this piece of information cannot be guessed, this special information allows the server to trust that the connection is legitimate.

With Vertebra, you can also "wrap a real person in a Vertebra API". This very curious feature sounds interesting, but what does it actually mean?

As part of the workflow system, a human interaction agent will be made to allow integration of humans into the process via XMPP chat protocol.  This way, currently human systems can be modeled and driven via XMPP chat, with automation points clearly defined for future automation.

To give more detail, in Vertebra, all of your code is exposed in the form of "operations".  An operation is a programmer's tool, offered to them in the familiar guise of a function call.  While this is helpful for interfacing with machines, that's rarely the hardest part of writing an application.  Developing an human interface is equally troubling.  We have put some work towards solving that problem as well.

Our first attempt at smoothing out the interface problem came with providing a way to build command-line tools.  While this was fruitful, Ezra went out of his way to identify the benefits of leveraging XMPP's chat system to provide control over IM.  This led to the idea that having some sort of gateway agent was beneficial and gave a step above the command line.

When I thought about it more, I realized that this model became very powerful if it is inverted as well.  Specifically, in addition to allowing people to trigger operations through said agent, what if we allowed them to "receive" operations?  This concept matured into what we call "The Meat Agent".  Perhaps it's a bit disparaging to refer to people as "meat", but the purpose of this agent is to allow people to be orchestrated by your system as well as allowing them to orchestrate operations in the cloud.

When you think about it, this is done every day.  Ticket-tracking systems are ways of orchestrating your technicians.  A shared calendar is a way of orchestrating your business or sales people.  Even automated e-mails serve this purpose.  In Vertebra, we've formulated this as a first class concept.  People can be part of your machine's workflow.

More can be found on the Vertebra website, for more technical information, visit the Vertebra repository on GitHub.

• Older >