Bio Matt DeBergalis cofounded the Meteor project and is the COO of Meteor Development Group. Before Meteor, he founded and ran ActBlue, the largest political fundraising platform in the world which raised more than 350 million USD in grassroots contributions. Before that, he was a kernel hacker, contributed to the NeXT port of NetBSD, worked on the NFSv4 and DAFS specs while at Network Appliance.
Software is Changing the World. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.
1. My name is Peter Bell and I am here with Matt Debergalis at Qcon New York. Matt thanks for taking the time to be interviewed. You’ve been speaking about Meteor, what is Meteor and why would somebody care about it, what problems does it solve?
Google Plus is another example of software written this way it’s actually an app running in your browser it’s not a website. So, for Google and Facebook you can build these things by hand, it turns out for most developers, developing software in this style is incredibly difficult, it takes a lot of custom engineering work, you end up solving many of the low level problems, and we can talk about some of these, each time you build an application. And it leaves people in a tough spot, they want to build rich, engaging experiences but they also don’t have months of spare engineering resources to build them with. So Meteor is the solution, Meteor is an off the shelf platform that lets more developers build rich real-time applications that run on the client.
2. There are a number of other frameworks out there that try to solve the problem of creating rich client apps so some of the things that come to mind is the whole swathe of clients MVC frameworks so you’ve got Backbone and Ember and Angular, and Sammy, and Batman, Knockout and all of these MVC frameworks. What is the difference between Meteor and something like Backbone, a client MVC framework?
We look at the application as a whole, and I think it’s important to have a solution to the full technical challenge of building a rich client application. So there is code on the client but there is also code on the server that we can’t forget about because the server is still in this architecture the privileged environment where your security code runs, where your batch processes run, where your emails that get sent out come from the server, we can’t ignore that half. And the challenge as I’ve mentioned of writing an app that has these characteristics is that there are so many moving parts now. It used to be easy to write a web app, right? You’d use Rails and you’d use MySQL and the book was about an inch thick and then you were done.
But today you need all of the sorts of things you mentioned. So we need something on the server, maybe that’s Node.js, if we’re at the leading edge or something like Rails or Django, we need something on the client, and you mentioned a number of options that solve overlapping parts of the problem, we also need a protocol for those two pieces to communicate, and in the web era we use REST of course, but in a real time application that’s not sufficient because we need a way for the server to be able to tell the client about new data, the basic problem is that the clients don’t reload pages, they don’t refresh themselves automatically as you click around so we need a way to deliver new information into the client so that you are not always looking at stale data. And most of the libraries that you mentioned don’t attack that problem, so you are still left as the developer designing a protocol so that the server can send data to the client, and so that the client can ask the server to make changes, as users fill out forms and modify information.
So for example you can put together a client and a server with a custom protocol running over SocketIO but what you lose in that case is tooling to inspect the protocol when it goes over the wire. You don’t have the equivalent of Curl, you don’t have a proxy server, you don’t have a way for a third party app to integrate with what you’ve built, because they would have to agree on your private SocketIO encoding. In Meteor we try to solve this is a standard way, so all apps use a protocol we call DDP, apps can connect to each other so you can subscribe to data from a third application, we are starting to build tools and people in the community and we should also talk about that, are also building tools that allow you to inspect what’s happening to be able to integrate with other components at the same time. So it’s both a faster development experience when you have a platform as a whole, but it’s also a broader experience in the sense that you have more capabilities than if you built these components by hand.
4. To get an idea, if I wanted to start with Meteor what is some of the use cases, often there are some kind of problems that are a better fit on one framework than another, what is some of the heuristic that I would say “Oh, if I have this kind of app to build, I should really look particularly hard at starting with Meteor”?
It is a challenge and it’s even harder as you build these distributed applications because one of the things you want to test is the interaction between the client and the server. And setting up that environment and doing it in twenty browsers at the same time, so that you can qualify mobile Safari and you can qualify an older version of IE, and the newest versions of Chrome, that becomes a testing challenge both, as you said, the low level unit tests but also as a whole the application. That’s a great example of something that we have seen tremendous interest from the community about and I should say Meteor is open source, and we have thousands of developers world wide building applications on Meteor, the most exciting thing about that is that parts of the core that we haven’t yet built as the core development team in San Francisco are parts that other have jumped into and so we have seen a number of really interesting efforts around integrating some of the popular unit testing frameworks with Meteor and that’s been I would say probably one of the most popular topics on the mailing lists. But also more forward thinking questions about what does a complete acceptance level testing framework for an application of this style look like. We have a public roadmap for the project, at roadmap.meteor.com, you’ll find testing is one of the things we are going to be building soon. And a lot of what’s there and the specifics of how we’ll do it, comes out the conversations we have at events like QCon and on lists like Meteor Talk where people can flesh out ideas and play with concepts about what might be the best way to approach a problem like that.
Our own website is built in Meteor it’s important to eat your own dog food early, it’s an early tech preview, it’s what we call it and so you'll want to be comfortable with quickly changing technology stacks. In particular some of the APIs in Meteor are still in flux because we want to explore in the context of these real production apps that people are building, what the best approach is. So as one example of that we have one library called Spark today in Meteor, which is how we automatically manage the DOM. So redrawing the screen in as efficient a way as possible, as the underlying data in the local database or other parts of the applications change. We are in the middle of rewrite of that library and the reason is that we have discovered as we have worked on larger and more complex apps some in-house and some that members of the community have written that there is a cleaner way we think to define the programmatic API around modern DOM manipulation. And so there is a tension because on the one hand we do have, as I said, thousands of developers that want to be able to know that these applications that they write today will be against the correct APIs on the other hand and we hear this very clearly from the community they want those APIs to be as excellent as possible and so we are moving forward but we are doing it in a way that allows us to step back and examine each of these design choices as we go and evaluate “Is this the best thing we can do for the developer experience? Is this the easiest possible way to write a modern app that doesn’t compromise on technical quality, that doesn’t compromise on user experience”. If it’s not, then our inclination is to take a second or third look before we bake it in.
6. That makes a lot of sense and just to understand how much of the stack you are changing, if I am building an application using Meteor am I still going to have HTML? Is the CSS still going to be there? And for simple DOM manipulations in page would I still be using something like jQuery or would Meteor replace some or all of those elements?
7. Now, obviously you could use this to build applications that run on the desktop and presumably you can build mobile websites as well that would run in mobile Safari. Could you also use this with something like Cordova, PhoneGap, to package up and create web applications, mobile applications?
Yes, Meteor works well with PhoneGap and we have a number of people in the ecosystem that have built applications like that and have explained how they’ve done it and so we are very grateful for anyone who takes the time to write a blog post, send out technical information on the list, that’s how the ecosystem grows, because the core team is focused on those key APIs that get us to a stable 1.0 and it’s the energy that comes from someone who says “You know what? I want to see if I can get this working in PhoneGap and when I’m done I am going to explain how I did it so that others can follow”. It’s really exciting and gratifying to see that.
8. Nice and then to go to the other side, from a hosting perspective, for the server side code, is this something you would throw up on Heroku, spin up an EC2 instance, what are some of the common patterns you are seeing in the community for hosting the server side code that relates to a Meteor app?
We’ve seen people take a number of different approaches Meteor and any real-time application with web sockets is a little challenging to host on some of the legacy hosting platforms that don’t support the real-time stack. And so one of the challenges for deployment which we’ll have to face in the next year or two, is moving the web infrastructure that we’ve built for request/response-based HTTP apps to a technology platform that supports this real-time web socket based application. So we provide a free hosting service called Meteor Deploy that’s designed that way that anyone can use, really as a way to share applications quickly, certainly we have people building on Heroku, we have people building on EC2 as you said. One of our growing interests is software that will help developers host applications very quickly and effectively on whatever environment they want.
So I think the trend is very much toward hosting in the cloud but hosting on infrastructure that you have some control over and so we would like to make it very possible for an organization that is building a Meteor application wants to host it alongside their existing web-based applications at Amazon, Rackspace, wherever it may be and solve most of the challenging problems around, how do I plug my web socket down to my stack, how do I run multiple copies of Meteor as I need to scale, how do I manage like what is the equivalent of a proxy server for the web sockets and how do I scale to millions of clients and so on. These are all things that properly belong in the open source framework that we are building so we will work on that as well.
We try to support the browsers that are in the field I believe IE8 and above is supported. I think it’s important for the framework itself to be fairly permissive about what is supported. I don’t know that every application developer will take advantage of that of course but there are some who need that and so it’s important for us to do that. The other half of your question is about mobile browsers and of course there is this huge explosion of mobile browsers so we think about challenges like for example there are some mobile networks where web sockets aren’t well supported and so we want to bake into the framework as a default behavior under what circumstances do we use a web socket, under what circumstances do we fall back to some sort of long polling strategy, each of those strategies have a different implication for battery life on your phone, a different implication for whether or not you see a spinny when you are using the app, and so those are things we focus on so that application developers can just build the application and know that it will work on the widest possible variety of platforms.
Peter: Matt, thank you very much for taking the time to be with us here, I am Peter Bell for QCon.