Virtual Panel: The Node.js Ecosystem - Frameworks, Libraries and Best Practices
Node.js author Ryan Dahl has given a very good brief explanation of what Node.js is during his JSConf 2010 presentation[PDF]:
- Built on Google’s V8
- Evented, non-blocking I/O. Similar to EventMachine or Twisted.
- CommonJS module system.
Node.js has been gaining lots of attention lately and Peter Griess Principal Engineer at Yahoo! Mail has mentioned that his company is looking into using it in production.
Currently there may be about 300 projects related to Node.js on GitHub and InfoQ has contacted the creators of some of the most popular ones for this panel.
The participants were:
- TJ Holowaychuk from Express, a Sinatra inspired Node.js web development framework,
- Guillermo Rauch from Socket.IO, an simple HTTP Socket interface implementation and server,
- Matthew Eernisse from Geddy, a web development framework for Node.js. similar to frameworks like Merb, Rails, Pylons, or Django,
- Astro from node-xmpp, an idiomatic XMPP library for Node.js
- Peteris Krumins and James Halliday from StackVM, a startup that tries to make virtual machines more accessible over the web using Node.js,
InfoQ: Would you like to tell us a little bit about your project? What problem does it try to solve and how does it go about it?
TJ (Express): As you already know Express is heavily inspired by Sinatra, initially this was mainly to aid in making developers feel comfortable on the new platform with new idioms. That being said it has evolved quite a bit in 1.x, it now utilizes the popular middleware framework Connect (which I co-authored), and removes some previous dependencies that were not a good fit for the project.
Some frameworks for node are interesting in their own right, however most attempt to be the "be-all end-all" library. My vision of Express is about options, not limitations.
Geddy has simple, resource-based routing, and proper content-negotation, so it's super-simple to use it for lightweight Web services that serve structured data.
Astro (node-xmpp): My goal was creating an XMPP library that can be naturally used in a Node.js environment. One previous library was based on Strophe.js for the browser and just didn't match the conventions of Node.js. It didn't use EventEmitter, didn't make use of the SRV-capable DNS resolver, and most importantly, didn't support both XMPP client and component connections.
InfoQ: What are the main components of your project, how do they interact with each other and with Node.js?
TJ (Express): The Connect middleware framework plays an important role in Express 1.x, components previously known as "Plugins" are now Connect "middleware". This abstraction means the community can form powerful frameworks with all the commonalities in a much shorter period of time. The core components and features of Express as of 1.0beta are:
- Robust routing
- View system including partials and collection support
- Environment based configuration
- Session based persistent "flash" notifications
- Content negotiation
- Response utilities
- Redirection utilities
- express(1) for generating skeleton apps
Matthew (Geddy): Geddy is an MVC framework. You define routes that map URLs to actions on a *controller.* You can create *models* to define the objects you want to use in your controllers. Finally, you respond to requests by rendering a *view,* which can be some sort of template, or just formatted data.
Astro (node-xmpp): Because Node.js is all about scalable server software performance was very important to me. That's why I first set out to evaluate the available XML SAX parser libararies and then started creating node-expat myself. That is just a C++ binding to libexpat, one particular fast parser library.
node-xmpp itself was modelled a bit like xmpp4r, the Ruby XMPP library, because I was used to use that a lot. Client & Component both derive from Connection which does the XML stream processing. Connection is based off a vanilla TCP client stream so a user still has access to those features: raw data management, send queue management et cetera.
Peteris & James (StackVM): Our main component is the StackVM itself (http://github.com/pkrumins/stackvm/) which uses several other Node.js modules that we have written for Node.js:
- dnode (http://github.com/substack/dnode) DNode simplifies communication among StackVM processes and the web browser by providing a simple, object-oriented interface for remote function calls with transparently wrapped callbacks to pass data around.
- node-bufferlist (http://github.com/substack/node-bufferlist) With bufferlist, we can build binary stream parsers in an asynchronous way.
- node-rfb (http://github.com/substack/node-rfb) This module uses node-bufferlist to talk the client side of the RFB protocol used by VNC.
- node-png (http://github.com/pkrumins/node-png) We use this module to render PNG images for the screen updates captured by node-rfb.
- node-jpeg (http://github.com/pkrumins/node-jpeg) This module takes output of node-rfb and produces JPEG screen updates.
- node-video (http://github.com/pkrumins/node-video) With this module, we can record screencasts of the screen updates captured by node-rfb.
- node-base64 (http://github.com/pkrumins/node-base64) This module takes output of node-png, node-jpeg, base64 encodes it and sends it to browser (because there is no easy way to stream binary data).
Then we also use Socket.IO (http://github.com/LearnBoost/Socket.IO) and Socket.IO-node (http://github.com/LearnBoost/Socket.IO-node) modules written by others to communicate between the browser and StackVM.
So here is how it all works: StackVM uses node-bufferlist and node-rfb to open a connection to a virtual machine’s VNC port. As screen updates come in, node-rfb forwards them to node-png or node-jpeg, which generate images. Then they are forwarded to the browser via Socket.IO. DNode abstracts over the Socket.IO transport to route messages between the server and browser in a more asynchronous way.
InfoQ: What were your main reasons for choosing to work with Node.js? What are the key benefits for your day-to-day work?
Guillermo (Socket.IO): Node makes writing realtime applications really easy. The main reason is that in Node, you're in the driver seat because it allows you to create your own HTTP server in a few lines of code. That kind of power cannot be achieved in typical stacks like the common Apache+PHP one, because PHP is only executed when the request is completed, and is expected to run for a short period of time, produce a response and exit.
On the other hand, in Node requests can extend for long periods of time without sacrificing performance or scalability, as it's a fully asynchronous, single-threaded, single-process framework.
Astro (node-xmpp): I want to master concurrency in IO-intensive applications. The canonical choice would be programming with threads, but they're expensive. Erlang and GHC tackle this by mapping light-weight green threads onto one native thread per CPU core. Despite of this feature developers still have to bear with synchronization and avoiding race conditions.
In asynchronous programming you're not developing serial "threads" of code, and thus it doesn't feel that natural. On the other hand being relieved of all the synchronization mess really felt more easy to me. Once you've got that concept it's just getting used to the "staircasing", that is nesting functions in nested functions.
InfoQ: Do you think Node.js is mature enough for mainstream adoption? What do you think about the surrounding ecosystem?
TJ (Express): That is a tough question, compared to many other communities it is still quite small. Personally I feel the project (and the community) has great potential, and chances are it's Django, Drupal, or Ruby on Rails equiv will eventually come along and pump even more traffic into contributions.
Guillermo (Socket.IO): Node is getting really mature really quickly. Today, it has almost half as many followers on github (http://github.com/ry/node) as the Rails framework (http://github.com/rails/rails), which is hands down one of the most well known frameworks for developing web applications. And this was achieved in a very short period of time.
Matthew (Geddy): It's mature enough for some purposes, but it's still way to early to do the kind of general-purpose development that people today use something like Ruby or Python for. Give it a year or two.
Node.js is in motion and you should often test your code against the newest version. I think it can get problematic if you're going to release code that should work for everybody regardless of whether their Node.js version is fresh or a few months old already.
Peteris & James (StackVM): If it isn’t mature enough in one regard or another, we’ll help to make it more mature. It’s open source, so if we find a bug or we need some new feature we can hack it together ourselves if necessary. The community-driven ecosystem is great: open source, plenty of ideas, many engaged developers who know their stuff, tons of documentation, blog articles, and new Node.js libraries on GitHub every day.
The API does sometimes change between versions slightly but Ryan Dahl has promised a new era of stability in the upcoming version 0.2.
InfoQ: Are you using any other Node.js libraries/frameworks in your project or everyday work? Is there any other Node.js library/framework [besides your own] that you think is especially useful and that other developers should look into?
TJ (Express): On a day to day basis I use a number of my own libraries, as well as the following works which I have great respect for the developers and their projects:
Guillermo (Socket.IO): We're using node on a daily basis. Aside from Socket.IO, I maintain Mongoose (http://github.com/learnboost/mongoose), an ORM for the NoSQL document-oriented database MongoDB, and other smaller components.
Logan is a small x-env (cross-environment) test runner for testing code that has to run in Node.js, in the browser (and also TheRubyRacer, which embeds V8 in a Ruby process):
Astro (node-xmpp): There are a few to mention. First, the package manager npm by Isaac Schlueter. You don't want to miss the convenience of a package manager anywhere.
Not to miss Senchalab's Connect web framework. The Node.js built-in HTTP support is its killer feature, and Connect is going to be what Rack is for the Ruby world. Of course, that won't look as straight-forward because it's all asynchronous.
Peteris & James (StackVM): The node package manager, npm (http://npmjs.org/) is a great way to use and discover libraries written by other developers. We use Socket.IO, which abstracts over websockets and provides fallbacks to support websocket-style communication for browsers that don’t have websockets yet. We’ve been experimenting with node-compress to speed up data transfer to the browser on top of Socket.IO too.
InfoQ: If a team wants to start working with Node.js, what are the main pitfalls that they should look out for? Any tips or “best practises”?
TJ (Express): Generally the same as with an environment. GIT is a must in my opinion, I have never been satisfied (in fact repulsed) but other SCMs. Setting up a continuous integration server, even a simple one like CIJoe http://github.com/defunkt/cijoe is a must, and tests tests tests! I cannot stress testing enough, for all my node projects I use the "Expresso" project http://github.com/visionmedia/expresso which supports code coverage reporting, letting you know exactly what code you missed, these simple steps should keep a project rolling.
Guillermo (Socket.IO): My recommendations are:
- Solid understanding of all the different layers that make a website work. That means understanding the low level aspects of it (solid knowledge of HTTP, and protocols and networking in general would help tremendously)
- Try to "forget" about some different ways of achieving the same things in other languages, and learning the "Node" way of doing them (for example, problems that you would solve using threads in Java, you solve differently -and easily- in Node)
Matthew (Geddy): It is still pretty early with Node.js, so there are still API changes that break code. Be prepared to make changes to your code -- or use a framework like Geddy or Express, where the framework author can keep a consistent API through Node's changes.
Peteris & James (StackVM): Blocking I/O is probably the biggest pitfall to be aware of, since the entire program will wait for that call to finish before doing anything else. Use blocking calls sparingly and only for initialization, especially when writing network services where performance matters. Asynchronous I/O and event-driven programming requires a different mindset from most programming. Embrace it.
Node's adoption of the CommonJS module pattern is actually my favorite organizational utility I have used in a language before. Typically most environments consist of a flat namespace, so in reality you still run into namespace collisions all the time, the CommonJS module system is great at preventing this.
- A very simple yet powerful module system (based on CommonJS)
- Lots of testing frameworks (Expresso, Vows, etc)
Astro (node-xmpp): I belong to the fraction that would recommend static typing for larger projects.
I'm also in favour of Test-Driven Development; not sure if the Node.js community has chosen a de-facto standard library for testing.
Peteris & James (StackVM): Node.js has a module system which helps a ton with project scalability, but large software projects tend to be far buggier and harder to test than small ones in any language. With StackVM, we’re trying to avoid the trap of a large codebase for as long as possible by factoring out functionality into reusable, open-source modules. So far, this approach has kept the main StackVM codebase more flexible and the supporting libraries are far easier to test in isolation. People using our modules in the community have even already identified some bugs for us. Probably the most useful tool for Node.js right now is npm, the node package manager. Using npm, we can easily make use of libraries that other programmers have written and we can share our own modules more easily amongst ourselves and the Node.js community.
InfoQ: Are there things that you think are missing and should be added to Node.js? How do you think it should evolve?
TJ (Express): I think it should continue as-is, being a low level foundation for frameworks to grow on. It should be considered a "blank canvas" to work from, with very little imposed upon the developer,
Guillermo (Socket.IO): SSL support in the HTTP server is incomplete and buggy. Besides that, it just comes down to get people using it, realizing the best patterns from looking at the common usage, and fixing bugs.
Matthew (Geddy): People are working on a package manager (NPM), so hopefully once that solidifies we'll have an easy way to install libraries and specify dependencies for projects we're building.
There is a wide array of embryonic projects attempting to deal with datastores and persistence, but no obvious best solution. We still need a solid ORM that works as much as it can to provide a common API for both relational and non-relational stores.
Astro (node-xmpp): In particular I would like to drop the node-base64 dependency from node-xmpp. A patch exposing Base64 encoding/decoding had been posted, but didn't get merged.
It's also interesting to many people trying to get web frameworks right.
Asynchronous programming can sometimes be a hassle and I understand if developers sometimes omit the callback references just to keep the control flow more straight-forward.
Peteris & James (StackVM): Whenever we think up some potential new library that we’d like to see in Node.js, we hack it up ourselves and release it so other programmers don’t need to waste time reinventing the wheel. Plenty of other programmers do the same thing, and the supporting ecosystem of libraries evolves on its own naturally. The Node.js core is intended to be small with the bulk of the functionality provided by supporting libraries, which is a good way for a community-driven project to distribute the workload and encourage participation.
InfoQ: What is the future roadmap for your project?
TJ (Express): At this point and time just keeping Express lean and mean. I am always accepting new ideas and patches for focused features that fit well with the project. Developers could even now consider Express a "frameworks framework", as itself could easily become a building block for larger more opinionated frameworks. Lastly I would like to thank Ciaran and Aaron for their great contributions, and my employer Sencha for allowing myself and Tim to give back to the community.
Guillermo (Socket.IO): Right now I'm hard at work at making sure it can be as fast, lean and well-tested as possible.
Matthew (Geddy): There is still a lot of work to be done on Geddy:
1. ORM query API
The persistence piece of the current model code is very much just a stake in the ground. ORMs are very complicated, and anything that tries to interface with both relational and non-relational stores is going to be even harder to build.
2. A plugin API
Geddy needs an easy way to allow snap-in functionality built by third parties.
Authentication is the perfect example of what should be implemented as a plugin.
4. Better modularity
Geddy was written to provide an integrated, "out of the box" experience that lets developers start writing real applications with a minimum of fuss. However, it should be easier to use components from Geddy "a la carte," or replace one specific piece with something else.
5. Support for other template libraries than simple EJS
People like to use crazy stuff like Haml or Moustache. Geddy should let them go crazy with that stuff.
6. Test coverage
Geddy has extensive tests for its router, and a solid suite of tests for models and validations, but the rest of the code isn't covered. Geddy's code needs to be refactored to be more testable so we can have more complete coverage.
Astro (node-xmpp): I was pretty happy about all the Github watchers I got in no time after I published the project. Other than that I'm still waiting for actual contributions.
My very own goals are met because I just wanted the XML parsing and authentication done, not to implement any particular XMPP XEPs on top of the library. That's what I leave to the application programmer for now.
Peteris & James (StackVM): In the shorter term, we’re focused on adding more features and tuning performance. Further out, we’ll probably be more focused on hardware and scaling.
Shane Hastie on Distributed Agile Teams, Product Ownership and the Agile Manifesto Translation Program
Shane Hastie Apr 17, 2015