BT

Versatile RESTful APIs Beyond XML

Posted by James Stewart on Mar 13, 2007 |

The world of RESTful resources that Rails firmly entered with version 1.2 naturally uses XML as its lingua franca. But there's no reason that it can't be multi-lingual, and thanks to the versatility of Rails it's easy to support other standards alongside XML in our RESTful applications, potentially opening them up to a wider audience and/or reducing their bandwidth requirements.

Setting Up Our Controller

I'm going to focus on an app that only deals with one resource, events. It will provide the standard set of CRUD actions, with a multi-lingual twist. To get started create a new Rails app:

% Rails lingua
% cd lingua

and use the scaffold_resource generator to create an event resource:

% script/generate scaffold_resource Event

Open up app/controllers/events_controller.rb and take a look, and you'll see plenty of use of respond_to to add XML output for each action, such as the 'show' action:

def show
@event = Event.find(params[:id])

respond_to do |format|
format.html # show.rhtml
format.xml { render :xml => @event.to_xml }
end
end

(One thing I won't be covering in this article is authentication/authorization. I highly recommend the restful_authentication plugin as a starting point for that).

Introducing JSON

JSON is a standard that has picked up traction of late, particularly thanks to the maturing of javascript as a language for UI development and the growth in take up of AJAX. Based on serialized javascript, JSON has reached a point where many regard it as a significantly better way to serialize and transmit simple data structures than XML, and it's certainly less verbose.

Getting Rails to output JSON is remarkably easy, since ActiveRecord will already serialize its records in that format. If all we want is JSON output then we can just update the actions with:

def show
@event = Event.find(params[:id])

respond_to do |format|
format.html # show.rhtml
format.xml { render :xml => @event.to_xml }
format.json { render :text => @event.to_json }
end
end

and now a GET on /events/1.json will return the event in JSON.

But to be truly multi-lingual we're going to want to be able to understand (parse) JSON, not just speak it. There are various ways we could do that. First up we'll need to be able to parse JSON from Ruby.

To most intents and purposes, valid JSON is also valid YAML. So given Rails' early embrace of YAML, the very simplest parser would be the YAML parser, which is already available as a symbol:

ActionController::Base.param_parsers[Mime::YAML] = :yaml

But JSON allows for a few extra constructs (particularly comments) that mean this will miss input from some less-than-careful JSON creators. The off-the-shelf JSON gems break Rails' JSON generation because in providing methods to both parse and generate JSON, they redefine the to_json method. My solution has been to take one of them, extract just the parsing functions, and store them in my lib folder as json.rb. That code's too lengthy to paste here, but you can find it at:

http://jystewart.net/code/json/json.txt

Once we have a way to parse the JSON, we then need to get that into Rails. The most obvious approach is to add a before_filter to our controller that intercepts JSON input, parses it out and hands it to the action as a standard parameters hash. We can do that with something like:

before_filter :intercept_json, :if => Proc.new { |p| p.content_type == Mime::JSON }

def intercept_json
params[:event] = JSON::parse(params[:event])
end

But if we were to add another resource, that wouldn't be so DRY, and if we add in support for more formats, we'll end up with lots of code in each controller. Rails handles XML input transparently to the controller, and it would be nice to be able to do that for JSON too. Thankfully, we can!

Rails 1.2 also introduced pluggable param_parsers, which let us define how each MIME type should be parsed. In fact, that's how the XML parsing is set up, with code like:

ActionController::Base.param_parsers[Mime::XML] = :xml_simple

Specific parsers can be added for any mime type, with the standard Rails parameter parsing (from CGI) catching any that fall through the gaps. We can add our own parsers in our environment.rb. To add JSON all we need is:

ActionController::Base.param_parsers[Mime::JSON] = Proc.new do |data|
JSON::parse(post)
end

And now our params hash will be the unserialized JSON request.

And that's all it takes. Once we actually take the step of creating our database, our events controller now presents a RESTful resource which can be read/written not just in XML, but also using JSON, and because JSON is so simple to use in javascript, we can very easily separate our behaviour layer from our application logic, using JSON to communicate between the two, or since most major languages offer lightweight JSON libraries, offer this as a lower-bandwidth way to receive input for our RESTful API.

Introducing Microformats

Microformats, like JSON, have been gaining more and more attention of late, but they come from a quite different angle. In a web development world where we finally have good support for standards-based use of HTML, it becomes more realistic to make our HTML (structurally, and in the use of IDs and class names) semantically rich. Instead of making our right-hand navigation bar simply the right most block in a table, or even labelling it with the ID "right-sidebar", we can mark it as 'secondary-navigation' and begin to say something about the content rather than the presentation.

Microformats seek to standardise some of those semantics for commonly used elements, providing a set of class names for events (hCalendar, derived from the iCalendar standard), personal and business details (hCard, derived from vcards), news stories (hAtom, derived from the atom syndication format) and more.

Adding microformat output is even easier than generating JSON, as we just need to make sure we apply the class name conventions in our .rhtml view files. Parsing them, however, is a trickier business as (being HTML) microformats don't have a distinctive, accepted mime type to distinguish them from standard requests. I'll come back to that, but first let's look at how to parse the input if we have already identified it.

Rubyists are lucky to have one of the most versatile microformat parsing libraries around, in the form of Chris Wanstrath's mofo gem. Mofo is based on why's hpricot library, and provides a DSL for describing microformats. It's distributed with classes offering definitions of the most common microformats, and so to parse out, say, all the hCalendar events in an HTML string all we need to do is invoke it with:

require 'mofo' data =  events = HCalendar.find(:all, :text => data)

We can even read out all the microformats mofo knows about with:

require 'mofo'
data =
parsed_data = Microformat.find(:all, :text => data)

Mofo will then return objects with all of the microformat data easily accessible.

hpricot and mofo are very efficient libraries, but nevertheless invoking them to parse all of our input would be too much, particularly if we have a standard web interface to take into account. There has been a little movement on identifying microformats using profile URIs, but support for them is far from common or stable. Hopefully as microformats mature, there will be more standardisation of how to handle these cases.

For now the easiest approach is to attempt to parse the input with the standard CGI parser, and test that return to see if the POST body is supposed to be understood as a single string, or as separate parameters. I use a general Proc called microformat_interceptor:

microformat_interceptor = Proc.new do |data|
parsed = CGI.parse(data)

if parsed.collect { |key, value| key + '=' + value[0] }.first == data
Microformat.find(:all, :text => data)
else
CGIMethods.parse_request_parameters(parsed)
end
end

and then assign this for both text/html and application/x-www-form-urlencoded mime types:

Mime::FORM = Mime::Type.lookup("application/x-www-form-urlencoded")

ActionController::Base.param_parsers[Mime::FORM] = microformat_interceptor
ActionController::Base.param_parsers[Mime::HTML] = microformat_interceptor

Since our action is expecting a single event as its input there will still be some work to extract the first event from the array this will generate, and then we will need to patch our Event model to accept HCalendar in its constructor methods, but presuming our model's columns share their names with the microformat attribute names, that's not a significant issue.

So given that we already support standard POSTS, XML, and JSON, why would we want to support microformats in a Rails application? On the frontend, it's a no-brainer. If we can publish our data in a standardised format that makes sense to humans, web browsers, and other parsers, we stand to gain better rankings from search engines, have our data included in indexes, and otherwise spread our message around the web.

But for accepting data, it also makes a lot of sense as it broadens our appeal. Learning microformats should take a matter of minutes for your HTML guru, whereas getting to grips with a new XML schema or JSON would be a much bigger task. Now the pages they generate in one system can simply be fed directly into your web service as the input. Data can be created once for web pages and for communication between systems, and as more and more publishing tools begin to support microformats, the audience for your services will just become bigger.


About the Author

James Stewart is a web developer primarily working with Rails. Currently living in the USA, he is in the midst of moving home to the UK. He blogs regularly on web development matters at http://jystewart.net/process/.

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Treating JSON as YAML by Aslak Hellesøy

Rails just added support for JSON parsing by simply massaging the JSON into valid YAML and then using the YAML parser: jystewart.net/process/2007/03/improvements-to-r... Does anyone have an opinion on the differences between Rails' new approach and James' parser? I would think the need for James parser is now obsoleted?

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

1 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT