BT

Agile Asset Management with Ruby DSLs

Posted by Jeremy Voorhis on May 22, 2006 |

A domain specific language is a language-oriented tool designed to solve a specific set of programming tasks. Common traits are very close to the problem domain within which the language is intended to work, and operating at a very high level of abstraction. In his article about DSLs, Martin Fowler classifies DSLs as either external or internal (see link). An external DSL is a programming language that is either compiledor interpreted. An internal DSL is built within a general-purpose programming language. Effectively, an internal DSL is a very high-level API to a general-purpose programming language. This article is a story about how implementing an internal DSL for Ruby contributed to a PLANET ARGON development project.

The Problem

The goal of one of my recent PLANET ARGON development projects was to build a one-off content management system with Ruby on Rails that supports 18 languages and manages roughly one thousand image files. Many of these files are pieces of professional photography, each one weighing in at well over 1MB. Others are small, detailed pieces of line art bearing various national flags. The only thing these files had in common was that they were not ready to be used in production directly.

Our application has an Image model. It uses a file upload plugin that we wrote in-house to persist image data. We had begun managing the image transformations with after_save hooks that used the RMagick image manipulation API inside of our model. This would have worked well, but our requirements were continuously being adjusted. Some images required 4 variations to be produced. Others required two. Some of them had to be uploaded to a third party bandwidth provider. Sometimes we got the dimensions wrong. At one point, we had written a batch script to upload these source images so the after_save hooks could reprocess the images. Soon enough,the edge cases came and our callbacks became engorged with stacks of conditional statements. As you may have guessed, this got old fast. We needed a tool that would help us meet our client’s needs in a sustainable manner.

The Solution

One day, our client posted another reference image and a set of instructions for how they created the image in PhotoShop, to our Basecamp. I had had enough.Thinking about a more elegant solution to solve the problem, I remembered when Jason Watkins had told me about his experience with a class of tools, called asset compilers, when he worked in the video game industry. An asset compiler is a build system which transforms a set of media files into the media files needed for the final build of a video game. I had recently been automating a lot of repetitive tasks with Rake (Rake is a build system implemented in pure Ruby), so creating an asset compiler with Rake was very attractive. If the duty of creating the production images could be delegated to an external tool, our application would not have to change whenever our client’s requirements changed. To sweeten the deal, this tool would allow us to rebuild the production images with a single command when requirements changed.

I had begun by using Jim Weirich's (the author of Rake) RDoc task library which is distributed with Rake as a template. One usage pattern which Jim has established for Rake is a class which you instantiate with the appropriate options to describe the set of Rake tasks needed to automate your build. For example, here is an example of how to use the RDoc task library in your Rakefile to build documentation for a Rails application, the plugins it uses and the Rails framework itself.

  Rake::RDocTask.new('my_docs') do |rdoc|
# configuration
rdoc.rdoc_dir = 'doc/my_docs'
rdoc.main = "doc/README_FOR_APP"
rdoc.title = 'Comprehensive Documentation'

# app documentation
rdoc.rdoc_files.include('doc/README_FOR_APP')
rdoc.rdoc_files.include('app/**/*.rb')
rdoc.rdoc_files.include('lib/**/*.rb')

# plugins documentation
rdoc.rdoc_files.include('vendor/plugins/*/lib/**/*.rb')

# framework documentation
# ...snip...
end

Instantiating a Rake::RDocTask object within your Rakefile as per this example will create Rake tasks to build, delete and rebuild your documentation. The rdoc block parameter yielded by Rake::RDocTask.new is actually the Rake::RDocTask object which is being instantiated, and the options which is being instantiated,and rdoc’s attributes are defined in Rake::RDocTask by attr_accessors.

I began by extracting the code to build the production images from our Image model into a Rake task library. After creating a working task library which operated similarly to the RDoc task library, my code looked something like this:

  ImageTask.new :bronze_thumbnail do |t|
t.src_files = image_src 'images/*.jpg'
t.build_path = build 'greyscale_thumbnail'
t.remote_dirs << REMOTE_DIR[:greyscale_thumbnail]
t.transformation do |img|
# apply bronze image effect
img = img.quantize 256, Magick::GRAYColorspace
img = img.colorize 0.25, 0.25, 0.25, '#706000'
# crop to thumbnail size
...snip...
end
end

And that code defined the following Rake tasks, as described by running rake -T:

  rake assets:bronze_thumbnail:build                    # Build the bronze_thumbnail files
rake assets:bronze_thumbnail:clobber # Remove bronze_thumbnail files
rake assets:bronze_thumbnail:rebuild # Force a rebuild of the bronze_thumbnail files
rake assets:build # Build all assets
rake assets:clobber # Clobber all assets
rake assets:rebuild # Rebuild all assets

I realized how flexible my tool would be if the code describing these image transformations closely resembled the PhotoShop instructions given to us by our client – an internal DSL that defines Rake tasks for image transformations.

I created some unit tests for my task library to ensure tasks were being created properly and begun some test-driven refactoring. Because I was creating an internal DSL, which in my case is a Ruby API, the creation of the DSL could be purely test driven. I began by replacing the example code in my unit test which instantiates the task library’s classes with example code for my unimplemented DSL and revised it several times until I was satisfied.

  define_image_transformation 'thumbnailize' do
crop_to '62x62', :north
end

define_image_transformation 'bronze' do
greyscale
lighten
# r g b tint
tint 0.25, 0.25, 0.25, '#706000'
end

image_task 'bronze_thumbnail' do
from images 'images/*.jpg'
to build 'greyscale_thumbnail'
remote_dirs << REMOTE_DIR[:greyscale_thumbnail]
transformation do
bronze
thumbnailize
end
end

I ran my unit tests and watched a raft of error messages float by. This was good. I now knew what I wanted, and what steps were necessary to achieve my goal. I then renamed a couple of methods (src became from, for example) and created some helper methods to help instantiate the task library class. I also wrapped the lower-level RMagick code with a a library that used designer-friendly method names and created define_image_transformation which defines high-level application-specific image transformations by aggregating those RMagick wrappers. By making the distinction between what a graphics program like PhotoShop can do and what our client wants to do, I was able to ensure my tool would be fit for extraction and reuse across future projects – the high-level image transformations lived in my app’s lib/tasks directory, but the lower-level wrappers lived in vendor/asset_compiler alongside thetask library.

The Outcome

Currently, our lib/tasks/assets.rake file contains 12 calls to image_task, including the example shown above. Although the language may not be suitable for lay programmers (according to Fowler’s article,lay programmers are not professional programmers, but they can make use of simple programming tools to automate business tasks), it is in sync with our client’s language and neatly models their ideas and goals.

Because Ruby DSLs are Ruby code, I strongly recommend following the test-driven process which I described in this article: code an example of the Ruby internal DSL you wish to use, write test cases for the desired outcome of that example and code until all tests are passing. Also consider the scope of your DSL in the beginning. Because I emphasized a separation between the functionality of an image transformation tool and the needs of our application, building a powerful, reusable and extensible framework around our DSL became natural. I feel that this was the right decision because I know I am going to use this library again for other projects very soon. If your needs are more particular you may not need multiple layers of abstraction. If you intend to reuse your internal DSL, ensure that no application-specific business logic sneaks in, but provide a mechanism for capturing that business logic.

Finally, there is no canonical text on how you should create an internal DSL with any language. The consensus between all articles I have read on the topic is that you must learn by doing. Pay attention to how clients and domain experts communicate their requirements and focus on capturing those requirements with executable code. Even if you can’t transform your domain expert into a lay programmer, they should be able to see their requirements clearly during a code review or a pair-programming session, and your goal should be to capture their requirements as quickly as you receive them.

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

nice one by Alex Popescu

Very interesting and pragmatic approach. Still, I would like to see why and mostly how Ruby is making things simpler. In this case the DSL is equivalent with creating an API that follows correctly your domain. And I think that doing this is not something that can be done with only dynamic languages. Take a look at Hibernate Criteria: quite a similar thing for working with SQL.

BR,

./alex
--
.w( the_mindstorm )p.

Re: nice one by Obie Fernandez

Alex,

It's just one of those things you have to experience to understand fully. :)

Cheers,
Obie

Re: nice one by Alex Popescu

I've been playing with Ruby for almost 2 years (though I might not have the Ruby mindset), and I cannot say this was one of the things that were simpler due to Ruby. Finally, I would say that it is a matter of how proficient you are and that's all that matters.

./alex
--
.w( the_mindstorm )p.

Re: nice one by Jeremy Voorhis

Alex,

Yes and no. I know of no other system that lets you create, for example, a specialized build system for such a task as quickly and with so little code, though I may be mistaken. Ruby's mutability is what really makes it work the way it does.

Though it may be a matter of proficiency, I don't think the Java example is as quite as expressive as the Ruby DSL examples in my article, and other places like Jay Field's blog, but I think that is more because of Java's lack of mutability. I wouldn't be at all surprised to see a similar example in C or perhaps a functional language. It's all about the language's willingness to be built up until it is the best possible tool to express your business logic.

Re: nice one by Alex Popescu

Jeremy,

I agree with most of the things you say. Still, I am quite sure I can build the same thing with Ant for example. IMO the argument of less lines (argument that is overused IMO when speaking about Ruby) is not as important as readability. Though, I agree on Ruby expresiveness. When talking about mutability, are you refering to the fact that many things in Ruby are expressions, so allowing statement chaining, while on Java this is more difficult?

./alex
--
.w( the_mindstorm )p.

Re: nice one by Jeremy Voorhis

"less lines... is not as important as readability."

I wholeheartedly agree! See my post about code acceptability at www.jvoorhis.com/articles/2006/05/03/is-your-co...

By Ruby being mutable, I am definitely not referring to message chaining. That is something that is all too easy in both Java and Ruby, and in depending on the context, it can make your code harder to read and harder to change regardless of what language you are using. What I mean when I talk about Ruby's mutability is that any class in the core and standard libraries can be opened up and modified at runtime. Also, there isn't the same one-to-one mapping between method definitions and messages that are passed - the object that receives the message can decide what to do with it.

I am quite sure you could build such a tool as this with Ant. The benefit of creating a build system with Ruby/Rake, however, is that you have the full power of a general-purpose programming language available to you. This is quite different from a general-purpose programming language shoehorned into XML. And I suspect that James Duncan Davidson would agree.

Re: nice one by Alex Popescu

What I mean when I talk about Ruby's mutability is that any class in the core and standard libraries can be opened up and modified at runtime. Also, there isn't the same one-to-one mapping between method definitions and messages that are passed - the object that receives the message can decide what to do with it.


This concept is very interesting. It may show a lot of benefits, but also may bring a lot of risk. And it makes me think about AOP (by the way with AOP Java becomes as mutable as Ruby ;-)). While, with AOP things may be well documented so that over time these changes are still visible, I am not very sure how this can be done in Ruby. If I start aliasing core classes/objects, than in 5 years maintaining a big project may become a real problem. But, this is something that I think we cannot decide now and we just sit back, enjoy what we are doing and see what the future will bring.


I am quite sure you could build such a tool as this with Ant. The benefit of creating a build system with Ruby/Rake, however, is that you have the full power of a general-purpose programming language available to you. This is quite different from a general-purpose programming language shoehorned into XML. And I suspect that James Duncan Davidson would agree.


Just a short clarification: I wouldn't do it with only-XML, but by creating/using different tasks and than just chaining them through XML. Benefit: full power of a programming language no need of recompilation (at least in Java ;-) ).

./alex
--
.w( the_mindstorm )p.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

7 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT