Facilitating the spread of knowledge and innovation in professional software development



Choose your language

InfoQ Homepage News Starfish Brings Google-Style Distributed Processing to Ruby

Starfish Brings Google-Style Distributed Processing to Ruby


Google developed a tool called MapReduce in C++ to make it easy to perform parallel computations over large datasets using many machines over a network. The system works with tasks that are easily turned into an algorithm where a certain step is repeated over all elements of an array or dataset. This step can then be performed in parallels by multiple machines before being combined down to a useful conclusion at the end. Google uses this tool to collate and analyze large sets of data (such as Web page contents) rapidly with their server farm.

Lucas Carlson has taken the MapReduce concept and brought it to Ruby. Recently, Lucas made a presentation called "Ridiculously easy ways to distribute processor intensive tasks using Rinda and DRb" where he looked at using Ruby's DRb system to get separate machines to run tasks such as log processing or database management tasks. By taking on the concepts from MapReduce, however, Lucas has taken it a step further and made it possible to replace standard map calls with calls that can processed in parallel with a new library called Starfish.

You can learn more and see some examples in Lucas's article about Starfish.

We need your feedback

How might we improve InfoQ for you

Thank you for being an InfoQ reader.

Each year, we seek feedback from our readers to help us improve InfoQ. Would you mind spending 2 minutes to share your feedback in our short survey? Your feedback will directly help us continually evolve how we support you.

Take the Survey

Rate this Article


Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Article is slightly disingenuous

    by James Richardson,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    The use of CORBA as the java version of map/reduce is a little wide of the mark. Granted, its a vendor implementation (non-free) but the Tangosol InvocableMap does this stuff in about two lines of code too...

  • Re: Article is slightly disingenuous

    by Kenji Nakamura,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hadoop is the open-source Java implementation of MapReduce and GFS(Google File System). Check it out...

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p


Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.