BT

InfoQ Homepage News Starfish Brings Google-Style Distributed Processing to Ruby

Starfish Brings Google-Style Distributed Processing to Ruby

Bookmarks

Google developed a tool called MapReduce in C++ to make it easy to perform parallel computations over large datasets using many machines over a network. The system works with tasks that are easily turned into an algorithm where a certain step is repeated over all elements of an array or dataset. This step can then be performed in parallels by multiple machines before being combined down to a useful conclusion at the end. Google uses this tool to collate and analyze large sets of data (such as Web page contents) rapidly with their server farm.

Lucas Carlson has taken the MapReduce concept and brought it to Ruby. Recently, Lucas made a presentation called "Ridiculously easy ways to distribute processor intensive tasks using Rinda and DRb" where he looked at using Ruby's DRb system to get separate machines to run tasks such as log processing or database management tasks. By taking on the concepts from MapReduce, however, Lucas has taken it a step further and made it possible to replace standard map calls with calls that can processed in parallel with a new library called Starfish.

You can learn more and see some examples in Lucas's article about Starfish.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Article is slightly disingenuous

    by James Richardson /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    The use of CORBA as the java version of map/reduce is a little wide of the mark. Granted, its a vendor implementation (non-free) but the Tangosol InvocableMap does this stuff in about two lines of code too...

  • Re: Article is slightly disingenuous

    by Kenji Nakamura /

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hadoop is the open-source Java implementation of MapReduce and GFS(Google File System). Check it out...

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
Country/Zone:
State/Province/Region:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.