BT

Starfish Brings Google-Style Distributed Processing to Ruby

| by Peter Cooper Follow 0 Followers on Aug 24, 2006. Estimated reading time: 1 minute |

Google developed a tool called MapReduce in C++ to make it easy to perform parallel computations over large datasets using many machines over a network. The system works with tasks that are easily turned into an algorithm where a certain step is repeated over all elements of an array or dataset. This step can then be performed in parallels by multiple machines before being combined down to a useful conclusion at the end. Google uses this tool to collate and analyze large sets of data (such as Web page contents) rapidly with their server farm.

Lucas Carlson has taken the MapReduce concept and brought it to Ruby. Recently, Lucas made a presentation called "Ridiculously easy ways to distribute processor intensive tasks using Rinda and DRb" where he looked at using Ruby's DRb system to get separate machines to run tasks such as log processing or database management tasks. By taking on the concepts from MapReduce, however, Lucas has taken it a step further and made it possible to replace standard map calls with calls that can processed in parallel with a new library called Starfish.

You can learn more and see some examples in Lucas's article about Starfish.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Article is slightly disingenuous by James Richardson

The use of CORBA as the java version of map/reduce is a little wide of the mark. Granted, its a vendor implementation (non-free) but the Tangosol InvocableMap does this stuff in about two lines of code too...

Re: Article is slightly disingenuous by Kenji Nakamura

Hadoop is the open-source Java implementation of MapReduce and GFS(Google File System). Check it out...

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

2 Discuss
BT