Google developed a tool called MapReduce in C++ to make it easy to perform parallel computations over large datasets using many machines over a network. The system works with tasks that are easily turned into an algorithm where a certain step is repeated over all elements of an array or dataset. This step can then be performed in parallels by multiple machines before being combined down to a useful conclusion at the end. Google uses this tool to collate and analyze large sets of data (such as Web page contents) rapidly with their server farm.
Lucas Carlson has taken the MapReduce concept and brought it to Ruby. Recently, Lucas made a presentation called "Ridiculously easy ways to distribute processor intensive tasks using Rinda and DRb" where he looked at using Ruby's DRb system to get separate machines to run tasks such as log processing or database management tasks. By taking on the concepts from MapReduce, however, Lucas has taken it a step further and made it possible to replace standard map calls with calls that can processed in parallel with a new library called Starfish.
You can learn more and see some examples in Lucas's article about Starfish.
Community comments
Article is slightly disingenuous
by James Richardson,
Re: Article is slightly disingenuous
by Kenji Nakamura,
Article is slightly disingenuous
by James Richardson,
Your message is awaiting moderation. Thank you for participating in the discussion.
The use of CORBA as the java version of map/reduce is a little wide of the mark. Granted, its a vendor implementation (non-free) but the Tangosol InvocableMap does this stuff in about two lines of code too...
Re: Article is slightly disingenuous
by Kenji Nakamura,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hadoop is the open-source Java implementation of MapReduce and GFS(Google File System). Check it out...