Amazon Rolls Out Hadoop Based MapReduce to EC2

Posted by Scott Delap on Apr 02, 2009

Sections

Operations & Infrastructure,

Enterprise Architecture,

Development,

Architecture & Design

Topics

Grid Computing ,

Cloud Computing ,

Java

Tags

EC2 ,

Hadoop ,

MapReduce

There have been tutorials available for quite a while detailing how to run the popular Apache Hadoop MapReduce framework on Amazon EC2.  Today Amazon raised the bar providing official support via Amazon Elastic MapReduce.  From the product page:

Amazon Elastic MapReduce automatically spins up a Hadoop implementation of the MapReduce framework on Amazon EC2 instances, sub-dividing the data in a job flow into smaller chunks so that they can be processed (the “map” function) in parallel, and eventually recombining the processed data into the final solution (the “reduce” function). Amazon S3 serves as the source for the data being analyzed, and as the output destination for the end results.

Amazon Elastic MapReduce pricing is on top of existing EC2 charges at the rate of 15%.  The FAQ has a full list of details on pricing and usage.  The official AWS blog also provides coverage:

...Processing in Elastic MapReduce is centered around the concept of a Job Flow. Each Job Flow can contain one or more Steps. Each step inhales a bunch of data from Amazon S3, distributes it to a specified number of EC2 instances running Hadoop (spinning up the instances if necessary), does all of the work, and then writes the results back to S3. Each step must reference application- specific "mapper" and/or "reducer" code (Java JARs or scripting code for use via the Streaming model). We've also included the Aggregate Package with built-in support for a number of common operations such as Sum, Min, Max, Histogram, and Count. You can get a lot done before you even start to write code!

We're providing three distinct access routes to Elastic MapReduce. You have complete control via the Elastic MapReduce API, you can use the Elastic MapReduce command-line tools, or you can go all point-and-click with the Elastic MapReduce tab within the AWS Management Console! Let's take a look at each one...

ZDNet's Dana Gardner speculates on the implications of of Amazon's new offering for the business intelligence market.

• Older >