InfoQ

News

MapReduce A Step Backwards: Is Comparison to Relational Databases Fair?

Posted by Scott Delap on Jan 18, 2008 03:40 PM

Community
Java
Topics
Grid Computing
Tags
Hadoop ,
MapReduce
A recent article on the Database Column by David J. DeWitt and Michael Stonebraker attempts to compare the increasingly popular MapReduce programming paradigm to a relational database. The article goes so far as to say:

...As a data processing paradigm, MapReduce represents a giant step backwards. The database community has learned the following three lessons from the 40 years that have unfolded since IBM first released IMS in 1968....Given the experimental evaluations to date, we have serious doubts about how well MapReduce applications can scale. Moreover, the MapReduce implementers would do well to study the last 25 years of parallel DBMS research literature.

The article goes on to list criteria such as:

  • MapReduce is a poor implementation (in comparison to B-trees)
  • MapReduce is not novel
  • MapReduce is missing features (such as loading and indexing)
  • MapReduce is incompatible with the DBMS tools

The blogsphere has quickly called foul on the comparison and its reasoning. Greg Jorgensen provides a detailed rebuttal. Among the items he notes are that MapReduce is not a database but an algorithmic technique for distributed processing and should not be compared to one. Jorgensen proposes that a better comparison would have been to SimpleDB:

...What the authors really want to gripe about is distributed “cloud” data management systems like Amazon’s SimpleDB; in fact if you change “MapReduce” to “SimpleDB” the original article almost makes sense...

Rich Skrenta comments on the angle of disruption:

...The thing that disrupts you is always uglier and worse in some way. Less features, less developed. But if there's a 10X price win in there somewhere, the cheap rickety thing wins in the end. Think Linux vs. AT&T Unix, or mysql vs. Oracle...

Lengthy debate and comment on the topic can also be found on reddit and ycombinator.

Sad... by Nikita Ivanov Posted Jan 18, 2008 5:39 PM
more research needed. by Zubin Wadia Posted Jan 19, 2008 2:42 AM
Stooopid by Kevin Teague Posted Jan 19, 2008 9:46 PM
Holy crap... by Kurt Christensen Posted Jan 20, 2008 10:47 AM
Re: Holy crap... by Kurt Christensen Posted Jan 20, 2008 10:53 AM
  1. Back to top

    Sad...

    Jan 18, 2008 5:39 PM by Nikita Ivanov

    I think authors either do not understand Map/Reduce (doubtfully) or clearly misplacing where it’s used and for what scenarios. What I think authors are also missing is the combination of data portioning and affinity map/reduce that is becoming prevailing design pattern for grid applications. I blogged about it here in more details. All in all, it is sad to read such a misguided piece… Best, Nikita Ivanov. GridGain – Grid Computing Made Simple

  2. Back to top

    more research needed.

    Jan 19, 2008 2:42 AM by Zubin Wadia

    MapReduce is not a DB. I don't see the parallel here to Teradata or any others of similar ilk. Swapping it with SimpleDB or BigTable is a more logical perspective. Also - if they were referring to BigTable, then in fact, it does support indexes and doesn't do brute force searches.

  3. Back to top

    Stooopid

    Jan 19, 2008 9:46 PM by Kevin Teague

    The authors would have done well to read the introduction to the MapReduce paper they cited:

    Prior to our development of MapReduce, the authors and many others at Google implemented hundreds of special-purpose computations that process large amounts of raw data, such as crawled documents, Web request logs, etc., to compute various kinds of derived data, such as inverted indices, various representations of the graph structure of Web documents, summaries of the number of pages crawled per host, and the set of most frequent queries in a given day. Most such computa- tions are conceptually straightforward. However, the input data is usu- ally large and the computations have to be distributed across hundreds or thousands of machines in order to finish in a reasonable amount of time. The issues of how to parallelize the computation, distribute the data, and handle failures conspire to obscure the original simple com- putation with large amounts of complex code to deal with these issues.
    MapReduce is for doing computation on raw data. In Google's case this data is usually crawled from the web. Google likely stores some of the data they glean from raw data they process using MapReduce in a ... DBMS. *sigh*

  4. Back to top

    Holy crap...

    Jan 20, 2008 10:47 AM by Kurt Christensen

    Yeah, I definitely think that the relational database goo-roos who annually inflict billions of dollars in monetary damages on unsuspecting IT departments are in a grand position to tell Google how to do search. Perhaps David and Michael would also like to offer me parenting advice...?

  5. Back to top

    Re: Holy crap...

    Jan 20, 2008 10:53 AM by Kurt Christensen

    Not that I parent as well as Google does search. Oh, you know what I meant...

Educational Content

Bindings, Platforms, and Innovation

This presentation focuses on the Internet and separating myth from fact, history from the future, and the mundane from the imaginative. Bob Frankston presents a vision of what could and should be.

Orchestrating Long Running Activities with JBoss / JBPM

This article explores the use of JBoss and jBPM to implement design solutions that effectively address the issue of orchestrating long running activities.

Neo4j - The Benefits of Graph Databases

This presentation covers the use of graph databases as an optimal solution for data that is difficult to fit in static tables, rapidly evolving data or data that has a lot of optional attributes.

Realistic about Risk: Software development with Real Options

This session introduces Real Options and shows how it can help in running your project. Real Options is a decision-making process that can be used to manage risk.

Communication Flexibility Using Bindings

This article discusses the use of bindings on services and references (including the instance of non-configured bindings) as the means to implement SCA communications in a Web and SOA environment.

Writing DSLs in Groovy

After a short introduction to DSLs, Scott Davis plays with the keyboard showing how to approach the creation of a DSL by typing working snippets of Groovy code that get executed.

Scaling Agile with C/ALM (Collaborative Application Lifecycle Management)

IBM Rational and InfoQ present, Scaling Agile with C/ALM, an eBook showing organizations how to become “finely tuned software delivery machines” by enabling team integration and scaling.

Concurrent Programming with Microsoft F#

Amanda Laucher presents a real life enterprise application written in F#. She shows actual code snippets, explaining design decisions and suggesting how to use some of the F# constructs.