BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News NoSQL Benchmark Compares Aerospike, Cassandra, Couchbase and MongoDB

NoSQL Benchmark Compares Aerospike, Cassandra, Couchbase and MongoDB

This item in japanese

Lire ce contenu en français

Bookmarks

A recent set of benchmarks compares Aerospike, Cassandra, Couchbase and MongoDB to see how they fare when it comes to insert throughput, maximum throughput, latency and behavior during a failover.

Thumbtack Technology has released two benchmark whitepapers with results from the comparison of several key-value stores: Ultra-High Performance NoSQL Benchmarking: Analyzing Durability and Performance Tradeoffs (PDF) and NoSQL Failover Characteristics: Aerospike, Cassandra, Couchbase, MongoDB (PDF). Both benchmarks attempted to test “consumer-facing applications which require extremely high throughput and low latency, and whose information can be represented using a key-value schema.”

Thumbtack used an improved version of Yahoo! Cloud Serving Benchmark (YCSB) one that is supposed to overcome some limitations reached when using very high volumes and multiple clients. The YCSB changes have been documented in the first whitepaper and committed back to the community.

The NoSQL databases tested were Aerospike, Cassandra, Couchbase (1.8 and 2.0), and MongoDB. The first is a commercial product, and the last is a document data store not a key-value store, but it was included because “in our experience clients often consider it for similar kinds of applications.” All databases were optimized using recommendations from the vendors backing them. The test systems used SSD storage rather than rotating disks. The whitepapers contain detailed information regarding the methodology used, the client and workload configuration, the hardware configuration, etc.

Thumbtack acknowledged having “strategic and/or commercial relationships with Aerospike, Couchbase, and 10gen” and the hardware used was rented from Aerospike.

We are including some of the results of the benchmarks.

Insert Throughput

The databases were loaded with initial working sets using YCSB’s loading routing which performs a number of inserts. Couchbase had good results when the working sets were loaded into memory, but it had problems loading on SSD when Couchbase 1.8 did not finish the operation and a smaller set and asynchronous mode had to be used for Couchbase 2.0. That explains the gradient blue used for it. Aerospike came second.

image

Maximum Throughput

This test used a “a strong durability model, using a dataset that, when replicated, would be significantly larger than the server’s RAM. This test is intended to model usage for transactional data that requires strong durability guarantees.”

Couchbase does not appear on the graphic because it could not complete the test using synchronous replication.

image

When asynchronous replication was applied, the result in memory was:

image

Latency/Throughput

The benchmarks also measured read and update latency at different levels of traffic. The next graphic includes a full view and a zoomed one for each of those:

image

Failover

Thumbtack attempted to see what happens when a node goes down, a hardware failure being simulated:

image

The downtime was also measured, i.e. the time needed for the cluster the become responsive again after a failure, all databases showing reasonable values:

image

The Thumbtack benchmarks contains more results for different cases but were not included here.

Another NoSQL benchmark was published in October 2012 when Cassandra, HBase, MongoDB, and Riak were compared. MySQL was also included in those tests for a reference against SQL technologies.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Interesting results

    by peter lin,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    The summary is interesting. Having spent the last few years studying the source code of several NoSql databases, comparing simple Key/Value, Column Family and Document databases doesn't make sense to me. Even if some use cases can use either type of NoSql database, it's missing some important details. Simple Key/Value databases are not well suited to time series data, nor is document database.

    The design of the key has a significant impact on query performance. Storing everything as JSON works for some use cases, but many cases it's totally not appropriate. Hopefully readers will take time to understand the differences and not jump to irrational conclusions not supported by these benchmark results.

  • Re: Interesting results

    by Vic Cekvenich,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Another comparison for more input:
    blog.michaelckennedy.net/2010/04/29/mongodb-vs-...

  • Aerospike wins

    by David Karnowski,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    > Thumbtack acknowledged having “strategic and/or commercial relationships with Aerospike, Couchbase, and 10gen” and the hardware used was rented from Aerospike.

    I'm a nosql newb, but it does seem a coincidence that the one product I've never heard of wins this when the testing company stands to gain if more people adopt Aerospike. No conflict of interest here at all is there? You guys shouldn't have published this. It makes you look bad.

  • Re: Aerospike wins

    by Cameron Purdy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    This is par for the course in our industry.

    Peace,

    Cameron Purdy | Oracle

  • Re: Interesting results

    by Akmal Chaudhri,

    Your message is awaiting moderation. Thank you for participating in the discussion.

  • Re: Aerospike wins

    by Josh Millgram,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    It is always to be skeptical of benchmarks. The best benchmark is the one you do yourself on your own equipment.

    But, Thumbtack is an *integrator* -- so they use all the products they evaluate for their customers. Notice they disclosed relationships with ALL the vendors. But their results did not state that ALL the vendors were the best. If they were truly biased they would have declared a three-way tie between Aerospike, Couchbase and MongoDB (10gen). In fact, they dinged Couchbase pretty hard in spite of the fact that they have a relationship with them!

    So, I am not sure we should slam them for just being honest.

  • Re: Aerospike wins

    by Ronaldo Pontes,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Is Thumbtack just being honest or do they earn more money promoting Aerospike vs Couchbase?

    Benchmarks made by people with conflict of interest shouldn't be published.

    This leads me to wonder if InfoQ is making money with these kinds of reports...

  • Defying the laws of physics

    by d t,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I can't help noticing that the top performers exceed the performance of maximum sequential write throughput of an average SSD, and the bottom performers are below the random IO performance of an average SSD. What SSD was used? Was all the data persisted in the insert workload or is this a test around which datastore can buffer data in-memory the fastest?

  • other databases

    by Juan Pi.,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hello.

    Please, could you add Hyperdex, MonetDB and Redis to your benchmarks?
    Thank you.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT