BT
x Your opinion matters! Please fill in the InfoQ Survey about your reading habits!

NoSQL Benchmark Compares Aerospike, Cassandra, Couchbase and MongoDB

by Abel Avram on Apr 12, 2013 |

A recent set of benchmarks compares Aerospike, Cassandra, Couchbase and MongoDB to see how they fare when it comes to insert throughput, maximum throughput, latency and behavior during a failover.

Thumbtack Technology has released two benchmark whitepapers with results from the comparison of several key-value stores: Ultra-High Performance NoSQL Benchmarking: Analyzing Durability and Performance Tradeoffs (PDF) and NoSQL Failover Characteristics: Aerospike, Cassandra, Couchbase, MongoDB (PDF). Both benchmarks attempted to test “consumer-facing applications which require extremely high throughput and low latency, and whose information can be represented using a key-value schema.”

Thumbtack used an improved version of Yahoo! Cloud Serving Benchmark (YCSB) one that is supposed to overcome some limitations reached when using very high volumes and multiple clients. The YCSB changes have been documented in the first whitepaper and committed back to the community.

The NoSQL databases tested were Aerospike, Cassandra, Couchbase (1.8 and 2.0), and MongoDB. The first is a commercial product, and the last is a document data store not a key-value store, but it was included because “in our experience clients often consider it for similar kinds of applications.” All databases were optimized using recommendations from the vendors backing them. The test systems used SSD storage rather than rotating disks. The whitepapers contain detailed information regarding the methodology used, the client and workload configuration, the hardware configuration, etc.

Thumbtack acknowledged having “strategic and/or commercial relationships with Aerospike, Couchbase, and 10gen” and the hardware used was rented from Aerospike.

We are including some of the results of the benchmarks.

Insert Throughput

The databases were loaded with initial working sets using YCSB’s loading routing which performs a number of inserts. Couchbase had good results when the working sets were loaded into memory, but it had problems loading on SSD when Couchbase 1.8 did not finish the operation and a smaller set and asynchronous mode had to be used for Couchbase 2.0. That explains the gradient blue used for it. Aerospike came second.

image

Maximum Throughput

This test used a “a strong durability model, using a dataset that, when replicated, would be significantly larger than the server’s RAM. This test is intended to model usage for transactional data that requires strong durability guarantees.”

Couchbase does not appear on the graphic because it could not complete the test using synchronous replication.

image

When asynchronous replication was applied, the result in memory was:

image

Latency/Throughput

The benchmarks also measured read and update latency at different levels of traffic. The next graphic includes a full view and a zoomed one for each of those:

image

Failover

Thumbtack attempted to see what happens when a node goes down, a hardware failure being simulated:

image

The downtime was also measured, i.e. the time needed for the cluster the become responsive again after a failure, all databases showing reasonable values:

image

The Thumbtack benchmarks contains more results for different cases but were not included here.

Another NoSQL benchmark was published in October 2012 when Cassandra, HBase, MongoDB, and Riak were compared. MySQL was also included in those tests for a reference against SQL technologies.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Interesting results by peter lin

The summary is interesting. Having spent the last few years studying the source code of several NoSql databases, comparing simple Key/Value, Column Family and Document databases doesn't make sense to me. Even if some use cases can use either type of NoSql database, it's missing some important details. Simple Key/Value databases are not well suited to time series data, nor is document database.

The design of the key has a significant impact on query performance. Storing everything as JSON works for some use cases, but many cases it's totally not appropriate. Hopefully readers will take time to understand the differences and not jump to irrational conclusions not supported by these benchmark results.

Re: Interesting results by Vic Cekvenich

Another comparison for more input:
blog.michaelckennedy.net/2010/04/29/mongodb-vs-...

Aerospike wins by David Karnowski

> Thumbtack acknowledged having “strategic and/or commercial relationships with Aerospike, Couchbase, and 10gen” and the hardware used was rented from Aerospike.

I'm a nosql newb, but it does seem a coincidence that the one product I've never heard of wins this when the testing company stands to gain if more people adopt Aerospike. No conflict of interest here at all is there? You guys shouldn't have published this. It makes you look bad.

Re: Aerospike wins by Cameron Purdy

This is par for the course in our industry.

Peace,

Cameron Purdy | Oracle

Re: Aerospike wins by Josh Millgram

It is always to be skeptical of benchmarks. The best benchmark is the one you do yourself on your own equipment.

But, Thumbtack is an *integrator* -- so they use all the products they evaluate for their customers. Notice they disclosed relationships with ALL the vendors. But their results did not state that ALL the vendors were the best. If they were truly biased they would have declared a three-way tie between Aerospike, Couchbase and MongoDB (10gen). In fact, they dinged Couchbase pretty hard in spite of the fact that they have a relationship with them!

So, I am not sure we should slam them for just being honest.

Re: Aerospike wins by Ronaldo Pontes

Is Thumbtack just being honest or do they earn more money promoting Aerospike vs Couchbase?

Benchmarks made by people with conflict of interest shouldn't be published.

This leads me to wonder if InfoQ is making money with these kinds of reports...

Defying the laws of physics by d t

I can't help noticing that the top performers exceed the performance of maximum sequential write throughput of an average SSD, and the bottom performers are below the random IO performance of an average SSD. What SSD was used? Was all the data persisted in the insert workload or is this a test around which datastore can buffer data in-memory the fastest?

other databases by Juan Pi.

Hello.

Please, could you add Hyperdex, MonetDB and Redis to your benchmarks?
Thank you.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

9 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT