NoSQL Benchmark Compares Aerospike, Cassandra, Couchbase and MongoDB
A recent set of benchmarks compares Aerospike, Cassandra, Couchbase and MongoDB to see how they fare when it comes to insert throughput, maximum throughput, latency and behavior during a failover.
Thumbtack Technology has released two benchmark whitepapers with results from the comparison of several key-value stores: Ultra-High Performance NoSQL Benchmarking: Analyzing Durability and Performance Tradeoffs (PDF) and NoSQL Failover Characteristics: Aerospike, Cassandra, Couchbase, MongoDB (PDF). Both benchmarks attempted to test “consumer-facing applications which require extremely high throughput and low latency, and whose information can be represented using a key-value schema.”
Thumbtack used an improved version of Yahoo! Cloud Serving Benchmark (YCSB) one that is supposed to overcome some limitations reached when using very high volumes and multiple clients. The YCSB changes have been documented in the first whitepaper and committed back to the community.
The NoSQL databases tested were Aerospike, Cassandra, Couchbase (1.8 and 2.0), and MongoDB. The first is a commercial product, and the last is a document data store not a key-value store, but it was included because “in our experience clients often consider it for similar kinds of applications.” All databases were optimized using recommendations from the vendors backing them. The test systems used SSD storage rather than rotating disks. The whitepapers contain detailed information regarding the methodology used, the client and workload configuration, the hardware configuration, etc.
Thumbtack acknowledged having “strategic and/or commercial relationships with Aerospike, Couchbase, and 10gen” and the hardware used was rented from Aerospike.
We are including some of the results of the benchmarks.
The databases were loaded with initial working sets using YCSB’s loading routing which performs a number of inserts. Couchbase had good results when the working sets were loaded into memory, but it had problems loading on SSD when Couchbase 1.8 did not finish the operation and a smaller set and asynchronous mode had to be used for Couchbase 2.0. That explains the gradient blue used for it. Aerospike came second.
This test used a “a strong durability model, using a dataset that, when replicated, would be significantly larger than the server’s RAM. This test is intended to model usage for transactional data that requires strong durability guarantees.”
Couchbase does not appear on the graphic because it could not complete the test using synchronous replication.
When asynchronous replication was applied, the result in memory was:
The benchmarks also measured read and update latency at different levels of traffic. The next graphic includes a full view and a zoomed one for each of those:
Thumbtack attempted to see what happens when a node goes down, a hardware failure being simulated:
The downtime was also measured, i.e. the time needed for the cluster the become responsive again after a failure, all databases showing reasonable values:
The Thumbtack benchmarks contains more results for different cases but were not included here.
Another NoSQL benchmark was published in October 2012 when Cassandra, HBase, MongoDB, and Riak were compared. MySQL was also included in those tests for a reference against SQL technologies.
The design of the key has a significant impact on query performance. Storing everything as JSON works for some use cases, but many cases it's totally not appropriate. Hopefully readers will take time to understand the differences and not jump to irrational conclusions not supported by these benchmark results.
Re: Interesting results
I'm a nosql newb, but it does seem a coincidence that the one product I've never heard of wins this when the testing company stands to gain if more people adopt Aerospike. No conflict of interest here at all is there? You guys shouldn't have published this. It makes you look bad.
Re: Aerospike wins
Cameron Purdy | Oracle
Re: Interesting results
Re: Aerospike wins
But, Thumbtack is an *integrator* -- so they use all the products they evaluate for their customers. Notice they disclosed relationships with ALL the vendors. But their results did not state that ALL the vendors were the best. If they were truly biased they would have declared a three-way tie between Aerospike, Couchbase and MongoDB (10gen). In fact, they dinged Couchbase pretty hard in spite of the fact that they have a relationship with them!
So, I am not sure we should slam them for just being honest.
Keith Adams Dec 06, 2013