Twitter's Manhattan: A Real-time, Multi-tenant Distributed Database

by Michael Hausenblas on May 15, 2014 |

Twitter Engineering has released details about Manhattan, its real-time, multi-tenant distributed database. 

As for the motivation, Peter Schuller, Manhattan's lead engineer, puts it this way:

Over the last few years, we found ourselves in need of a storage system that could serve millions of queries per second, with extremely low latency in a real-time environment. Availability and speed of the system became the utmost important factor.

Manhattan, reportedly being in production for over two years, exposes a key/value interface over storage as a service as well as an upcoming graph-based interface to interact with edges. On top of Manhattan, the Twitter Engineering team has created additional services, such as a batch Hadoop importing or the time series counters service.

The system is design to handle 6,000 Tweets a second and more from 240 million users worldwide as Cade Metz (Editor Wired Business and Wired Enterprise) reports, as well as emphasising Twitter's need for a strongly consistent database:

Like many web outfits, Twitter once relied heavily on Cassandra. Originally developed by Facebook, Cassandra is one of many NoSQL databases designed to store data across hundreds or even thousands of machines. This worked well enough, but Twitter soon found it was too difficult to expand the system to new sets of machines.

Derrick Harris of Gigaom spoke with the trio of Twitter engineers (Chris Goffinet, Peter Schuller and Boaz Avital) who built Twitter’s Manhattan, extracting three lessons in database design, also speaking to the imminent question if Manhattan will be made available to a wider audience, beyond Twitter's internal user base:

If and when Twitter decides to open source Manhattan, Schuller added, the aim would be to make it as useful as possible to a broad range of users whose applications might look nothing like those running inside Twitter.

It will be interesting to see what Twitter plans to do with Manhattan, given its strong Open Source commitment. For the immediate future, the team has announced a white paper outlining more technical details on Manhattan.

Rate this Article


Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

General Feedback
Marketing and all content copyright © 2006-2016 C4Media Inc. hosted at Contegix, the best ISP we've ever worked with.
Privacy policy

We notice you're using an ad blocker

We understand why you use ad blockers. However to keep InfoQ free we need your support. InfoQ will not provide your data to third parties without individual opt-in consent. We only work with advertisers relevant to our readers. Please consider whitelisting us.