BT

Your opinion matters! Please fill in the InfoQ Survey!

Twitter's Manhattan: A Real-time, Multi-tenant Distributed Database

| by Michael Hausenblas Follow 0 Followers on May 15, 2014. Estimated reading time: 1 minute |

Twitter Engineering has released details about Manhattan, its real-time, multi-tenant distributed database. 

As for the motivation, Peter Schuller, Manhattan's lead engineer, puts it this way:

Over the last few years, we found ourselves in need of a storage system that could serve millions of queries per second, with extremely low latency in a real-time environment. Availability and speed of the system became the utmost important factor.

Manhattan, reportedly being in production for over two years, exposes a key/value interface over storage as a service as well as an upcoming graph-based interface to interact with edges. On top of Manhattan, the Twitter Engineering team has created additional services, such as a batch Hadoop importing or the time series counters service.

The system is design to handle 6,000 Tweets a second and more from 240 million users worldwide as Cade Metz (Editor Wired Business and Wired Enterprise) reports, as well as emphasising Twitter's need for a strongly consistent database:

Like many web outfits, Twitter once relied heavily on Cassandra. Originally developed by Facebook, Cassandra is one of many NoSQL databases designed to store data across hundreds or even thousands of machines. This worked well enough, but Twitter soon found it was too difficult to expand the system to new sets of machines.

Derrick Harris of Gigaom spoke with the trio of Twitter engineers (Chris Goffinet, Peter Schuller and Boaz Avital) who built Twitter’s Manhattan, extracting three lessons in database design, also speaking to the imminent question if Manhattan will be made available to a wider audience, beyond Twitter's internal user base:

If and when Twitter decides to open source Manhattan, Schuller added, the aim would be to make it as useful as possible to a broad range of users whose applications might look nothing like those running inside Twitter.

It will be interesting to see what Twitter plans to do with Manhattan, given its strong Open Source commitment. For the immediate future, the team has announced a white paper outlining more technical details on Manhattan.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT