Twitter Engineering has released details about Manhattan, its real-time, multi-tenant distributed database.
As for the motivation, Peter Schuller, Manhattan's lead engineer, puts it this way:
Over the last few years, we found ourselves in need of a storage system that could serve millions of queries per second, with extremely low latency in a real-time environment. Availability and speed of the system became the utmost important factor.
Manhattan, reportedly being in production for over two years, exposes a key/value interface over storage as a service as well as an upcoming graph-based interface to interact with edges. On top of Manhattan, the Twitter Engineering team has created additional services, such as a batch Hadoop importing or the time series counters service.
The system is design to handle 6,000 Tweets a second and more from 240 million users worldwide as Cade Metz (Editor Wired Business and Wired Enterprise) reports, as well as emphasising Twitter's need for a strongly consistent database:
Like many web outfits, Twitter once relied heavily on Cassandra. Originally developed by Facebook, Cassandra is one of many NoSQL databases designed to store data across hundreds or even thousands of machines. This worked well enough, but Twitter soon found it was too difficult to expand the system to new sets of machines.
Derrick Harris of Gigaom spoke with the trio of Twitter engineers (Chris Goffinet, Peter Schuller and Boaz Avital) who built Twitter’s Manhattan, extracting three lessons in database design, also speaking to the imminent question if Manhattan will be made available to a wider audience, beyond Twitter's internal user base:
If and when Twitter decides to open source Manhattan, Schuller added, the aim would be to make it as useful as possible to a broad range of users whose applications might look nothing like those running inside Twitter.
It will be interesting to see what Twitter plans to do with Manhattan, given its strong Open Source commitment. For the immediate future, the team has announced a white paper outlining more technical details on Manhattan.