BT

Your opinion matters! Please fill in the InfoQ Survey!

Facebook Chat Architecture

| by Gavin Terrill Follow 0 Followers on May 16, 2008. Estimated reading time: 2 minutes |

On the Facebook engineering blog, Software Engineer Eugene Letuchy recently posted details of the engineering decisions behind Facebook Chat:

when your feature's userbase will go from 0 to 70 million practically overnight, scalability has to be baked in from the start

Eugene identified a number of challenges for that size user base, starting with presence notification:

The naive implementation of sending a notification to all friends whenever a user comes online or goes offline has a worst case cost of O(average friendlist size * peak users * churn rate) messages/second, where churn rate is the frequency with which users come online and go offline, in events/second. This is wildly inefficient to the point of being untenable, given that the average number of friends per user is measured in the hundreds, and the number of concurrent users during peak site usage is on the order of several millions.

Another challenge was delivering messages in real time. Facebook choose a technique whereby the client pulls updates from the server, similar to Comet's XHR Long Polling Process:

The method we chose to get text from one user to another involves loading an iframe on each Facebook page, and having that iframe's Javascript make an HTTP GET request over a persistent connection that doesn't return until the server has data for the client. 

Eugene goes on to mention that "Having a large-number of long-running concurrent requests makes the Apache part of the standard LAMP stack a dubious implementation choice".

Facebook choose a combination of C++ and Erlang to implement clustered and partitioned subsystems. The C++ module is used to log chat messages, while Erlang "holds online users' conversations in-memory and serves the long-polled HTTP requests". epoll, a new system call introduced in Linux 2.6, was used to drive the Erlang module. Eugene states why the decision was made to go with Erlang:

In short, because the problem domain fits Erlang like a glove. Erlang is a functional concurrency-oriented language with extremely low-weight user-space "processes", share-nothing message-passing semantics, built-in distribution, and a "crash and recover" philosophy proven by two decades of deployment on large soft-realtime production systems.

Thrift, the open source framework (released by Facebook on April fool's day last year) for "scalable cross-language services development", was used to tie together the various technologies used in Facebook Chat, and now features bindings for Erlang.

An interesting approach was used to roll out the service - the so called "dark launch":

The secret for going from zero to seventy million users overnight is to avoid doing it all in one fell swoop. We chose to simulate the impact of many real users hitting many machines by means of a "dark launch" period in which Facebook pages would make connections to the chat servers, query for presence information and simulate message sends without a single UI element drawn on the page.

The choice of Erlang by the Facebook engineers is a significant endorsement for the language. Yariv Sadan, long time Erlang evangelist, notes:

This announcement should remove any doubts that Erlang is *the* platform for building scalable realtime (aka Comet) applications.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT