BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News Challenges Building Facebook Live Streams

Challenges Building Facebook Live Streams

Bookmarks

Facebook Live started in a hackathon two years ago, and was launched to users eight months later. One of the challenges has been dealing with the unpredictable number of viewers of a single stream which can vary extensively, Sachin Kulkarni noted in his presentation at the recent QCon London conference, where he described the architecture and design challenges when building Facebook live streams.

From a high-level view of the infrastructure, a live stream starts when a client connects to the closest PoP (Point of Presence), which in turn will forward the connection to a full data centre where encoding is done. From the data centre, the stream is then forwarded to different PoPs and on to playback clients. Kulkarni, director of video infrastructure at Facebook, describes a PoP as responsible for caching and terminating incoming client connections and passing these on to a data centre over Facebook’s own network, which is more reliable and reduces the roundtrip time.

Looking at scaling challenges, Kulkarni notes that both the number of concurrent unique streams and number of viewers of all streams together are quite predictable and therefore pose a limited challenge. The real problem lies in the number of viewers of a single stream, which can be anything from just a few up to a very high number. Because it is unpredictable, you can’t plan for it. Caching and stream distribution are two ways they have solved this problem.

Comparing live streaming with normal videos, Kulkarni notes some other challenges:

  • Populating a cache ahead of time is not possible since content is created in real time, preventing any form of precaching.
  • Planning for live events and scaling resources ahead of time is problematic.
  • Predicting concurrent stream/viewer spikes caused by world events is difficult.

One major reliability challenge with live streaming is network problems. To deal with those, Kulkarni proposes three solutions:

  • Adapting to a lower bandwidth by using an adaptive bitrate, thus lowering the video quality, is typically used on the playback side, but they are also using it on the ingestion side.
  • Temporary connectivity loss is handled by temporary buffering on the client.
  • In worst case scenarios when the bandwidth is not high enough they can switch to audio only broadcasts or playbacks, noting that hearing what people say is more important than seeing them.

Lessons Kulkarni has learned from the project include:

  • Large services can grow from small beginnings. It’s better to write some code instead of forever discussing the architecture.
  • Reliability and scalability must be built into the design, including designing for planned and unplanned outages.
  • Make compromises to enable shipping of large projects.
  • Keep the architecture flexible for future features. This will allow a team to move faster than the infrastructure is able to change.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Too generic

    by Arindam Mukherjee,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Umm. What's the point of this article? What's the key takeaway. Seemed a little incoherent. Rehash of some traditional wisdom.

  • Re: Too generic

    by Jan Stenberg,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    It's a summary of Sachin Kulkarni's talk at QCon London and for me the key takeaway is that it was possible to build a service like Facebook Live during quite a short period and be able to handle the load from the high number of users that Facebook have. If you’re interested, the presentation will be available here on InfoQ but I don’t know when.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT