Facebook Introduces Chef To Manage Its Web-Tier
Facebook rolled out Private Chef, Opscode's commercial infrastructure automation product, to manage its web-tier. To ensure that Chef meets Facebook's scalability requirements they helped design the latest version of Chef Server, which is a total re-write using Erlang.
Facebook claims they are happy with using Chef so far and will present details at the ChefConf in late April. Phil Dibowitz, production engineer at Facebook, told The Register that Chef was the only automation solution flexible enough to bend to their scale dynamics without requiring them to change their workflow.
Chef seems to gain traction in large scale deployments:
- Dreamhost, a big web hosting company, is using Private Chef across three data centers.
- Cycle Computing has spun up over 10,000 Amazon AWS nodes using a single Chef 11 server to identify potential leads agains a cancer target for a big pharma company.
- edmunds.com has completely automated its Hadoop infrastructure consisting of 2500 hosts. For edmunds.com this was the first highly visible success story for introducing infrastructure automation.
It's impressive that Opscode has gotten to this scale, but we've been at this scale for the last three or four years
He names Zynga with 50.000 servers and an upcoming deployment at the CERN nuclear physics research lab in Switzerland with up to 300,000 servers to prove his point.
Edmunds.com, Chef, and Hadoop
One slight correction in your bullet though I'd like to make. We are indeed managing around 2,500 hosts with Chef, but not all of those are for our Hadoop clusters. We only have 70-80 hosts in our Hadoop infrastructure managed with Chef at the moment. (Just a slight correction, but perhaps an important detail.) I don't see us getting into the 2,500 Hadoop cluster size for some time. ;)
Sr Director, Production Engineering