Facebook rolled out Private Chef, Opscode's commercial infrastructure automation product, to manage its web-tier. To ensure that Chef meets Facebook's scalability requirements they helped design the latest version of Chef Server, which is a total re-write using Erlang.
Facebook claims they are happy with using Chef so far and will present details at the ChefConf in late April. Phil Dibowitz, production engineer at Facebook, told The Register that Chef was the only automation solution flexible enough to bend to their scale dynamics without requiring them to change their workflow.
Chef seems to gain traction in large scale deployments:
- Dreamhost, a big web hosting company, is using Private Chef across three data centers.
- Cycle Computing has spun up over 10,000 Amazon AWS nodes using a single Chef 11 server to identify potential leads agains a cancer target for a big pharma company.
- edmunds.com has completely automated its Hadoop infrastructure consisting of 2500 hosts. For edmunds.com this was the first highly visible success story for introducing infrastructure automation.
Luke Kanies from Puppet Labs, makers of Puppet and competing with Opscode in the infrastructure automation area, says:
It's impressive that Opscode has gotten to this scale, but we've been at this scale for the last three or four years
He names Zynga with 50.000 servers and an upcoming deployment at the CERN nuclear physics research lab in Switzerland with up to 300,000 servers to prove his point.
Community comments
Edmunds.com, Chef, and Hadoop
by John Martin,
Edmunds.com, Chef, and Hadoop
by John Martin,
Your message is awaiting moderation. Thank you for participating in the discussion.
Matthias - First of all, thank you for the link to our webinar with Opscode about Edmunds' use of Chef for building/managing our Hadoop infrastructure. It's been a wonderful experience for us at Edmunds. As you've pointed out, that particular project was the first visible success story for Chef. We're very happy to be involved in the Los Angeles Chef User community and recently presented on our homegrown application deployment tool - Kingpin - which is built entirely around Chef. We're continuing our adoption and really excited to talk with the community at large about our adventures with such an awesome tool. I'd expect to hear much more from us in the near future about how we're taking advantage of it as our CM of choice.
One slight correction in your bullet though I'd like to make. We are indeed managing around 2,500 hosts with Chef, but not all of those are for our Hadoop clusters. We only have 70-80 hosts in our Hadoop infrastructure managed with Chef at the moment. (Just a slight correction, but perhaps an important detail.) I don't see us getting into the 2,500 Hadoop cluster size for some time. ;)
Thanks again!
John Martin
Sr Director, Production Engineering
Edmunds.com