BT

InfoQ Homepage News Corona Improves Hadoop Scalability At Facebook

Corona Improves Hadoop Scalability At Facebook

Leia em português

Bookmarks

Facebook has open sourced Corona, an in-house developed improvement to Hadoop MapReduce scheduling software.

Corona separates the two critical tasks of Cluster management and Job Tracking. This is very similar in concept to Apache YARN which is also an improved version of MapReduce scheduler/resource manager. Facebook Engineering has published an article explaining Corona along with the background. They have also explained the reason for not reusing YARN –

It’s worth noting that we considered Apache YARN as a possible alternative to Corona. However, after investigating the use of YARN on top of our version of HDFS (a strong requirement due to our many petabytes of archived data) we found numerous incompatibilities that would be time-prohibitive and risky to fix. Also, it is unknown when YARN would be ready to work at Facebook-scale workloads.

One of the main differences in Facebook’s version of Hadoop is AvatarNode, that creates a hot-standby for every Node in the cluster. This creates a highly-available NameNode and even allows software upgrades to happen without downtime. This is critical for the company as it currently processes hundreds of petabytes of data in it's data warehouse, with half-a-petabyte of new data coming in every day. 

Corona can currently run MapReduce Jobs, but Facebook intends to use it for scheduling Jobs from other types of applications as well, such as Peregrine

Apache Hadoop is a set of projects that allows large-scale distributed computing. You can refer to several resources related to Hadoop on InfoQ

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
Country/Zone:
State/Province/Region:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.