BT

New Early adopter or innovator? InfoQ has been working on some new features for you. Learn more

Corona Improves Hadoop Scalability At Facebook

| by Roopesh Shenoy Follow 0 Followers on Nov 15, 2012. Estimated reading time: 1 minute |

Facebook has open sourced Corona, an in-house developed improvement to Hadoop MapReduce scheduling software.

Corona separates the two critical tasks of Cluster management and Job Tracking. This is very similar in concept to Apache YARN which is also an improved version of MapReduce scheduler/resource manager. Facebook Engineering has published an article explaining Corona along with the background. They have also explained the reason for not reusing YARN –

It’s worth noting that we considered Apache YARN as a possible alternative to Corona. However, after investigating the use of YARN on top of our version of HDFS (a strong requirement due to our many petabytes of archived data) we found numerous incompatibilities that would be time-prohibitive and risky to fix. Also, it is unknown when YARN would be ready to work at Facebook-scale workloads.

One of the main differences in Facebook’s version of Hadoop is AvatarNode, that creates a hot-standby for every Node in the cluster. This creates a highly-available NameNode and even allows software upgrades to happen without downtime. This is critical for the company as it currently processes hundreds of petabytes of data in it's data warehouse, with half-a-petabyte of new data coming in every day. 

Corona can currently run MapReduce Jobs, but Facebook intends to use it for scheduling Jobs from other types of applications as well, such as Peregrine

Apache Hadoop is a set of projects that allows large-scale distributed computing. You can refer to several resources related to Hadoop on InfoQ

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT