Google announced last week the release of open source MapReduce framework for C, called MR4C, that allows developers to run native code in Hadoop framework.
MR4C framework brings together the performance and flexibility of natively developed algorithms with the scalability and throughput provided by Hadoop execution framework. The goal of the project is to abstract the details of MapReduce framework and allow users to focus on developing custom algorithms.
The framework was originally developed by Skybox team for satellite image processing and geospatial data science use cases. The team wanted to leverage the image processing libraries developed in C and C++ along with job tracking and cluster management capabilities of Hadoop well-suited for scalable data handling.
In MR4C, algorithms are stored in native shared objects that access data from a local file or an uniform resource identifier (URI). And the input/output datasets, runtime parameters, and any external libraries are configured using JavaScript Object Notation (JSON) files. Splitting mappers and allocating resources can be configured with Apache YARN based tools (for Hadoop v2) or at the cluster level for MapReduce Version 1 (MRv1). Workflows of multiple algorithms can be connected together using an automatically generated configuration. The framework also supports callbacks for logging and progress reporting which can be viewed using the Hadoop JobTracker interface. The workflows can be tested on a local machine using the same interface employed on the target Hadoop cluster.
For more details of the framework, check out the documentation and source code at the MR4C GitHub website. If you are interested in contributing to the project, the team has created a web page to help the project contributors.