BT

Intel Open-Sources BigDL, Distributed Deep Learning Library for Apache Spark

| by Alexandre Rodrigues Follow 0 Followers on Jan 13, 2017. Estimated reading time: 1 minute |

Intel has open-sourced BigDL, a distributed deep learning library that runs on Apache Spark. It leverages existing Spark clusters to run deep learning computations and simplifies the data loading from big datasets stored in Hadoop.

Tests show a significant speedup performance running on Xeon servers compared to other open source frameworks Caffe, Torch or TensorFlow. The speed is comparable with a mainstream GPU and BigDL is able to scale to tens of Xeon servers.

The BigDL library supports Spark versions 1.5, 1.6 and 2.0 and allows for deep learning to be embedded in existing Spark based programs. It contains methods to convert Spark RDDs to a BigDL DataSet and can be used directly with Spark ML Pipelines.

For model training, BigDL applies a synchronous mini-batch SGD (Stochastic Gradient Descent) executed in a single Spark task across multiple executors. Each executor runs a multi-threaded engine and processes a part of the micro-batch data. In the current version, all the training and validation data is loaded into memory.

BigDL is implemented in Scala and is modeled after Torch. Like Torch, it provides a Tensor class, that uses Intel MKL library for computations. Intel MKL, short for Math Kernel Library, consists of a library with a set of routines optimized for calculations, ranging from FFT (Fast Fourier Transform) to matrix multiplications, that are heavily used for deep learning model training. Other concepts borrowed from Torch are Module, inspired on Torch’s nn package, that represents individual neural network layers, Table and Criterion.

BigDL provides an AWS EC2 image and examples for text classification using convolutional neural networks, image classification and how to load models pre-trained in Torch or Caffe into Spark for predictions computation. The main community requests are Python support and MKL-DNN, deep learning extensions for MKL.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT