BT

如何利用碎片时间提升技术认知与能力? 点击获取答案

MXNet实践

| 作者 侠天 关注 5 他的粉丝 发布于 2016年4月13日. 估计阅读时间: 29 分钟 | QCon上海2018 关注大数据平台技术选型、搭建、系统迁移和优化的经验。

当下如此多的深度学习框架,为什么要选择MXNet?

MXNet是个深度学习的框架,支持从单机到多GPU、多集群的计算能力。对于其它深度学习框架来说,使用者得考虑什么时候开始计算,数据怎样同步。而这些在MXNet中都是“懒”操作(类似于Spark中的计算),MXNet等到资源合适就自己计算。

MXNet特点:

  • 基于赋值表达式建立计算图,类似于Tensorflow、Teano、Torch和Caffe;
  • 支持内存管理,并对两个不交叉的变量重复使用同一内存空间;
  • MXNet核心使用C++实现,并提供C风格的头文件。支持Python、R、Julia、Go和Javascript;
  • 支持Torch;
  • 支持移动设备端发布

命令式编程 vs 符号式编程

MXNet尝试将两种编程模式无缝的结合起来。命令式API作为一个接口暴露出来使用,类似于Numpy;符号表达式API提供用户像Theano和Tensorflow一样定义计算图。

命令式编程

NDarray编程接口,类似于Numpy.ndarray/Torch.tensor,主要是因为其提供张量运算(或称之为多维数组);与Numpy不同的地方在于NDarray可以把对象的指针指向内存(CPU或GPU),它能自动的将需要执行的数据分配到多台gpu和cpu上进行计算,从而完成高速并行。

举个例子:在CPU和GPU上构建一个零张量。

import mxnet as mx
cpu_tensor = mx.nd.zeros((10,), ctx=mx.cpu())
gpu_tensor = mx.nd.zeros((10,), ctx=mx.gpu(0))

类似于Numpy,MXNet可以做基本的数学计算。
ctx = mx.cpu() # which context to put memory on.
a = mx.nd.ones((10, ), ctx=ctx)
b = mx.nd.ones((10, ), ctx=ctx)
c = (a + b) / 10.
d = b + 1

图1

与Numpy不同的地方在于,在数组上的操作都是惰性的,并且这些操作都会返回一个NDAarry实例(将来用于计算的)。MXNet真正厉害之处在于这些操作是如何计算的。一个计算计划生成后并不立即执行,而当此计算所依赖的数据都准备好才开始真正的计算。例如,计算D值时你需要知道B值而不需要C值,所以先计算B值。但计算C值和D值就无关顺序,并且可以利用并发计算。

在NDArray上读取值可以调用numpy_d = d.asnumpy()。这个调用是会触发d值的计算,然后转化成Numpy数组。

符号式编程

命令式编程有其优势,但其缺点在于计算都得提前知道计算完成的时间,并且得用户提前写好,而符号式编程试图解决这个问题。符号式程序的不同之处在于,当执行代码时,程序并没有产生真正的计算,而是生成了一张计算图/符号图(computation graph/symbolic graph)来描述整个计算过程。

举一个类似上面的例子:

import mxnet as mx
a = mx.sym.Variable("A") # represent a placeholder. These can be inputs, weights, or anything else.
b = mx.sym.Variable("B")
c = (a + b) / 10
d = c + 1

当定义好符号图后,可以检查计算图输入和输出:

d.list_arguments()
# ['A', 'B']

d.list_outputs()
# ['_plusscalar0_output'] This is the default name from adding to scalar.

计算图允许图形操作:

# define input shapes
inp_shapes = {'A':(10,), 'B':(10,)}
arg_shapes, out_shapes, aux_shapes = d.infer_shape(**inp_shapes)

arg_shapes # the shapes of all the inputs to the graph. Order matches d.list_arguments()
# [(10, ), (10, )]

out_shapes # the shapes of all outputs. Order matches d.list_outputs()
# [(10, )]

aux_shapes # the shapes of auxiliary variables. These are variables that are not
 trainable such as batch normalization population statistics. For now, they are save to ignore.
# []

无状态图

不同于其它框架的,MXNet图是完全无状态的,计算图代表着有实参和输出的函数。在MXNet中,权重或者模型参数和数据输入(数据灌入)之间没有区别,它们都是计算图的实参。例如,一个逻辑回归计算图有三个实参:输入数据,权重变量,误差。

为了触发实际的图计算,可以调用Executor,然后绑定符号表达式和指定的输入变量。

input_arguments = {}
input_arguments['A'] = mx.nd.ones((10, ), ctx=mx.cpu())
input_arguments['B'] = mx.nd.ones((10, ), ctx=mx.cpu())
executor = d.bind(ctx=mx.cpu(),
                  args=input_arguments, # this can be a list or a 
dictionary mapping names of inputs to NDArray
                  grad_req='null') # don't request gradients

Executor申请所有需要的内存和临时变量来进行计算。一旦绑定好,Executor会一直
同一段内存空间做计算输入和输出。在执行一个符号表达式前,需要对所有的自由变
量进行赋值,执行完forward后softmax的输出。
import numpy as np
# The executor
executor.arg_dict
# {'A': NDArray, 'B': NDArray}

executor['A'][:] = np.random.rand(10,) # Note the [:]. This sets the contents 
of the array instead of setting the array to a new value instead of overwriting the variable.
executor['B'][:] = np.random.rand(10,)
executor.forward()
executor.outputs
# [NDArray]
output_value = executor.outputs[0].asnumpy()

跟命令式编程类似,调用forward也是lazy的,在计算完成之前就回返回计算图。

计算流图比较重要的是它能根据相应的输入数据自动优化,这在backward函数中是自动进行的。Exector需要一定的空间来存放梯度输出,如下所示:

# allocate space for inputs
input_arguments = {}
input_arguments['A'] = mx.nd.ones((10, ), ctx=mx.cpu())
input_arguments['B'] = mx.nd.ones((10, ), ctx=mx.cpu())
# allocate space for gradients
grad_arguments = {}
grad_arguments['A'] = mx.nd.ones((10, ), ctx=mx.cpu())
grad_arguments['B'] = mx.nd.ones((10, ), ctx=mx.cpu())

executor = d.bind(ctx=mx.cpu(),
                  args=input_arguments, # this can be a list or a dictionary
 mapping names of inputs to NDArray
                  args_grad=grad_arguments, # this can be a list or a dictionary
 mapping names of inputs to NDArray
                  grad_req='write') # instead of null, tell the executor to write
 gradients. This replaces the contents of grad_arguments with the gradients computed.

executor['A'][:] = np.random.rand(10,)
executor['B'][:] = np.random.rand(10,)

executor.forward()
# in this particular example, the output symbol is not a scalar or loss symbol.
# Thus taking its gradient is not possible.
# What is commonly done instead is to feed in the gradient from a future computation.
# this is essentially how backpropagation works.
out_grad = mx.nd.ones((10,), ctx=mx.cpu())
executor.backward([out_grad]) # because the graph only has one output, only one output grad is needed.

executor.grad_arrays
# [NDarray, NDArray]

命令式和符号式混合编程

MXNet真正厉害之处在于可以结合命令式编程和符号式编程两种风格。例如,你可以用符号表达式来构件一个神经网络,给定的batch_size,绑定一个executor来计算,然后用命令式API来计算梯度。所有这些操作都是等实际的数据到达才开始计算。

现在来点更复杂的例子:一个简单、全连接神经网络,带有一层隐藏层的MINIST。

图2

import mxnet as mx
import numpy as np
# First, the symbol needs to be defined
data = mx.sym.Variable("data") # input features, mxnet commonly calls this 'data'
label = mx.sym.Variable("softmax_label")

# One can either manually specify all the inputs to ops (data, weight and bias)
w1 = mx.sym.Variable("weight1")
b1 = mx.sym.Variable("bias1")
l1 = mx.sym.FullyConnected(data=data, num_hidden=128, name="layer1", weight=w1, bias=b1)
a1 = mx.sym.Activation(data=l1, act_type="relu", name="act1")

# Or let MXNet automatically create the needed arguments to ops
l2 = mx.sym.FullyConnected(data=a1, num_hidden=10, name="layer2")

# Create some loss symbol
cost_classification = mx.sym.SoftmaxOutput(data=l2, label=label)

# Bind an executor of a given batch size to do forward pass and get gradients
batch_size = 128
input_shapes = {"data": (batch_size, 28*28), "softmax_label": (batch_size, )}
executor = cost_classification.simple_bind(ctx=mx.gpu(0),
                                           grad_req='write',
                                           **input_shapes)
# The above executor computes gradients. When evaluating test data we don't need this.
# We want this executor to share weights with the above one, so we will use bind
# (instead of simple_bind) and use the other executor's arguments.
executor_test = cost_classification.bind(ctx=mx.gpu(0),
                                         grad_req='null',
                                         args=executor.arg_arrays)

# initialize the weights
for r in executor.arg_arrays:
    r[:] = np.random.randn(*r.shape)*0.02

# Using skdata to get mnist data. This is for portability. Can sub in any data loading you like.
from skdata.mnist.views import OfficialVectorClassification

data = OfficialVectorClassification()
trIdx = data.sel_idxs[:]
teIdx = data.val_idxs[:]
for epoch in range(10):
  print "Starting epoch", epoch
  np.random.shuffle(trIdx)

  for x in range(0, len(trIdx), batch_size):
    # extract a batch from mnist
    batchX = data.all_vectors[trIdx[x:x+batch_size]]
    batchY = data.all_labels[trIdx[x:x+batch_size]]

    # our executor was bound to 128 size. Throw out non matching batches.
    if batchX.shape[0] != batch_size:
        continue
    # Store batch in executor 'data'
    executor.arg_dict['data'][:] = batchX / 255.
    # Store label's in 'softmax_label'
    executor.arg_dict['softmax_label'][:] = batchY
    executor.forward()
    executor.backward()

    # do weight updates in imperative
    for pname, W, G in zip(cost_classification.list_arguments(), executor.arg_arrays, executor.grad_arrays):
        # Don't update inputs
        # MXNet makes no distinction between weights and data.
        if pname in ['data', 'softmax_label']:
            continue
        # what ever fancy update to modify the parameters
        W[:] = W - G * .001

  # Evaluation at each epoch
  num_correct = 0
  num_total = 0
  for x in range(0, len(teIdx), batch_size):
    batchX = data.all_vectors[teIdx[x:x+batch_size]]
    batchY = data.all_labels[teIdx[x:x+batch_size]]
    if batchX.shape[0] != batch_size:
        continue
    # use the test executor as we don't care about gradients
    executor_test.arg_dict['data'][:] = batchX / 255.
    executor_test.forward()
    num_correct += sum(batchY == np.argmax(executor_test.outputs[0].asnumpy(), axis=1))
    num_total += len(batchY)
  print "Accuracy thus far", num_correct / float(num_total)

MXNet提供大量的higher level APIs封装,减少直接使用forward模型的复杂性。例如,使用FeedForward类构建同样的模型,但是需要编写的代码量减小,编写代码也变的简单。

import mxnet as mx
import numpy as np
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__) # get a logger to accuracies are printed

data = mx.sym.Variable("data") # input features, when using FeedForward this must be called data
label = mx.sym.Variable("softmax_label") # use this name aswell when using FeedForward

# When using Forward its best to have mxnet create its own variables.
# The names of them are used for initializations.
l1 = mx.sym.FullyConnected(data=data, num_hidden=128, name="layer1")
a1 = mx.sym.Activation(data=l1, act_type="relu", name="act1")
l2 = mx.sym.FullyConnected(data=a1, num_hidden=10, name="layer2")

cost_classification = mx.sym.SoftmaxOutput(data=l2, label=label)

from skdata.mnist.views import OfficialVectorClassification

data = OfficialVectorClassification()
trIdx = data.sel_idxs[:]
teIdx = data.val_idxs[:]

model = mx.model.FeedForward(symbol=cost_classification,
                             num_epoch=10,
                             ctx=mx.gpu(0),
                             learning_rate=0.001)

model.fit(X=data.all_vectors[trIdx],
          y=data.all_labels[trIdx],
          eval_data=(data.all_vectors[teIdx], data.all_labels[teIdx]),
          eval_metric="acc",
          logger=logger)

结论

希望这个教程能给大家一个很好的MXNet使用介绍。一般情况,和大部分其它框架一样,MXNet需要在编写代码的复杂度和完成新技术的灵活性之间找个平衡。

这里只是MXNet的一些皮毛,想更深入的了解请看文档


感谢杜小芳对本文的审校。

给InfoQ中文站投稿或者参与内容翻译工作,请邮件至editors@cn.infoq.com。也欢迎大家通过新浪微博(@InfoQ@丁晓昀),微信(微信号:InfoQChina)关注我们。

评价本文

专业度
风格

您好,朋友!

您需要 注册一个InfoQ账号 或者 才能进行评论。在您完成注册后还需要进行一些设置。

获得来自InfoQ的更多体验。

告诉我们您的想法

允许的HTML标签: a,b,br,blockquote,i,li,pre,u,ul,p

当有人回复此评论时请E-mail通知我

翻译 by song wei

是不是应该给出原地址比较好

允许的HTML标签: a,b,br,blockquote,i,li,pre,u,ul,p

当有人回复此评论时请E-mail通知我

允许的HTML标签: a,b,br,blockquote,i,li,pre,u,ul,p

当有人回复此评论时请E-mail通知我

1 讨论

登陆InfoQ,与你最关心的话题互动。


找回密码....

Follow

关注你最喜爱的话题和作者

快速浏览网站内你所感兴趣话题的精选内容。

Like

内容自由定制

选择想要阅读的主题和喜爱的作者定制自己的新闻源。

Notifications

获取更新

设置通知机制以获取内容更新对您而言是否重要

BT