Akid: A Library for Neural Network Research and Production from a Dataism Approach

1.7 MB·

by  Shuai Li

Akid: A Library for Neural Network Research and
Production From A Dataism Approach
Shuai Li
ChineseUniversityofHongKong
[email protected]
ABSTRACT
7 Neuralnetworksarearevolutionarybutimmaturetechnique
1
that is fast evolving and heavily relies on data. To benefit
0
from the newest development and newly available data, we
2
want the gap between research and production as small as

n possibly. On the other hand, differing from traditional ma-
a chinelearningmodels,neuralnetworkisnotjustyetanother
J statistic model, but a model for the natural processing en-

3 gine—thebrain. Inthiswork,wedescribeaneuralnetwork
library named akid. It provides higher level of abstraction

] for entities (abstracted as blocks) in nature upon the ab-
G
stractiondoneonsignals(abstractedastensors)byTensor-
L flow,characterizingthedataismobservationthatallentities
. in nature processes input and emit out in some ways. It
s
includesafullstackofsoftwarethatprovidesabstractionto
c Figure 1: Illustration of stack abstraction of akid.
[ letresearchersfocusonresearchinsteadofimplementation,
while at the same time the developed program can also be

1 putintoproductionseamlesslyinadistributedenvironment,
v and be production ready. At the top application stack, it [9] [14], it is still rather an immature technique that is fast
9 provides out-of-box tools for neural network applications. moving and in short of understanding [11]. Temporarily its
0 true value lies in the capability to solve perception related
Lower down, akid provides a programming paradigm that
6 data analytic problems in industry, e.g. self-driving cars,
lets user easily build customized models. The distributed
0 detection of lung cancer etc. On the other hand, Neural
computing stack handles the concurrency and communica-
0 Networkisatechniquethatheavilyreliesonalargevolume
tion, thus letting models be trained or deployed to a single
. ofdata. Itiscriticalforbusinessesthatusesuchatechnique
1 GPU, multiple GPUs, or a distributed environment with-
toleverageonnewlyavailabledataassoonaspossible,which
0 out affecting how a model is specified in the programming
helpsformapositivefeedbackloopthatreinforcesthequal-
7 paradigm stack. Lastly, the distributed deployment stack
ity of service.
1 handles how the distributed computing is deployed, thus
Accordingly, to benefit from the newest development and
: decouplingtheresearchprototypeenvironmentwiththeac-
v newlyavailabledata,wewantthegapbetweenresearchand
tualproductionenvironment,andisabletodynamicallyal-
i productionassmallaspossible. Inthispackage,weexplore
X locatecomputingresources,sodevelopment(Devs)andop-
technology stack abstraction that enable fast research pro-
r erations(Ops)couldbeseparated. Ithasbeenopensource,
totyping and are production ready.
a andpleaserefertohttp://akid.readthedocs.io/en/latest/for
akidtriestoprovideafullstackofsoftwarethatprovides
documentation.
abstractiontoletresearchersfocusonresearchinsteadofim-
plementation,whileatthesametimethedevelopedprogram
Keywords can also be put into production seamlessly in a distributed
environment, and be production ready.
neural network; library; block; distributed computing
At the top application stack, it provides out-of-box tools
for neural network applications. Lower down, akid pro-
1. INTRODUCTION
videsprogrammingparadigmthatletsuserseasilybuildcus-
Neuralnetwork,whichisacornerstonetechniqueofapool tomized models, which is the major intellectual innovation
of techniques under the name of Deep Learning nowadays, of akid that provides higher level of abstraction for entities
seems to have the potential to lead to another technology in nature (abstracted as blocks) upon the abstraction done
revolution. Ithasincurredwideenthusiasminindustry,and on signals (abstracted as tensors) by Tensorflow. The dis-
seriousconsiderationinpublicsectorandimpactevaluation tributedcomputingstackhandlestheconcurrencyandcom-
ingovernment. However,thoughbeingaremarkablebreak- munication,thuslettingmodelsbetrainedordeployedtoa
through in high dimensional perception problems academi- single GPU, multiple GPUs, or a distributed environment
callyandintellectuallystimulatingandpromising[8][13][5] without affecting how a model is specified in the program-
ming paradigm stack. Lastly, the distributed deployment learning has reached wide popularity, but is mostly a quick
stack handles how the distributed computing is deployed, solutiontodoresearchinneuralnetworkswhentheintegra-
thus decoupling the research prototype environment with tion with community and general purpose production pro-
the actual production environment, and is able to dynam- grammingarenotpressing. CaffeiswritteninC++,whose
ically allocate computing resources, so development (Devs) friendly front-end, aka the text network configuration file,
andoperations(Ops)couldbeseparated. Anillustrationof loses its affinity when the model goes more than dozens of
the four stack is shown in Figure 1. layer.
Fromafeaturepointofviewasalibrary,itaimstoenable DeepLearning4J is an industrial solution to neural net-
fast prototyping and production ready at the same time by workswritteninJavaandScala,andistooheavyweightfor
offering the following features: research prototyping.
Perhaps the most similar package existing with akid is
• supports fast prototyping
Keras,whichbothaimtoprovideamoreintuitiveinterface
– built-in data pipeline framework that standard- torelativelylow-levellibrary,i.e. Tensorflow. akidisdiffer-
izes data preparation and data augmentation. ent from Keras in at least two fundamental aspects. First,
akidmimicshowsignalspropagateinnaturebyabstracting
– arbitrary connectivity schemes (including multi-
everything as a semantic block, which holds many states,
input and multi-output training), and easy re-
thus is able to provide a wide range of functionalities in a
trieval of parameters and data in the network
easily customizable way, while Keras uses a functional API
– meta-syntaxtogenerateneuralnetworkstructure
that directly manipulates tensors, which is a lower level of
before training
abstraction, e.g. it has to do class attributes traverse to re-
– support for visualization of computation graph, trievelayerweightswithafixedvariablenamewhileinakid
weightfilters,featuremaps,andtrainingdynam- variableareretrievedbynames. Second,Kerasmostlyonly
ics statistics. provides an abstraction to build a neural network topology,
which is roughly the programming paradigm stack of akid,
• be production ready
while akid provides unified abstraction that includes appli-
– built-in support for distributed computing cation stack, programming stack, and distributed comput-
ing stack. A noticeable improvement is that Keras needs
– compatibility to orchestrate with distributed file
the user to handle communication and concurrency, while
systems,dockercontainers,anddistributedoper-
the distributed computing stack of akid hides them.
ating systems such as Kubernetes.
The name comes from the Kid saved by Neo in Matrix, 3. AKIDSTACK
and the metaphor to build a learning agent, which we call
Now we go technical to discuss each stack provided by
kid in human culture.
akid. Themajornovelintellectualdesignof akidisthepro-
TherestofthepaperdiscussesrelatedworksinSection2,
grammingparadigmthatprovidesahigherlevelabstraction
each stack of akid in detail in Section 3.
upon signal/tensor. We introduce it first, then we discuss
the application stack, and distributed computing and de-
2. RELATEDWORKS
ployment stack.
akid differs from existing packages from the perspective
3.1 ProgrammingParadigm
that it does not aim to be yet another wrapper for another
machine learning model. Subtle it seems. The fundamen-
3.1.1 Itisallaboutsignalprocessingblocks
tal difference lies in the design. It aims to reproduce how
signal propagates in nature by introducing Block. If Tensor
in Tensorflow can be viewed as the abstraction for signals
in nature, Block can be viewed as the abstraction for en-
tities in nature, which all process inputs in some way, and
emit output. It also aims to integrate technology stacks to
solve both research prototyping and industrial production
byclearlydefiningthebehaviorforeachstack. Wecompare
akid with existing packages in the following briefly. Note
that since Tensorflow is used as the computation backend,
we do not discuss speed here, which is not our concern for
akid.
Theano [12], Torch [3], Caffe [7], MXNet [2] are packages
that aim to provide a friendly front end to complex com-
putation back-end that are written in C++. Theano is a
pythonfrontendtoacomputationalgraphcompiler,which
has been largely superseded by Tensorflow in the compila- Figure 2: Illustration of the arbitrary connectivity
tion speed, flexibility, portability etc, while akid is built on supported by akid. Forward connection, branch-
of Tensorflow. MXNet is a competitive competitor to Ten- ing and mergine, and feedback connection are sup-
sorflow. Torchissimilarwiththeano,butwiththefront-end ported.
languagetobeLua,thechoiceofwhichismostlymotivated
from the fact that it is much easier to interface with C us- akid builds another layer of abstraction on top of Ten-
ing Lua than Python. It has been widely used before deep sor: Block. Tensor can be taken as the media/formalism
signal propagates in digital world, while Block is the data class OneLayerBrain(Brain):
processing entity that processes inputs and emits outputs. def __init__(self, **kwargs):
Itcoincideswithabranchof“ideology”calleddataismthat super(OneLayerBrain, self).__init__(**kwargs)
takes everything in this world is a data processing entity. self.attach(
An interesting one that may come from A Brief History of ConvolutionLayer(ksize=[5, 5],
Tomorrow by Yuval Noah Harari. strides=[1, 1, 1, 1],
Best designs mimic nature. akid tries to reproduce how padding="SAME",
signals in nature propagates. Information flow can be ab- out_channel_num=32,
stractedasdatapropagatingthroughinter-connectedblocks, name="conv1")
each of which processes inputs and emits outputs. For ex- )
ample, a vision classification system is a block that takes self.attach(ReLULayer(name="relu1"))
image inputs and gives classification results. Everything is self.attach(
a Block in akid. PoolingLayer(ksize=[1, 5, 5, 1],
A block could be as simple as a convonlutional neural strides=[1, 5, 5, 1],
network layer that merely does convolution on the input padding="SAME",
data and outputs the results; it also be as complex as an name="pool1")
acyclic graph that inter-connects blocks to build a neural )
network,orsequentiallylinkedblocksystemthatdoesdata
augmentation. self.attach(InnerProductLayer(
Compared with pure symbol computation approach, like out_channel_num=10, name="ip1"))
the one in tensorflow, a block is able to contain states as- self.attach(SoftmaxWithLossLayer(
sociated with this processing unit. Signals are passed be- class_num=10,
tween blocks in form of tensors or list of tensors. Many inputs=[
heavyliftinghasbeendoneintheblock(Blockanditssub- {"name": "ip1", "idxs": [0]},
classes), e.g. pre-condition setup, name scope maintenance, {"name": "system_in", "idxs": [1]}],
copy functionality for validation and copy functionality for name="loss"))
distributed replicas, setting up and gathering visualization
Itassemblesaconvolutionlayer,aReLULayer,apooling
summaries, centralization of variable allocation, attaching
layer, an inner product layer and a loss layer. To attach a
debugging ops now and then etc.
block(layer)thatdirectlytakestheoutputsoftheprevious
akidoffersvariouskindsofblocksthatareabletoconnect
attached layer as inputs, just directly attach the block. If
tootherblocksinanarbitraryway,asillustratedinFigure2.
inputs exists, the brain will fetch corresponding tensors by
It is also easy to build one’s own blocks. The Kid class is
name of the block attached and indices of the outputs of
essentially an assembler that assemblies blocks provided by
that layer. See the loss layer above for an example. Note
akidtomainlyfulfillthetasktotrainneuralnetworks. Here
thateventhoughtherearemultipleinputsforthebrain,the
we show how to build an arbitrary acyclic graph of blocks
first attached layer of the brain will take the first of these
using class Brain, to illustrate how to use blocks in akid.
input by default, given the convention that the first tensor
Abrainisthedataprocessingenginetoprocessdatasup-
is the data, and the remaining tensors are normally labels,
plied by Sensor to fulfill certain tasks. More specifically,
which is not used till very late.
• it builds up blocks to form an arbitrary network Asanexampletobuildmorecomplexconnectivityscheme,
residualunitscanbebuiltusingBrainasshowninFigure3.
• offers sub-graphs for inference, loss, evaluation, sum-
maries
3.1.2 Self-modifyingbrains—parametertuning
• provides access to all data and parameters within
akid offers automatic parameter tuning through defining
Touseabrain,dataasalistshouldbefedin,ashowitis
template using tune function. A function tune that takes
doneinwithanyotherblock. Somepre-specifiedbrainsare
a Brain jinja2 template class and a parameters to fill the
availableunderakid.models.brains. Anexamplecouldbe:
template in runtime.
# ... first get a feed sensor The tune function would use all available GPUs to train
sensor.setup() networks with all given different set of parameters. If avail-
brain = OneLayerBrain(name="brain") ableGPUsarenotenough,theonesthatcannotbetrained
input = [sensor.data(), sensor.labels()] will wait till some others finish, and get its turn.
brain.setup(input) Tunable parameters are divided into two sets, network
hyper parameters, net_paras_list, and optimization hy-
Noteinthiscase,data()andlabels()of sensorreturns perparameters,opt_paras_list. Eachsetisspecifiedbya
tensors. It is not always the case. If it does not, saying list whose item is a dictionary that holds the actual value
return a list of tensors, you need do things like: of whatever hyper parameters defined as jinja2 templates.
Each item in the list corresponds to a tentative training in-
input = [sensor.data()]
stance. Networkparasandoptimizationparascombinewith
input.extend(sensor.labels())
eachotherexponentially(orinCartesianProductwayifwe
Act accordingly. could use Math terminology), which is to say if you have
Similarly, all blocks work this way. two items in network parameter list, and two in optimiza-
A brain provides easy ways to connect blocks. For exam- tion parameters, the total number of training instances will
ple, a one layer brain can be built through the following: be four.
{"type": "relu"}],
"bn": True})
net_paras_list.append({
"activation": [
{"type": "maxout", "group_size": 2},
{"type": "maxout", "group_size": 2},
{"type": "maxout", "group_size": 2},
{"type": "maxout", "group_size": 5}],
"bn": True})
opt_paras_list = []
opt_paras_list.append({"lr": 0.025})
opt_paras_list.append({"lr": 0.05})
def setup(graph):
brain.attach(cnn_block(
ksize=[8, 8],
init_para={
"name": "uniform",
"range": 0.005},
wd={"type": "l2", "scale": 0.0005},
out_channel_num=384,
pool_size=[4, 4],
pool_stride=[2, 2],
activation={{ net_paras["activation"][1] }},
keep_prob=0.5,
bn={{ net_paras["bn"] }}))
tune(setup, opt_paras_list, net_paras_list)
3.2 Applicationstack
At the top of the stack, akid could be used as a part of
application without knowing the underlying mechanism of
neural networks.
akid provides full machinery from preparing data, aug-
Figure 3: A residual unit. On the left is the branch menting data, specifying computation graph (neural net-
thatbuildsuppatternscomplexity, andontheright workarchitecture),choosingoptimizationalgorithms,speci-
is the stem branch that shortcuts any layers to any fyingparalleltrainingscheme(dataparallelismetc),logging
layers. They merge at the at the start and at the and visualization.
end of the branching points.
3.2.1 Neuralnetworktraining—Aholisticexample
Tocreatebettertoolstotrainneuralnetworkhasbeenat
GiventheavailableGPUnumbers,asemaphoreiscreated the core of the original motivation of akid. Consequently,
tocontrolaccesstoGPUs. Alockiscreatedtocontrolaccess in this section, we describe how akid can be used to train
to the mask to indicator which GPU is available. After a neural networks. Currently, all the other features resolve
processhasmodifiedthegpumask,itreleasesthelockimme- around this.
diately,sootherprocesscouldaccessit. Butthesemaphore The snippet below builds a simple neural network, and
isstillnotrelease,sinceitisusedtocontrolaccesstotheac- trains it using MNIST, the digit recognition dataset.
tualGPU.Atraininginstancewillbelaunchedinasubshell
using the GPU acquired. The semaphore is only released from akid import AKID_DATA_PATH
after the training has finished. from akid import FeedSensor
For example, to tune the activation function and learn- from akid import Kid
ingratesofanetwork,firstwesetupnetworkparametersin from akid import MomentumKongFu
net_paras_list,optimizationparametersinopt_paras_list, from akid import MNISTFeedSource
build a network in the setup function, then pass all of it to
tune: from akid.models.brains import LeNet
net_paras_list = [] brain = LeNet(name="Brain")
net_paras_list.append({ source = MNISTFeedSource(
"activation": [ name="Source",
{"type": "relu"}, url='http://yann.lecun.com/exdb/mnist/',
{"type": "relu"}, work_dir=AKID_DATA_PATH + '/mnist',
{"type": "relu"}, center=True,
3.2.2 Visualization
As a library gearing upon research, it also has rich fea-
tures to visualize various components of a neural network.
It has built-in training dynamics visualization, more specif-
ically, distribution visualization on multi-dimensional ten-
sors,e.g.,weights,activation,biases,gradients,etc,andline
graph visualization on on scalars, e.g., training loss, vali-
dation loss, learning rate decay, regularization loss in each
layer,sparsityofneuronactivationetc,andfilterandfeature
map visualization for neural networks.
Distribution and scalar visualization are built in for typ-
ical parameters and measures, and can be easily extended,
and distributedly gathered. Distribution visualizations are
shown in Figure 5, and scalar visualizations are shown in
Figure 6.
Figure 4: Computational graph of the simple neural
networkbuiltforMNISTdigitrecognitionexample.
scale=True,
num_train=50000,
num_val=10000)
sensor = FeedSensor(name='Sensor', source_in=source)
s = Kid(sensor,
brain,
MomentumKongFu(name="Kongfu"),
max_steps=100)
kid.setup()
kid.practice()
It builds a computation graph as shown in Figure 4.
Figure 5: Visualization of how distribution of multi-
The underlying stories are described in the following sec-
dimensional tensors change over time. Each line on
tion, which also debriefs the design motivation and vision
the chart represents a percentile in the distribution
behind akid.
over the data: for example, the bottom line shows
akid is a kid who has the ability to keep practicing to
how the minimum value has changed over time, and
improve itself. The kid perceives a data Source with its
the line in the middle shows how the median has
Sensor and certain learning methods (nicknamed KongFu)
changed. Reading from top to bottom, the lines
toimproveitself(itsBrain),tofulfillacertainpurpose. The
have the following meaning: [maximum, 93%, 84%,
worldistimedbyaclock. Itrepresentshowlongthekidhas
69%, 50%, 31%, 16%, 7%, minimum] These per-
been practicing. Technically, the clock is the conventional
centiles can also be viewed as standard deviation
training step.
boundaries on a normal distribution: [maximum,
Tobreakthingsdone,SensortakesaSourcewhicheither
µ+1.5σ, µ+σ, µ+0.5σ, µ, µ-0.5σ, µ-σ, µ-1.5σ, min-
provides data in form of tensors from Tensorflow or numpy
imum] so that the colored regions, read from inside
arrays. Optionally, it can make jokers on the data using
to outside, have widths [σ, 2σ, 3σ] respectively.
Joker, meaning doing data augmentation. The data pro-
cessingengine,whichisadeepneuralnetwork,isabstracted
as a Brain. Brain is the name we give to the data process- akidsupportsvisualizationofallfeaturemapsandfilters
ingsysteminlivingbeings. ABrainincarnatesone ofdata with control on the layout through Observer class. When
processing system topology, or in the terminology of neural havingfinishedcreatingaKid,passittoObserver,andcall
visualization as the following.
network, network structure topology, such as a sequentially
linked together layers, to process data. Available topology
from akid import Observer
is defined in module systems. The network training meth-
ods,whicharefirstorderiterativeoptimizationmethods,is
o = Observer(kid)
abstracted as a class KongFu. A living being needs to keep
# Visualize filters as the following
practicingKongFutogetbetterattasksneededtosurvive.
o.visualize_filters()
A living being is abstracted as a Kid class, which as-
# Or visualize feature maps as the following
semblies all above classes together to play the game. The
o.visualize_activation()
metaphor means by sensing more examples, with certain
genreofKongFu(differenttrainingalgorithmsandpolicies), Variouslayoutsareprovidedwhendrawingthefilters. Ad-
thedataprocessingengineof theKid, thebrain, shouldget ditional features are also available. The post-preprocessed
betteratdoingwhatevertaskitisdoing,lettingitbeimage visualizationresultsoffiltersareshowninFigure8,andthat
classification or something else. of feature maps are shown Figure 7.
Theendcomputationalgraphconstructedisillustratedin
Figure 9.
3.4 DistributedDeployment
Thedistributeddeploymentstackhandlestheactualpro-
ductionenvironment,thusdecouplesthedevelopment/prototyping
environment and production environment. We leverage on
recentdevelopmentsofdistributedsystemintheopensource
community to build a distributed deployment solution for
akid. More specifically, we investigate and test out three
cornerstone techniques that provides network file system,
i.e. Glusterfs, containerization, i.e. Docker, and distributed
scheduler, i.e. Kubernetes, functionality. More would come
whenwegetthechancetotestthemoutinarealproduction
Figure 6: Visualization of how important scalar environment.
measures change over time.
4. CONCLUSION
We have described akid, a neural network library that
providesafour-layerstacktoenablefastresearchprototyp-
ing and be production ready. It has a clean and intuitive
application facing interface, nature inspired programming
paradigm, and abstracts away distributing computing, de-
Figure 7: Visualization of feature maps learned. couples developments and operations.
3.3 DistributedComputation
The distributed computing stack is responsible to handle
concurrency and communication between different comput-
ing nodes, so the end user only needs to deal with how to
build a power network. All complexity has been hidden in
the class Engine. The usage of Engine is just to pick and
use.
Morespecifically,akidoffersbuilt-indataparallelscheme
informofclassEngine. Currently,theenginemainlyworks
with neural network training, which is be used with Kid by
specifying the engine at the construction of the kid.
As an example, we could do data parallelism on multiple
towers using:
kid = kids.Kid(
sensor,
brain,
MomentumKongFu(
lr_scheme={
"name": LearningRateScheme.placeholder}),
engine={"name": "data_parallel", "num_gpu": 2},
log_dir="log",
max_epoch=200)
Figure 8: Visualization of filters learned.
Figure9: Illustrationofcomputationalgraphconstructedbyadataparallelengine. Itpartitionsamini-batch
of data into subsets, as indicated by the data_split blue blocks, and passes the subsets to replicates of neural
network models at different computing towers, as indicated by the gray blocks one level above blue blocks,
then after the inference results have been computed, the results and the labels (from the splitted data block)
will be passed to the optimizers in the same tower, as indicated by red and orange blocks named opt, to
compute the gradients. Lastly, the gradients will be passed to a tower that computes the average of the
gradients, and pass them back to neural networks of each computing towers to update their parameters.
5. REFERENCES for consistent development and deployment. Linux
[1] B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and Journal, (239):2, 2014.
J. Wilkes. Borg, Omega, and Kubernetes. Queue, [11] C. Szegedy, W. Zaremba, and I. Sutskever. Intriguing
14(1):70–93, jan 2016. properties of neural networks. arXiv preprint arXiv:
[2] T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, ..., pages 1–10, 2013.
T. Xiao, B. Xu, C. Zhang, and Z. Zhang. MXNet: A [12] T. T. D. The Theano Development Team, R. Al-Rfou,
Flexible and Efficient Machine Learning Library for G. Alain, A. Almahairi, C. Angermueller,
Heterogeneous Distributed Systems. In NIPS, D. Bahdanau, N. Ballas, F. Bastien, J. Bayer,
Workshop on Machine Learning Systems, dec 2016. A. Belikov, A. Belopolsky, Y. Bengio, A. Bergeron,
[3] R. Collobert, R. Collobert, S. Bengio, and J. Bergstra, V. Bisson, J. B. Snyder, N. Bouchard,
J. Marithoz. Torch: A Modular Machine Learning N. Boulanger-Lewandowski, X. Bouthillier,
Software Library. Technical report, 2002. A. de Br´ebisson, O. Breuleux, P.-L. Carrier, K. Cho,
[4] J. D. J. Deng, W. D. W. Dong, R. Socher, L.-J. L. J. Chorowski, P. Christiano, T. Cooijmans, M.-A.
L.-J. Li, K. L. K. Li, and L. F.-F. L. Fei-Fei. Coˆt´e, M. Coˆt´e, A. Courville, Y. N. Dauphin,
ImageNet: A large-scale hierarchical image database. O. Delalleau, J. Demouth, G. Desjardins, S. Dieleman,
CVPR, 2009. L. Dinh, M. Ducoffe, V. Dumoulin, S. E. Kahou,
D. Erhan, Z. Fan, O. Firat, M. Germain, X. Glorot,
[5] K. He, X. Zhang, S. Ren, and J. Sun. Delving Deep
I. Goodfellow, M. Graham, C. Gulcehre, P. Hamel,
into Rectifiers: Surpassing Human-Level Performance
I. Harlouchet, J.-P. Heng, B. Hidasi, S. Honari,
on ImageNet Classification. In ICCV, 2015.
A. Jain, S. Jean, K. Jia, M. Korobov, V. Kulkarni,
[6] K. He, X. Zhang, S. Ren, and J. Sun. Identity
A. Lamb, P. Lamblin, E. Larsen, C. Laurent, S. Lee,
MappingsinDeepResidualNetworks.InECCV,2016.
S. Lefrancois, S. Lemieux, N. L´eonard, Z. Lin, J. A.
[7] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev,
Livezey, C. Lorenz, J. Lowin, Q. Ma, P.-A. Manzagol,
J. Long, R. Girshick, S. Guadarrama, and T. Darrell.
O. Mastropietro, R. T. McGibbon, R. Memisevic,
Caffe: Convolutional Architecture for Fast Feature
B. van Merri¨enboer, V. Michalski, M. Mirza,
Embedding. Technical report, jun 2014.
A. Orlandi, C. Pal, R. Pascanu, M. Pezeshki,
[8] A. Krizhevsky, I. Sutskever, and G. E. Hinton.
C. Raffel, D. Renshaw, M. Rocklin, A. Romero,
ImageNet Classification with Deep Convolutional
M. Roth, P. Sadowski, J. Salvatier, F. Savard,
Neural Networks. In NIPS, 2012.
J. Schlu¨ter, J. Schulman, G. Schwartz, I. V. Serban,
[9] S.Mallat.Understandingdeepconvolutionalnetworks. D. Serdyuk, S. Shabanian, E´. Simon, S. Spieckermann,
Philosophical transactions. Series A, Mathematical,
S. R. Subramanyam, J. Sygnowski, J. Tanguay, G. van
physical, and engineering sciences, 374(2065), 2016.
Tulder, J. Turian, S. Urban, P. Vincent, F. Visin,
[10] MerkelandDirk.Docker: lightweightLinuxcontainers
H.de Vries, D.Warde-Farley, D.J.Webb, M.Willson,
K. Xu, L. Xue, L. Yao, S. Zhang, and Y. Zhang.
Theano: A Python framework for fast computation of
mathematical expressions. Technical report, may 2016.
[13] W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer,
A. Stolcke, D. Yu, and G. Zweig. Achieving Human
Parity in Conversational Speech Recognition.
Technical report, oct 2016.
[14] M. Zeiler and R. Fergus. Visualizing and
understanding convolutional networks. ECCV, 2014.