Two-Bit Networks for Deep Learning on Resource-Constrained Embedded Devices

1.2 MB·

by  Wenjia Meng

Two-Bit Networks for Deep Learning on implemented with additions, subtractions, and shifts, which are very
Resource-Constrained Embedded Devices efficientandhardware-friendly.
We propose a training algorithm for DNNs based on Stochastic
WenjiaMeng,ZonghuaGu,MingZhang,ZhaohuiWu GradientDescent.Duringeachiteration,asetofreal-valuedweightsare
discretizedintotwo-bitvalues,whichareusedbythefollowingforward
WiththerapidproliferationofInternetofThingsandintelligentedge passandbackwardpass.Then,thereal-valuedweightsareupdatedwith
devices,thereisanincreasingneedforimplementingmachinelearning gradients computed by the backward pass. During inference, only the
algorithms, including deep learning, on resource-constrained mobile two-bit weights are used. Experimental results show that or method
embeddeddeviceswithlimitedmemoryandcomputationpower.Typical achievesbetterperformanceonImageNetthanotherweightcompression
large Convolutional Neural Networks (CNNs) need large amounts methods.
of memory and computational power, and cannot be deployed on
embeddeddevicesefficiently.WepresentTwo-BitNetworks(TBNs)for
Two-BitsNetworks: InaCNNwithLlayers,eachlayerl∈{1,...,L}
modelcompressionofCNNswithedgeweightsconstrainedto(-2,-1,
1,2),whichcanbeencodedwithtwobits.Ourapproachcanreducethe performsaconvolution∗onitsinputtensorIl∈Rcl×wiln×hlinandeach
memoryusageandimprovecomputationalefficiencysignificantlywhile of its Kl convolution filters Wl,k∈Rcl×wl×hl, where (cl,wl ,hl )
achieving good performance in terms of classification accuracy, thus and (cl,wl,hl) represents shape of the input tensor and thienfilitner,
representingareasonabletradeoffbetweenmodelsizeandperformance.
respectively, including channels, width, and height. Let nl,k=cl,k×
wl,k×hl,k denotethenumberofelementsinWl,k,andWl,k denote
i
Introduction: Deep Neural Networks (DNNs) are widely used for the ith element of Wl,k, with i∈{1,...,nl,k}. For brevity, we drop
a variety of machine learning tasks currently, including computer indexeslandkwhentheyareunnecessary.
vision, natural language processing, and others. Convolutional Neural Eachreal-valuedconvolutionfilterW isapproximatedwithabinary
Networks (CNNs) are especially suitable for computer vision filter W˜ ∈{−2,−1,1,2}c×w×h and a scaling factor α∈R+, so that
applications, including object recognition, classification, detection, W≈αW˜.Aconvolutionoperationcanbeapproximatedby:
image segmentation, etc. Smart mobile devices equipped with high-
7 I∗W≈I∗(αW˜)=(αI)⊕W˜ (1)
resolutioncamerashaveopenedthedoortopotentialmobilecomputer
1
visionapplications.However,atypicalCNNhasmillionsofparameters where⊕denotesaconvolutionoperationwithoutmultiplication.
0
and perform billions of arithmetic operations for each inference, e.g., Ideally,theTBNshouldmimicitsfull-precisioncounterpartclosely,
2
AlexNet[1]has61Mparameters(249MBofmemorywithfloatingpoint providedthatquantizationerrorbetweenW anditsapproximationαW˜

nweights) and performs 1.5B high-precision operations to classify one isminimized.WeseektominimizetheL2-normofthequantizationerror
aimage. The enormous demand for memory storage and computation foreachconvolutionfilter:
Jpower hinders their deployment on resource-constrained embedded
4 devices with limited memory and computation power. It is an active α∗,W˜∗=argmin(cid:107)W −αW˜ (cid:107)22
research topic to adapt DNNs/CNNs for deployment on embedded α,W˜
G] devAicensu.mberofauthorshaveproposedtechniquesforreducingsizeof s.t. α>0,W˜i∈{−2,−1,1,2},i∈{1,...,n} (2)
LDNNs/CNNs, including model pruning [6], i.e., removing edges with Theoptimizationcanbedividedintotwosteps.First,thereal-valued
small weight magnitudes, and weight compression, i.e., reducing the weights are discretized to find the two-bit weights. Then, the optimal
.
sprecision of weights by using a small number of bits to edge weight scalingfactorisfoundtominimizethequantizationerror,giventhetwo-
cand/or activation. These two approaches are orthogonal and can be bitweights.Forsimplicity,weadoptdeterministicdiscretization:
[applied separately or together. We focus on weight compression in

85v2 bthBahniiinigndsaharprByayairCpniaetaohcrrn.tmyinvTNeeahtectieittcoln[[co43sow]].mfofBbporiuortt-ttrahwtartiiaiondBinnitinhiannglatgrreaayfanCfiDscolDaiNnetNnenNesNccytwt.owiCatsihntmohduabrblobBlitanhimrnaiaarebyrumiynxoNwarereeyyttigfawchola.toesntippg-rrha1eitncssehtoniraaetnnev1ddde, W˜i=12−−12 iiiiffff0WW−<ii1><W≤1−iW.≤1,i1≤, 0, (3)
4
good performance on small datasets such as MNIST, CIFAR-10 and Substitutethetwo-bitweights(3)intotheexpressionforquantization
0
SVHN, but performed worse than their full-precision counterparts by error,theexpressioncanbesimplifiedinto:
0
a wide margin on large-scale datasets like ImageNet. Rastegari et al.
1.[2] presented Binary Weight Networks and XNOR-Net, two efficient J(α)=(cid:107)W −αW˜ (cid:107)22
0approximations to standard CNNs, which are shown to outperform
(cid:88) (cid:88)
7BinaryConnectandBinaryNetbylargemarginsonImageNet.InBinary- = ||Wi|−α|2+ ||Wi|−2α|2
1WeightNetworks,theconvolutionalfiltersareapproximatedwithbinary i:|Wi|≤1 i:|Wi|>1
v:vlaaylueressa(-r1e,1b)i;nianryX.NHOowRe-Nveert,wtohrekres,ibsosthtilfillaterssigannidficinanptutpteorfcoornmvaonlucteiognaapl =(|B1|+4|B2|)α2−2((cid:88) |Wi|+2 (cid:88) |Wi|)α+C (4)
i
Xbetween these network models and their full-precision counterparts. i∈B1 i∈B2
Also, binarization of both activations and weights generally leads to (cid:12) (cid:12)
ardornalmy.aticperformancedegradationcomparedtobinarizationofweights w(cid:80)hnie=re1WBi21=is{1a≤coin≤stnan(cid:12)|tWiin|d≤ep1e}n,deBnt2o=f{α1,≤ain≤dn|(cid:12)W|Wi|i|d>en1o}t,esCth=e
To strike a balance between model compression rate and model magnitudeofWi.
TakingthederivativeofJ(α)w.r.t.αandsetting dJ(α) tozero,we
capacity,Lietal.[5]presentedTernaryWeightNetworkswithweights dα
obtaintheoptimalscalingfactor:
constrainedto(-1,0,1),eachweightencodedwithtwobits.Compared
wacihthievDeNbNettmeropdeerlfsowrmitahncbeindauryetwoeiingchrtesa,sTederwnaeriyghWtpeirgehcitsNioent.wHoorkwsevcearn, α∗=(cid:80)i∈B1|Wi|+2(cid:80)i∈B2|Wi| (5)
TernaryWeightNetworksmakeuseofonlythreevalues(-1,0,1)outof |B1|+4|B2|
thefourpossiblevaluesthatcanbeencodedwithtwobits.
TrainingTwo-BitNetworks: Wedescribedetailsofthetrainingalgorithm
Inthispaper,weproposeTwo-BitNetworks(TBNs)tofurtherexplore
for Two-Bit Networks based on Stochastic Gradient Descent (SGD).
the tradeoff between model compression rate and model capacity. (We
Algorithm1showsthepseudocodeforeachtrainingiteration.Inorder
focus on CNNs in this paper, although our techniques can be adapted
to keep track of tiny weight updates of each iteration, we adopt a
to apply to general DNNs, including Recurrent Neural Networks.) We
similartrickas[3]andmaintainasetofreal-valuedconvolutionfilters
constrain weights to four values (-2, -1, 1, 2), which can be encoded
throughoutthetrainingprocess.First,approximatefiltersarecomputed
with two bits. Compared with existing weight compression methods,
from the real-valued filters for all convolutional layers (Lines 1-7).
TBNs make more efficient use of weight bit-width to achieve higher
Note that fully-connected layers can be treated as convolutional layers
model capacity and better performance. Arithmetic operations can be
[7]. The real-valued weights are discretized into two-bit weights (Line
3), and an optimal scaling factor is computed for each filter (Line 4).
Then, a forward pass is run on the network inputs (Line 8), followed
Fig.1ValidationaccuracycomparisonoftheTwo-BitNetworkmethodand
state-of-the-artweightcompressionmethodsontheImageNetdataset
Fig. 2 Memory size comparisons between Two-Bits Networks and double-
precisionnetworksfordifferentCNNarchitectures.
byabackwardpass,whichback-propagateserrorsthroughthenetwork
to compute gradients w.r.t. the approximate filters (Line 9). Unlike reductioninmemorysizerequirementmakesTwo-BitNetworkssuitable
conventionalCNNs,theforwardandbackwardpassesuseapproximate fordeploymentonembeddeddevices.
filtersinsteadofthereal-valuedfilters.Finally,thereal-valuedfiltersare
updatedwiththegradients(Line10).Duringinference,onlythetwo-bit Conclusion: We have presented Two-Bit Networks for model
filtersandtheoptimalscalingfactorsareused. compression of CNNs, which achieves a good tradeoff between
model size and performance compared with state-of-the-art weight
compressionmethods.Comparedtotherecentworkonbinaryweights
Algorithm1SGDTrainingforTwo-BitNetworks
and/oractivations,ourmethodachieveshighermodelcapacityandbetter
Input: A minibatch of inputs X and targets Y, current real-valued performancewithslightlylargermemorysizerequirement.
weightsW andlearningrateη
Output: Updatedreal-valuedweightsWnewandlearningrateηnew
1: forl=1toLdo Acknowledgment: This work is partially supported by NSFC Grant
2: forkthfilterinlthlayerdo #61672454.TheTitan-XGPUusedforthisresearchwasdonatedby
3: Computetwo-bitfilterW˜l,kfromWl,kby(3) theNVIDIACorporation.
4: Computeoptimalscalingfactorαl,k∗by(5)
5: ApproximatefilterWˆl,k=αl,k∗W˜l,k Wenjia Meng, Zonghua Gu, Ming Zhang and Zhaohui Wu (College of
ComputerScience,ZhejiangUniversity,Hangzhou,China,310027)
6: endfor
7: endfor E-mail:[email protected]
8: Yˆ=TwoBitForward(X,{W˜l,k},{αl,k∗})//Convolutionsby(1)
9: { ∂C }=TwoBitBackward(∂C,{Wˆl,k}) References
∂Wˆl,k ∂Yˆ
10: Wnew=UpdateParameters(W,{ ∂C },η)
∂Wˆl,k 1 Krizhevsky, A., Sutskever, I., Hinton, G.E., "Imagenet
11: Updatelearningrateηnewaccordingtoanylearningratescheduling
classification with deep convolutional neural networks,",
function
Advances in neural information processing systems. (2012)
1097´lC1105.
2 M.Rastegari,V.Ordonez,J.Redmon,andA.Farhadi,"XNOR-
Net:ImageNetClassificationUsingBinaryConvolutionalNeural
Experiments: Weusethewell-knownImageNetdataset(ILSVRC2012)
toevaluateperformanceofTwo-BitNetworks.ImageNetisacomputer Networks,"CoRR,vol.abs/1603.0,2016.
visionbenchmarkdatasetwithalargenumberoflabeledimages,divided 3 M. Courbariaux, Y. Bengio, and J.-P. David, "BinaryConnect:
intoatrainingsetandavalidationset.Thetrainingsetconsistsof∼1.2M Training Deep Neural Networks with binary weights during
imagesfrom1Kcategories,includinganimals,plants,andothercommon propagations," in Advances in Neural Information Processing
items;thevalidationsetcontains50Kimages. Systems28,2015,pp.3123-3131.
WeuseDeepResidualNetworks(DRNs)[8]astheCNNarchitecture 4 M. Courbariaux and Y. Bengio, "Binary Neural Networks:
in our experiments, which achieved state-of-the-art performance on
Training Deep Neural Networks with Weights and Activations
ImageNet. Compared with other CNN architectures, DRNs can have a
Constrainedto+1or-1,"CoRR,vol.abs/1602.0,2016.
large number of convolutional layers (from 18 to 152), and a shortcut
5 F. Li and B. Liu, "Ternary Weight Networks," CoRR, vol.
connectionthatperformslinearprojectionexistsalongsideeachgroupof
abs/1605.04711,2016.
twoconsecutiveconvolutionallayers,inordertoreformulatethelayers
aslearningfunctionswithreferencetothelayerinputs.Forsimplicity,we 6 S.Han,J.Pool,J.Tran,andW.Dally,"LearningbothWeightsand
adoptResNet-18,whichhas18convolutionallayersandisthesmallest ConnectionsforEfficientNeuralNetwork,"inAdvancesinNeural
modelpresentedintheirpaper. InformationProcessingSystems28,2015,pp.1135-1143.
The experiments are conducted with Torch7 [9] on an NVIDIA 7 J. Long, E. Shelhamer, and T. Darrell, "Fully Convolutional
TitanX.Attrainingtime,imagesarerandomlycroppedwith224×224 NetworksforSemanticSegmentation,"inTheIEEEConference
windows.Werunthetrainingalgorithmfor58epochswithbatchsizeof onComputerVisionandPatternRecognition(CVPR),2015.
256.WeuseSGDwithmomentumof0.9toupdateparametersandbatch
8 K.He,X.Zhang,S.Ren,andJ.Sun,"DeepResidualLearningfor
normalization[10]tospeedupconvergence.Theweightdecayissetto
ImageRecognition,"CoRR,vol.abs/1512.03385,2015.
0.0001. The learning rate starts at 0.1 and is divided by 10 at epoches
9 R.Collobert,K.Kavukcuoglu,andC.Farabet,"Torch7:Amatlab-
30,40,and50.Atinferencetime,weusethe224×224centercropsfor
like environment for machine learning," in BigLearn, NIPS
forwardpropagation.
We compare our method with state-of-the-art weight compression Workshop,2011,no.EPFL-CONF-192376.
methods,includingTernaryWeightNetwork[5],BinaryWeightNetwork 10 S. Ioffe and C. Szegedy, "Batch Normalization: Accelerating
and XNOR-Net [2]. Fig. 1 shows performance results (classification Deep Network Training by Reducing Internal Covariate Shift,"
accuracy) on ImageNet. Our method outperforms the other weight Proc.32ndInt.Conf.Mach.Learn.,pp.448-456,2015.
compressionmethods,withtop-5accuracyof84.5%andtop-1accuracy
of62.6%.Weattributetheimprovedperformancetotheincreasedmodel
capacity due to more efficient use of the two-bit representation for
weights.
Fig. 2 shows memory size requirement of Two-Bit Networks
compared to double-precision floating point representation for three
differentarchitectures(AlexNet,ResNet-18andVGG-19).Thedramatic
2