apprentice: using knowledge distillation techniques to improve low-precision net

17 Pages·2017·0.36 MB·English

PublishedasaconferencepaperatICLR2018
APPRENTICE: USING KNOWLEDGE DISTILLATION
TECHNIQUES TO IMPROVE LOW-PRECISION NET-
WORK ACCURACY
AsitMishra&DebbieMarr
AcceleratorArchitectureLab
IntelLabs
{asit.k.mishra,debbie.marr}@intel.com
ABSTRACT
Deeplearningnetworkshaveachievedstate-of-the-artaccuraciesoncomputervi-
sion workloads like image classification and object detection. The performant
systems,however,typicallyinvolvebigmodelswithnumerousparameters. Once
trained,achallengingaspectforsuchtopperformingmodelsisdeploymentonre-
sourceconstrainedinferencesystems—themodels(oftendeepnetworksorwide
networks or both) are compute and memory intensive. Low-precision numerics
and model compression using knowledge distillation are popular techniques to
lower both the compute requirements and memory footprint of these deployed
models. In this paper, we study combination of these two techniques and show
that the performance of low-precision networks can be significantly improved
byusingknowledgedistillationtechniques. Ourapproach,Apprentice,achieves
state-of-the-artaccuraciesusingternaryprecisionand4-bitprecisionforvariants
of ResNet architecture on ImageNet dataset. We present three schemes using
which one can apply knowledge distillation techniques to various stages of the
train-and-deploypipeline.
1 INTRODUCTION
Background: Today’shighperformingdeepneuralnetworks(DNNs)forcomputervisionapplica-
tionscompriseofmultiplelayersandinvolvenumerousparameters. ThesenetworkshaveO(Giga-
FLOPS)computerequirementsandgeneratemodelswhichareO(Mega-Bytes)instorage(Canziani
etal.,2016). Further,thememoryandcomputerequirementsduringtrainingandinferencearequite
different(Mishraetal.,2017). Trainingisperformedonbigdatasetswithlargebatch-sizeswhere
memoryfootprintofactivationsdominatesthemodelmemoryfootprint. Ontheotherhand,batch-
size during inference is typically small and the model’s memory footprint dominates the runtime
memoryrequirements.
Because of complexity in compute, memory and storage requirements, training phase of the net-
works is performed on CPU and/or GPU clusters in a distributed computing environment. Once
trained, a challenging aspect is deployment of trained models on resource constrained inference
systemssuchasportabledevicesorsensornetworks,andforapplicationsinwhichreal-timepredic-
tionsarerequired. Performinginferenceonedge-devicescomeswithsevereconstraintsonmemory,
compute and power. Additionally, ensemble based methods, which one can potentially use to get
improvedaccuracypredictions,becomeprohibitiveinresourceconstrainedsystems.
Quantization using low-precision numerics (Vanhoucke et al., 2011; Zhou et al., 2016; Lin et al.,
2015;Miyashitaetal.,2016;Guptaetal.,2015;Zhuetal.,2016;Rastegarietal.,2016;Courbariaux
et al., 2015; Umuroglu et al., 2016; Mishra et al., 2017) and model compression (Buciluaˇ et al.,
2006; Hinton et al., 2015; Romero et al., 2014) have emerged as popular solutions for resource
constraineddeploymentscenarios. Withquantization,alow-precisionversionofnetworkmodelis
generatedanddeployedonthedevice. Operatinginlowerprecisionmodereducescomputeaswell
asdatamovementandstoragerequirements. However,majorityofexistingworksinlow-precision
DNNssacrificeaccuracyoverbaselinefull-precisionnetworks. Withmodelcompression,asmaller
1
PublishedasaconferencepaperatICLR2018
lowmemoryfootprintnetworkistrainedtomimicthebehaviouroftheoriginalcomplexnetwork.
Duringthistraining, aprocesscalled, knowledgedistillationisusedto“transferknowledge”from
thecomplexnetworktothesmallernetwork.WorkbyHintonetal.(2015)showsthattheknowledge
distillation scheme can yield networks at comparable or slightly better accuracy than the original
complexmodel. However, tothebestofourknowledge, allpriorworksusingmodelcompression
techniquestargetcompressionatfull-precision.
Our proposal: In thispaper, westudy the combinationof networkquantization with modelcom-
pressionandshowthataccuraciesoflow-precisionnetworkscanbesignificantlyimprovedbyusing
knowledge distillation techniques. Previous studies on model compression use a large network as
theteachernetworkandasmallnetworkasthestudentnetwork. Thesmallstudentnetworklearns
from teacher network using distillation process. The network architecture of the student network
is typically different from that of the teacher network – for e.g. Hinton et al. (2015) investigate a
studentnetworkthathasfewernumberofneuronsinthehiddenlayerscomparedtotheteachernet-
work. Inourwork,thestudentnetworkhassimilartopologyasthatofteachernetwork,exceptthat
thestudentnetworkhaslow-precisionneuronscomparedtotheteachernetworkwhichhasneurons
operatingatfull-precision.
We call our approach Apprentice1 and study three schemes which produce low-precision net-
worksusingknowledgedistillationtechniques.Eachofthesethreeschemesproducestate-of-the-art
ternaryprecisionand4-bitprecisionmodels.
In the first scheme, a low-precision network and a full-precision network are jointly trained from
scratch using knowledge distillation scheme. Later in the paper we describe the rationale behind
thisapproach. Usingthisscheme, anewstate-of-the-artaccuracyisobtainedforternaryand4-bit
precisionforResNet-18,ResNet-34andResNet-50onImageNetdataset. Infact,usingthisscheme
theaccuracyofthefull-precisionmodelalsoslightlyimproves. Thisschemethenservesasthenew
baselinefortheothertwoschemesweinvestigate.
In the second scheme, we start with a full-precision trained network and transfer knowledge from
this trained network continuously to train a low-precision network from scratch. We find that the
low-precision network converges faster (albeit to similar accuracies as the first scheme) when a
trainedcomplexnetworkguidesitstraining.
Inthethirdscheme,westartwithatrainedfull-precisionlargenetworkandanapprenticenetwork
thathasbeeninitialisedwithfull-precisionweights. Theapprenticenetwork’sprecisionislowered
andisfine-tunedusingknowledgedistillationtechniques. Wefindthatthelow-precisionnetwork’s
accuracymarginallyimprovesandsurpassestheaccuracyobtainedviathefirstscheme.Thisscheme
thensetsthenewstate-of-the-artaccuraciesfortheResNetmodelsatternaryand4-bitprecision.
Overall, the contributions of this paper are the techniques to obtain low-precision DNNs using
knowledge distillation technique. Each of our scheme produces a low-precision model that sur-
passes the accuracy of the equivalent low-precision model published to date. One of our schemes
alsohelpsalow-precisionmodelconvergefaster. Weenvisiontheseaccuratelow-precisionmodels
to simplify the inference deployment process on resource constrained systems and even otherwise
oncloud-baseddeploymentsystems.
2 MOTIVATION FOR LOW-PRECISION MODEL PARAMETERS
Loweringprecisionofmodelparameters: Resourceconstrainedinferencesystemsimposesignif-
icantrestrictionsonmemory,computeandpowerbudget. Withregardtostorage,model(orweight)
parametersandactivationmapsoccupymemoryduringtheinferencephaseofDNNs. Duringthis
phase memory is allocated for input (IFM) and output feature maps (OFM) required by a single
layer in the DNN, and these dynamic memory allocations are reused for other layers. The total
memoryallocationduringinferenceisthenthemaximumofIFMandmaximumofOFMmemory
requiredacrossallthelayersplusthesumofallweighttensors(Mishraetal.,2017). Wheninfer-
ence phase for DNNs is performed with a small batch size, the memory footprint of the weights
1Dictionarydefinesapprenticeasapersonwhoislearningatradefromaskilledemployer,havingagreedto
workforafixedperiodatlowwages.Inourwork,theapprenticeisalow-precisionnetworkwhichislearning
theknowledgeofahighprecisionnetwork(skilledemployer)duringafixednumberofepochs.
2
PublishedasaconferencepaperatICLR2018
exceeds the footprint of the activation maps. This aspect is shown in Figure 1 for 4 different net-
works (AlexNet (Krizhevsky et al., 2012), Inception-Resnet-v2 (Szegedy et al., 2016), ResNet-50
andResNet-101(Heetal.,2015))running224x224imagepatches. Thusloweringtheprecisionof
the weight tensors helps lower the memory requirements during deployment. One other aspect of
loweringmemoryfootprintisthattheworkingsetsizeoftheworkloadstartstofitonchipandby
reducingaccessestoDRAM(off-chip)memory, thecomputecorestartstoseebetterperformance
andenergysavings(DRAMaccessesareexpensiveinlatencyandenergy).
Benefit of low-precision compute: Low-precision
compute simplifies hardware implementation. For
example, the compute unit to perform the convolu- % Ws % ACTs
tion operation (multiplication of two operands) in-
volves a floating-point multiplier when using full- print 2.3% 8.7% 20.4% 5.9% 33.5% 3.5% 22.4%
precision weights and activations. The floating- ot
point multiplier can be replaced with a much sim- y fo 67.2%
or 97.7% 91.3% 94.1% 96.5%
pler circuitry (xnor and popcount logic elements) m 79.6% 77.6%
e 66.5%
whenusingbinaryprecisionforweightsandactiva- M
% 32.8%
tions(Courbariaux&Bengio,2016;Rastegarietal.,
2016;Courbariauxetal.,2015). Similarly,whenus- 1 8 1 8 1 8 1 8
ing ternary precision for weights and full-precision Alexnet IRv2 ResNet-50 ResNet-101
for activations, the multiplier unit can be replaced
withasigncomparatorunit. Simplerhardwarealso
Figure 1: Memory footprint of activations
helps lower the inference latency and energy bud-
(ACTs) and weights (W) during inference for
get. Thus, operating in lower precision mode re-
mini-batchsizes1and8.
ducescomputeaswellasdatamovementandstorage
requirements.
Thedrawbackoflow-precisionmodels,however,isdegradedaccuracy.Wediscusslaterinthepaper
thenetworkaccuraciesobtainedusingmethodsproposedinliterature. Theseaccuraciesserveasthe
startingpointandbaselineswecomparetoinourwork.
3 RELATED WORK
Low-precisionnetworks: Low-precisionDNNsareanactiveareaofresearch. Mostlow-precision
networks acknowledge the over parameterization aspect of today’s DNN architectures and/or the
aspect that lowering the precision of neurons post-training often does not impact the final perfor-
mance. Reducing precision of weights for efficient inference pipeline has been very well studied.
WorkslikeBinaryconnect(BC)(Courbariauxetal.,2015),Ternary-weightnetworks(TWN)(Li&
Liu,2016),fine-grainedternaryquantization(Mellempudietal.,2017)andINQ(Zhouetal.,2017)
targetprecisionreductionofnetworkweights. Accuracyisalmostalwaysaffectedwhenquantizing
theweightssignificantlybelow8-bitsofprecision.ForAlexNetonImageNet,TWNloses5%Top-1
accuracy.SchemeslikeINQ,workinSungetal.(2015)andMellempudietal.(2017)dofine-tuning
toquantizethenetworkweights.
WorkinXNOR-NET(Rastegarietal.,2016),binaryneuralnetworks(Courbariaux&Bengio,2016),
DoReFa(Zhouetal.,2016)andtrainedternaryquantization(TTQ)(Zhuetal.,2016)targettraining
pipeline.WhileTTQtargetsweightquantization,mostworkstargetingactivationquantizationshow
that quantizing activations always hurt accuracy. XNOR-NET approach degrades Top-1 accuracy
by 12% and DoReFa by 8% when quantizing both weights and activations to 1-bit (for AlexNet
on ImageNet). Work by Gupta et al. (2015) advocates for low-precision fixed-point numbers for
training. Theyshow16-bitstobesufficientfortrainingonCIFAR10dataset. WorkbySeideetal.
(2014)quantizesgradientsinadistributedcomputingsystem.
Knowledgedistillationmethods: Thegeneraltechniqueindistillationbasedmethodsinvolvesus-
ingateacher-studentstrategy,wherealargedeepnetworktrainedforagiventaskteachesshallower
student network(s) on the same task. The core concepts behind knowledge distillation or transfer
technique have been around for a while. Buciluaˇ et al. (2006) show that one can compress the
information in an ensemble into a single network. Ba & Caurana (2013) extend this approach to
studyshallow, butwide, fullyconnectedtopologiesbymimickingdeepneuralnetworks. Tofacil-
3
PublishedasaconferencepaperatICLR2018
itate learning, the authors introduce the concepts of learning on logits rather than the probability
distribution.
Hintonetal.(2015)proposeaframeworktotransferknowledgebyintroducingtheconceptoftem-
perature. ThekeyideaistodividethelogitsbyatemperaturefactorbeforeperformingaSoftmax
function. Byusingahighertemperaturefactortheactivationsofincorrectclassesareboosted. This
thenfacilitatesmoreinformationflowingtothemodelparametersduringback-propagationopera-
tion. FitNets(Romeroetal.,2014)extendthisworkbyusingintermediatehiddenlayeroutputsas
target values for training a deeper, but thinner, student model. Net2Net (Chen et al., 2015a) also
uses a teacher-student network system with a function-preserving transformation approach to ini-
tialize the parameters of the student network. The goal in Net2Net approach is to accelerate the
trainingofalargerstudentnetwork. Zagoruyko&Komodakis(2016)useattentionasamechanism
fortransferringknowledgefromonenetworktoanother. Inasimilartheme,Yimetal.(2017)pro-
poseaninformationmetricusingwhichateacherDNNcantransferthedistilledknowledgetoother
studentDNNs. InN2Nlearningwork, Ashoketal.(2017)proposeareinforcementlearningbased
approachforcompressingateachernetworkintoanequallycapablestudentnetwork. Theyachieve
acompressionfactorof10xforResNet-34onCIFARdatasets.
Sparsity and hashing: Few other popular techniques for model compression are pruning (LeCun
etal.,1990;Hanetal.,2015a;Wenetal.,2016;Hanetal.,2015b),hashing(Weinbergeretal.,2009)
and weight sharing (Chen et al., 2015b; Denil et al., 2013). Pruning leads to removing neurons
entirelyfromthefinaltrainedmodelmakingthemodelasparsestructure. Withhashingandweight
sharing schemes a hash function is used to alias several weight parameters into few hash buckets,
effectively lowering the parameter memory footprint. To realize benefits of sparsity and hashing
schemesduringruntime, efficienthardwaresupportisrequired(e.g. supportforirregularmemory
accesses(Hanetal.,2016;Venkateshetal.,2016;Parasharetal.,2017)).
4 KNOWLEDGE DISTILLATION
We introduce the concept of knowledge distillation in this section. Buciluaˇ et al. (2006), Hinton
etal.(2015)andUrbanetal.(2016)analyzethistopicingreatdetail.
Figure2showstheschematicoftheknowledgedistillationsetup. Givenaninputimagex,ateacher
DNNmapsthisimagetopredictionspT. TheC classpredictionsareobtainedbyapplyingSoftmax
function on the un-normalized log probability values z (the logits), i.e. pT = ezkT/(cid:80)CezjT. The
j
same image is fed to the student network and it predicts pA = ezkA/(cid:80)CezjA. During training, the
j
costfunction,L,isgivenas:
L(x;W ,W )=αH(y,pT)+βH(y,pA)+γH(zT,pA) (1)
T A
where,W andW aretheparametersoftheteacherandthestudent(apprentice)network,respec-
T A
tively, y isthegroundtruth, H(·)denotesalossfunctionand, α, β andγ areweightingfactorsto
prioritizetheoutputofacertainlossfunctionovertheother.
Inequation1,loweringthefirsttermofthecostfunctiongivesabetterteachernetworkandlowering
the second term gives a better student network. The third term is the knowledge distillation term
whereby the student network attempts to mimic the knowledge in the teacher network. In Hinton
etal.(2015),thelogitsoftheteachernetworkaredividedbyatemperaturefactorτ. Usingahigher
valueforτ producesasofterprobabilitydistributionwhentakingtheSoftmaxofthelogits. Inour
studies,weusecross-entropyfunctionforH(·),setα = 1,β = 0.5andγ = 0.5and,performthe
transferlearningprocessusingthelogits(inputstotheSoftmaxfunction)oftheteachernetwork. In
ourexperimentswestudytheeffectofvaryingthedepthoftheteacherandthestudentnetwork,and
theprecisionoftheneuronsinthestudentnetwork.
5 OUR APPROACH - APPRENTICE NETWORK
Low-precisionDNNstargetthestorageandcomputeefficiencyaspectsofthenetwork. Modelcom-
pressiontargetsthesameefficiencyparametersfromthepointofviewofnetworkarchitecture.With
4
PublishedasaconferencepaperatICLR2018
Teacher network
WT
Filter bank zT softmax pT
Hard
Input image Knowledge label
x Apprentice network distillation
!
WA
Filter bank zA softmax pA
Figure2: Schematicoftheknowledgedistillationsetup.Theteachernetworkisahighprecisionnetworkand
theapprenticenetworkisalow-precisionnetwork.
Apprentice we combine both these techniques to improve the network accuracy as well as the
runtimeefficiencyofDNNs. Usingtheteacher-studentsetupdescribedinthelastsection,weinves-
tigatethreeschemesusingwhichonecanobtainalow-precisionmodelforthestudentnetwork. The
firstscheme(scheme-A)jointlytrainsboththenetworks-full-precisionteacherandlow-precision
studentnetwork. Thesecondscheme(scheme-B)trainsonlythelow-precisionstudentnetworkbut
distills knowledge from a trained full-precision teacher network throughout the training process.
Thethirdscheme(scheme-C)startswithatrainedfull-precisionteacherandafull-precisionstudent
networkbutfine-tunesthestudentnetworkafterloweringitsprecision.Beforewegetintothedetails
ofeachoftheseschemes, wediscusstheaccuracynumbersobtainedusinglow-precisionschemes
describedinliterature. Theseaccuracyfiguresserveasthebaselineforcomparativeanalysis.
5.1 TOP-1ERRORWITHPRIORPROPOSALSFORLOW-PRECISIONNETWORKS
Wefocusonsub8-bitsprecisionforinferencedeployments,specificallyternaryand4-bitsprecision.
WefoundTTQ(Zhuetal.,2016)schemeachievingthestate-of-the-artaccuracywithternarypre-
cisionforweightsandfull-precision(32-bitsfloating-point)foractivations. OnImagenet-1K(Rus-
sakovskyetal.,2015), TTQachieves33.4%Top-1errorratewithaResNet-18model. Weimple-
mented TTQ scheme for ResNet-34 and ResNet-50 models trained on Imagenet-1K and achieved
28.3%and25.6%Top-1errorrates,respectively. Thisschemeisourbaselinefor2-bitsweightand
full-precisionactivations. For2-bitsweightand8-bitsactivation,wefindworkbyMellempudietal.
(2017)toachievethebestaccuraciesreportedinliterature.ForResNet-50,Mellempudietal.(2017)
obtain 29.24% Top-1 error. We consider this work to be our baseline for 2-bits weight and 8-bits
activationmodels.
For4-bitsprecision,wefindWRPNscheme(Mishraetal.,2017)toreportthehighestaccuracy. We
implemented this scheme for 4-bits weight and 8-bits activations. For ResNet-34 and ResNet-50
modelstrainedonImagenet-1K,weachieve29.7%and28.4%Top-1errorrates,respectively.
5.2 SCHEME-A:JOINTTRAININGOFTEACHER-STUDENTNETWORKS
Inthefirstschemethatweinvestigate,afull-precisionteachernetworkisjointlytrainedwithalow-
precisionstudentnetwork. Figure2showstheoveralltrainingframework. WeuseResNettopology
forboththeteacherandstudentnetwork. Whenusingacertaindepthforthestudentnetwork, we
picktheteachernetworktohaveeitherthesameorlargerdepth.
In Buciluaˇ et al. (2006) and Hinton et al. (2015), only the student network trains while distilling
knowledgefromtheteachernetwork. Inourcase,wejointlytrainwiththerationalethattheteacher
networkwouldcontinuouslyguidethestudentnetworknotonlywiththefinaltrainedlogits,butalso
onwhatpaththeteachertakestowardsgeneratingthosefinalhigheraccuracylogits.
5
PublishedasaconferencepaperatICLR2018
Weimplementpre-activationversionofResNet(Heetal.,2016)inTensorFlow(Abadietal.,2015).
ThetrainingprocesscloselyfollowstherecipementionedinTorchimplementationofResNet-we
useabatchsizeof256andnohyper-parametersarechangedfromwhatismentionedintherecipe.
For the teacher network, we experiment with ResNet-34, ResNet-50 and ResNet-101 as options.
Forthestudentnetwork, weexperimentwithlow-precisionvariantsofResNet-18, ResNet-34and
ResNet-50.
For low-precision numerics, when using ternary precision we use the ternary weight network
scheme (Li & Liu, 2016) where the weight tensors are quantized into {−1,0,1} with a per-layer
scalingcoefficientcomputedbasedonthemeanofthepositivetermsintheweighttensor. Weuse
theWRPNscheme(Mishraetal.,2017)toquantizeweightsandactivationsto4-bitsor8-bits. We
donotlowertheprecisionofthefirstlayerandthefinallayerintheapprenticenetwork.Thisisbased
ontheobservationinalmostallpriorworksthatloweringtheprecisionoftheselayersdegradesthe
accuracy dramatically. While training and during fine-tuning, the gradients are still maintained at
full-precision.
Table 1: Top-1 validation set error rate (%) on ImageNet-1K for ResNet-18 stu-
dent network as precision of activations (A) and weight (W) changes. The last
threecolumnsshowerrorratewhenthestudentResNet-18ispairedwithResNet-34,
ResNet-50andResNet-101.
ResNet-18 ResNet-18 ResNet-18 ResNet-18
Baseline withResNet-34 withResNet-50 withResNet-101
32A,32W 30.4 30.2 30.1 30.1
32A,2W 33.4 31.7 31.5 31.8
8A,4W 33.6 29.6 29.6 29.9
8A,2W 33.9 32.0 32.2 32.4
Results with ResNet-18: Table 1 shows the ef-
fect of lowering precision on the accuracy (Top-
1 error) of ResNet-18 with baseline (no teacher) Difference (!) in Top-1 error for Res-18 from baseline
and with ResNet-34, ResNet-50 and ResNet-101
asteachers. Inthetable, Adenotestheprecision 8A, 2W -3.5%
-1.5%
oftheactivationmaps(inbits)andW denotesthe
precisionoftheweights.ThebaselineTop-1error 8A, 4W -3.2%
0.8%
for full-precision ResNet-18 is 30.4%. By low-
ering the precision without using any help from 32A, 2W -3.0%
-1.0%
a teacher network, the accuracy drops by 3.5%
when using ternary and 4-bits precision (the col- 32A, 32W 0.0%
0.3%
umn corresponding to “Res-18 Baseline” in the
-6% -4% -2% 0% 2%
table). With distillation based technique, the ac-
!from 32A, 32W without Apprentice !from 32A, 32W with Apprentice
curacy of low-precision configurations improves
significantly. In fact, the accuracy of the full-
precision ResNet-18 also improves when paired Figure 3: Difference in Top-1 error rate for low-
withalargerfull-precisionResNetmodel(therow precisionvariantsofResNet-18with(bluebars)and
corresponding to “32A, 32W” in Table 1). The without (red bars) distillation scheme. The differ-
ence is calculated from the accuracy of ResNet-18
best full-precision accuracy was achieved with a
with full-precision numerics. Higher % difference
studentResNet-18andResNet-101astheteacher
denotesabetternetworkconfiguration.
(improvement by 0.35% over the baseline). The
gapbetweenfull-precisionResNet-18andthebest
achievedternaryweightResNet-18isonly1%(improvementof2%overpreviousbest). With“8A,
4W”, we find the accuracy of the student ResNet-18 model to beat the baseline accuracy. We hy-
pothesize regularization with low-precision (and distillation) to be the reason for this. “8A, 4W”
improvingtheaccuracybeyondbaselinefigureisonlyseenforResNet-18.
Figure3showsthedifferenceinTop-1errorrateachievedbyourbestlow-precisionstudentnetworks
(when trained under the guidance of a teacher network) versus not using any help from a teacher
network. Forthisfigure, thedifferenceinTop-1errorofthebestlow-precisionstudentnetworkis
calculatedfromthebaselinefull-precisionnetwork(i.e. ResNet-18with30.4%Top-1error),i.e. we
wanttoseehowclosealow-precisionstudentnetworkcancometoafull-precisionbaselinemodel.
6
PublishedasaconferencepaperatICLR2018
Wefindourlow-precisionnetworkaccuraciestosignificantlyclosethegapbetweenfull-precision
accuracy(andforsomeconfigurationsevenbeatthebaselineaccuracy).
Hintonetal.(2015)mentionimprovingthebaselinefull-precisionaccuracywhenastudentnetwork
ispairedwithateachernetwork. TheymentionimprovingtheaccuracyofasmallmodelonMNIST
dataset. We show the efficacy of distillation based techniques on a much bigger model (ResNet)
withmuchlargerdataset(ImageNet).
Table 2: Top-1 validation set error rate (%) on ImageNet-1K for ResNet-34 stu-
dent network as precision of activations (A) and weight (W) changes. The last
threecolumnsshowerrorratewhenthestudentResNet-34ispairedwithResNet-34,
ResNet-50andResNet-101.
ResNet-34 ResNet-34 ResNet-34 ResNet-34
Baseline withResNet-34 withResNet-50 withResNet-101
32A,32W 26.4 26.3 26.1 26.1
32A,2W 28.3 27.6 27.2 27.2
8A,4W 29.7 27.0 26.9 26.9
8A,2W 30.8 28.8 28.8 28.5
Table 3: Top-1 validation set error rate (%) on ImageNet-1K for
ResNet-50 student network as precision of activations (A) and
weight(W)changes. Thefinaltwocolumnsshowerrorratewhen
thestudentResNet-50ispairedwithResNet-50andResNet-101.
ResNet-50 ResNet-50 ResNet-50
Baseline withResNet-50 withResNet-101
32A,32W 23.8 23.7 23.5
32A,2W 26.1 25.4 25.3
8A,4W 28.5 25.5 25.3
8A,2W 29.2 27.3 27.2
Difference (!) in Top-1 error for Res-34 from baseline Difference (!) in Top-1 error for Res-50 from baseline
8A, 2W -4.4% 8A, 2W-5.5%
-2.1% -3.4%
8A, 4W -3.3% 8A, 4W -4.7%
-0.5% -1.5%
32A, 2W -1.9% 32A, 2W -2.3%
-0.8% -1.5%
32A, 32W 0.0% 32A, 32W 0.0%
0.3% 0.3%
-6% -4% -2% 0% 2% -6% -4% -2% 0% 2%
!from 32A, 32W without Apprentice !from 32A, 32W with Apprentice !from 32A, 32W without Apprentice !from 32A, 32W with Apprentice
(a)ApprenticeversusbaselineaccuracyforResNet-34.(b)ApprenticeversusbaselineaccuracyforResNet-50.
Figure 4: DifferenceinTop-1errorrateforlow-precisionvariantsofResNet-34andResNet-50with(blue
bars)andwithout(redbars)distillationscheme.Thedifferenceiscalculatedfromtheaccuracyofthebaseline
network(ResNet-34for(a)andResNet-50for(b))operatingatfull-precision. Higher%differencedenotesa
betternetworkconfiguration.
ResultswithResNet-34andResNet-50: Table2andTable3showtheeffectofloweringprecision
ontheaccuracyofResNet-34andResNet-50,respectively,withdistillationbasedtechnique. With
a student ResNet-34 network, we use ResNet-34, ResNet-50 and ResNet-101 as teachers. With
a student ResNet-50 network, we use ResNet-50 and ResNet-101 as teachers. The Top-1 error
for full-precision ResNet-34 is 26.4%. Our best 4-bits weight and 8-bits activation ResNet-34 is
within0.5%ofthisnumber(26.9%errorratewithResNet-34studentandResNet-50teacher). This
7
PublishedasaconferencepaperatICLR2018
significantly improves upon the previously reported error rate of 29.7%. 4-bits weight and 8-bits
activation for ResNet-50 gives us a model that is within 1.5% of full-precision model accuracy
(25.3%vs. 23.8%). Figure4aandFigure4bshowthedifferenceinTop-1errorachievedbyourbest
low-precisionResNet-34andResNet-50studentnetworks,respectively,andcompareswithresults
obtainedusingmethodsproposedinliterature. OurApprenticeschemesignificantlyclosesthegap
betweenfull-precisionbaselinenetworksandlow-precisionvariantsofthesamenetworks. Inmost
casesweseeourschemetobetterthepreviouslyreportedaccuracynumbersby1.5%-3%.
Discussion: Inscheme-A,weuseateachernetworkthatisalwaysaslargeorlargerinnumberof
parameters than the student network. We experimented with a ternary ResNet-34 student network
whichwaspairedwithafull-precisionResNet-18. TheternarymodelforResNet-34isabout8.5x
smaller in size compared to the full-precision ResNet-18 model. The final trained accuracy of the
ResNet-34 ternary model with this setup is 2.7% worse than that obtained by pairing the ternary
ResNet-34 network with a ResNet-50 teacher network. This suggests that the distillation scheme
worksonlywhentheteachernetworkishigherinaccuracythanthestudentnetwork(andnotneces-
sarilybiggerincapacity). Further,thebenefitfromusingalargerteachernetworksaturatesatsome
point. This can be seen by picking up a precision point, say “32A, 2W” and looking at the error
ratesalongtherowinTable1,2and3.
Oneconcern,wehadintheearlystagesofourinvestigation,withjointtrainingofalow-precision
smallnetworkandahighprecisionlargenetworkwastheinfluenceofthesmallnetwork’saccuracy
on the accuracy of the large network. When using the joint cost function, the smaller network’s
probabilityscoresarematchedwiththepredictionsfromtheteachernetwork.Thejointcostisadded
asatermtothetotallossfunction(equation1).Thisledustopositthatthelargernetwork’slearning
capabilitywillbeaffectedbytheinherentimpairmentinthesmallerlow-precisionnetwork.Further,
since the smaller student network learns form the larger teacher network, a vicious cycle might
formwherethestudentnetwork’saccuracywillfurtherdropbecausetheteachernetwork’slearning
capabilityisbeingimpeded. However, inpractice, wedidnotseethisphenomenonoccurring-in
eachcasewheretheteachernetworkwasjointlytrainedwithastudentnetwork,theaccuracyofthe
teachernetworkwasalwayswithin0.1%to0.2%oftheaccuracyoftheteachernetworkwithoutit
jointlysupervisingastudentnetwork. Thiscouldbebecauseofourchoiceofα,β andγ values.
InSection4,wementionedabouttemperature,τ,forSoftmaxfunctionandhyper-parametersα=1,
β =0.5andγ =0.5.Since,wetraindirectlyonthelogitsoftheteachernetwork,wedidnothaveto
experimentwiththeappropriatevalueofτ. τ isrequiredwhentrainingonthesofttargetsproduced
by the teacher network. Although we did not do extensive studies experimenting with training on
soft targets as opposed to logits, we find that τ = 1 gives us best results when training on soft
targets. Hintonetal.(2015)mentionthatwhenthestudentnetworkissignificantlysmallerthanthe
teachernetwork,smallvaluesofτ aremoreeffectivethanlargevalues.Forfewofthelow-precision
configurations, weexperimentedwithα = β = γ = 1, and, α = 0.9, β = 1andγ = 0.1or0.3.
Eachoftheseconfigurations, yieldedalowerperformancemodelcomparedtoouroriginalchoice
fortheseparameters.
Forthethirdterminequation1,weexperimentedwithamean-squarederrorlossfunctionandalsoa
lossfunctionwithlogitsfromboththestudentandtheteachernetwork(i.e. H(zT,zA)). Wedidnot
findanyimprovementinaccuracycomparedtoouroriginalchoiceofthecostfunctionformulation.
Athoroughinvestigationofthebehaviorofthenetworkswithothervaluesofhyper-parametersand
differentlossfunctionsisanagendaforourfuturework.
Overall,wefindthedistillationprocesstobequiteeffectiveingettingushighaccuracylow-precision
models. Allourlow-precisionmodelssurpasspreviouslyreportedlow-precisionaccuracyfigures.
Forexample,TTQschemeachieves33.4%Top-1errorrateforResNet-18with2-bitsweight. Our
bestResNet-18model,usingscheme-A,with2-bitsweightachieves∼31.5%errorrate,improving
themodelaccuracyby∼2%overTTQ.Similarly,theschemeinMellempudietal.(2017)achieves
29.2% Top-1 error with 2-bits weight and 8-bits activation. The best performing Apprentice net-
workatthisprecisionachieves27.2%Top-1error.ForScheme-BandScheme-C,whichwedescribe
next,Scheme-Aservesasthenewbaseline.
8
PublishedasaconferencepaperatICLR2018
5.3 SCHEME-B:DISTILLINGKNOWLEDGEFROMATEACHER
Inthisscheme,westartwithatrainedteachernetwork. ReferringbacktoFigure2,theinputimage
ispassedtoboththeteacherandthestudentnetwork,exceptthatthelearningwithback-propagation
happensonlyinthelowprecisionstudentnetworkwhichistrainedfromscratch. Thisisthescheme
used by Buciluaˇ et al. (2006) and Hinton et al. (2015) for training their student networks. In this
scheme,thefirstterminequation1zeroesoutandonlythelasttwotermsintheequationcontribute
towardthelossfunction.
1 1
0.9 0.9
ResNet-34student with Scheme-A ResNet-34student with Scheme-A
0.8 ResNet-50 teacher, Scheme-B 0.8 ResNet-50 teacher, Scheme-B
0.7 2W 32A 0.7 4W 8A
or 0.6 or 0.6
1 err 0.5 1 err 0.5
Top- 0.4 Top- 0.4
0.3 0.3
0.2 0.2
0.1 0.1
0 0
05101520253035404550556065707580859095100105 05101520253035404550556065707580859095100105
Epochs Epochs
1 1
0.9 0.9
ResNet-50student with Scheme-A ResNet-50student with Scheme-A
0.8 ResNet-101 teacher, Scheme-B 0.8 ResNet-101 teacher, Scheme-B
0.7 2W 32A 0.7 4W 8A
or 0.6 or 0.6
1 err 0.5 1 err 0.5
Top- 0.4 Top- 0.4
0.3 0.3
0.2 0.2
0.1 0.1
0 0
05101520253035404550556065707580859095100105 05101520253035404550556065707580859095100105
Epochs Epochs
Figure5: Top-1errorrateversusepochsoffourstudentnetworksusingscheme-Aandscheme-B.
With scheme-B, one can pre-compute and store the logit values for the input images on disk and
access them during training the student network. This saves the forward pass computations in the
teachernetwork. Scheme-Bmightalsohelpthescenariowhereastudentnetworkattemptstolearn
the “dark knowledge” from a teacher network that has already been trained on some private or
sensitivedata(inadditiontothedatathestudentnetworkisinterestedintrainingon).
Withscheme-A,wehadthehypothesisthatthestudentnetworkwouldbeinfluencedbynotonlythe
“darkknowledge”intheteachernetworkbutalsothepaththeteacheradoptstolearntheknowledge.
With scheme-B we find, that the student network gets to similar accuracy numbers as the teacher
networkalbeitatfewernumberofepochs.
With this scheme, the training accuracies are similar to that reported in Table 1, 2 and 3. The
low-precisionstudentnetworks,however,learninfewernumberofepochs. Figure5plotstheTop-1
errorratesforfewoftheconfigurationsfromourexperimentsuite.Ineachoftheseplots,thestudent
networkinscheme-Bconvergesaround80th-85thepochcomparedtoabout105epochsinscheme-
A.Ingeneral,wefindthestudentnetworkswithscheme-Btolearninabout10%-20%fewerepochs
thanthestudentnetworkstrainedusingscheme-A.
5.4 SCHEME-C:FINE-TUNINGTHESTUDENTMODEL
Scheme-Cisverysimilartoscheme-B,exceptthatthestudentnetworkisprimedwithfullprecision
training weights before the start of the training process. At the beginning of the training process,
theweightsandactivationsareloweredandthestudentnetworkissortoffine-tunedonthedataset.
9
PublishedasaconferencepaperatICLR2018
Similartoscheme-B,onlythefinaltwotermsinequation1comprisethelossfunctionandthelow-
precisionstudentnetworkistrainedwithback-propagationalgorithm.Since,thenetworkstartsfrom
agoodinitialpoint,comparativelylowlearningrateisusedthroughoutthetrainingprocess.Thereis
noclearrecipeforlearningrates(andchangeoflearningratewithepochs)whichworksacrossallthe
configurations.Ingeneral,wefindtrainingwithalearningrateof1e-3for10to15epochs,followed
by1e-4foranother5to10epochs,followedby1e-5foranother5epochstogiveusthebestaccuracy.
Someconfigurationsrunforabout40to50epochsbeforestabilizing. Fortheseconfigurations,we
findtrainingusingscheme-Bwithwarmstartup(trainthestudentnetworkatfull-precisionforabout
25-30 epochs before lowering the precision) to be equally good. Wu (2016) investigate a similar
scheme for binary precision on AlexNet. Our experiments show that distillation is an overkill for
AlexNet and one can get comparable accuracies using techniques proposed in (Tang et al., 2017;
Mishra et al., 2017). Further, Wu (2016) hypothesize that distillation scheme will work on larger
networks,weshowinthispaperhowtomakeitwork. Tannetal.(2017)useasimilarschemefor
AlexNet and mention starting from a non-global optimal checkpoint gives better accuracy, though
wedidnotfindthisobservationtoholdinourexperiments.
We find the final accuracy of the models obtained using scheme-C to be (marginally) better than
those obtained using scheme-A or scheme-B. Table 4 shows error rates of few configurations of
low-precisionstudentnetworkobtainedusingscheme-A(orscheme-B)andscheme-C.ForResNet-
50studentnetwork,theaccuracywithternaryweightsisfurtherimprovedby0.6%comparedtothat
obtained using scheme-A. Note that the performance of ternary networks obtained using scheme-
A are already state-of-the-art. Hence, for ResNet-50 ternary networks, 24.7% Top-1 error rate is
thenewstate-of-the-art. Withthis, ternaryResNet-50iswithin0.9%ofbaselineaccuracy(23.8%
vs. 24.7%). Similarly, with 4-bits weight and 8-bits activations, ResNet-50 model obtained using
scheme-C is 0.4% better than that obtained with scheme-A (closing the gap to be within 1.3% of
full-precisionResNet-50accuracy).
Table 4: Top-1 ImageNet-1K validation set error rate (%) with scheme-A and scheme-C for
ResNet-34andResNet-50studentnetworkswithternaryand4-bitsprecision.
32A,2W 32A,2W
withscheme-AorB withscheme-C
ResNet-34studentwithResNet-50teacher 27.2 26.9
ResNet-50studentwithResNet101teacher 25.3 24.7
8A,4W 8A,4W
withscheme-AorB withscheme-C
ResNet-34studentwithResNet-50teacher 26.9 26.8
ResNet-50studentwithResNet101teacher 25.5 25.1
Scheme-Cisusefulwhenonealreadyhasatrainednetworkwhichcanbefine-tunedusingknowl-
edgedistillationschemestoproducealow-precisionvariantofthetrainednetwork.
5.5 DISCUSSION-TERNARYPRECISIONVERSUSSPARSITY
Asmentionedearlier,low-precisionisaformofmodelcompression. Therearemanyworkswhich
target network sparsification and pruning techniques to compress a model. With ternary preci-
sion models, the model size reduces by a factor of 2/32 compared to full-precision models. With
Apprentice,weshowhowonecangetaperformantmodelwithternaryprecision. Manyworkstar-
getingnetworkpruningandsparsificationtargetafull-precisionmodeltoimplementtheirscheme.
To be comparable in model size to ternary networks, a full-precision model needs to be sparsified
by93.75%. Further,tobeeffective,asparsemodelneedstostoreakeyforeverynon-zerovaluede-
notingthepositionofthevalueintheweighttensor. Thisaddsstorageoverheadandasparsemodel
needstobeabout95%sparsetobeat-parinmemorysizeasa2-bitmodel. Notethatternarypreci-
sionalsohasinherentsparsity(zeroisatermintheternarysymboldictionary)–wefindourternary
modelstobeabout50%sparse. InworkbyWenetal.(2016)andHanetal.(2015b),sparsification
of full-precision networks is proposed but the sparsity achieved is less than 93.75%. Further, the
networkaccuracyusingtechniquesinboththeseworksleadtolargerdegradationinaccuracycom-
paredtoourternarymodels. Overall,webelieve,ourternaryprecisionmodelstobestate-of-the-art
10