Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic ...

14 Pages·2017·3.52 MB·English

DeepIoT: Compressing Deep Neural Network Structures for
Sensing Systems with a Compressor-Critic Framework
ShuochaoYao YiranZhao AstonZhang
UniversityofIllinoisUrbana UniversityofIllinoisUrbana UniversityofIllinoisUrbana
Champaign Champaign Champaign
LuSu TarekAbdelzaher
StateUniversityofNewYorkat UniversityofIllinoisUrbana
Buffalo Champaign
ABSTRACT CCSCONCEPTS
Recentadvancesindeeplearningmotivatetheuseofdeepneutral •Human-centeredcomputing→Ubiquitousandmobilecom-
networksinsensingapplications,buttheirexcessiveresourceneeds puting; •Computing methodologies →Machine learning;
onconstrainedembeddeddevicesremainanimportantimpediment. •Computer systems organization →Embedded and cyber-
Arecentlyexploredsolutionspaceliesincompressing(approximat- physicalsystems;
ingorsimplifying)deepneuralnetworksinsomemannerbefore
useonthedevice.Weproposeanewcompressionsolution,called KEYWORDS
DeepIoT, that makes two key contributions in that space. First, DeepLearning,ModelCompression,StructureCompression,Mobile
unlikecurrentsolutionsgearedforcompressingspecifictypesof Computing,InternetofThings
neuralnetworks,DeepIoTpresentsaunifiedapproachthatcom-
pressesallcommonlyuseddeeplearningstructuresforsensing ACMReferenceformat:
ShuochaoYao,YiranZhao,AstonZhang,LuSu,andTarekAbdelzaher.
applications,includingfully-connected,convolutional,andrecur-
2017.DeepIoT:CompressingDeepNeuralNetworkStructuresforSensing
rentneuralnetworks,aswellastheircombinations.Second,unlike
SystemswithaCompressor-CriticFramework.InProceedingsofSenSys’17,
solutionsthateithersparsifyweightmatricesorassumelinearstruc-
Delft,Netherlands,November6–8,2017,14pages.
turewithinweightmatrices,DeepIoTcompressesneuralnetwork
DOI:10.1145/3131672.3131675
structures into smaller dense matrices by finding the minimum
numberofnon-redundanthiddenelements,suchasfiltersanddi-
1 INTRODUCTION
mensionsrequiredbyeachlayer,whilekeepingtheperformance
Thispaperismotivatedbytheprospectofenablinga“smarter”
ofsensingapplicationsthesame.Importantly,itdoessousingan
andmoreuser-friendlycategoryofevery-dayphysicalobjectsca-
approachthatobtainsaglobalviewofparameterredundancies,
pableofperformingcomplexsensingandrecognitiontasks,suchas
whichisshowntoproducesuperiorcompression.Thecompressed
thoseneededforunderstandinghumancontextandenablingmore
modelgeneratedbyDeepIoTcandirectlyuseexistingdeeplearn-
naturalinteractionswithusersinemergingInternetofThings(IoT)
inglibrariesthatrunonembeddedandmobilesystemswithout
applications.
furthermodifications.Weconductexperimentswithfivedifferent
Present-daysensingapplicationscoverabroadrangeofareas
sensing-relatedtasksonIntelEdisondevices.DeepIoToutperforms
includinghumaninteractions[27,59],contextsensing[6,34,39,
allcomparedbaselinealgorithmswithrespecttoexecutiontime
46,54],crowdsensing[55,58],objectdetectionandtracking[7,
andenergyconsumptionbyasignificantmargin. Itreducesthe
29, 42, 53]. The recent commercial interest in IoT technologies
sizeofdeepneuralnetworksby90%to98.9%. Itisthusableto promisesaproliferationofsmartobjectsinhumanspacesatamuch
shorten execution time by 71.4% to 94.5%, and decrease energy broaderscale.Suchobjectswillideallyhaveindependentmeansof
consumptionby72.2%to95.7%.Theseimprovementsareachieved interactingwiththeirsurroundingstoperformcomplexdetection
withoutlossofaccuracy. Theresultsunderscorethepotentialof
andrecognitiontasks,suchasrecognizingusers,interprettingvoice
DeepIoTforadvancingtheexploitationofdeepneuralnetworkson
commands,andunderstandinghumancontext.Thepaperexplores
resource-constrainedembeddeddevices.
thefeasibilityofimplementingsuchfunctionsusingdeepneural
networksoncomputationally-constraineddevices,suchasIntel’s
suggestedIoTplatform:theEdisonboard1.
Theuseofdeepneuralnetworksinsensingapplicationshasre-
Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalor centlygainedpopularity.Specificneuralnetworkmodelshavebeen
classroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributed designedtofusemultiplesensorymodalitiesandextracttemporal
forprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitation
relationshipsforsensingapplications.Thesemodelshaveshown
onthefirstpage.CopyrightsforcomponentsofthisworkownedbyothersthanACM
mustbehonored.Abstractingwithcreditispermitted.Tocopyotherwise,orrepublish, significantimprovementsonaudiosensing[31],trackingandlocal-
topostonserversortoredistributetolists,requirespriorspecificpermissionand/ora ization[8,38,52,56],humanactivityrecognition[37,56],anduser
[email protected].
identification[31,56].
SenSys’17,Delft,Netherlands
©2017ACM. 978-1-4503-5459-2/17/11...$15.00
DOI:10.1145/3131672.3131675 1https://software.intel.com/en-us/iot/hardware/edison
SenSys’17,November6–8,2017,Delft,Netherlands ShuochaoYao,YiranZhao,AstonZhang,LuSu,andTarekAbdelzaher
Trainingtheneuralnetworkcanoccuronacomputationally sensingapplicaiton.Thecompressor-criticframeworkemulatesthe
capablenodeand,assuch,isnotofconcerninthispaper.Thekey ideaofthewell-knownactor-criticalgorithmfromreinforcement
impedimenttodeployingdeep-learning-basedsensingapplications learning[28],optimizingtwonetworksinaniterativemanner.
liesinthehighmemoryconsumption,executiontime,andenergy WeevaluatetheDeepIoTframeworkontheIntelEdisoncomput-
demandassociatedwithstoringandusingthetrainednetworkon ingplatform[1],whichIntelmarketsasanenablerplatformforthe
thetargetdevice. Thisleadstoincreasedinterestincompressing computingelementsofembedded“things”inIoTsystems.Wecon-
neuralnetworkstoenableexploitationofdeeplearningonlow-end ducttwosetsofexperiments.Thefirstsetconsistsofthreetasksthat
embeddeddevices. enableembeddedsystemstointeractwithhumanswithbasicmodal-
WeproposeDeepIoTthatcompressescommonlyuseddeepneu- ities,includinghandwrittentext,vision,andspeech,demonstrating
ralnetworkstructuresforsensingapplicationsthroughdeciding superioraccuracyofourproducedneuralnetworks,comparedto
theminimumnumberofelementsineachlayer. Previousillumi- othersofsimilarsize. Thesecondsetprovidestwoexamplesof
natingstudiesonneuralnetworkcompressionsparsifylargedense applyingcompressedneuralnetworkstosolvinghuman-centric
parametermatricesintolargesparsematrices[4,22,24].Incontrast, contextsensingtasks;namely,humanactivityrecognitionanduser
DeepIoTminimizesthenumberofelementsineachlayer,which identification,inaresource-constrainedscenario.
resultsinconvertingparametersintoasetofsmalldensematrices. We compare DeepIoT with other state-of-the-art magnitude-
Asmalldensematrixdoesnotrequireadditionalstorageforelement based[22]andsparse-coding-based[4]neuralnetworkcompression
indicesandisefficientlyoptimizedforprocessing[19]. DeepIoT methods.TheresourceconsumptionofresultingmodelsontheIntel
greatlyreducestheeffortofdesigningefficientneuralstructuresfor Edisonmoduleandthefinalperformanceofsensingapplicationsare
sensingapplicationsbydecidingthenumberofelementsineach estimatedforallcompressedmodels.Inallexperiments,DeepIoT
layerinamannerinformedbythetopologyoftheneuralnetwork. isshowntooutperformtheotheralgorithmsbyalargemarginin
DeepIoT borrows the idea of dropping hidden elements termsofcompressionratio,memoryconsumption,executiontime,
from a widely-used deep learning regularization method called andenergyconsumption. Intheseexperiments,whencompared
dropout[44]. Thedropoutoperationgiveseachhiddenelement withthenon-compressedneuralnetworks,DeepIoTisabletore-
a dropout probability. During the dropout process, hidden ele- ducethemodelsizeby90%to98.9%,shortentherunningtimeby
mentscanbeprunedaccordingtotheirdropoutprobabilities.Then 71.4%to94.5%,anddecreasetheenergyconsumptionby72.2%to
a“thinned”networkstructurecanbegenerated. However,these 95.7%. Whencomparedwiththestate-out-the-artbaselinealgo-
dropoutprobabilitiesareusuallysettoapre-definedvalue,such rithm,DeepIoTisabletoreducethemodelsizeby11.6%to83.2%,
as0.5. Suchpre-definedvaluesarenottheoptimalprobabilities, shortentherunningtimeby60.9%to87.9%,anddecreasetheenergy
therebyresultinginalessefficientexplorationofthesolutionspace. consumptionby64.1%to88.7%.Importantly,theseimprovements
Ifwecanobtaintheoptimaldropoutprobabilityforeachhidden areachievedwithoutlossofaccuracy. Experimentsdemonstrate
element,itbecomespossibleforustogeneratetheoptimalslimnet- thepromiseofDeepIoTinenablingresource-constrainedembedded
workstructurethatpreservestheaccuracyofsensingapplications devicestobenefitfromadvancesindeeplearning.
whilemaximallyreducingtheresourceconsumptionofsensingsys- Therestofthispaperisorganizedasfollows.Section2introduces
tems.AnimportantpurposeofDeepIoTisthustofindtheoptimal related work on optimizating sensing applications for resource-
dropoutprobabilityforeachhiddenelementintheneuralnetwork. constraineddevices.WedescribethetechnicaldetailsofDeepIoT
Noticethat,dropoutcanbeeasilyappliedtoallcommonlyused inSection3.WedescribesystemimplementationinSection4.The
neural network structures. In fully-connected neural networks, evaluationispresentedinSection5.Finally,wediscusstheresults
neuronsaredroppedineachlayer[44]; inconvolutionalneural inSection6andconcludeinSection7.
networks,filtersaredroppedineachlayer[14];andinrecurrent
2 RELATEDWORK
neuralnetworks,dimensionsarereducedineachlayer[15].This
meansthatDeepIoTcanbeappliedtoallcommonly-usedneural Akeydirectioninembeddedsensingliteraureistoenablerun-
networkstructuresandtheircombinations. ningprogressivelymoreinterestingapplicationsunderthemore
Toobtaintheoptimaldropoutprobabilitiesfortheneuralnet- pronouncedresourceconstraintsofembeddedandmobiledevices.
work,DeepIoTexploitsthenetworkparametersthemselves.From Brouwersetal. reducedtheenergyconsumptionofWi-Fibased
theperspectiveofmodelcompression, ahiddenelementthatis localizationwithanincrementalscanningstrategy[5].Hesteretal.
connectedtoredundantmodelparametersshouldhaveahigher proposedanultra-low-powerhardwarearchitectureandacompan-
probabilitytobedropped. AcontributionofDeepIoTliesinex- ionsoftwareframeworkforenergy-efficientsensingsystem[25].
ploitinganovelcompressorneuralnetworktosolvethisproblem. Ferrari et al. and Schuss et al. focused on low-power wireless
Ittakesmodelparametersofeachlayerasinput,learnsparameter communicationprotocols[13,41].Wangetal.enabledenergyeffi-
redundancies,andgeneratesthedropoutprobabilitiesaccordingly. cientreliablebroadcastbyconsideringpoorlycorrelatedlinks[49].
Since there are interconnections of parameters among different Saifullahetal. designedascalableandenergy-efficientwireless
layers,wedesignthecompressorneuralnetworktobearecurrent sensornetwork(WSN)overwhitespaces[40].Alsheikhetal.dis-
neuralnetworkthatcangloballysharetheredundancyinformation cussed about data compression in WSN with machine learning
andgeneratedropoutprobabilitieslayerbylayer. techniques[3].
Thecompressorneuralnetworkisoptimizedjointlywiththeorig- Recentstudiesfocusedoncompressingdeepneuralnetworksfor
inalneuralnetworktobecompressedthroughacompressor-critic embeddedandmobiledevices. Hanetal. proposedamagnitude-
frameworkthattriestominimizethelossfunctionoftheoriginal based compression method with fine-tuning, which illustrated
CompressingDeepNeuralNetworkStructuresforSensingSystems SenSys’17,November6–8,2017,Delft,Netherlands
DeepIoT beforeintroducingthetechnicaldetails,wefirstuseanexampleof
Original Neural Network compressinga3-layerfully-connectedneuralnetworkstructureto
illustratetheoverallpipelineofDeepIoT.Thedetailedillustrationis
Fully-Connected3
Compressor Neural Network showninFigure1.Thebasicstepsofcompressingneuralnetwork
Dropout Layer2 p(2)
structuresforsensingapplicationswithDeepIoTcanbesummarized
Fully-Connected2 Parameter2 W(2) RNN a(1s)foIlnlosewrst.operationsthatrandomlyzeroingouthiddenelements
Dropout Layer1 p(1)
withprobabilitiesp(l) calleddropout(redboxesinFigure1)
Fully-Connected1 Parameter1 W(1) RNN ilanytoerisnetexrcnluadlelaiyneprustolfaytheersoarnigdinoaultnpueutrlaaylenrestwthoartkh.aThveetihnetefirxneadl
dimensionforasensingapplication.Thisstepwillbedetailed
Figure1:OverallDeepIoTsystemframework.Orangeboxes
inSection3.1.
representdropoutoperations.Greenboxesrepresentparam-
(2) Constructthecompressorneuralnetwork.Ittakestheweight
etersoftheoriginalneuralnetwork.
promisingcompressionresults[24].Thismethodremovesweight matricesW(l) (greenboxesinFigure1)fromthelayerstobe
connectionswithlowmagnitudeiteratively;however,itrequires compressedintheoriginalneuralnetworkasinputs,learnsand
additionalimplementationofsparsematrixwithmoreresourcecon- sharestheparameterredundanciesamongdifferentlayers,and
sumption.Inaddition,theaggressivepruningmethodincreasesthe generatesoptimaldropoutprobabilitiesp(l),whichisthenfed
potentialriskofirretrievablenetworkdamage.Guoetal.proposed backtothedropoutoperationsintheoriginalneuralnetwork.
acompressionalgorithmwithconnectionsplicing,whichprovided ThisstepwillbedetailedinSection3.2.
thechanceofrehabilitationwithacertainthreshold[22]. How- (3) Iterativelyoptimizethecompressorneuralnetworkandthe
ever,thealgorithmstillfocusesonweightlevelinsteadofstructure originalneuralnetworkwiththecompressor-criticframework.
level.Otherthanthemagnitude-basedmethod,anotherseriesof Thecompressorneuralnetworkisoptimizedtoproducebetter
worksfocusedonthefactorization-basedmethodthatreducedthe dropoutprobabilitiesthatcangenerateamoreefficientnetwork
neuralnetworkcomplexitybyexploitinglow-rankstructuresin structurefortheoriginalneuralnetwork.Theoriginalneural
parameters. Dentonetal. exploitedvariousmatrixfactorization networkisoptimizedtoachieveabetterperformancewiththe
methodswithfine-tunningtoapproximatetheconvolutionaloper- moreefficientstructureforasensingapplication.Thisstepwill
ationsinordertoreducetheneuralnetworkexecutiontime[12]. bedetailedinSection3.3.
Laneetal. appliedsparsecodingbasedandmatrixfactorization Fortherestofthispaper,allvectorsaredenotedbyboldlower-
basedmethodtoreducecomplexityoffully-connectedlayerand caseletters(e.g.,xandy),andmatricesandtensorsarerepresented
convolutionallayerrespectively[4].However,factorization-based byboldupper-caseletters(e.g.,XandY).Foracolumnvectorx,the
methodsusuallyobtainlowercompressionratiocomparedwith jthelementisdenotedbyx[j].ForatensorX,thetthmatrixalong
magnitude-basedmethods,andthelow-rankassumptionmayhurt thethirdaxisisdenotedbyX··t,andtheotherslicingdenotations
thefinalnetworkperformance. Wangetal. appliedtheinforma-
aredefinedsimilarly.Thesuperscriptl inx(l) andX(l) denotethe
tionoffrequencydomainformodelcompression[51]. However, vectorandtensorforthelthlayeroftheneuralnetwork.Weuse
additionalimplementationisrequiredtospeed-upthefrequency-
calligraphicletterstodenotesets(e.g.,XandY).ForanysetX,|X|
domainrepresentations,andthemethodisnotsuitableformodern
denotesthecardinalityofX.
CNNswithsmallconvolutionfiltersizes.Hintonetal.proposeda
teacher-studentframeworkthatdistilledtheknowledgeinanen-
3.1 DropoutOperationsintheOriginalNeural
sembleofmodelsintoasinglemodel[26].However,theframework
Network
focusedmoreoncompressingmodelensembleintoasinglemodel
Dropoutiscommonlyusedasaregularizationmethodthatprevents
insteadofstructurecompression.
featureco-adaptingandmodeloverfitting.Theterm“dropout”refers
Our paper is partly inspired by deep reinforcement learning.
todroppingoutunits(hiddenandvisible)inaneuralnetwork.Since
Withtheaidofdeepneuralnetworks,reinforcementleaninghas
DeepIoTisastructurecompressionframework,wefocusmainly
achieved great success on Atari games [33], Go chess [43], and
ondroppingouthiddenunits.Thedefinitionsofhiddenunitsare
multichannelaccess[50].
distinctindifferenttypesofneuralnetworks,andwewilldescribe
Tothebestofourknowledge,DeepIoTisthefirstframeworkfor
themindetail. Thebasicideaisthatweregardneuralnetworks
neuralnetworkstructurecompressingbasedondropoutoperations
withdropoutoperationsasbayesianneuralnetworkswithBernoulli
andreducingparameterredundancies,wheredropoutoperations
variationaldistributions[14,15,44].
provideDeepIoTthechanceofrehabilitationwithacertainprobabil-
Forthefully-connectedneuralnetworks, thefully-connected
ity.DeepIoTgeneratesamoreconcisenetworkstructurefortrans-
operationwithdropoutcanbeformulatedas
planting large-scale neural networks onto resource-constrained
embeddeddevices.
z(l) ∼Bernoulli(p(l)),
[j] [j]
W˜ (l)=W(l)diag(cid:16)z(l)(cid:17),
3 SYSTEMFRAMEWORK (1)
WeintroduceDeepIoT,aneuralnetworkstructurecompression Y(l)=X(l)W˜ (l)+b(l),
frameworkforsensingapplications. Withoutlossofgenerality, X(l+1)=f(cid:16)Y(l)(cid:17).
SenSys’17,November6–8,2017,Delft,Netherlands ShuochaoYao,YiranZhao,AstonZhang,LuSu,andTarekAbdelzaher
Referto(1). Thenotationl = 1,···,Listhelayernumberin Asshownin(2),DeepIoTusesthesamevectorofBernoulliran-
thefully-connectedneuralnetwork. Foranylayerl,theweight domvariablesz(l) tocontrolthedroppingoperationsamongdiffer-
matrixisdenotedasW(l) ∈Rd(l−1)×d(l);thebiasvectorisdenoted enttimestepsineachlayer,whileindividualBernoullirandomvari-
asb(l) ∈ Rd(l); andtheinputisdenotedasX(l) ∈ R1×d(l−1). In ablesareusedfordifferentstepsintheoriginalLSTMdropout[57].
addition,f(·)isanonlinearactivationfunction. Fromtheperspectiveofstructurecompression,DeepIoTtriesto
Asshownin(1),eachhiddenunitiscontrolledbyaBernoulli prunethenumberofhiddendimensionsusedinLSTMblocks.The
randomvariable.Intheoriginaldropoutmethod,thesuccessprob- dropoutoperationofotherrecurrentneuralnetworkarchitectures,
suchasGatedRecurrentUnit(GRU),canbedesignedsimilarly.
abilitiesofp(l) canbesettothesameconstantp forallhidden
[j]
units[44],butDeepIoTusestheBernoullirandomvariablewith
3.2 CompressorNeuralNetwork
individualsuccessprobabilitiesfordifferenthiddenunitsinorder
Nowweintroducethearchitectureofthecompressorneuralnet-
tocompresstheneuralnetworkstructureinafinergranularity.
work.AswedescribedinSection1,ahiddenelementintheoriginal
Fortheconvolutionalneuralnetworks,thebasicfully-connected
neuralnetworkthatisconnectedtoredundantmodelparameters
operationisreplacedbytheconvolutionoperation[14].However,
shouldhaveahigherprobabilitytobedropped.Thereforewedesign
theconvolutioncanbereformulatedasalinearoperationasshown
(cid:40) (cid:41) thecompressorneuralnetworktotaketheweightsofanoriginal
in(1).Foranylayerl,wedenoteK(l) = K(l) fork =1,···,c(l)
k neuralnetwork{W(l)}asinputs,learntheredundanciesamong
asthesetofconvolutionalneuralnetwork(CNN)’skernels,where
Kk(l) ∈Rh(l)×w(l)×c(l−1) isthekernelofCNNwithheighth(l),width tehleemseenwtesitghhatts,caanndbegeevneenratutealdlyroupsoeudttporcoobmabpirleitsisesth{epo(lr)i}gifnoarlhniedudreanl
w(l),andchannelc(l−1).Theinputtensoroflayerl isdenotedas networkstructure.
Xˆ(l) ∈ Rhˆ(l−1)×wˆ(l−1)×c(l−1) with heighthˆ(l−1), widthwˆ(l−1), and A straightforward solution is to train an individual fully-
channelc(l−1). connected neural network for each layer in the original neural
Next,weconvertconvolvingthekernelswiththeinputintoper- network.However,sincethereareinterconnectionsamongweight
formingmatrixproduct.Weextracth(l)×w(l)×c(l−1)dimensional redundanciesindifferentlayers,DeepIoTusesavariantLSTMasthe
patchesfromtheinputXˆ(l)withstridesandvectorizethem.Collect structureofcompressortoshareandusetheparameterredundancy
thesevectorizednpatchestobetherowsofournewinputrepre- informationamongdifferentlayers.
sentationX(l) ∈Rn×(h(l)w(l)c(l−1)).Thevectorizedkernelsformthe AccordingtothedescriptioninSection3.1,theweightinlayer
columnsoftheweightmatrixW(l) ∈R(h(l)w(l)c(l−1))×c(l). l offully-connected,convolutional,orrecurrentneuralnetwork
Withthistransformation,dropoutoperationscanbeappliedto canallberepresentedasasinglematrixW(l) ∈Rdf(l)×dd(lro)p,where
convolutionalneuralnetworksaccordingto(1).Thecompositionof d(l) denotesthedimensionthatdropoutoperationisappliedand
poolingandactivationfunctionscanberegardedasthenonlinear drop
function f(·) in(1). Insteadofdroppingouthiddenelementsin df(l)denotesthedimensionoffeatureswithineachdropoutelement.
eachlayer,wedropoutconvolutionalkernelsineachlayer.From Here,weneedtonoticethattheweightmatrixofLSTMatlayerl
theperspectiveofstructurecompression,DeepIoTtriestoprune canbereshapedasW(l) ∈R4·(n(l−1)+n(l))×n(l),whered(l) =n(l)
drop
thenumberofkernelsusedintheconvolutionalneuralnetworks.
Fortherecurrentneuralnetwork,wetakeamulti-layerLong anddf(l) = 4· (n(l−1) +n(l)). Hence, w(cid:40)etak(cid:41)eweightsfromthe
ShortTermMemorynetwork(LSTM)asanexample. TheLSTM originalnetworklayerbylayer,W = W(l) withl = 1,···,L,
operationwithdropoutcanbeformulatedas astheinputofthecompressorneuralnetwork.Insteadofusinga
vanillaLSTMasthestructureofcompressor,weapplyavariant
z(l) ∼Bernoulli(p(l)), l-stepLSTMmodelshownas
[j] [j]
(cid:42)(cid:46)(cid:46)(cid:46)(cid:46)(cid:44)ogfi(cid:43)(cid:47)(cid:47)(cid:47)(cid:47)(cid:45)=(cid:42)(cid:46)(cid:46)(cid:46)(cid:46)(cid:44)ssstaiiigggnmmmh(cid:43)(cid:47)(cid:47)(cid:47)(cid:47)(cid:45)W(l)(cid:42)(cid:44)ht(hl−t(l−1))1(cid:12)(cid:12)zz((ll−)1)(cid:43)(cid:45), (2) (cid:42)(cid:46)(cid:46)(cid:46)(cid:46)(cid:44)vvvvдoif(cid:124)(cid:124)(cid:124)(cid:124)(cid:43)(cid:47)(cid:47)(cid:47)(cid:47)(cid:45)=Wc(l)W(l)Wi(l), (cid:42)(cid:46)(cid:46)(cid:46)(cid:46)(cid:44)uuuuдofi(cid:43)(cid:47)(cid:47)(cid:47)(cid:47)(cid:45)=Whhl−1,
ct(l)=f (cid:12)ct(l−)1+i(cid:12)g, (cid:42)i(cid:43) (cid:42)sigm(cid:43)(cid:42)(cid:42)vi(cid:43) (cid:42)ui(cid:43)(cid:43)
h(l)=o(cid:12)tanh(cid:16)c(l)(cid:17). (cid:46)(cid:46)(cid:46)f(cid:47)(cid:47)(cid:47)=(cid:46)(cid:46)(cid:46)sigm(cid:47)(cid:47)(cid:47)(cid:46)(cid:46)(cid:46)(cid:46)(cid:46)(cid:46)vf(cid:47)(cid:47)(cid:47)+(cid:46)(cid:46)(cid:46)uf(cid:47)(cid:47)(cid:47)(cid:47)(cid:47)(cid:47),
t t (cid:46)o(cid:47) (cid:46)sigm(cid:47)(cid:46)(cid:46)vo(cid:47) (cid:46)uo(cid:47)(cid:47) (3)
(cid:44)g(cid:45) (cid:44)tanh(cid:45)(cid:44)(cid:44)vд(cid:45) (cid:44)uд(cid:45)(cid:45)
Thenotationl =1,···,Listhelayernumberandt =1,···,T cl =f (cid:12)cl−1+i(cid:12)g,
isthestepnumberintherecurrentneuralnetwork.Element-wise (cid:16) (cid:17)
multiplicationisdenotedby(cid:12). Operatorssiдmandtanhdenote hl =o(cid:12)tanh cl ,
(cid:16) (cid:17)
sigmoidfunctionandhyperbolictangentrespectively.Thevector p(l)=pt =sigm Wo(l)hl ,
hist(lt)h∈eiRnnp(ul)tifsotrhtehoeuwtphuotleofnsetuepratlanteltawyoerrkl.aThtsetevpetc.toThrhet(0m) a=trxixt z[(lj]) ∼Bernoulli(p[(lj])).
W(l) ∈ R4n(l)×(n(l−1)+n(l)) istheweightmatrixatlayerl. Welet Referto(3),wedenotedc asthedimensionofthevariantLSTM
p[(0j]) =1,sinceDeepIoTonlydropshiddenelements. hidden state. Then W(l) ∈ Rdf(l)×dd(lro)p, Wc(l) ∈ R4×df(l), Wi(l) ∈
CompressingDeepNeuralNetworkStructuresforSensingSystems SenSys’17,November6–8,2017,Delft,Netherlands
Rdd(lro)p×dc,Wh ∈R4dc×dc,andWo(l) ∈Rdd(lro)p×dc.Thesetoftraining Thereforeanunbiasedestimatorfor(5)canbe
param(cid:40)etersofthecompresso(cid:41)rneuralnetworkisdenotedasϕ,where ∇(cid:69)ϕL=∇ϕlogµϕ(W)·L(cid:16)y,FW(x|z)(cid:17) z∼µϕ. (6)
ϕ = W(l),W(l),W ,W(l) . ThematrixW(l) istheinputmatrix
forstepcl inthiecomhpresosorneuralnetwork,whichisalsothelth ThegradientoverW(l) ∈Wis
(cid:88) (cid:16) (cid:17)
layer’sparametersoftheoriginalneuralnetworkin(1)or(2). ∇W(l)L= µϕ(W)·∇W(l)L y,FW(x|z)
ComparedwiththevanillaLSTMthatrequiresvectorizingthe z (cid:102) (cid:103) (7)
(cid:16) (cid:17)
originalweightmatrixasinputs,thevariantLSTMmodelpreserves =Ez∼µϕ ∇W(l)L y,FW(x|z) .
thestructureoforiginalweightmatrixanduseslesslearningpa- Similarly,anunbiasedestimatorfor(7)canbe
rameterstoextracttheredundancyinformationamongthedropout
(cid:16) (cid:17)
elements.Inaddition,W(l)andW(l)convertoriginalweightmatrix ∇(cid:71)W(l)L=∇W(l)L y,FW(x|z) z∼µϕ. (8)
c i
W(l) withdifferentsizesintofixed-sizerepresentations.Thebinary Nowweprovidemoredetailsof∇(cid:69)Lin(6).Althoughtheesti-
ϕ
vectorz(l) isthedropoutmaskandprobabilityp(l) isthedropout mator(6)isanunbiasedestimator,ittendstohaveahighervariance.
probabilitiesforthelthlayerintheoriginalneuralnetworkused Ahighervarianceofestimatorcanmaketheconvergenceslower.
in(1)and(2), whichisalsothestochasticdropoutpolicylearnt Therefore,variancereductiontechniquesaretypicallyrequiredto
throughobservingtheweightredundanciesoftheoriginalneural maketheoptimizationfeasibleinpractice[21,32].
network. One variance reduction technique is to subtract a constantc
(cid:16) (cid:17)
3.3 Compressor-CriticFramework from learning signalL y,FW(x|z) in (5), which still keeps the
expectationofthegradientunchanged[32]. Therefore,wekeep
In Section 3.1 and Section 3.2, we have introduced customized
(cid:16) (cid:17)
dropoutoperationsappliedontheoriginalneuralnetworksthat trackofthemovingaverageofthelearningsignalL y,FW(x|z)
needtobecompressedandthestructureofcompressorneuralnet- denotedbyc,andsubtractcfromthegradientestimator(6).
workusedtolearndropoutprobabilitiesbasedonparameterredun- Theothervariancereductiontechniqueiskeepingtrackofthe
dancies.Inthissubsection,wewilldiscussthedetailofcompressor- movingaverageo√fthesignalvariancev,anddividesthelearning
criticcompressingprocess.Itoptimizestheoriginalneuralnetwork signalbymax(1, v)[21].
and the compressor neural network in an iterative manner and Combingtheaforementionedtwovariancereductiontechniques,
enablesthecompressorneuralnetworktograduallycompressthe thefinalestimator(6)forgradientoverϕbecomes
originalneuralnetworkwithsoftdeletion. (cid:16) (cid:17)
itcWrieticd.enItotteaktehsexoraigsininapluntesuaranldngeetwneorrakteasspFrWed(ixc|tzio),nasnbdawseedcoanll ∇(cid:69)ϕL=∇ϕlogµϕ(W)· L ym,FaWx(1(,x√|zv))−c z∼µϕ, (9)
binarydropoutmaskszandmodelparametersWthatrefertoaset wherec andv arethemovingaverage(cid:16)ofmeanan(cid:17)dthemoving
ofweightsW ={W(l)}.WeassumethatFW(x|z)isapre-trained averageofvarianceoflearningsignalL y,FW(x|z) respectively.
model.Wedenotethecompressorneuralnetworkbyz∼µ (W).It AfterintroducingthebasicoptimizationprocessinDeepIoT,now
ϕ
takestheweightsofthecriticasinputsandgeneratestheprobability wearereadytodeliverthedetailsofthecompressingprocess.Com-
distributionofthemaskvectorzbasedonitsownparametersϕ.In paredwithpreviouscompressingalgorithmsthatgraduallydelete
ordertooptimizethecompressortodropouthiddenelementsin weightswithoutrehabilitation[24],DeepIoTapplies“soft”deletion
thecritic,DeepIoTfollowstheobjectivefunction bygraduallysuppressingthedropoutprobabilitiesofhiddenele-
(cid:102) (cid:16) (cid:17)(cid:103) mentswithadecayfactorγ ∈ (0,1). Duringtheexperimentsin
L=Ez∼µϕ L y,FW(x|z) Section5,wesetγ asthedefaultvalue0.5. Sinceitisimpossible
= (cid:88) µϕ(W)·L(cid:16)y,FW(x|z)(cid:17), (4) tomaketheoptimalcompressiondecisionsfromthebeginning,
suppressingthedropoutprobabilitiesinsteadofdeletingthehidden
z∼{0,1}|z|
elementsdirectlycanprovidethe“deleted”hiddenelementschanges
whereL(·,·) istheobjectivefunctionofthecritic. Theobjective
torecover. Thislessaggressivecompressionprocessreducesthe
function can be interpreted as the expected loss of the original
potentialriskofirretrievablenetworkdamageandlearningineffi-
neuralnetworkoverthedropoutprobabilitiesgeneratedbythe
ciency.
compressor.
Duringthecompressingprocess,DeepIoTgraduallyincreases
DeepIoToptimizesthecompressorandcriticinaniterativeman-
ner.Itreducestheexpectedlossasdefinedin(4)byapplyingthe the threshold of dropout probabilityτ from 0 with step ∆. The
gradientdescentmethodoncompressorandcriticiteratively.How- hidden elements with dropout probability, p(l) that is less than
[j]
ever, since there are discrete sampling operations, i.e., dropout the thresholdτ will be given decay on dropout probability, i.e.,
operations,withinthecomputationalgraph,backpropagationisnot pˆ(l) ←γ ·p(l).Therefore,theoperationincompressor(3)canbe
directlyapplicable.Thereforeweapplyanunbiasedlikelihood-ratio [j] [j]
updatedas
estimatortocalculatethegradientoverϕ[17,36]: (cid:18) (cid:19)
∇ϕL=(cid:88)∇ϕµϕ(W)·L(cid:16)y,FW(x|z)(cid:17) z[(lj]) ∼Bernoulli p[(lj]) ·γ1p[(lj])≤τ , (10)
z where1istheindicatorfunction;γ ∈ (0,1) isthedecayfactor;
=(cid:88)µϕ(W)∇ϕlogµϕ(W)·L(cid:16)y,FW(x|z)(cid:17) (5) andτ ∈[0,1)isthethreshold.Sincetheoperationofsuppressing
z (cid:102) (cid:103) dropoutprobabilitywiththepre-defineddecayfactorγ isdifferen-
(cid:16) (cid:17)
=Ez∼µϕ ∇ϕlogµϕ(W)·L y,FW(x|z) . tiable,wecanstilloptimizetheoriginalandthecompressorneural
SenSys’17,November6–8,2017,Delft,Netherlands ShuochaoYao,YiranZhao,AstonZhang,LuSu,andTarekAbdelzaher
Algorithm1Compressor-predictorcompressingprocess 4.2 Software
1: Input:pre-trainedpredictorFW(x|z) Alltheoriginalneuralnetworksforallsensingapplicationsmen-
2: Initialize:compressorµϕ(W)withparameterϕ,movingaveragec,moving tionedinSection5aretrainedontheworkstationwithNVIDIA
averageofvariancev
3: whileµϕ(W)isnotconvergentdo GeForceGTXTitanX.Forallbaselinealgorithmsmentionedin
4: z∼µϕ(W) (cid:16) (cid:16) (cid:17)(cid:17) Section5, thecompressingprocessesarealsoconductedonthe
5: c←movingAvg L y,FW(x|z) workstation.Thecompressedmodelsareexportedandloadedinto
6: v←movingVar(cid:16)L(cid:16)y,FW(x|z)(cid:17)(cid:17) theflashstorageonIntelEdisonforexperiments.
7: ϕ←ϕ−β·∇ϕlogµϕ(W)·(cid:16)L(cid:16)y,FW(x|z)(cid:17)−c(cid:17)/max(1,√v) WeinstalledtheUbilinuxoperationsystemonIntelEdisoncom-
8: endwhile puting platform [2]. Far fairness, all compressed deep learning
9: τ =0 modelsarerunthroughTheano[48]withonlyCPUdeviceonIntel
10: whilethepercentageofleftnumberofparametersinFW(x|z)islargerthanα Edison. Thematrixmultiplicationoperationsandsparsematrix
do
11: z∼µϕ(W) (cid:16) (cid:16) (cid:17)(cid:17) multiplicationoperationsareoptimizedbyBLASandSparseBLAS
12: c←movingAvg L y,FW(x|z) respectivelyduringtheimplementation. Noadditionalrun-time
(cid:16) (cid:16) (cid:17)(cid:17)
13: v←movingVar L y,FW(x|z) optimizationisappliedforanycompressedmodelandinallexperi-
(cid:16) (cid:16) (cid:17) (cid:17) √
14: ϕ←ϕ−β·∇ϕlogµϕ(W)· L y,FW(x|z) −c /max(1, v) ments.
(cid:16) (cid:17)
15: W←W−β·∇WL y,FW(x|z)
16: updatethresholdτ:τ ←τ+∆foreveryTrounds 4.3 Architecture
17: endwhile Given the original neural network structure and parameters as
18: zˆ[(lj])=1p[(lj]) >τ wellasthedeviceresourceinformation,DeepIoTcanautomatically
19: whileFW(x|zˆ)isnotcon(cid:16)vergentdo (cid:17) generateacompressedneuralnetworkthatisreadytoberunon
20: W←W−β·∇WL y,FW(x|zˆ)
21: endwhile embeddeddeviceswithsensorinputs.Thesystemfirstobtainsthe
memoryinformationfromtheembeddeddeviceandsetsthefinal
networkthrough(8)and(9). Thecompressionprocesswillstop compressedsizeoftheneuralnetworktofitinapre-configured
whenthepercentageofleftnumberofparametersinFW(x|z)is fractionofavailablememory,fromwhichtheneededcompression
smallerthanauser-definedvalueα ∈ (0,1). ratioiscomputed. Intheexperiments,wemanuallysettheratio
Afterthecompression,DeepIoTfine-tunesthecompressedmodel toexploitthecapabilityofDeepIoT.Thisratio,togetherwiththe
FW(x|zˆ),withafixedmaskzˆ,whichisdecidedbytheprevious parametersoftheoriginalmodelarethenusedtoautomatically
thresholdτ. Therefore the mask generation step in (10) will be generatethecorrespondingcompressorneuralnetworktocom-
updatedas presstheoriginalneuralnetwork.Theresultingcompressedneural
zˆ(l) =1p(l) >τ. (11) networkistransferredtotheembeddeddevice.Thismodelcanbe
[j] [j] thencalledlocallywithadatainputtodecideontheoutput.The
We summarize the compressor-critic compressing process of semanticsofinputandoutputarenotknowntothemodel.
DeepIoTinAlgorithm1.
Thealgorithmconsistsofthreeparts. Inthefirstpart(Line3 4.4 PerformanceSummary
toLine8),DeepIoTfreezesthecriticFW(x|z)andinitializesthe Welisttheresourceconsumptionnumbersofallcompressedmod-
compressor µϕ(W) accordingto(9). Inthesecondpart(Line9 elswithoutlossofaccuracygeneratedbyDeepIoTandtheircor-
toLine17),DeepIoToptimizesthecriticandcompressorjointly
responding original model in Table 1 with the form of (origi-
with the gradients calculated by (8) and (9). At the same time,
nal/compressed/reductionpercentage).Thesemodelsareexplained
DeepIoTgraduallycompressesthepredictorbysuppressingdropout
inmoredetailintheevaluation,Section5.
probabilitiesaccordingto(10).Inthefinalpart(Line18toLine21),
DeepIoTfine-tunesthecriticwiththegradientcalculatedby(8)and
Table1:Resourceconsumptionsofmodelimplementations
adeterministicdropoutmaskisgeneratedaccordingto(11).After
onIntelEdison
thesethreephases,DeepIoTgeneratesabinarydropoutmaskzˆand
thefine-tuningparametersofthecriticW.Withthesetworesults, Model Size(MB) Time(ms) Energy(mJ)
wecaneasilyobtainthecompressedmodeloftheoriginalneural LeNet5 1.72/0.04/97.6% 50.2/14.2/71.4% 47.1/12.5/73.5%
network. VGGNet 118.8/2.9/97.6% 1.5K/82.2/94.5% 1.7K/74/95.6%
Bi-LSTM 76.0/7.59/90.0% 71K/9.6K/86.5% 62.9K/8.1K/87.1%
DeepSense1 1.89/0.12/93.7% 130/36.7/71.8% 99.6/27.7/72.2%
4 IMPLEMENTATION DeepSense2 1.89/0.02/98.9% 130/25.1/80.7% 105.1/18.1/82.8%
Inthissection,webrieflydescribethehardware,software,archi-
tecture,andperformancesummaryofDeepIoT. AlthoughthemodelsgeneratedbyDeepIoTdonotusesparsema-
trixrepresentations,otherbaselinealgorithms,aswillbeintroduced
4.1 Hardware inSection5,mayusesparsematricestorepresentmodels.When
OurhardwareisbasedonIntelEdisoncomputingplatform[1].The theproportionofnon-zeroelementsinthesparsematrixislarger
IntelEdisoncomputingplatformispoweredbytheIntelAtomSoC than20%,sparsematrixmultiplicationscanevenrunslowerthan
dual-coreCPUat500MHzandisequippedwith1GBmemoryand theirnon-sparsecounterpart.Therefore,thereisatradeoffbetween
4GBflashstorage.Forfairness,allneuralnetworkmodelsarerun memoryconsumptionandexecutiontimeforsparsematriceswith
solelyonCPUduringexperiments. alargeproportionofnon-zeroelements.Inaddition,convolution
CompressingDeepNeuralNetworkStructuresforSensingSystems SenSys’17,November6–8,2017,Delft,Netherlands
operationsconductedonCPUarealsoformulatedandoptimizedas 5.2 SupportingHuman-CentricInteraction
matrixmultiplications,asmentionedinSection3.1.Therefore,the Modalities
tradeoffstillexists.ForallbaselinealgorithmsinSection5,weim-
Threebasicinteractionmodalitiesamonghumansaretext,vision,
plementboththesparsematrixversionandthenon-sparsematrix
andspeech.Inthissection,wedescribethreedifferentexperiments
version.Duringalltheexperimentswithbaselinealgorithms,we
thattestimplementationsofthesebasicinteractionmodalitieson
“cheat”,intheirfavor,bychoosingtheversionthatperformsbetter
low-enddevicesusingtrainedandcompressedneuralnetworks.
accordingtothecurrentevaluationmetrics.
Wetrainstate-of-artneuralnetworksontraditionalbenchmark
5 EVALUATION datasetsasoriginalmodels.Then,wecompresstheoriginalmodels
Inthissection,weevaluateDeepIoTthroughtwosetsofexper-
usingDeepIoTandthethreebaselinealgorithmsdescribedabove.
iments. Thefirstsetismotivatedbytheprospectofenablingfu-
Finally,wetesttheaccuracyandresourceconsumptionthatresult
turesmarterembedded“things”(physicalobjects)tointeractwith
fromusingthesecompressedmodelsontheembeddeddevice.
humansusinguser-friendlymodalitiessuchasvisualcues,hand-
writtentext, andspeechcommands, whilethesecondevaluates 5.2.1 HandwrittendigitsrecognitionwithLeNet5. Thefirsthu-
human-centriccontextsensing,suchashumanactivityrecogni- maninteractionmodalityisrecognizinghandwrittentext.Inthis
tionanduseridentification.Inthefollowingsubsections,wefirst experiment,weconsiderameaningfulsubsetofthat;namelyrec-
describethecomparisonbaselinesthatarecurrentstateoftheart ognizinghandwrittendigitsfromvisualinputs.Anexampleappli-
deepneuralnetworkcompressiontechniques.Wethenpresentthe cationthatusesthiscapabilitymightbeasmartwalletequipped
firstsetofexperimentsthatdemonstrateaccuracyandresourcede- withacameraandatipcalculator.WeuseMNIST2asourtraining
mandsobservedifIoT-stylesmartobjectsinteractedwithusersvia andtestingdataset.TheMNISTisadatasetofhandwrittendigits
naturalhuman-centricmodalitiesthankstodeepneuralnetworks thatiscommonlyusedfortrainingvariousimageprocessingsys-
compressed,fortheresource-constrainedhardware,withthehelp tems.Ithasatrainingsetof60000examples,andatestsetof10000
ofourDeepIoTframework. Finally,wepresentthesecondsetof examples.
experimentsthatdemonstrateaccurancyandresourcedemands We test our algorithms and baselines on the LeNet-5 neural
whenapplyingDeepIoTtocompressdeepneuralnetworkstrained networkmodel.Thecorrespondingnetworkstructureisshownin
forhuman-centriccontextsensingapplications.Inbothcases,we Table2. NoticethatweomitallthepollinglayersinTable2for
showsignificantadvantagesintheaccuracy/resourcetrade-offover simplicity,becausetheydonotcontaintrainingparameters.
thecomparedstate-of-the-artcompressionbaselines. ThefirstcolumnofTable2representsthenetworkstructureof
LeNet-5,where“convX”representstheconvolutionallayerand“fcY”
5.1 BaselineAlgorithms
representsthefully-connectedlayer.Thesecondcolumnrepresents
WecompareDeepIoTwithotherthreebaselinealgorithms:
thenumberofhiddenunitsorconvolutionalkernelsweusedin
(1) DyNS: This is a magnitude-based network pruning algo- eachlayer.Thethirdcolumnrepresentsthenumberofparameters
rithm [22]. The algorithm prunes weights in convolutional usedineachlayerandintotal.TheoriginalLeNet-5istrainedand
kernelsandfully-connectedlayerbasedonthemagnitude.Itre- achievesanerrorrateof0.85%inthetestdataset.
trainsthenetworkconnectionsaftereachpruningstepandhas WethenapplyDeepIoTandtwootherbaselinealgorithms,DyNS
theabilitytorecovertheprunedweights.Forconvolutionaland andSparseSep,tocompressLeNet-5.Notethat,wedonotuseDyNS-
fully-connectedlayers,DyNSsearchestheoptimalthresholds Extbecausethenetworkdoesnotcontainarecurrentlayer. The
separately. networkstatisticsofthecompressedmodelareshowninTable2.
(2) SparseSep:Thisisasparse-codingandfactorizationbasedal- DeepIoTis designedto prunethe number of hiddenunits fora
gorithm[4].Thealgorithmsimplifiesthefully-connectedlayer moreefficientnetworkstructure.Therefore,weillustrateboththe
byfindingtheoptimalcode-bookandcodebasedonasparse remainingnumberofhiddenunitsandtheproportionofthere-
codingtechnique. Fortheconvolutionallayer,thealgorithm mainingnumberofparametersinTable2.BothDeepIoTandDyNS
compressesthemodelwithmatrixfactorizationmethods.We cansignificantlycompressthenetworkwithouthurtingthefinal
greedily search for the optimal code-book and factorizaiton performance.SparseSepshowsanacceptabledropofperformance.
numberfromthebottomtothetoplayer. ThisisbecauseSparseSepisdesignedwithoutfine-tuning.Ithas
(3) DyNS-Ext:Theprevioustwoalgorithmsmainlyfocusoncom- thebenefitofnotfine-tuningthemodel,butitsuffersthelossin
pressingconvolutionalandfully-connectedlayers.Therefore thefinalperformanceatthesametime.
wefurtherenhanceandextendthemagnitude-basedmethod Thedetailedtradeoffbetweentestingaccuracyandmemorycon-
usedinDyNStorecurrentlayersandcallthisalgorithmDyNS- sumptionbythemodelisillustratedinFig2a. Wecompressthe
Ext.JustlikeDeepIoT,DyNS-Extcanbeappliedtoallcommonly originalneuralnetworkwithdifferentcompressionratiosandre-
useddeepnetworkmodules,includingfully-connectedlayers, codethefinaltestingaccuracy.Inthezoom-inillustration,DeepIoT
convolutionallayers,andrecurrentlayers.Ifthenetworkstruc- achievesatleast×2bettertradeoffcomparedwiththetwobaseline
turedoesnotcontainrecurrentlayers,weapplyDyNSinstead methods.Thisismainlyduetotworeasons.Oneisthatthecompres-
ofDyNS-Ext. sorneuralnetworkinDeepIoTobtainsaglobalviewofparameter
For magnitude-based pruning algorithms, DyNS and DyNS-Ext, redundanciesandisthereforebettercapableofeliminatingthem.
hiddenelementswithzeroinputconnectionsorzerooutputcon- TheotheristhatDeepIoTprunesthehiddenunitsdirectly,which
nectionswillbeprunedtofurthercompressthenetworkstructure.
Inaddition,allmodelsuse32-bitfloatswithoutanyquantization. 2http://yann.lecun.com/exdb/mnist/
SenSys’17,November6–8,2017,Delft,Netherlands ShuochaoYao,YiranZhao,AstonZhang,LuSu,andTarekAbdelzaher
Table2:LeNet5onMNISTdataset
Layer HiddenUnits Params DeepIoT(HiddenUnits/Params) DyNS SparseSep
conv1(5×5) 20 0.5K 10 50.0% 24.2% 84%
conv2(5×5) 50 25K 20 20.0% 20.7% 91%
fc1 500 400K 10 0.8% 1.0% 78.75%
fc2 10 5K 10 2.0% 16.34% 70.28%
total 431K 1.98% 2.35% 72.39%
TestError 0.85% 0.85% 0.85% 1.05%
100 DeepIoT 100 DeepIoT 100 DeepIoT
DyNS DyNS DyNS
SparseSep SparseSep SparseSep
80 80 80
%) %) %)
Accuracy (60 1680000 Accuracy (60 1068000 Accuracy (60 1680000
40 40 40 40 40 40
200 0.02 0.04 0.06 0.08 0.1 206 8 10 12 14 204 6 8 10 12 14
200 0.2 0.4 0.6 0.8 1 20 10 20 30 40 50 20 10 20 30 40 50
Proportion of memory consumption by the model Execution Time (ms) Inference Energy (mJ)
(a)Thetradeoffbetweentestingaccuracyand(b)Thetradeoffbetweentestingaccuracyand(c)Thetradeoffbetweentestingaccuracyand
memoryconsumptionbymodels. executiontime. energyconsumption.
Figure2:SystemperformancetradeoffforLeNet5onMNISTdataset
enablesustorepresentthecompressedmodelparameterswitha layerswithoutmakingthelow-rankassumptionthatcanhurtthe
smalldensematrixinsteadofalargesparsematrix. Thesparse networkperformance.
matrixconsumesmorememoryfortheindicesofmatrixelements. TheevaluationofenergyconsumptiononIntelEdisonisillus-
AlgorithmssuchasDyNSgeneratemodelsrepresentedbysparse tratedinFig.2c. Foreachcompressedmodel,werunitfor5000
matricesthatcauselargermemoryconsumption. timesandmeasurethetotalenergyconsumptionbyapowermeter.
Theevaluationresultsonexecutiontimeofcompressedmodels Then,wecalculatetheexpectedenergyconsumptionforone-time
onIntelEdison,areillustratedinFig.2b.Weruneachcompressed executionandusetheone-timeenergyconsumptiontogenerate
modelonIntelEdisonfor5000timesandusethemeanvaluefor thetradeoffcurvesinFig.2c.
generatingthetradeoffcurves. Notsurprisingly,DeepIoTstillachievesthebesttradeoffinthe
DeepIoTstillachievesthebesttradeoffcomparedwithothertwo evaluationonenergyconsumptionbyasignificantmargin. Itre-
baselinesbyasignificantmargin.DeepIoTtakes14.2mstomakea ducesenergyconsumptionby73.7%comparedwiththeoriginal
singleinference,whichreducesexecutiontimeby71.4%compared networkwithoutlossofaccuracy.Beingsimilarastheevaluationon
withtheoriginalnetworkwithoutlossofaccuracy.HoweverSpars- executiontime,energyconsumptionfocusesmoreonthenumberof
eSeptakeslessexecutiontimecomparedwithDyNSatthecostof operationsthanthemodelsize.Therefore,SparseSepcantakeless
acceptableperformancedegradation(around0.2%degradationon energyconsumptioncomparedwithDyNSatthecostofacceptable
testerror).Themainreasonforthisobservationisthat,eventhough lossonperformance.
fully-connectedlayersoccupythemostmodelparameters,most
executiontimeisusedbytheconvolutionoperations. SparseSep 5.2.2 ImagerecognitionwithVGGNet. Thesecondhumanin-
teractionmodalityisthroughvision.Duringthisexperiment,we
usesamatrixfactorizationmethodtocovertthe2dconvolutional
useCIFAR103 asourtrainingandtestingdataset. TheCIFAR-10
kernelintotwo1dconvolutionalkernelsontwodifferentdimen-
datasetconsistsof6000032x32colourimagesin10classes,with
sions[47].Althoughthismethodmakeslow-rankassumptionon
6000imagesperclass.Thereare50000trainingimagesand10000
convolutionalkernel,itcanspeedupconvolutionoperationsifthe
testimages.Itisastandardtestingbenchmarkdatasetfortheimage
sizeofconvolutionalkernelislarge(5×5inthisexperiment). It
recognitiontasks.Whilenotnecessarilyrepresentativeofseeing
cansometimesspeeduptheoperationevenwhentwo1dkernels
objectsinthewild,itoffersamorecontrolledenvironmentforan
havemoreparametersintotalcomparedwiththeoriginal2dker-
apples-to-applescomparison.
nel.HoweverDyNSappliesamagnitude-basedmethodthatprunes
During this evaluation, we use the VGGNet structure as our
mostoftheparametersinfully-connectedlayers.Forconvolutional
originalnetworkstructure. Itisahugenetworkwithmillionsof
layers,DyNSdoesnotreducethenumberofconvolutionalopera-
parameters. VGGNetischosentoshowthatDeepIoTisableto
tionseffectively,andsparsematrixmultiplicationislessefficient
compressrelativedeepandlargenetworkstructure.Thedetailed
comparedwithregularmatrixwiththesamenumberofelements.
structureisshownTable3.
DeepIoTdirectlyreducesthenumberofconvolutionalkernelsin
eachlayer,whichreducesthenumberofoperationsinconvolutional
3https://www.kaggle.com/c/cifar-10
CompressingDeepNeuralNetworkStructuresforSensingSystems SenSys’17,November6–8,2017,Delft,Netherlands
Table3:VGGNetonCIFAR-10dataset
Layer HiddenUnits Params DeepIoT(HiddenUnits/Params) DyNS SparseSep
conv1(3×3) 64 1.7K 27 42.2% 53.9% 93.1%
conv2(3×3) 64 36.9K 47 31.0% 40.1% 57.3%
conv3(3×3) 128 73.7K 53 30.4% 52.3% 85.1%
conv4(3×3) 128 147.5K 68 22.0% 67.0% 56.8%
conv5(3×3) 256 294.9K 104 21.6% 71.2% 85.1%
conv6(3×3) 256 589.8K 97 15.4% 65.0% 56.8%
conv7(3×3) 256 589.8K 89 13.2% 61.2% 56.8%
conv8(3×3) 512 1.179M 122 8.3% 36.5% 85.2%
conv9(3×3) 512 2.359M 95 4.4% 10.6% 56.8%
conv10(3×3) 512 2.359M 64 2.3% 3.9% 56.8%
conv11(2×2) 512 1.049M 128 3.1% 3.0% 85.2%
conv12(2×2) 512 1.049M 112 5.5% 1.7% 85.2%
conv13(2×2) 512 1.049M 149 6.4% 2.4% 85.2%
fc1 4096 2.097M 27 0.19% 2.2% 95.8%
fc2 4096 16.777M 371 0.06% 0.39% 135%
fc3 10 41K 10 9.1% 18.5% 90.2%
total 29.7M 2.44% 7.05% 112%
TestAccuracy 90.6% 90.6% 90.6% 87.1%
100 100 100
90
80 80
%)80 %) %)
Accuracy (6700 Accuracy (60 Accuracy (60
40 40
50 DeepIoT DeepIoT DeepIoT
DyNS DyNS DyNS
SparseSep SparseSep SparseSep
40 0 0.2 0.4 0.6 0.8 1 1.2 200 500 1000 1500 200 500 1000 1500
Proportion of memory consumption by the model Execution Time (ms) Inference Energy (mJ)
(a)Thetradeoffbetweentestingaccuracyand(b)Thetradeoffbetweentestingaccuracyand(c)Thetradeoffbetweentestingaccuracyand
memoryconsumptionbymodels. executiontime. energyconsumption.
Figure3:SystemperformancetradeoffforVGGNetonCIFAR-10dataset
InTable3,weillustratethedetailedstatisticsofbestcompressed timecomparedwithSparseSepinthisexperiment.Therearetwo
modelthatkeepstheoriginaltestingaccuracyforthreealgorithms. reasonsforthis. OneisthatVGGNetusesmallerconvolutional
WeclearlyseethatDeepIoTbeatstheothertwobaselinealgorithms kernel compared with LeNet-5. Therefore factorizing 2d kernel
byasignificantmargin.ThisshowsthatthecompressorinDeepIoT intotwo1dkernelhelpslessonreducingcomputationtime.The
canhandlenetworkswithrelativelydeepstructure.Thecompressor otherpointisthatSparseSepfailstocompresstheoriginalnetwork
usesavariantoftheLSTMarchitecturetosharetheredundancy intoasmallsizewhilekeepingtheoriginalperformance. Aswe
informationamongdifferentlayers.Comparedwithotherbaselines mentionedbefore,itisbecauseSparseSepavoidsthefine-tuning.
consideringonlylocalinformationwithineachlayer,sharingthe Fig.3cshowsthetradeoffbetweentestingaccuracyandenergy
globalinformationamonglayershelpsuslearnabouttheparameter consumptionfordifferentmodels. DeepIoTreducesenergycon-
redundancyandcompressthenetworkstructure.Inaddition,we sumptionby95.7%comparedwiththeoriginalVGGNetwithout
observeperformancelossinthecompressednetworkgeneratedby lossofaccuracy.Itgreatlyhelpsustodevelopalong-standingappli-
SparseSep. ItismainlyduetothefactthatSparseSepavoidsthe cationwithdeepneuralnetworkinenergy-constrainedembedded
fine-tuningstep.Thisexperimentshowsthatfine-tuning(Line18 devices.
toLine21inAlgorithm1)isimportantformodelcompression.
Fig.3ashowsthetradeoffbetweentestingaccuracyandmem-
oryconsumptionfordifferentmodels.DeepIoTachievesabetter 5.2.3 SpeechrecognitionwithdeepBidirectionalLSTM. Thethird
performancebyevenalargermargin,becausethemodelgenerated humaninteractionmodalityisspeech.Thesensingsystemcantake
byDeepIoTcanstillberepresentedbyastandardmatrix,while thevoicesofusersfromthemicrophoneandautomaticallyconvert
othermethodsthatuseasparsematrixrepresentationrequiremore whatuserssaidintotext. Theprevioustwoexperimentsmainly
memoryconsumption. focusonthenetworkstructurewithconvolutionallayersandfully-
Fig.3bshowsthetradeoffbetweentestingaccuracyandexe- connectedlayers.WeseehowDeepIoTandthebaselinealgorithms
cutiontimefordifferentmodels. DeepIoTstillachievesthebest workontherecurrentneuralnetworkinthissection.
tradeoff. DeepIoTtakes 82.2ms fora prediction, which reduces Inthisexperiment,weuseLibriSpeechASRcorpus[35]asour
94.5%executiontimewithoutthelossofaccuracy.Differentfrom trainingandtestingdataset.TheLibriSpeechASRcorpusisalarge-
theexperimentwithLeNet-5onMNIST,DyNSuseslessexecution scalecorpusofreadEnglishspeech.Itconsistsof460-hourtraining
dataand2-hourtestingdata.
SenSys’17,November6–8,2017,Delft,Netherlands ShuochaoYao,YiranZhao,AstonZhang,LuSu,andTarekAbdelzaher
Table4:DeepbidirectionalLSTMonLibriSpeechASRcorpus
Layer HiddenUnit Params DeepIoT(HiddenUnits/Params) DyNS-Ext
LSTMf1 LSTMb1 512 512 1.090M 1.090M 55 20 10.74% 3.91% 34.9% 18.2%
LSTMf2 LSTMb2 512 512 2.097M 2.097M 192 71 4.03% 0.54% 37.2% 23.1%
LSTMf3 LSTMb3 512 512 2.097M 2.097M 240 76 17.58% 2.06% 43.1% 27.9%
LSTMf4 LSTMb4 512 512 2.097M 2.097M 258 81 23.62% 2.35% 52.3% 40.2%
LSTMf5 LSTMb5 512 512 2.097M 2.097M 294 90 28.93% 2.78% 72.6% 61.8%
fc1 29 59.3K 29 37.5% 69.0%
total 19.016M 9.98% 37.1%
Worderrorrate(WER) 9.31 9.20 9.62
35 35 35
DeepIoT DeepIoT DeepIoT
DyNS−Ext DyNS−Ext DyNS−Ext
30 30 30
Rate25 Rate25 Rate25
Error 20 Error 20 Error 20
Word 15 Word 15 Word 15
10 10 10
5 5 5
0 0.2 0.4 0.6 0.8 1 0 20 40 60 80 0 10 20 30 40 50 60 70
Proportion of memory consumption by the model Execution Time (s) Inference Energy (J)
(a)Thetradeoffbetweenworderrorrateand(b)Thetradeoffbetweenworderrorrateand(c)Thetradeoffbetweenworderrorrateand
memoryconsumptionbymodels. executiontime. energyconsumption.
Figure4:SystemperformancetradeofffordeepbidirectionalLSTMonLibriSpeechASRcorpus
WechoosedeepbidirectionalLSTMastheoriginalmodel[20]in Fig.4bshowsthetradeoffbetweenworderrorrateandexecution
thisexperiment.Ittakesmelfrequencycepstralcoefficient(MFCC) time.DeepIoTreducesexecutiontimeby86.4%withoutdegradation
featuresofvoicesasinputs,andusestwo5-layerlongshort-term onWERcomparedwiththeoriginalnetwork.Withtheevaluation
memory (LSTM) in both forward and backward direction. The onIntelEdision,theoriginalnetworkrequires71.15secondsin
outputoftwoLSTMarejointlyusedtopredictthespokentext.The average to recognize one human speak voice example with the
detailednetworkstructureisshowninthefirstcolumnofTable4, averagelengthof7.43seconds.Thecompressedstructuregenerated
where“LSTMf”denotestheLSTMinforwarddirectionand“LSTMb” byDeepIoTreducestheaverageexecutiontimeto9.68seconds
denotestheLSTMinbackwarddirection. withoutperformanceloss,whichimprovesresponsivenessofhuman
Twobaselinealgorithmsarenotapplicabletotherecurrentneu- voicerecognition.
ralnetwork,sowecomparedDeepIoTonlywithSyNS-Extinthis Fig.4cshowsthetradeoffbetweenworderrorrateandenergy
experiment.Theworderrorrate(WER),definedastheeditdistance consumption.DeepIoTreducesenergyby87%comparedwiththe
betweenthetruewordsequenceandthemostprobablewordse- original network. It performs better than DyNS-Ext by a large
quencepredictedbytheneuralnetwork,isusedastheevaluation margin.
metricforthisexperiment.
5.3 SupportingHuman-CentricContext
Weshowthedetailedstatisticsofbestcompressedmodelthat
keepstheoriginalWERinTable4.DeepIoTachievesasignificantly Sensing
bettercompressionratecomparedwithDyNS-Ext,andthemodel Inadditiontoexperimentsaboutsupportingbasichuman-centric
generatedbyDeepIoTevenhasalittleimprovementonWER.How- interaction modalities, we evaluate DeepIoT on several human-
ever,comparedwiththeprevioustwoexamplesonconvolutional centric context sensing applications. We compress the state-of-
neuralnetwork,DeepIoTfailstocompressthemodeltolessthan the-artdeeplearningmodel,DeepSense,[56]fortheseproblems
5%oftheoriginalparametersintherecurrentneuralnetworkcase andevaluatetheaccuracyandothersystemperformanceforthe
(stilla20-foldreductionthough).Themainreasonisthatcompress- compressed networks. DeepSense contains all commonly used
ingrecurrentnetworksneedstopruneboththeoutputdimension modules,includingconvolutional,recurrent,andfully-connected
andthehiddendimension. Ithasbeenshownthatdroppinghid- layers, whichisalsoagoodexampletotesttheperformanceof
dendimensioncanharmthenetworkperformance[57].However compressionalgorithmsonthecombinationofdifferenttypesof
DeepIoTisstillsuccessfulincompressingnetworktolessthan10% neuralnetworkmodules.
ofparameters. Twohuman-centriccontextsensingtasksweconsiderarehetero-
Fig.4ashowsthetradeoffbetweenworderrorrateandmemory geneoushumanactivityrecognition(HHAR)anduseridentification
consumptionbycompressedmodels.DeepIoTachievesaround×7 withbiometricmotionanalysis(UserID).TheHHARtaskrecognizes
bettertradeoffcomparedwithmagnitude-basedmethod,DyNS-Ext. humanactivitieswithmotionsensors,accelerometerandgyroscope.
Thismeanscompressingrecurrentneuralnetworksrequiresmore “Heterogeneous”meansthatthetaskisfocusonthegeneralization
informationaboutparameterredundancieswithinandamongeach abilitywithhumanwhohasnotappearedinthetrainingset.The
layer.Compressionusingonlylocalinformation,suchasmagnitude UserIDtaskidentifiesusersduringaperson’sdailyactivitiessuch
information,willcausedegradationinthefinalperformance. aswalking,running,siting,andstanding.