Neurogenesis-Inspired Dictionary Learning: Online Model Adaption in a Changing World

6 MB·

by  Sahil Garg

Neurogenesis-InspiredDictionaryLearning
NEUROGENESIS-INSPIRED DICTIONARY LEARNING:
ONLINE MODEL ADAPTION IN A CHANGING WORLD
SahilGarg
TheDepartmentofComputerScience,UniversityofSouthernCalifornia,LosAngeles,CAUSA
[email protected]
IrinaRish,GuillermoCecchi,AurelieLozano
7 IBMThomasJ.WatsonResearchCenter,YorktownHeights,NYUSA
1 rish, gcecchi, aclozano @us.ibm.com
0 { }
2

b
e ABSTRACT
F
In this paper, we focus on online representation learning in non-stationary envi-
9
1 ronments which may require continuous adaptation of model’s architecture. We
proposeanovelonlinedictionary-learning(sparse-coding)frameworkwhichin-

] corporatestheadditionanddeletionofhiddenunits(dictionaryelements),andis
G inspired by the adult neurogenesis phenomenon in the dentate gyrus of the hip-
L pocampus,knowntobeassociatedwithimprovedcognitivefunctionandadapta-
. tiontonewenvironments.Intheonlinelearningsetting,wherenewinputinstances
s arrivesequentiallyinbatches,the“neuronalbirth”isimplementedbyaddingnew
c
units with random initial weights (random dictionary elements); the number of
[
newunitsisdeterminedbythecurrentperformance(representationerror)ofthe

2 dictionary,highererrorcausinganincreaseinthebirthrate.“Neuronaldeath”is
v implementedbyimposingl /l -regularization(groupsparsity)onthedictionary
1 2
6 within the block-coordinate descent optimization at each iteration of our online
0
alternatingminimizationscheme,whichiteratesbetweenthecodeanddictionary
1
updates. Finally, hidden unit connectivity adaptation is facilitated by introduc-
6
ing sparsity in dictionary elements. Our empirical evaluation on several real-life
0
datasets(imagesandlanguage)aswellasonsyntheticdatademonstratesthatthe
.
1 proposed approach can considerably outperform the state-of-art fixed-size (non-
0 adaptive) online sparse coding of Mairal et al. (2009) in the presence of non-
7 stationarydata.Moreover,weidentifycertainpropertiesofthedata(e.g.,sparse
1
inputs with nearly non-overlapping supports) and of the model (e.g., dictionary
:
v sparsity)associatedwithsuchimprovements.
i
X
r 1 INTRODUCTION
a
Theabilitytoadapttoachangingenvironmentisessentialforsuccessfulfunctioninginbothnatural
andartificialintelligentsystems.Inhumanbrains,adaptationisachievedvianeuroplasticity,which
takesdifferentforms,includingsynapticplasticity,i.e.changingconnectivitystrengthamongneu-
rons,andneurogenesis,i.e.thebirthandmaturationofnewneurons(accompaniedwiththedeathof
someneworoldneurons).Particularly,adultneurogenesis(Kempermann,2006)(i.e.,neurogenesis
in the adult brain) in the dentate gyrus of the hippocampus is associated with improved cognitive
functions such as pattern separation (Sahay et al., 2011), and is often implicated as a “candidate
mechanismforthespecificdynamicandflexibleaspectsoflearning”(Stuchlik,2014).
Inthemachine-learningcontext,synapticplasticityisanalogoustoparametertuning(e.g.,learning
neural net weights), while neurogenesis can be viewed as an online model selection via addition
(and deletion) of hidden units in specific hidden-variable models used for representation learning
(where hidden variables represent extracted features), from linear and nonlinear component anal-
ysis methods such as PCA, ICA, sparse coding (dictionary learning), nonlinear autoencoders, to
deepneuralnetsandgeneralhidden-factorprobabilisticmodels.However,optimalmodelselection
inlarge-scalehidden-variablemodels(e.g.,adjustingthenumberoflayers,hiddenunits,andtheir
1
Neurogenesis-InspiredDictionaryLearning
connectivity),isintractableduetoenormoussearchspacesize.Growingamodelgraduallycanbea
morefeasiblealternative;afterall,everyrealbrain’s“architecture”developmentprocessstartswith
asinglecell.Furthermore,theprocessofadaptingthemodel’sarchitecturetodynamicallychanging
environmentsisnecessaryforachievingalifelong,continuallearning.Finally,anonlineapproach
todynamicallyexpandingandcontractingmodel’sarchitecturecanserveasapotentiallymoreef-
fectivealternativetothestandardoff-linemodelselection(e.g.,MDL-basedoff-linesparsecoding
(Ramirez & Sapiro, 2012)), as well as to the currently popular network compression (distillation)
approaches(Hintonetal.,2015;Srivastavaetal.,2014;Ba&Caruana,2014;Buciluetal.,2006),
where a very large-scale architecture, such as a deep neural network with millions of parameters,
mustbefirstselectedinad-hocwaysandtrainedonlargeamountsofdata,onlytobecompressed
latertoamorecompactandsimplermodelwithsimilarlygoodperformance;wehypothesizethat
adaptivegrowthandreductionofthenetworkarchitectureisaviablealternativetothedistillation
approach,althoughdevelopingsuchanalternativeremainsthetopicoffurtherresearch.
Inthispaper,wefocusondictionarylearning,a.k.a.sparsecoding(Olshausen&Field,1997;Kreutz-
Delgado et al., 2003; Aharon et al., 2006; Lee et al., 2006) – a representation learning approach
which finds a set of basis vectors (atoms, or dictionary elements) and representations (encodings)
of the input samples as sparse linear combinations of those elements1. More specifically, our ap-
proachbuildsuponthecomputationallyefficientonlinedictionary-learningmethodofMairaletal.
(2009),wherethedatasamplesareprocessedsequentially,oneatatime(orinsmallbatches).Online
approachesareparticularlyimportantinlarge-scaleapplicationswithmillionsofpotentialtraining
samples, where off-line learning can be infeasible; furthermore, online approaches are a natural
choiceforbuildingsystemscapableofcontinual,lifelonglearning.
Herein, we propose a novel online dictionary learning approach inspired by adult neurogenesis,
whichextendsthestate-of-artmethodofMairaletal.(2009)tononstationaryenvironmentsbyin-
corporatingonlinemodeladaption,i.e.theadditionanddeletionofdictionaryelements(i.e.,hidden
units) in response to the dynamically changing properties of the input data2. More specifically, at
eachiterationofonlinelearning(i.e.,foreverybatchofdatasamples),weaddagroupofrandom
dictionaryelements(modelingneuronalbirth),wherethegroupsizedependsonthecurrentrepre-
sentationerror,i.e.themismatchbetweenthenewinputsamplesandtheirapproximationbasedon
thecurrentdictionary:highererrortriggersmoreneurogenesis.Theneuronaldeath,whichinvolves
removing “useless”dictionary elements,is implementedas an l /l group-sparsity regularization;
1 2
this step is essential in neurogenesis-inspired learning, since it reduces a potentially uncontrolled
growth of the dictionary, and helps to avoid overfitting (note that neuronal death is also a natural
partoftheadultneurogensisprocess,whereneuronalsurvivaldependsonmultiplefactors,includ-
ingthecomplexityofalearningenvironment(Kempermann,2006)).Moreover,weintroducespar-
sity in dictionary elements, which reflects sparse connectivity between hidden units/neurons and
theirinputs;thisisamorebiologicallyplausibleassumptionthanthefully-connectedarchitecture
ofstandarddictionarylearning,anditalsoworksbetterinourexperiments.Thus,adaptationinour
model involves not only the addition/deletion of the elements, but adapting their connectivity as
well.
Wedemonstrateonbothsimulateddataandontworeal-lifedatasets(naturalimagesandlanguage
processing) that, in presence of a non-stationary input, our approach can significantly outperform
non-adaptive,fixed-dictionary-sizeonlinemethodofMairaletal.(2009).Moreover,weidentifycer-
taindatapropertiesandparametersettingsassociatedwithsuchimprovements.Finally,wedemon-
stratethatthenovelapproachnotonlyimprovestherepresentationaccuracy,butalsocanboostthe
classificationaccuracybasedontheextractedfeatures.
Note that, although the group-sparsity constraint enforcing deletion of some dictionary elements
wasintroducedearlierinthegroup-sparsecodingmethodofBengioetal.(2009),itwasonlyim-
plementedandtestedintheoff-lineratherthanonlinesetting,and,mostimportantly,itwasnotac-
1Notethatthecorrespondingneuralnetworkinterpretationofsparsecodingframeworkisa(single-hidden-
layer)linearautoencoderwithsparsityconstraints:thehiddenunitsareassociatedwithdictionaryelements,
eachelementrepresentedbyaweightvectorassociatedwithunit’soutgoinglinksintheoutputlayer,andthe
sparsevectorofhiddenunitactivationscorrespondingtotheencodingofaninput.
2Anearlyversionofourneurogeneticonlinedictionarylearningapproachwaspresentedasaposteratthe
2011SocietyforNeurosciencemeeting(Rishetal.,2011),althoughitdidnotappearbeforeasapeer-reviewed
publication.
2
Neurogenesis-InspiredDictionaryLearning
companiedbytheneurogenesis.Ontheotherhand,whilesomepriorworkconsideredonlinenode
additioninhidden-variablemodels,andspecifically,inneuralnetworks,fromcascadecorrelations
(Fahlman&Lebiere,1989)totherecentworkbyDraelosetal.(2016a;b),nomodelpruningwas
incorporatedinthoseapproachesinordertobalancethemodelexpansion.Overall,wearenotaware
ofanypriorworkwhichwouldproposeandsystematicallyevaluate,empiricallyandtheoretically,a
dynamicprocessinvolvingbothadditionanddeletionofhiddenunitsintheonlinemodelselection
setting,eitherinsparsecodingorinaneuralnetworksetting.
Tosummarize,themaincontributionsofthispaperareasfollows:
we propose a novel online model-selection approach to dictionary learning3, inspired by
•
theadultneurogenesisphenomenon;ourmethodsignificantlyoutperformsthestate-of-art
baseline,especiallyinnon-stationarysettings;
we perform an extensive empirical evaluation, on both synthetic and real data, in order
•
to identify the conditions when the proposed adaptive approach is most beneficial, both
fordatareconstructionandforclassificationbasedonextractedfeatures;weconcludethat
these conditions include a combination of sparse dictionary elements (and thus a more
biologically plausible sparse network connectivity as opposed to fully connected units),
accompaniedbysufficientlydensecodes;
furthermore,weprovideanintuitivediscussion,aswellastheoreticalanalysisofcertain
•
combinations of the input data properties and the algorithm’s parameters when the pro-
posedapproachismostbeneficial;
from the neuroscientific perspective, we propose a computational model which supports
•
earlier empirical observations indicating that adult neurogenesis is particularly beneficial
inchangingenvironments,andthatcertainamountofneuronaldeath,whichaccompanies
theneuronalbirth,isanimportantcomponentofanefficientneurogenesisprocess;
overall, to the best of our knowledge, we are the first to perform an in-depth evaluation
•
oftheinterplaybetweenthebirthanddeathofhiddenunitsinthecontextofonlinemodel
selectioninrepresentationlearning,and,morespecifically,inonlinedictionarylearning.
This paper is organized as follows. In Sec. 2, we summarize the state-of-art non-adaptive (fixed-
size) online dictionary learning method of Mairal et al. (2009). Thereafter, in Sec. 3, we describe
ouradaptiveonlinedictionarylearningalgorithm.InSec.4,wepresentourempiricalresultsonboth
syntheticandrealdatasets,includingimagesandlanguagedata.Next,inSec.5,weprovidesome
theoretical, as well as an intuitive analysis of settings which can benefit most from our approach.
Finally,weconcludewithasummaryofourcontributionsinSec.6.Theimplementationdetailsof
thealgorithmsandadditionalexperimentalresultsaredescribedintheAppendix.
2 BACKGROUND ON DICTIONARY LEARNING
Traditional off-line dictionary learning (Olshausen & Field, 1997; Aharon et al., 2006; Lee et al.,
2006) aims at finding a dictionary D Rm×k, which allows for an accurate representation of a
training data set X = x1, ,xn ∈Rm , where each sample xi is approximated by a linear
combination xi Dα{i of t·h·e· colum∈ns of}D, called dictionary elements d1, ,dk Rm .
≈ { ··· ∈ }
Hereα istheencoding(codevector,orsimplycode)ofx inthedictionary.Dictionarylearning
i i
isalsoreferredtoassparsecoding,sinceitisassumedthatthecodevectorsaresparse,i.e.havea
relativelysmallnumberofnonzeros;theproblemisformulatedasminimizingtheobjective
n
1 1
f (D)= x Dα 2+λ α (1)
n n 2|| i− i||2 c|| i||1
i=1
X
wherethefirsttermisthemeansquareerrorlossincurredduetoapproximatingtheinputsamples
bytheirrepresentationsinthedictionary,andthesecondtermisthel -regularizationwhichenforces
1
thecodestobesparse.Thejointminimizationoff (D)withrespecttothedictionaryandcodesis
n
non-convex;thus,acommonapproachisalternatingminimizationinvolvingconvexsubproblemsof
findingoptimalcodeswhilefixingadictionary,andviceversa.
3TheMatlabcodeisavailableathttps://github.com/sgarg87/neurogenesis_inspired_
dictionary_learning.
3
Neurogenesis-InspiredDictionaryLearning
However,theclassicaldictionarylearningdoesnotscaletoverylargedatasets;moreover,itisnot
immediatelyapplicabletoonlinelearningfromacontinuousstreamofdata.Theonlinedictionary
learning(ODL)methodproposedbyMairaletal.(2009)overcomesbothoftheselimitations,and
servesasabasisforourproposedapproach,presentedinAlg.1inthenextsection.Whilethehigh-
lightedlinesinAlg.1representourextensionofODL,thenon-highlightedonesarecommontoboth
approaches,andarediscussedfirst.ThealgorithmsstartwithsomedictionaryD0,e.g.arandomly
initialized one (other approaches include using some of the inputs as dictionary elements (Mairal
etal.,2010;Bengioetal.,2009)).Ateachiterationt,bothonlineapproachesconsiderthenextinput
sample x (more generally, a batch of samples) as in the step 3 of Alg. 1 and compute its sparse
t
codeα bysolvingtheLASSO(Tibshirani,1996)problem(thestep4inAlg.1),withrespecttothe
t
current dictionary. In Alg. 1, we simply use D instead of D(t) to simplify the notation. Next, the
standardODLalgorithmcomputesthedictionaryupdate,D(t),byoptimizingthesurrogateobjec-
tivefunctionfˆ(D)whichisdefinedjustastheoriginalobjectiveineq.(1),forn=t,butwithone
t
importantdifference:unliketheoriginalobjective,whereeachcodeα forsamplex iscomputed
i i
with respect to the same dictionary D, the surrogate function includes the codes α ,α , ,α
1 2 t
···
computed at the previous iterations, using the dictionaries D(0),...,D(t−1), respectively; in other
words, it does not recompute the codes for previously seen samples after each dictionary update.
This speeds up the learning without worsening the (asymptotic) performance, since the surrogate
objectiveconvergestotheoriginalonein(1),undercertainassumptions,includingdatastationarity
(Mairal et al., 2009). Note that, in order to prevent the dictionary entries from growing arbitrarily
large,Mairaletal.(2009;2010)imposethenormconstraint,i.e.keepthecolumnsofDwithinthe
convex set C = {D ∈ Rm×k s.t. ∀j dTjuj ≤ 1}. Then the dictionary update step computes
D(t) =argmin fˆ(D),ignoringl -regularizeroverthecodewhichisfixedatthisstep,as
D t 1
∈C
t
1 1 1
argmin x Dα 2 =argmin Tr(DTDA) Tr(DTB), (2)
D t 2|| i− i||2 D 2 −
∈C i=1 ∈C
X
whereA= t α αT andB = t x αT arethe“bookkeeping”matrices(wealsocallthem
i=1 i i i=1 i i
“memories”ofthemodel),compactlyrepresentingtheinputsamplesandencodinghistory.Ateach
iteration,oncPethenewinputsamplPex isencoded,thematricesareupdatedasA A+α αT
andB B+x αT (seethestep11oifAlg.1).In(Mairaletal.,2009;2010),abl←ockcoorditnatte
← t t
descentisusedtooptimizetheconvexobjectiveineq.2;ititeratesoverthedictionaryelementsina
fixedsequence,optimizingeachwhilekeepingtheothersfixedasshownineq.(3)(essentially,the
steps14and17inAlg.1;theonlydifferenceisthatourapproachwilltransformu intow inorder
j j
toimposeadditionalregularizerbeforecomputingstep17),untilconvergence.
uj bj − k6=jdkajk; dj uj (3)
← a ← max(1, u )
P jj || j||2
Herein,whentheoff-diagonalentriesa inAareaslargeasthediagonala ,thedictionaryele-
jk jj
mentsget“tied”toeachother,playingcomplementaryrolesinthedictionary,therebyconstraining
theupdatesofeachother.
It is important to note that, for the experiment settings where we consider dictionary elements to
be sparse in our algorithm NODL (discussed next in Sec. 3), we will actually use as a baseline
algorithmamodifiedversionofthefixed-sizeODL,whichallowsforsparsedictionaryelements,i.e.
includes the sparsification step 15 in Alg. 1, thus optimizing the following objective in dictionary
updatestepinsteadoftheoneineq.(2):
t
1 1
argmin x Dα 2+ λ d . (4)
D t 2|| i− i||2 j|| j||1
∈C i=1 j
X X
From now on, ODL will refer to the above extended version of the fixed-size method of Mairal
etal.(2009)whereverwehavesparsityindictionaryelements(otherwise,thestandardmethodof
Mairal et al. (2009) is the baseline); in our experiments, dictionary sparsity of both the baseline
and the proposed method (discussed in the next section) will be matched. Note that Mairal et al.
(2010)mentionthattheconvergenceguarantiesforODLholdevenwiththesparsityconstraintson
dictionaryelements.
4
Neurogenesis-InspiredDictionaryLearning
3 OUR APPROACH: NEUROGENIC ONLINE DICTIONARY LEARNING (NODL)
Our objective is to extend the state-of-art online dictionary learning, designed for stationary input
distributions,toamoreadaptiveframeworkcapableofhandlingnonstationarydataeffectively,and
learningtorepresentnewtypesofdatawithoutforgettinghowtorepresenttheoldones.Towardsthis
end, we propose a novel algorithm, called Neurogenetic Online Dictionary Learning (see Alg. 1),
whichcanflexiblyextendandreduceadictionaryinresponsetothechangesinaninputdistribution,
andpossiblytotheinherentrepresentationcomplexityofthedata.Themainchanges,ascomparedto
thenon-adaptive,fixed-dictionary-sizealgorithmofMairaletal.(2009),arehighlightedinAlg.1;
the two parts involve (1) neurogenesis, i.e. the addition of dictionary elements (hidden units, or
“neurons”)and(2)thedeathofoldand/ornewelementswhichare“lessuseful”thanotherelements
forthetaskofdatareconstruction.
At each iteration in Alg. 1, the next batch of samples is received and the corresponding codes, in
the dictionary, are computed; next, we add k new dictionary elements sampled at random from
n
Rm (i.e., kn random linear projections of the input sample). The choice of the parameter kn is
important;oneapproachistotuneit(e.g.,bycross-validation),whileanotheristoadjustitdynam-
ically,basedonthedictionaryperformance:e.g.,iftheenvironmentischanging,theolddictionary
maynotbeabletorepresentthenewinputwell,leadingtodeclineintherepresentationaccuracy,
which triggers neurogenesis. Herein, we use as the performance measure the Pearson correlation
betweenanewsampleanditsrepresentationinthecurrentdictionaryr(xt,D(t−1)αt),i.e.denoted
aspc(xt,D(t−1),αt)(forabatchofdata,theaverageoverpc(.)istaken).Ifitdropsbelowacertain
pre-specifiedthresholdγ (where0 γ 1),theneurogenesisistriggered(thestep5inAlg.1).
≪ ≤
Thenumberk ofnewdictionaryelementsisproportionaltotheerror1 p (),sothatworseper-
n c
− ·
formancewilltriggermoreneurogenesis,andviceversa;themaximumnumberofnewelementsis
bounded by c (the step 6 in Alg. 1). We refer to this approach as conditional neurogenesis as it
k
involvestheconditionalbirthofnewelements.Next,k randomelementsaregeneratedandadded
n
tothecurrentdictionary(thestep7),andthememorymatricesA,B areupdated,respectively,to
accountforlargerdictionary(thestep8).Finally,thesparsecodeisrecomputedforx (or,allthe
t
samplesinthecurrentbatch)withrespecttotheextendeddictionary(thestep9).
Thenextstepisthedictionaryupdate,whichuses,similarlytothestandardonlinedictionarylearn-
ing, the block-coordinate descent approach. However, the objective function includes additional
regularizationterms,ascomparedto(2):
t
1 1
D(t)=argmin x Dα 2+λ d + λ d . (5)
D t 2|| i− i||2 g || j||2 j|| j||1
∈C i=1 j j
X X X
Thefirsttermisthestandardreconstructionerror,asbefore.Thesecondterm,l /l -regularization,
1 2
promotesgroupsparsityoverthedictionaryentries,whereeachgroupcorrespondstoacolumn,i.e.
a dictionary element. The group-sparsity (Yuan & Lin, 2006) regularizer causes some columns in
D to be set to zero (i.e. the columns less useful for accurate data representation), thus effectively
eliminatingthecorrespondingdictionaryelementsfromthedictionary(“killing”thecorresponding
hidden units). As it was mentioned previously, Bengio et al. (2009) used the l /l -regularizer in
1 2
dictionarylearning,thoughnotinonlinesetting,andwithoutneurogenesis.
Finally,thethirdtermimposesl -regularizationondictionaryelementsthuspromotingsparsedic-
1
tionary,besidesthesparsecoding.Introducingsparsityindictionaryelements,correspondingtothe
sparseconnectivityofhiddenunitsintheneuralnetrepresentationofadictionary,ismotivatedby
both their biological plausibility (neuronal connectivity tends to be rather sparse in multiple brain
networks),andbythecomputationaladvantagesthisextraregularizationcanprovide,asweobserve
laterinexperimentssection(Sec.4).
AsintheoriginalalgorithmofMairaletal.(2009),theaboveobjectiveisoptimizedbytheblock-
coordinatedescent,whereeachblockofvariablescorrespondstoadictionaryelement,i.e.,acolumn
inD;theloopinsteps12-19oftheAlg.1iteratesuntilconvergence,definedbythemagnitudeof
change between the two successive versions of the dictionary falling below some threshold. For
eachcolumnupdate,thefirstandthelaststeps(thesteps14and17)arethesameasintheoriginal
methodofMairaletal.(2009),whilethetwointermediatesteps(thesteps15and16)areimplement-
ingadditionalregularization.Bothsteps15and16(sparsityandgroupsparsityregularization)are
5
Neurogenesis-InspiredDictionaryLearning
Algorithm1NeurogeneticOnlineDictionaryLearning(NODL)
Reqnueiruer:ogDenaetasisstrteharemshxo1ld,,xγ2;,m··a·x,nxunmb∈erRomfn;eiwniteialelmdiecntitosnaadrdyeDdpe∈rRdamta×bka;tccho,ncdkit;iognroaulpsparsityregularization
parameter,λg;numberofnon-zerosinadictionaryelement,βd;numberofnon-zerosinacode,βc.
1: Initialize:A 0,B 0 % reset the ‘‘memory’’
% assumin←g sing←le data in a batch, for the simpler exposition
2: fort=1tondo
3: Inputxt % representing the tth batch of data
% Sparse coding of data:
4: αt=argα∈Rkmin21||xt−Dα||22+λc||α||1 % λc tuned to have βc non-zeros in αt
% Conditional neurogenesis: if accuracy below threshold, add more
elements (should not be more than the number of data in a batch
5: ifpc(xt,D,αt) γthen
76:: Dknn=(1in−itpica(l≤xizte,RDa,nαdt()k)nc)k, % the count of the births of neurons
D ←[D Dn]
←
A 0
8: A← 0 0 , B←[B 0],k←k+kn
(cid:20) (cid:21)
% Repeat sparse coding, now including the new dictionary elements
190:: endαitf=%arEgnαd∈Rokfmnineu12r||oxgte−neDsαis||22+λc||α||1
% ‘‘Memory’’ update:
11: A←A+αtαTt, B←B+xtαTt
% Dictionary update by block-coordinate descent with l1/l2 group sparsity
12: repeat
13: forj=1tokdo
14: uj ← bj−Pka6=jjjdkajk
% Sparsifying elements (optional):
15: vj ←Proxλj||.||1(uj)=sgn(uj)(|uj|−λj)+, % λj tuned to get βd non-zeros in vj
% Killing useless elements with l1/l2 group sparsity
16: wj ←vj 1−||vλjg||2 +
17: dj ← max((cid:16)1w,||jwj||2) (cid:17)
18: endfor
19: untilconvergence
20: endfor
21: return D
implementedusingthestandardproximaloperatorsasdescribedinJenattonetal.(2011).Notethat
weactuallyuseasinputthedesirednumberofnon-zeros,anddeterminethecorrespondingsparsity
parameterλ andλ usingabinarysearchprocedure(seeAppendix).
c j
Overall,thekeyfeaturesofouralgorithmistheinterplayofboththe(conditional)birthand(group-
sparsity)deathofdictionaryelementsinanonlinesetting.
3.1 DISCUSSIONOFIMPORTANTALGORITHMICDETAILS
A rationale behind sparsity of dictionary elements. We focus here on sparse dictionary ele-
ments, which, in the network terms, correspond to sparse connectivity between hidden units and
their inputs; one reason for this choice was that sparse connectivity appears to be a more biologi-
callyplausibleassumptionthanafully-connectedarchitectureimpliedbydensedictionary,inmany
brainareas,andspecificallybetweendentategyrusandCA3.Theotherreasonrelatestocomputa-
tionaladvantages.
Note that (Mairal et al., 2009) state that convergence guaranties for the original ODL algorithm
would also hold for the case of sparse dictionary elements. However, no empirical evaluation is
providedforthiscase;furthermore,wearenotawareofanypreviousworkonsparsecodingwhich
wouldinvolveandextensiveempiricalevaluationforsuchsetting.Priorfocusondenseratherthan
sparse dictionary elements is perhaps more natural when the input consists of a large number of
relatively small image patches, and thus each element also represents a small patch. In our work,
however,dictionaryisbeinglearnedonfullimages,andthusanonzeropatterninasparsedictionary
elementcorrespondstoasmallpatchwithinalargerimage,withmultiplesparseelements(patches)
coveringtheimage.Thus,ratherthanexplicitlyrepresentinganimageasasetofpatchesandthen
6
Neurogenesis-InspiredDictionaryLearning
learningadictionaryofdenseelementsforaccuraterepresentationofsuchpatches,adictionaryof
full-image-size, but sparse dictionary elements can be used to implicitly represents an image as a
linear combination of those elements, with possible overlap of non-zero pixels between elements;
the non-zero pixels in a sparse element of a dictionary are learned automatically. Computational
advantagesofusingsparsedictionariesaredemonstratedinourexperimentresults(Sec.4),where
classifierslearnedontopofrepresentationsextractedwithsparsedictionariesyieldsmallererrors.
ThememorymatrixAanditsproperties. ThematrixAkeepsthe“memory”oftheencodings
α for the previous data samples, in a sense, as it accumulates the sum of α αT matrices from
t t t
each iteration t. It turns out that the matrix A can have a significant effect on dictionary learning
in both ODL and NODL algorithms. As it is pointed out in (Mairal et al., 2009), the quadratic
surrogatefunctionin(2)isstrictlyconvexwithalower-boundedHessianAensuringconvergenceto
asolution.Fromthepracticalstandpoint,whenthematrixAhasahighconditionnumber(theratio
of the largest to smallest singular value in the singular value decomposition of a matrix), despite
its lower-bounded eigenvalues, the adaptation of a dictionary elements using the standard ODL
algorithmcanbedifficult,asweseeinourexperiments.Specifically,whenthedictionaryelements
are sparse, this effect is more pronounced, since the condition number of A becomes high due
to the complementary roles of sparse dictionary elements in the reconstruction process (see the
comparisonofAfromdenseelementsandsparseelementsin6(a)and6(b),respectively).Insuch
scenarios, the submatrix of A corresponding to the new elements in a dictionary, added by our
NODL algorithm, can have a better condition number, leading to an improved adaptation of the
dictionary.
CodeSparsity. Codesparsityiscontrolledbytheparameterβc,thenumberofnonzeros,which
determinesthecorrespondingregularizationweightλ instep4ofAlg.1;notethatλ isdetermined
c c
via binary search for each input sample separately, as shown in Algorithm 2, and thus may vary
slightlyfordifferentinstancesgivenafixedβ .
c
Selectinganappropriatelevelofcodesparsitydependsonthechoiceofotherparameters,suchasthe
inputbatchsize,sparsityofthedictionaryelements,theextentofnon-stationarityandcomplexity
of the data, and so on. When the dictionary elements are themselves sparse, denser codes may be
moreappropriate,sinceeachsparsedictionaryelementrepresentsonlyarelativelysmallsubsetof
imagepixels,andthusalargenumberofthosesubsetscoveringthewholeimagemaybeneededfor
anaccurateinputrepresentation.
Interestingly, using very sparse codes in combination with non-sparse dictionary elements in the
standard ODL approach can sometimes lead to creation of “dead” (zero l -norm) elements in the
2
dictionary,especiallyiftheinputbatchsizeissmall.ThisisavoidedbyourNODLalgorithm,since
such dead elements are implicitly removed via group sparsity at the dictionary update step, along
withthe“weak”(verysmalll -norm)elements.Also,averyhighcodesparsityincombinationwith
2
densedictionaryelementscanleadtoasignificantdecreaseinthereconstructionaccuracyforboth
ODL and our NODL when the online data stream is non-stationary. Such shortcomings were not
encounteredin(Mairaletal.,2009;2010),whereonlystationarydatastreamswerestudied,bothin
theoretical and empirical results. On the other hand, high sparsity in dictionary elements does not
seemtocauseadegradationinthereconstructionaccuracy,aslongasthecodesarenottoosparse.
Thechoiceandtuningofmetricforconditionalneuronalbirth. Inthe“conditionalbirth”ap-
proachdescribedabove,thenumberofnewelementsk isdeterminedbasedontheperformanceof
n
thecurrentdictionary,usingthePearsoncorrelationbetweentheactualandreconstructeddata,for
thecurrentbatch.Thisis,ofcourse,justoneparticularapproachtomeasuringdatanonstationarity
andtheneedforadaptation,butweconsideritareasonableheuristic.Lowreconstructionerrorin-
dicatesthattheolddictionaryisstillcapableofrepresentingthenewdata,andthuslessadaptation
might be needed, while a high error indicates that the data distribution might have changed, and
triggerneurogenesisinordertobetteradapttoanewenvironment.WechoosethePearsoncorrela-
tionasthemeasureofreconstructionaccuracysinceitsvalueiseasilyinterpretable,isalwaysinthe
range[0,1](unlike,forexample,themean-squareerror),whichsimplifiestuningthethresholdpa-
rameterγ.Clearly,onecanalsotryotherinterpretablemetrics,suchas,forexample,theSpearman
correlation.
7
Neurogenesis-InspiredDictionaryLearning
Tuningparameters:groupsparsityλg andothers. Thegroupsparsityregularizationparameter
λ controlstheamountofremoval(“death”)ofelementsinNODL:instep16oftheAlg.1,allele-
g
mentswithl -normbelowλ (i.e.,“weak”elements),aresettozero(“killed”).Sincethedictionary
2 g
elementsarenormalizedtohavel -normlessthanone,weonlyneedtoconsiderλ [0,1].(Note
2 g
∈
thatthestepofkillingdictionaryelementsprecedesthenormalizationstepinthealgorithm.Thus,
thetuningofλ isaffectedbythenormalizationoftheelementsfromthepreviousiteration.)Note
g
thatincreasingthesparsityofthedictionaryelelments,i.e.decreasingβ (thenumberofnozerosin
d
dictionaryelements)mayrequirethecorrespondingreductionofλ ,whileanincreaseintheinput
g
dimensionalitymmayalsorequireanincreaseintheλ parameter.Tuningtherestoftheparameters
g
isrelativelyeasy.Clearly,thebatchsizeshouldbekeptrelativelysmall,and,ideally,notexceedthe
“window of stationarity” size in the data (however, the frequency of the input distribution change
mayneedtobealsoestimatedfromthedata,andthusthebatchsizemayneedtobetunedadaptively,
whichisoutsideofthescopeofthispaper).Mairaletal.(2009)suggesttouseabatchsizeof256in
theirexperimentswhilegettingsimilarperformancewithvalues128and512.Astothemaximum
numberofnewelementsc addedateachiteration,itisreasonabletokeepitsmallerthanthebatch
k
size.
4 EXPERIMENTS
We now evaluate empirically the proposed approach, NODL, against ODL, the standard (non-
adaptive)onlinedictionarylearningofMairaletal.(2009).Moreover,inordertoevaluateseparately
theeffectsofeitheronlyadding,oronlydeletingdictionaryelements,wealsoevaluatetworestricted
versionsofourmethod:NODL+involvesonlyadditionbutnodeletion(equivalenttoNODLwith
nogroup-sparsity,i.e.λ =0),andNODL-which,viceversa,involvesdeletiononlybutnoaddition
g
(equivalenttoNODLwiththenumberofnewelementsc = 0).Theabovealgorithmsareevalu-
k
atedinanon-stationarysetting,whereasequenceoftrainingsamplesfromoneenvironment(first
domain)isfollowedbyanothersequencefromadifferentenvironment(seconddomain),inorderto
testtheirabilitytoadapttonewenvironmentswithout“forgetting”thepreviousones.
4.1 REAL-LIFEIMAGES
OurfirstdomainincludestheimagesofOxfordbuildings4 (urbanenvironment),whilethesecond
usesacombinationofimagesfromFlowers5andAnimals6imagedatabases(naturalenvironment);
examplesofbothtypesofimagesareshowninFig.1(a)and1(b).Weconvertedtheoriginalcolor
imagesintoblack&whiteformatandcompressedthemtosmallersizes,32x32and100x100.Note
that,unlike(Mairaletal.,2009),weusedfullimagesratherthanimagepatchesasourinputs.
(a) Urban:OxfordBuildings (b) Nature:FlowersandAnimals
Figure1: Theimagedatasetsfortheevaluationoftheonlinedictionarylearningalgorithms.
We selected 5700 images for training and another 5700 for testing; each subset contained 1900
images of each type (i.e., Oxford, Flowers, Animals). In the training phase, as mentioned above,
4http://www.robots.ox.ac.uk/˜vgg/data/oxbuildings/index.html
5http://www.robots.ox.ac.uk/˜vgg/data/flowers/102/
6http://www.robots.ox.ac.uk/˜vgg/data/pets/
8
Neurogenesis-InspiredDictionaryLearning
(a) LearnedDictionarySize (b) 1stdomain(Oxford) (c) 2nddomain(Flowers)
Figure2: ReconstructionaccuracyofNODLandODLon32x32images(sparsedictionary).
(a) 1stdomain(Oxford) (b) 2nddomain(Flowers) (c) ClassificationError
Figure3: ReconstructionaccuracyofNODLandODLon100x100imageswithsparsedictionary
elements(50non-zeros)andnon-sparsecodes.
eachonlinedictionarylearningalgorithmreceivesasequenceof1900samplesfromthefirst,urban
domain (Oxford), and then a sequence of 3800 samples from the second, natural domain (1900
Flowersand1900Animals,permutedrandomly).Ateachiteration,abatchof200imagesisreceived
asaninput.(Forcomparison,Mairaletal.(2009)usedabatchofsize256,thoughimagepatches
rather than full images.) The following parameters are used by our algorithm: Pearson correlation
threshold γ = 0.9, group sparsity parameter λ = 0.03 and λ = 0.07, for 32x32 and 100x100
g g
images,respectively.Theupperboundonthenumberofnewdictionaryelementsateachiterationis
c =50.(Weobservedthattheresultsareonlymildlysensitivetothespecifiedparametervalues.)
k
Oncethetrainingphaseiscompleted,theresultingdictionaryisevaluatedontestimagesfromboth
the first (urban) and the second (natural) domains; for the second domain, separate evaluation is
performed for flowers and animals. First, we evaluate the reconstruction ability of the resulting
dictionary D, comparing the actual inputs x versus approximations x = Dα, using the mean
∗
square error (MSE), Pearson correlation, and the Spearman correlation. We present the results for
Pearson correlations between the actual and reconstructed inputs, since all the three metrics show
consistentpatterns(forcompleteness,MSEresultsareshowninAppendix).Moreover,weevaluate
the dictionaries in a binary classification setting (e.g., flowers vs animals), using as features the
codesoftestsamplesinagivendictionary.Finally,weexploredawiderangeofsparsityparameters
forboththecodesandthedictionaryelements.
Ourkeyobservationsarethat:(1)theproposedmethodfrequentlyoftenoutperforms(orisatleast
asgoodas)itscompetitors,onboththenewdata(adaptation)andtheoldones(memory);(2)itis
mostbeneficialwhendictionaryelementsaresparse;(3)viceversa,whendictionaryelementsare
dense,neurogeneticapproachmatchesthebaseline,fixed-sizedictionarylearning.Wenowdiscuss
theresultsindetail.
SparseDictionaryElements
In Fig. 2, we present the results for sparse dictionaries, where each column (an element in the
dictionary)has5nonzerosoutofthe1024dimensions;thecodesarerelativelydense,withatmost
200nonzerosoutofk (thenumberofdictionaryelements),andk rangingfrom5to1000(i.e.the
codes are not sparse for k 200). Due to space limitations, we put in the Appendix (Sec. B.2)
≤
our results on a wider range of values for the dictionary and code sparsity (Fig. 12). In Fig. 2(a),
wecomparethedictionarysizefordifferentmethods:thefinaldictionarysizeaftercompletingthe
trainingphase(y-axis)isplottedagainsttheinitialdictionarysize(x-axis).Obviously,thebaseline
(fixed-size) ODL method (magenta plot) keeps the size constant, deletion-only NODL- approach
reduces the initial size (red plot), and addition-only NODL+ increases the size (light-blue plot).
9
Neurogenesis-InspiredDictionaryLearning
However,theinterplaybetweentheadditionanddeletioninourNODLmethod(dark-blue)produces
amoreinterestingbehavior:ittendstoadjusttherepresentationcomplexitytowardscertainbalanced
range,i.e.verysmallinitialdictionariesareexpanded,whileverylargeonesare,viceversa,reduced.
Our main results demonstrating the advantages of the proposed NODL method are shown next in
Fig.2(b)andFig.2(c),forthe“old”(Oxford)and“new”(Flowers)environment(domain),respec-
tively.(VerysimilarresultareshownforAnimalsaswell,intheAppendix).Thex-axisshowsthe
finaldictionarysize,andthey-axisisthereconstructionaccuracyachievedbythetraineddictionary
on the test samples, measured by Pearson correlation between the actual and reconstructed data.
NODLclearlyoutperformsthefixed-sizeODL,especiallyonsmallerdictionarysizes;remarkably,
thishappensonbothdomains,i.e.besidesimprovedadaptationtothenewdata,NODLisalsobetter
atpreservingthe“memories”oftheolddata,withoutincreasingtherepresentationcomplexity,i.e.
forthesamedictionarysize.
Interestingly, just deletion would not suffice, as deletion-only version, NODL-, is inferior to our
NODLmethod.Ontheotherhand,addition-only,orNODL+,methodisasaccurateasNODL,but
tendstoincreasethedictionarysizetoomuch.Theinterplaybetweentheadditionanddeletionpro-
cessesinourNODLseemstoachievethebestofthetwoworlds,achievingsuperiorperformance
whilekeepingthedictionarysizeundercontrol,inanarrowerrange(400to650elements),expand-
ing,asnecessary,smalldictionaries,whilecompressinglargeones7.
Wewillnowfocusoncomparingthetwomainmethods,thebaselineODLandtheproposedNODL
method.Theadvantagesofourapproachbecomeevenmorepronouncedonlargerinputsizes,e.g.
100x100images,insimilarsparse-dictionary,dense-codesettings.(Wekeepthedictionaryelements
atthesamesparsityrate,50nonzerosoutof10,000dimensions,andjustusecompletelynon-sparse
codes). In Fig. 3(a) and Fig. 3(b), we see that NODL considerably outperforms ODL on both the
first(Oxford)andthe(partofthe)seconddomain(Flowers);theresultsforAnimalsareverysimilar
andaregivenintheAppendixinFig.10.InAppendixSec.B.6,Fig.17depictsexamplesofactual
animalimagesandthecorrespondingreconstructionsbythefixed-sizeODLandourNODLmethods
(notincludedhereduetospacerestrictions).Abetterreconstructionqualityofourmethodcanbe
observed(e.g.,amorevisibledogshape,moredetailssuchasdog’slegs,asopposedtoacollection
clustersproducedbytheODLmethodsnotehoweverthatprinterresolutionmayreducethevisible
difference,andlookingattheimagesinonlineversionofthispaperisrecommended).
Moreover,NODLcanbealsobeneficialinclassificationsettings.Givenadictionary,i.e.asparselin-
earautoencodertrainedinanunsupervisedsetting,weusethecodes(i.e.,featurevectors)computed
on the test data from the second domain (Animals and Flowers) and evaluate multiple classifiers
learnedonthosefeaturesinordertodiscriminatebetweenthetwoclasses.InFig.3(c),weshowthe
logisticregressionresultsusing10-foldcross-validation;similarresultsforseveralotherclassifiers
arepresentedintheAppendix,Fig.10.Notethatwealsoperformfilter-basedfeaturesubsetselec-
tion, using the features statistical significance as measured by its p-value as the ranking function,
andselectingsubsetsoftopkfeatures,increasingkfrom1tothetotalnumberoffeatures(thecode
length,i.e.thenumberofdictionaryelements).Thex-axisinFig.3(c)showsthevalueofk,while
they-axisplotstheclassificationerrorrateforthefeaturesderivedbyeachmethod.Wecanseethat
ourNODLmethod(blue)yieldslowererrorsthanthebaselineODL(magenta)forrelativelysmall
subsetsoffeatures,althoughthedifferenceisnegligibleforthefullfeatureset.Overall,thissuggests
thatourNODLapproachachievesbetterreconstructionperformanceoftheinputdata,withoutextra
overfittinginclassificationsetting,sinceitgeneralizesatleastasgoodas,andoftenbetterthanthe
baselineODLmethod.
Non-sparsedictionaryelements
Whenexploringawiderangeofsparsitysettings(seeAppendix),weobservedquitedifferentresults
for non-sparse dictionaries as opposed to those presented above. Fig. 8(b) (in Appendix, due to
space constraints) summarizes the results for a particular setting of fully dense dictionaries (no
zero entries), but sparse codes (50 non-zeros out of up to 600 dictionary elements; however, the
codes are still dense when dictionary size is below 50). In this setting, unlike the previous one,
wedonotobserveanysignificantimprovementinaccuracyduetoneurogeneticapproach,neitherin
reconstructionnorinclassificationaccuracy;bothmethodsperformpracticallythesame.(Also,note
7Inourexperiments,wealsotrackwhichdictionaryelementsaredeletedbyourmethod;generally,bothold
andnewlyaddedelementsgetdeleted,dependingonspecificsettings.
10