Table Of ContentGaussiParno cessfeosr Machine Learning
AdaptiveC omputatioann d MachineL earning
ThomDaise ttEedriitcohr,
ChristBoipshheDorap v,Hi edc kerMmiacnh,Ja oerld aannMd,i chKaeealr Ansss,o cEidaitteo rs
BioinforTmhaMeta icchsiL:ne ea rnAipnpgro ach,
PieBrarlead niSd ¢ rBernu nak
ReinforLceeamrennAitnnI gn:t roduction,
RichSa.Sr udt taonnAd n drGe.wB arto
GraphiMcoadlef losMr a chiLneea rnianngDd i giCtoamlm unication,
BrendJ.aF nr ey
LearniinGn ragp hiMcoaedll s,
Mich1a.Je olr dan
CausaPtrideoinc,t ainoSdne ,a rscehc,oe nddi tion,
PetSepri rCtleasGr,lk y moaunrRd,i chSacrhde ines
PrincoifDp alteas Mining,
DavHiadn Hde,i kMkain nialnaPd,a dhrSamiyct h
BioinforTmhaMeta icchsiL:ne ea rnAipnpgro ascehc,oe nddi tion,
PieBrarlead niSd ¢ rBernu nak
LearnKienrgnC ella ssiTfiheeroasrn:yAd l gorithms,
RalHfe rbrich
LearnwiintKghe rneSlusp:pV oercttM oarc hiRneegsu,l arOipztaitmiioznaa,nt Bdie oyno,n d,
BernhSacrhd6 laknoAdpl fe xaJn.Sd meorl a
IntrodutcoMt aicohniL neea rning,
EtheAml paydin
GaussPiroacne sfsoeMrsa chiLneea rning,
CarEld waRrads musasneCdnh ristKo.1p .Wh ielrl iams
GaussiParno cessfeosr Machine Learning
CarEld waRrads mussen
ChristKo.1p .Wh ielrl iams
TheM ITP ress
CambriMdagses,a chusetts
LondEonng,l and
© 200M6a ssachIunssettiottfTsu ec then ology
Alrli grhetsse .rN vope adrotft hbioso mka yb er eprodiunac nefydo rbmya neyl ectorrmo encihca nical
mean(si nclpuhdoitnogc orpeycionrgod,rii nnfgo,r msattoiraoanngdr ee triweivtahlpo)eu rtm isisni on
writfirnotgmh peu bl.i sher
MITP rebsoso mkasy b ep urchaatss peedc qiuaaln tdiitsyc ofuobnrut ssi onres saslp erso motuis.oe nal
Foirn formpalteieaomsnae,si p lec [email protected] .reidtuto Se p ecSiaalDlee sp artment,
TheM ITP re5ss5H, a ywaSrtdr eet, CMaAm0 b12r4.i2 dge,
Typebsyet tha eu thuosrisg\n 1EXg 2•10
Thibso opkr inatnebddo unidnt hUen itSetda otfeA sm er.i ca
LibroafCr oyn grCeastsa loging-iDna-tPau blication
RasmusCsaerEnld, w a.r d
Gausspiraonc efsomsrae csh lienaer /nC ianrEgld waRrads musCsherni,s tKo.Ip .Wh ielrl .i ams
p.c m.- (Adapctoimvpeu taantmdia ocnh lienaer ning)
Inclbuidbelsi ographiacnaidln dre.ex feesrences
ISB0N2- 6128-2-5X3
1. Gaupsrsoicaens sepsr-oDcae.ts2 as.M i ancghlienaer ning-Mamtohde.em lastical
I.W illiCahmrsi,s tKo.Ip .Ih. IeT ri .tI l.IeS Ier.i es
QA724.47.0R2036
591.'32-dc22
2005504333
109 8 7 6 5 4 3 2 1
Thea ctusacli eonflc oeg iiscc o nverastpa rnets oennltwy i tthh inegist her
certiamipno,s soireb nlteid,ro eulbytn founolefw, h i(cfho rtu)nw aeht aevlteyo
reaso.onT nh erefotrreul eot gfhiotecrh wiosr iltsdh cea lcoufPl ruosb abilities,
whitcahk aecsc oouftn htme a gniotfut dhpeer obabwihliiictshoy, ro ugthot
bei,na r easomnaa'nbsml ien. d
- JamCelse Mrakx we[8l105l]
Contents
SerFioerse w.o rd xi
Pref.ac.e. . . xiii
SymbaonlNdso tation xvii
1 Introduction 1
1.1A PictoIrnitarlo dtuocB taiyoensM ioadne lling 3
1.2R oadm.a p. . . . . . . . . . . . . .5 . . . . . .
2 Regression 7
2.1W eight-Vseipw.a c.e. . . . . . . ... . . . . . 7
2.1.T1h eS tandLairndeM aord .el. . . .. . . 8
2.1.P2r ojecotfIi nopnuistn stF oe atSuprae.c e 11
2.2F unctioVnei-ws. p a.c e. . . . 13
2.3V arytihnHegy perparameters . 19
2.4D eciTshioenof roRyre gression 21
2.5A nE xampAlpep lic.a t.i o.n . 22
2.6S moothWieniggF,hu tn ctainoEdnq su ivaKleernnte ls 24
* 2.7I ncorpoErxaptliBinacgsiF itus n ctions 27
2.7.M1a rgiLniakle lihood 29
2.8H istaonrRdye laWtoerd.k 29
2.9E xercises 30
3 Classification 33
3.1C lassifiPcraotbilo.enm. s. . . ... . . 34
3.1.D1e ciTshioenof roCyrl assification 35
3.2L ineMaord efloCsrl assifi.c.a.t.i. o.n 37
3.3G aussPiraoncC elsass sifi.c a.t i.o n. . . 39
3.4T heL aplAapcper oxifmoatrthi Beoi nn aGrPyC lass.i fier 41
3.4.P1o ste.ri.o.r. 42
3.4.P2r edic.t i.o n.s . . 44
3.4.I3m plemen.t.a.ti on 45
3.4.M4a rgiLniakle lihood 47
* 3.5M ulti-LcalpalAsapscp er oximation 48
3..15 Implemen.t.a tion 51
3.6E xpectPartoipoang .a t.i on 52
3.6.P1r edic.t.io.n.s. 56
3.6.M2a rgiLniakle lihood 57
3.6.I3m plemen.t ation 57
3.7E xperi.m.en.t.s. .. . . 60
3.7.1 A Toy Problem 60
3.7.O2n e-dimeEnxsaimopnlael 62
3.7.B3i naHrayn dwrDiitgtCieltna ssifiEcxaatmip.ol ne 63
3.7.140 -cHlaansdsw rDiitgtCieltna ssifiEcxaatmipolne 70
3.8D iscnus..s.io. . . . . . . . . . . . . . . . . . . .7.2. . .
*Sectimoanrsk ebdy aasnt erisk caodnvtaanicne d mtahtaemtra iyab leo mitted onr eaa dfiirnsgt.
viii Contents
* 3.9A ppenMdoimxe:Dn etr ivations 74
3.1E0x ercises 75
4 Covariancfeu nctions 79
4.1P relimi.n a.r i.e s. . . . . . . . . . .7 9. . . . . . . .
* 4.1.M1e aSnq uaCroen tinauniDdti yff erentiability 81
4.2E xampolfCe osv ariFaunnccet .i on.s . . 81
4.2.S1t atiCoonvaarryiF aunnccet. io.n.s.. . 82
4.2.D2o Ptr odCuocvta riFaunnccet. i.on..s. 89
4.2.O3t hNeorn -staCtoivoanrairFayun nccet ions 90
4.2.M4a kiNnegwK ernferlosOm l d 94
4.3 EigenfunocfKt eironn.e lA.sn a.l y.s i.s . . . . .9 6
* 4.13 .AnA nalyEtxiacm p.l.e . . . . ... . . . 97
4.3.N2u merAipcparlo xiomfaE tiigoenn fu.n ctions 98
4.4K ernfeolNrso n-vecItnopruitasl 99
4.4.S1t rKienrgn els 100
4.4.F2i shKeerrn els 101
4.5E xercises 102
5 Model Selectiaonnd Adaptatioonf H yperparameters 105
5.1T heM odeSle lecPtrioobnl em 106
5.2 BaMyoedseSilea lne c.t.i.o. n . 108
53. Cross-va.l.i.da.t.i.o.n. . . . 111
5.4M odeSle lecftoGirPo R ne gression 112
54..1M argiLniakle li.h..o od 112
54..2C ross-va.li.d.a..t i.o n 116
54..3E xampalneDdsi scus.s ion 118
5.5M odeSle lecftoGirPo C nl assi.fi cation 124
* 5.5.D1e rivaotfti hvMeea sr giLniakle lfiohLroa opdl saa cpep'roxim1a2t5i on
* 5.5.D2e rivaotfti hvMeea sr giLniakle lfiohEroP .o d 127
5.35 .Cross-validation 127
5.5.E4x am.p le 128
5.6E xercises 128
6 Relationshibpest weenG Ps and OtherM odels 129
6.1R eprodKuecrinnHegil l bSepratc. e s. . . . . . . 192
6.2R egulari.z.a.ti.o.n. . . . . . . . . . . . . . 123
* 6.2.R1e gularDiezfiantbeiydDo inff ereOnpteiraalt ors 133
6.2.O2b taitnhiRene gg ulaSroilzue.td i .o n. . . . . 1. 53
6.2.T3h eR elatioofnt shRheei gpu larVieizwat toGi aouns sPiraonc ess
Predi.c t.i o.n . . . . . . . . . .1 53. . . . . . .
6.3S plMinoed e.l s. . . . . . . . . . . . 1. 3.6 . . . . . . .
* 6.3.A1 1 d-G aussPiraoncS epslsCi onnes tr.u ction 183
* 6.4S uppVoercttM oarc hi.ne.s. . . . 141
6.4.S1u ppVoercttC olra ssification 141
6.4.S2u ppVoercttR oerg ression 154
* 6.5L east-SCqluaasrseisfi. c a.t i.o n. . 146
6.5.P1r obabLielaisstt-iScCq luaasrseisfi cation 147
Contents ix
* 6.6R elevVaencctMeoa rc hines 149
6.7E xercises 105
7 TheoreticPaelr spectives 151
7.1T hEeq uivaKleernnte. l. . . . . . . . ... . . . . . 151
7.1.S1o mSep ecEixfiacm polfEe qsu ivaKleernnte ls 153
* 7.2A symptotic. .An.a.l.ys..i. s. . 155
7.2.C1o nsist.e.n.cy. . ... . . 155
7.2.E2q uivaalneOdnr cteh ogonality 157
* 7.3A verageL-eCaarsnCeiu nrgv es 195
* 7.4P AC-Bayesian Analysis 161
7.4.T1h PeA C Frame.w.o rk 162
7.4.P2A C-BayAensailayns is 136
7.4.P3A C-BayAensailaoynfsG iPCs l assifi.c ation 146
7.5C omparwiistOohtn h Seurp ervLiesaerdnM ientgh ods 165
* 7.6A ppenLdeiaxr:n Ciunrgfv oetr h Oer nstein-PUrholceensbsec k1 86
7.7E xercises 169
8 ApproximatioMne thodsf orL argeD atasets 171
8.1R educedA-prparnokx imoaftt hiGeor nasMm a trix 117
8.2G reeAdpyp roxi.ma.t.i.on. . . . ... . . . . . 147
8.3A pproximfaoGtrPi Ro wnistF hi xHeydp erparameters 157
8.3.S1u bsoefRt e gres.s ors 157
8.3.T2h eN ystrMoemt h.od. . ... . 177
83..3S ubsoefDt a tapo.in.t.s. .. . 177
83..4P rojectedA ppPrrooxcie.ms ast ion 187
83..5B ayesCioamnm itMtaeceh .i ne. . 108
8.3.I6t eraStoilvuoetf iL oinn eSayrs tems 181
8.3.C7o mparoifAs popnr oxiGmPRa Mteet hods 182
8.4A pproximfaotri oGnPsCF iwxHieytdph e rparameters 185
* 8.5A pproxitmhaMeta irnggiL niakle lainhdo oDidet rsi vatives 185
* 8.6A ppendEiqxu:i vaolfSe Rn ceaGn PdR usitnhgNe y strAopmp roximate
Kernel 187
8.7 Exercises 187
9 FurtheIrs sueasn d Conclusions 189
9.1M ultiOpultep. ut.s. . .. . . 190
9.2N oiMsoed ewlistD he pendencies 190
9.3 Non-GauLsiskiealni. h.o.o. d s 191
94. DerivOabtsievrev a.t.i.o.n. s 191
9.5P rediwcittiUhon nc erItnapiunt s 192
9.6M ixtuorfGe asu ssPiraonc esses 192
9.7G lobOapltimiz.a tion 193
9.8E valuaotfIi notne grals 193
9.9S tudset nP tr'ocess 194
91.0I nvari.a.nc.e..s . 194
91.1L ateVnatr iaMboldee ls 196
91.2C oncluasniFdou ntsuD rier ections 196
x Contents
AppendixA MathematicaBla ckground 199
A.1 JoiMnatr,g iannaCdlo nditPiroonbaalb. i lity 199
A.2 GaussIidaenn tities 200
A.3 MatrIidxe nt.i t.i e.s . . 210
A.3.1M atrDiexr ivatives 220
A.3.2M atrNioxr .ms. . 220
A.4 CholeDsekcyo mpo.s ition 220
A.5 EntraonpKdyu llbackD-iLveeirbgleenrc e 203
A.6 Limi.t.s. . . ... . . 204
A.7 MeasaunrIden tegration 204
A.7.1Lp Spac.e s 250
A.8 FourTriaenrs forms 250
A.9C onve.xi.t.y. . 260
AppendixB GaussiaMna rkovP rocesses 207
B.1 FourAinearl y.s.is. . . . . . . . . . 280
B.l1. SamplainnPdge riodization . . 290
B.2 ContinuoGuasu-stsMiiamarenk P orvo cesses 211
B.2.1C ontinuoGuMsP-sot nilR m.e . . . . 211
B.2.2T heS oluotfit ohCneo rrespSoDnEdo intn hgCe i rcle 213
B.3 DiscretGea-utsisMmiaear nkP orvo ce.s s.e s. . . . . 2. 1.4 . .
B.3.1D iscretGeM-Ptsoin Zm .e . . . . . . . . 2.1 4. . . . . .
B.3.2T heS oluotfit ohCneo rrespDoinffdeirnEegqn ucaeto inIP'o Nn 215
B.4 TheR elatiBoentswheiDepin s cretaen-dSt aimmpelC eodn tinuous-time
GMPs. . . . . . . . . . . . . . . 2.1 7. . .
B.5 MarkPorvo ceisnHs iegshD eirm ensions 218
AppendixC Datasetasn d Code 221
Bibliography 223
AuthorI ndex 239
SubjecItn dex 244
SerieFso reword
Theg oaolfb uilsdyisntgte hmacsta and atpott heeinrv ironamnedln etasr n
frotmh eeixrp erhiaesn caet rtersaecatrfecrdho memar nfisye lidnsc,l ucdoimng
putsecri eenncgei,n emeartihnegm,ap thiycssni,ec usr,o scainedcn ocgen,i tive
sci.eO nucotef t hriess ehaarsc hc awo imdeve a rioeflt eya rtneicnhgn tihqaute s
havteh peo tenttoti raaln smfaonrsymc ienatniidfin cd usfiterl.idR aeslc ently,
severreasle acrocmhm unhiatviceeo sn veornga ec do mmosneo tfi sssuuers
roundsiunpge rvised, aunnrdse uipnefrovrliceseaemrdepn,nri ton bg.l T ehmes
MITP ressesr oineA sd aptCiovmep utaatnidMo anc hiLneea rnsienetgko s
unitfhyme a ndyi vesrtsreao nfmd asc hlienaer rneisnegaa nrtdco hf oshtiegrh
qualrietsye aanridcn hn ovaaptpilviec. a tions
Oneo ft hmeo satc tidvier ecitnim oancsh ilneea rhnaibsne ge tnh dee
velopomfep nrta ctBiacyaelsm ieatnh ofdocsrh allelnegairnpngri onbg.l ems
GaussPiroacne sfsoeMrsa chiLneea rnpirnegs oennoetf ts h meo sitm portant
Bayesmiaacnh lienaer anpipnrgo abcahsoeensda p articeuffleacrmtleiytv heo d
foprl acaip nrgid oirs triobvuettrhis eop na ocffe u nc.tC iaornElsd waRrads
mussaennCd h rWiisl liaartmews oo ft hpei onienet rhasir se aan,td h ebioro k
descrtihbmeea st hemaftoiucnadlaa tnipdor nasc taipcpalli coafGt aiuosns ian
proceisnrs eegsr easnscdil oans sitfiacs.ak Ttshieoaynl ssoh ohwo wG aussian
procecsasbnee is n terparsae B taeyde sviearnso ifto hnwe e ll-ksnuopwpno rt
vecmtaocrh mienteh .oS dtsudeannrtdes s earwchhose trutsdh ybi oso wki blel
ablteoa ppGlayu sspiraoncm eestsh oidnc sr eawtaiyvtseos olavw ei drea nge
ofp robliensm csi eannceden gin.e ering
ThomDaise tterich
