Computational learning theory and natural learning systems - Making Learning Systems Practical

409 Pages·1997·43.531 MB·English

by  edited by Russell Greiner, Thomas Petsche and Stephen José Hanson.

ComputationLaela rninTgh eorya nd Natural

LearninSgy stems
CopyrighMtaetde rial
ComputationLaela rninTgh eorya nd Natural
LearninSgy stems
VolumIeV :M akinLge arniSnygs temPsr actical
editbyeR du ssGerleli Tnheorm,Pa est sacnhSdet epJhoesnHe a nson
A Bradford Book
TheM ITP ress
CambriMdagses,a chusetts
Lo no dn E,n ga 1nd CopyrighMtaetde rial
© 199M7as sachuIsnestttisot fTu etceh nology
Alrli grhetsse rNvoep da.ro tft hibso omka yb er eprodiunac neyfd o rbmya ny
electorrmo encihca nmiecaan(lsi nclpuhdoitnogc orpeycionrgod,ri i nngf,o rmation
storaangdre e triweivtahlpo)eu rtm isisnwi roint firnogtm h peu blisher.
Thsib oowkas s eitnIf Tfi.EJX byt haeu thaonrdws as prinatnedbd o unidnt hUen ited
StatoefAs m erica.
LibroafCr oyn grCeastsa loging-inD-aPtuab lication
Revifsoevrdo l4.)
Computatlieoanratnlhi enogar nydn atulreaalr snyisntge ms.
"AB radfBoorodk ."
Varieoduist foorlrsa tveorl umes.
Inclubdiebsl iographiacnadil n dreexfeesr.e nces
Contevn.1t .sC :o nstraanidpn rtoss pec2t.Is n-tve.r sectiotnhse oarbnyed t ween
experime3n.St e-lve.cg toiomndgo dels4-.Mv a.k ilnega rnsiynsgt ems practical
ISB0N- 262-5(8v11.2) 6--04- 262-(5v82.1) 3-30--72 62-(6v63.0) 9-6-2
0-262-5(7v14.1) 8 -8
1.C omputatLieoanranTlihe nogr y-Cong1r.He asnsseSostn.e, p hJeonsI eI.D. r astal,
GeorAg.Ie I RIi.v eRsotn,a Ll.d
Q325.5.1C96954 006.3'1 93-34468
CopyrighMtaetde rial
Contents
Preface IX
Introduction xi
Contributors xvii
I ScalinUgp
1 InitializNienugr alN etworksU singD ecisioTnr ees 3
ArunavBaa nerjee
2 RecurrenNte uralN etworksw ithC ontinuous
TopologyA daptationK,a lman FilterB asedTr aining 17
DragOabrna dovic
3 ImposingB oundso n theN umber ofC ategoriefso r
IncrementaClo nceptF ormation 35
LeonS hakrla nHda yHmi rsh
4 Scalintgo Domainsw ithI rrelevaFneta tures 51
PatriLcka ngalneSdyt pheaenS iage
II Robusta nd EfficientL earning
67
5 MixtureM odelsf orL earninfgr omI ncompletDea ta
ZoubiGnh ahramaannid MichIa.Je olr dan
87
6 Learnintgo C lassifIyn completEex amples
DaleSc huurmaannsd RusGsreleli ner
7 SuperviseLde arninUgs ingL abeleda nd Unlabeled
107
Examples
GeoffreTyo well
8 AbnormalD ataP ointsi nt heD ataS et: An
121
AlgorithFmo r RobustN euralN et Regression
YonLgi u
CopyrighMtaetde rial
vi Contents
9 DynamicM odelingo fC haotiTci me Seriebsy Neural
Networks 137
GustavDoe coa ndB erndS chiirmann
155
10 FastD istribution-specific Learning
DalSec huurmaanndsR usseGlrl einer
IIII mprovinagn d AnalyzinGge neralization
11 Explorintgh eD ecisioFno restA:n Empirical
InvestigatoifoO nc cam'sR azori nD ecisioTrne e
171
Induction
PatriMc.k M urphy andM ichaeJl.p azzani
12 N-LearnerPsr oblem:S ystemo fP AC Learners 189
NageswaSr.a Va.o anRdE . M. Oblow
13 The DiscriminatPiovwee ro fa DynamicaMlo del
211
Neuron
AnthonMy. Z adoar ndB arakA .P earlmutter
14 On Learnintgh eN euralN etworkA rchitectuAr e:
CaseS tudy 227
MostefGao lea
15 ProbabilisSteilcf -structuarnidn Lge arning 245
DaviGda rviann dP eteRra yner
16 A PracticAaplp roachf orE valuatinGge neralization
Performance 279
MarjoKrlieen in
IV RealW orldA pplications
17 What Makes DerivationAanla logyW ork: An
ExperiencRee port UsiAnPgU 303
SanjaByh ansanadlM eih diT . Harandi
CopyrighMtaetde rial
Contents vii
18 LearninVge rb TranslatioRnu lesf romA mbiguous
323
Examplesa nd a LargeS emantiHci erarchy
Hussein AYlamsuuhaiAlrokl iibmTa,a, k eifY uammazaankdi
ShigKeaon eda
19 EfficientL earninogf R egularE xpressiofnrso m
ApproximatEex amples 337
AlvBirsa zma
20 A Comparisioonf RBFa nd MLP Networksf or
Classfiicatioonf B iomagnetiFci elds 353
MartFi.S nc hlanKgla,u sA braham-RFuaclhh pNse,u neier
anJdo hanDn ebler
21 FastP erceptuaLle arninogf M otioni nH umans and
363
NeuralN etworks
LucMi.aV ainVae,n krataSruanmdaanr easnwJdao rhanG n.
Harris
References 379
Index 405
CopyrighteMda terial
Preface
Thivso lucmoen tapianpsef rrsot mh fien ailna s eroifwe osr kschaolplse d
"ComputatLieoanranlTi henogyr and'N atu'r LaelarnSiynsgt, e"mo rs
"CLN(Lp"r ounnoce"dk er).nT ehlefi" r,sh telSde ptem1b9e9ir0nP rince­
ton,N ewJe rs, ewysasp onsroed bSyi emeCnosr poraRtees ea(rCScRh)M ,I T
anPdr incUentvieorintys . Itpsur powsaets o e xpltohreeem ergiinntge rsec­
tioonfth eorelteiacrarnlei snegaa nrdnc aht ulreaarilnn gs yst(eCmOsL T,
AIa ndNe uraNelt )s. Ove1r0 p0ar ticipantsa ttentdohe eda1 r6o raaln d
25p oster preTsheesn etcaowtnoidro knsshh eolpid,nS e pmtbeer1 99in1
Berk,e Claelyifowranssip ao,n sboyrS eCdRM ,I Ta ndU .CB.er keleTyh. e
thiwrodr ks, hheolpidnS eptem1b9e9ir2nM aidsonW,i scownaissnop ,n ­
sorbeySd C RM,I Ta nUd. o fW isconFsiinnat.lh l,ity sh feo urwotrkhs hop,
wash elidne arSleyp tem1b9e9ir3nP rocveitno,w Mnassach(utsheotstes
ofy owuh ot ootkh beo arti dree ctahltele mper)a,at nuwdras es!opnseodr
solbeylS yCR .O ver5 0p aprsew erseu bmiftrtoe8mcd o unt(riinecsl uding
papefrrso1 m4U Ss tates);1 6o ft hewseers ee tleedfc oorarl p resentation
and1 7f opro stTehreswr.ee r ael s4io n viptreeds eionnst:a t
• ThomDaise tt(eOrrieocngS ht aatnedA rrPihsa rmac)e:u" Stciainclga l
UpM achiLenarenin g: PractiacnadlT heoretiIcsaslu es"
• LeBor eim(aUnCBe rkeley): "CurrTreeneRt e search"
• YannI eC un( AT&BTe lLla b):s" LarHgeet erogLeenaeronuMisan ­g
chinfeoPsre rceTpastkus" al
• RonR ive(MsIt)T: "RefloencM tacihoinnLseer naing"
Abou1t0 p0e opaltet en(indcleuddin gt h3e0 w hoar rivoend ourr neted
boa).t
Theusneu swuoarlk sahtotprsa rcetseeda ricnhteerrisent sh teie ndt ersec­
tioonft hrheiteos riycd ailinslctstty lsoe fl ernainregs earCcohm:p utational
LearninTgh eorhya usn dgeonrea renassian,Nc eeuralN etworreks ercahh as
seeexnp onengtrioawaltn hSdy mbolic Machine hLaseba ernoimcnega d om­
inanitn flueinntc heAe I fi elIdn.r ecentp eyrehaparapsrs,td ulety o t he
"concioruasigns"tei hsnaostc cudrurteeo d e set whorks(hnooeptt sh amatn y
ofth e progrcahmao ifrt shp ere vailliaerningng c onferenhcaevsea t teend d
aC LNLw orkshaotpl eaosntc), e hetrheav eb een ttaemptst oc o-locate
MachiLneea rnainnCdgO LTw,h ilteh per emnieeurrn awelok rc oenrfeen c
CopyrighMtaetde rial
x Preface
(NIPhSa)bs e commoer aen mdo rree preseonfbt easwttoi rvakett hien ­
teresoefct thieooanrn yedx perismyemnbtao,nl sdsi gnals.
Ovetrh yee a,wr osrksphaorpt ichiapvbaeen etensn courtaoeg xeadm ine
geneirsasilunl e esa rsnyisntgwe hmiscc,oh u plrdo vciodnes tfroatrih netosr y,
whialtte h sea mtei mteh eorreetsiucwlaetlris en terpirnte htceeo dn text
ofe xperiwmietanhct tslu eaalr snyisntge Tmhseo. r ganfiozlelroosnw ee d
simphleeu riascttiicpv:ue rlsytu oep iincts hi en tersoeftc hteifisoeenl ds.
Tom aximtihzleei keloiflh iovodedil syc uasnsdsi uobns taenxtcihvaen ge,
speakweirttshh eorteotpiwicecarslje u xtapwoistsehpd e akweorrski inn g
experidmoemnatiaWnle sa .l ssoou gahd ti vesreostfei nvistpeeda kers,
in
addititohonen elsi sted above:
to
Firswto rksho(p1 99E0r)i:Bc .B aum( NERee seaLracbhsT )h;o mas
Mitc(heeMlUlD) a;v Riudm ealrh(tS tanfLoersVdla)il;ei (aHnatr vard).
Secondw orksho(p1 991B)e:r naAr.Hd uob erm(aXnE ROPXA Re);
NilJ.sN ils(sSotna nfGoerodff)rH;ei yn t(oUnT. o ronDtaov)iH;da ussler
(UeSe).
ThirdW orkshop( 1992M)i:c hKaeealr (nAsT &BTe lLla bsR)i;c hard
1
Sutt(oGnT E)A;l aLna pe(dLeoAssl amoPsa)Lt;a ng(lNeAyS AAM ES2);
GraWcaehb a( UW.i scoantMs aidni son).
Thewsoer kswheorpiesn tentdope rdo vaif doer fuomtr h dee velopment
andc reaotfin oenwa reoasfr eseianrL ceha rnainnAdgd aptsiyvset ems.
Weh opteh ehya vfeo stteorleedr iantn hceeo rfiotsrht iesm portaanndc e
diveorsfeix tpye riampepnrtoaatlcot h hseets u odfsy u cccleesasrfsnuyilsn­g
temasn bdi aesmepdi raitcl iesastttows a ttchhee v oltvhienoogrfl y e arning
systems.
INowa tU niversity of Massachusetts.
2Nowa tI SLEa ndS tanfoUr�dWti9 hted Material
Introduction
Thiists h feo uarntfidhn avlo luimnaes eroifee dsi vtoeldu tmheaestx plore
three cleanntd socfLa epaer nSiynsgtR eemsse awrhcihsc,ph a tnhse oarnyd
experismyemnbtao,nl sdsi gnVaollsu.1mo eft hsiesr ies ionutgrre ond­uced
erfaolc aunsgd o aolfe xplotrhiienn tge rsoeftc htrieoen hidsitsotriinccatl ly
areoafls e arrneisnega rch: CLoemapruntTiahntegio( orCnyOa LlTN )e,u ­
raNle two(rNkNsa) n dA I'Msa chine L(eMaLrT)nh.ie sn egc ovnodl ume
focuosnes dp ecairfiecoa fis n terabcettiwtoehnee noa rnyed x periImne nt.
partitchueliraasehr e,a lhtihsyto oftr hye orweotritkch aahltaa sc ontinuity
with comtphleeoixrnciy ot myp ustceirea nncmdea intaal iinofsfie t osw n.
Fomra nyye atrhse hraeas l sboe eanv ereym pireinctaelr kpnroiwsanes
MachiLneea rnUinntgri.el c entthletysw,eto r enhdasgd e nerdailvleyr ged
anddo nlei tttoil nef oornmae n othVeorl.u2 mw ea asn a ttetmop tp rovide
cleeaxra mpolfhe oswt heocrabyne t eswtietdth hc ea reefxuple riamnedn t
howe mpirwiocrcakal bn e g uidienmd e aninwgafybusytl h eoRreyc.e ntly,
there hmaosr webo ertekon w atredsst tihnegoa rnydr a tioneaxlpiezriin­g
mentWse.h opteh evsoel umceosn tritbotu htatetrd e antdl ,e aisnst o me
smawlaly .
Thefi natlw vo olum(iensc ltuhdpeir negs oenneft)o counsc ertkaeiyn
areoafls e arsnyisntgre emsse tahrachtha vdee velroepceedn Vtollyu.m e
3p roviedxeadm polfhe oswr esearhcahvceeor nsc epttuhapelr iozbelde m
of" SeleGcotoiMdno gd elTsh.i"is sa c entprraolb laesam n,yl earning
appromaucshst p eciidfye,n otris feyl aem cotd eflr oams eotf p ossible
modeglisv,de anta ar isfoirmnt gh weo ralndsd o mper iboerl iaebfotsuh te
howt hdea tmaa yh avbee egne nerated.
Thivso luemxep ltohrteee sr roaf"i Mna kLienagr nSiynsgtP ermasc tical."
Althotuhgehar reme a nfya cttohramsta yr endale era rsnyisntgue sme less
ori mpractthisecu ablm,i stsoti hoinwsso rksihdoepn ttihfiregedee n eral
problecmrsi t(i1cS)ac la:lu ipfn rgos mm aplrlo blteorm esa loinsewtsii tch
as
larignep duitm en(sei.o>g n1. 0,a) n edx amp(lee.1sg0 .0,0( s2i))n; c reasing
efficieanncrdyo busotfnl eesasrm neitnhgo adnsd(; 3s )t ratteotg aikieens
obtaignoiongdeg n eralfirzoamt ilooinrsm miatlelds admaptlaTe hse.s e,
pluass e ctoinso pne ceixfiacm pfloertsmh, fe o ucra tegiontr hivieossl ume.
Theea rlhiiesrto ofMr ayc hLienaer nRiensge awracmsho tivbaytp erdo b­
lemtsh afto custehdbe a sciocm pleoxfli etayr cnoinncgde epstc riptions,
on
ratthhearon nt hesicra loirpn ogt ensttiaatlip srtoipcearFlto eirxe asm.p le,
Winst(on1 9i7ll5u)s tthraaitttw e ades a styoi dentthitefa yr gceotn cept
dp
ift hter aisneiitnn gc lu9d}j�e<Cm.liY1ifa ?9fiHex)a mptlheawste e rj usatt
xii Introduction
thdece isbiouonndr yab etwtehepeno siatnindve eg aet eistvsT.hi tse lling
exalmespe tth es tafgoebro tinhtr eesti np edgoag(yi .ien.t,e ltluitgoernst)
anlde artnhienogOr tyh.e rs (1M9i8tM0ci;hc ehla&ll S,st kepi1p ,9 8d3i)s ­
cusstehdedi fficulotfley a rnincgo mpcloenxc eptss iwgintnihtbfio icuaiatns
three preselnatunaagget.Ti hoents ree ntdesn dteosd u pnptlm aoreex per­
imetlnaa propachofeu ss idnigffi cruelatl -pwroorbllde mrso b(oceto.ingc­. ,
trool,p tichcaaralc treer ciotiogn,ns peercehc ognn)ui ntttiiholme i d-s1 980
wheNne urNeatlw oerxkpsl oidnettdohM ea chLienaenr gns cien(eR umel­
hart eta l1.9,8 6U)n.l iMakceh iLneea rnwihnigtc,eh n dteodd fie nei tso wn
probltehmfiese ,lo df)J" e uraNle trwko(si onr dteotr r ansicstce hnedc kered
pashta)td os olovrae tl,e asdto a sw easl lan ym ethoonds,so mhaerd p rob­
lem,s ucahss peercehc ognhietrtieho,oin fs;t e mne ancso mpetinagg ainst
approacesht hahta bde efinn e-hfoon2re 0yd e a(res.H gM.M,s )T.h iosr ien­
tattioowna driffidc uplrotb elmsh asse stt andtahraadtnsl y e armneitnhgo d
nowh atsom eeitno rdteob re t akseenr iyo.u sl
A currenfto ciunles ra ningr eseiasir dche ntwihfaymtia nkgae l se arning
tasdki fficaunltdth etnr aincghko w tehp erforomftah nelc eea rsnyisntge m
variesa st hiapssec ti ncreTahsie"sss .c aplrionbgl iestm h"me a ifno cus
ofa numboefpr a peirnts h ivosl umBaen:er je(e chap1t)she orws t hat
initialiaz niengu nraeults ian dge scioint recea ni mprovbeo tthhe s ystem's
accuraancedyffi cieInnacs iymi.lr av eiOnbr,a dovi(cch apter2 )e volhviess
systoenlmi bneyi, n cretahsneiu nmgbo efer lm eentsi nt hnee twork's hidden
layewrh enetvheec ru rrteonpto lcoagnync oatp ttuhrdeey namoiftc hse
nedwa tSah.kl a&r Hirsh( cha3p)tsh oewr h otw oi ncrtehaeesffi ec iency
ofa w ell-kcnnocoewpnt culauslt earpiprnoga cbhys uppliynifonrgm ation
abotuhtne u mboeferx p ectdec lusitnte hrdesom ai.nF inaLlalngyl,ey&
Sage( cha4p)ptr evoried a genearnaallo yfl seiasr rnaiotnfetgw oa lgorithms
inth ep resoefni cnec renausmbienorfgi rrelfeevaatnutr es.
Othearrt icledsi reactttlsayoc mkpe ro blteamh tm akessc agl airhdn- in
partitchurelo abru,s atnnedeffi scs ieonftc hyle e arsnyisntgAe gmain., f ew
ifa noyf t heea rlMya chLineea rning rceosnseiatdrhecrerhob eeudrsn setss
oft heliera rsnyisntgae gma ifnosertx ,a mpilrer,e loernv oainfstey a tures.
Thef osci,un stweaaosdn t, h nea tuorfteh e a lgoribtehhma'vosind o art a
obtaiinnt ehdbe e softw orlhdesn;tc hee esareyl m achlienaer snyisnstg e m
were givceonm plcenotsein,st at nedn oise-fsreetes data
almosatl ways -­
i.eev.e,ro yb servaitsic oonpm osoefda laln-d-otnhrleey l efveaantt-u res
alfle atvuarleua erisen cluadndee dac,vh alius1e0 0%c orrOefcc tuo.re s,
CopyrightMeadt erial