Computer Vision: Algorithms and Applications - Szeliski.org

979 Pages·2010·22.14 MB·English

by  Richard Szeliski

Computer Vision:
Algorithms and Applications
Richard Szeliski
September3,2010draft
c 2010Springer
(cid:13)
Thiselectronicdraftisfornon-commercialpersonaluseonly,
andmaynotbepostedorre-distributedinanyform.
Pleasereferinterestedreaderstothebook’sWebsiteat
http://szeliski.org/Book/.

Thisbookisdedicatedtomyparents,
ZdzisławandJadwiga,
andmyfamily,
Lyn,Anne,andStephen.
1 Introduction 1
Whatiscomputervision? Abriefhistory
• •
Bookoverview Samplesyllabus Notation
• •
2 Imageformation 29
n^
Geometricprimitivesandtransformations
•
Photometricimageformation
•
Thedigitalcamera
3 Imageprocessing 99
Pointoperators Linearfiltering
• •
Moreneighborhoodoperators Fouriertransforms
• •
Pyramidsandwavelets Geometrictransformations
• •
Globaloptimization
4 Featuredetectionandmatching 205
Pointsandpatches
•
Edges Lines
•
5 Segmentation 267
Activecontours Splitandmerge
• •
Meanshiftandmodefinding Normalizedcuts
• •
Graphcutsandenergy-basedmethods
6 Feature-basedalignment 309
2Dand3Dfeature-basedalignment
•
Poseestimation
Geometricintrinsiccali•bration
7 Structurefrommotion 343
Triangulation Two-framestructurefrommotion
• •
Factorization Bundleadjustment
• •
Constrainedstructureandmotion
8 Densemotionestimation 381
Translationalalignment Parametricmotion
• •
Spline-basedmotion Opticalflow
• •
Layeredmotion
9 Imagestitching 427
Motionmodels Globalalignment
• •
Compositing
10 Computationalphotography 467
Photometriccalibration Highdynamicrangeimaging
• •
Super-resolutionandblurremoval
•
Imagemattingandcompositing
•
Textureanalysisandsynthesis
11 Stereocorrespondence 533
Epipolargeometry Sparsecorrespondence
• •
Densecorrespondence Localmethods
• •
Globaloptimization Multi-viewstereo
•
12 3Dreconstruction 577
ShapefromX Activerangefinding
• •
Surfacerepresentations Point-basedrepresentations
• •
Volumetricrepresentations Model-basedreconstruction
• •
Recoveringtexturemapsandalbedos
13 Image-basedrendering 619
Viewinterpolation Layereddepthimages
• •
LightfieldsandLumigraphs Environmentmattes
• •
Video-basedrendering
14 Recognition 655
Objectdetection Facerecognition
• •
Instancerecognition Categoryrecognition
• •
Contextandsceneunderstanding
•
Recognitiondatabasesandtestsets

Preface
Theseedsforthisbookwerefirstplantedin2001whenSteveSeitzattheUniversityofWash-
ingtoninvitedmetoco-teachacoursecalled“ComputerVisionforComputerGraphics”. At
thattime,computervisiontechniqueswereincreasinglybeingusedincomputergraphicsto
createimage-basedmodelsofreal-worldobjects,tocreatevisualeffects,andtomergereal-
world imagery using computational photography techniques. Our decision to focus on the
applicationsofcomputervisiontofunproblemssuchasimagestitchingandphoto-based3D
modelingfrompersonalphotosseemedtoresonatewellwithourstudents.
Sincethattime,asimilarsyllabusandproject-orientedcoursestructurehasbeenusedto
teachgeneralcomputervisioncoursesbothattheUniversityofWashingtonandatStanford.
(ThelatterwasacourseIco-taughtwithDavidFleetin2003.) Similarcurriculahavebeen
adoptedatanumberofotheruniversitiesandalsoincorporatedintomorespecializedcourses
oncomputationalphotography.(Forideasonhowtousethisbookinyourowncourse,please
seeTable1.1inSection1.4.)
Thisbookalsoreflectsmy20years’experiencedoingcomputervisionresearchincorpo-
rateresearchlabs,mostlyatDigitalEquipmentCorporation’sCambridgeResearchLaband
atMicrosoftResearch. Inpursuingmywork, Ihavemostlyfocusedonproblemsandsolu-
tiontechniques(algorithms)thathavepracticalreal-worldapplicationsandthatworkwellin
practice. Thus,thisbookhasmoreemphasisonbasictechniquesthatworkunderreal-world
conditionsandlessonmoreesotericmathematicsthathasintrinsicelegancebutlesspractical
applicability.
Thisbookissuitableforteachingasenior-levelundergraduatecourseincomputervision
to students in both computer science and electrical engineering. I prefer students to have
either an image processing or a computer graphics course as a prerequisite so that they can
spendlesstimelearninggeneralbackgroundmathematicsandmoretimestudyingcomputer
visiontechniques. Thebookisalsosuitableforteachinggraduate-levelcoursesincomputer
vision(bydelvingintothemoredemandingapplicationandalgorithmicareas)andasagen-
eralreferencetofundamentaltechniquesandtherecentresearchliterature.Tothisend,Ihave
attemptedwhereverpossibletoatleastcitethenewestresearchineachsub-field,evenifthe
viii ComputerVision: AlgorithmsandApplications(September3,2010draft)
technicaldetailsaretoocomplextocoverinthebookitself.
Inteachingourcourses,wehavefounditusefulforthestudentstoattemptanumberof
smallimplementationprojects,whichoftenbuildononeanother,inordertogetthemusedto
workingwithreal-worldimagesandthechallengesthatthesepresent. Thestudentsarethen
askedtochooseanindividualtopicforeachoftheirsmall-group,finalprojects. (Sometimes
these projects even turn into conference papers!) The exercises at the end of each chapter
contain numerous suggestions for smaller mid-term projects, as well as more open-ended
problems whose solutions are still active research topics. Wherever possible, I encourage
studentstotrytheiralgorithmsontheirownpersonalphotographs,sincethisbettermotivates
them, often leads to creative variants on the problems, and better acquaints them with the
varietyandcomplexityofreal-worldimagery.
Informulatingandsolvingcomputervisionproblems,Ihaveoftenfounditusefultodraw
inspirationfromthreehigh-levelapproaches:
Scientific: build detailed models of the image formation process and develop mathe-
•
maticaltechniquestoinverttheseinordertorecoverthequantitiesofinterest(where
necessary,makingsimplifyingassumptiontomakethemathematicsmoretractable).
Statistical: useprobabilisticmodelstoquantifythepriorlikelihoodofyourunknowns
•
andthenoisymeasurementprocessesthatproducetheinputimages,theninferthebest
possibleestimatesofyourdesiredquantitiesandanalyzetheirresultinguncertainties.
Theinferencealgorithmsusedareoftencloselyrelatedtotheoptimizationtechniques
usedtoinvertthe(scientific)imageformationprocesses.
Engineering: develop techniques that are simple to describe and implement but that
•
are also known to work well in practice. Test these techniques to understand their
limitation and failure modes, as well as their expected computational costs (run-time
performance).
Thesethreeapproachesbuildoneachotherandareusedthroughoutthebook.
Mypersonalresearchanddevelopmentphilosophy(andhencetheexercisesinthebook)
haveastrongemphasisontestingalgorithms. It’stooeasyincomputervisiontodevelopan
algorithmthatdoessomethingplausibleonafewimagesratherthansomethingcorrect. The
bestwaytovalidateyouralgorithmsistouseathree-partstrategy.
First,testyouralgorithmoncleansyntheticdata,forwhichtheexactresultsareknown.
Second, add noise to the data and evaluate how the performance degrades as a function of
noise level. Finally, test the algorithm on real-world data, preferably drawn from a wide
varietyofsources, suchasphotosfoundontheWeb. Onlythencanyoutrulyknowifyour
algorithm can deal with real-world complexity, i.e., images that do not fit some simplified
modelorassumptions.
Preface ix
Inordertohelpstudentsinthisprocess,thisbookscomeswithalargeamountofsupple-
mentarymaterial,whichcanbefoundonthebook’sWebsitehttp://szeliski.org/Book. This
material,whichisdescribedinAppendixC,includes:
pointerstocommonlyuseddatasetsfortheproblems,whichcanbefoundontheWeb
•
pointerstosoftwarelibraries,whichcanhelpstudentsgetstartedwithbasictaskssuch
•
asreading/writingimagesorcreatingandmanipulatingimages
slidesetscorrespondingtothematerialcoveredinthisbook
•
aBibTeXbibliographyofthepaperscitedinthisbook.
•
The latter two resources may be of more interest to instructors and researchers publishing
new papers in this field, but they will probably come in handy even with regular students.
Someofthesoftwarelibrariescontainimplementationsofawidevarietyofcomputervision
algorithms, which can enable you to tackle more ambitious projects (with your instructor’s
consent).
Acknowledgements
I would like to gratefully acknowledge all of the people whose passion for research and
inquiryaswellasencouragementhavehelpedmewritethisbook.
Steve Zucker at McGill University first introduced me to computer vision, taught all of
his students to question and debate research results and techniques, and encouraged me to
pursueagraduatecareerinthisarea.
TakeoKanadeandGeoffHinton,myPh.D.thesisadvisorsatCarnegieMellonUniversity,
taught me the fundamentals of good research, writing, and presentation. They fired up my
interest in visual processing, 3D modeling, and statistical methods, while Larry Matthies
introducedmetoKalmanfilteringandstereomatching.
DemetriTerzopouloswasmymentoratmyfirstindustrialresearchjobandtaughtmethe
ropes of successful publishing. Yvan Leclerc and Pascal Fua, colleagues from my brief in-
terludeatSRIInternational,gavemenewperspectivesonalternativeapproachestocomputer
vision.
DuringmysixyearsofresearchatDigitalEquipmentCorporation’sCambridgeResearch
Lab,Iwasfortunatetoworkwithagreatsetofcolleagues,includingIngridCarlbom,Gudrun
Klinker,KeithWaters,RichardWeiss,Ste´phaneLavalle´e,andSingBingKang,aswellasto
supervisethefirstofalongstringofoutstandingsummerinterns,includingDavidTonnesen,
SingBingKang,JamesCoughlan,andHarryShum. ThisisalsowhereIbeganmylong-term
collaborationwithDanielScharstein,nowatMiddleburyCollege.
x ComputerVision: AlgorithmsandApplications(September3,2010draft)
AtMicrosoftResearch,I’vehadtheoutstandingfortunetoworkwithsomeoftheworld’s
bestresearchersincomputervisionandcomputergraphics,includingMichaelCohen,Hugues
Hoppe,StephenGortler,SteveShafer,MatthewTurk,HarryShum,Anandan,PhilTorr,An-
tonioCriminisi, GeorgPetschnigg, KentaroToyama, RaminZabih, ShaiAvidan, SingBing
Kang, Matt Uyttendaele, Patrice Simard, Larry Zitnick, Richard Hartley, Simon Winder,
DrewSteedly,ChrisPal,NebojsaJojic,PatrickBaudisch,DaniLischinski,MatthewBrown,
SimonBaker,MichaelGoesele,EricStollnitz,DavidNiste´r,BlaiseAguerayArcas,Sudipta
Sinha, Johannes Kopf, Neel Joshi, and Krishnan Ramnath. I was also lucky to have as in-
ternssuchgreatstudentsasPolinaGolland,SimonBaker,MeiHan,ArnoScho¨dl,RonDror,
AshleyEden,JinxiangChai,RahulSwaminathan,YanghaiTsin,SamHasinoff,AnatLevin,
Matthew Brown, Eric Bennett, Vaibhav Vaish, Jan-Michael Frahm, James Diebel, Ce Liu,
JosefSivic,GrantSchindler,ColinZheng,NeelJoshi,SudiptaSinha,ZeevFarbman,Rahul
Garg,TimCho,YekeunJeong,RichardRoberts,VarshaHedau,andDilipKrishnan.
WhileworkingatMicrosoft,I’vealsohadtheopportunitytocollaboratewithwonderful
colleaguesattheUniversityofWashington,whereIholdanAffiliateProfessorappointment.
I’m indebted to Tony DeRose and David Salesin, who first encouraged me to get involved
with the research going on at UW, my long-time collaborators Brian Curless, Steve Seitz,
Maneesh Agrawala, Sameer Agarwal, and Yasu Furukawa, as well as the students I have
hadtheprivilegetosuperviseandinteractwith,includingFre´dericPighin,Yung-YuChuang,
Doug Zongker, Colin Zheng, Aseem Agarwala, Dan Goldman, Noah Snavely, Rahul Garg,
and Ryan Kaminsky. As I mentioned at the beginning of this preface, this book owes its
inception to the vision course that Steve Seitz invited me to co-teach, as well as to Steve’s
encouragement,coursenotes,andeditorialinput.
I’m also grateful to the many other computer vision researchers who have given me so
manyconstructivesuggestionsaboutthebook,includingSingBingKang,whowasmyinfor-
malbookeditor,VladimirKolmogorov,whocontributedAppendixB.5.5onlinearprogram-
mingtechniquesforMRFinference,DanielScharstein,RichardHartley,SimonBaker,Noah
Snavely, Bill Freeman, Svetlana Lazebnik, Matthew Turk, Jitendra Malik, Alyosha Efros,
Michael Black, Brian Curless, Sameer Agarwal, Li Zhang, Deva Ramanan, Olga Veksler,
YuriBoykov,CarstenRother,PhilTorr,BillTriggs,BruceMaxwell,JanaKosˇecka´,EeroSi-
moncelli,AaronHertzmann,AntonioTorralba,TomasoPoggio,TheoPavlidis,BabaVemuri,
Nando de Freitas, Chuck Dyer, Song Yi, Falk Schubert, Roman Pflugfelder, Marshall Tap-
pen,JamesCoughlan,SammyRogmans,KlausStrobel,Shanmuganathan,AndreasSiebert,
YongjunWu,FredPighin,JuanCockburn,RonaldMallet,TimSoper,GeorgiosEvangelidis,
Dwight Fowler, Itzik Bayaz, Daniel O’Connor, and Srikrishna Bhat. Shena Deuchers did a
fantastic job copy-editing the book and suggesting many useful improvements and Wayne
Wheeler and Simon Rees at Springer were most helpful throughout the whole book pub-
lishing process. Keith Price’s Annotated Computer Vision Bibliography was invaluable in