Table Of ContentLecture Notes in Computer Science 7082
CommencedPublicationin1973
FoundingandFormerSeriesEditors:
GerhardGoos,JurisHartmanis,andJanvanLeeuwen
EditorialBoard
DavidHutchison
LancasterUniversity,UK
TakeoKanade
CarnegieMellonUniversity,Pittsburgh,PA,USA
JosefKittler
UniversityofSurrey,Guildford,UK
JonM.Kleinberg
CornellUniversity,Ithaca,NY,USA
AlfredKobsa
UniversityofCalifornia,Irvine,CA,USA
FriedemannMattern
ETHZurich,Switzerland
JohnC.Mitchell
StanfordUniversity,CA,USA
MoniNaor
WeizmannInstituteofScience,Rehovot,Israel
OscarNierstrasz
UniversityofBern,Switzerland
C.PanduRangan
IndianInstituteofTechnology,Madras,India
BernhardSteffen
TUDortmundUniversity,Germany
MadhuSudan
MicrosoftResearch,Cambridge,MA,USA
DemetriTerzopoulos
UniversityofCalifornia,LosAngeles,CA,USA
DougTygar
UniversityofCalifornia,Berkeley,CA,USA
GerhardWeikum
MaxPlanckInstituteforInformatics,Saarbruecken,Germany
Daniel Cremers Marcus Magnor
Martin R. Oswald Lihi Zelnik-Manor (Eds.)
Video Processing
and Computational Video
International Seminar
Dagstuhl Castle, Germany, October 10-15, 2010
Revised Papers
1 3
VolumeEditors
DanielCremers
TechnischeUniversitätMünchen,Germany
E-mail:[email protected]
MarcusMagnor
TechnischeUniversitätBraunschweig,Germany
E-mail:[email protected]
MartinR.Oswald
TechnischeUniversitätMünchen,Germany
E-mail:[email protected]
LihiZelnik-Manor
TheTechnion,IsraelInstituteofTechnology,Haifa,Israel
E-mail:[email protected]
ISSN0302-9743 e-ISSN1611-3349
ISBN978-3-642-24869-6 e-ISBN978-3-642-24870-2
DOI10.1007/978-3-642-24870-2
SpringerHeidelbergDordrechtLondonNewYork
LibraryofCongressControlNumber:2011938798
CRSubjectClassification(1998):I.4,I.2.10,I.5.4-5,F.2.2,I.3.5
LNCSSublibrary:SL6–ImageProcessing,ComputerVision,PatternRecognition,
andGraphics
©Springer-VerlagBerlinHeidelberg2011
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,re-useofillustrations,recitation,broadcasting,
reproductiononmicrofilmsorinanyotherway,andstorageindatabanks.Duplicationofthispublication
orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,1965,
initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.Violationsareliable
toprosecutionundertheGermanCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnotimply,
evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevantprotectivelaws
andregulationsandthereforefreeforgeneraluse.
Typesetting:Camera-readybyauthor,dataconversionbyScientificPublishingServices,Chennai,India
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
With the swift development of video imaging technology and the, drastic im-
provementsinCPUspeedandmemory,bothvideoprocessingandcomputational
video are becoming more and more popular. Similar to the digital revolution in
photography of fifteen years ago, today digital methods are revolutionizing the
waytelevisionandmoviesarebeingmade.Withtheadventofprofessionaldigital
movie cameras, digital projector technology for movie theaters, and 3D movies,
the movie and television production pipeline is turning all-digital, opening up
numerous new opportunities for the way dynamic scenes are acquired, video
footage can be edited, and visual media may be experienced.
This book provides a compilation of selected articles resulting from a wor-
shopon“VideoProcessingandComputationalVideo”,heldatDagstuhlCastle,
Germany in October 2010. During this workshop, 43 researchers from all over
the worlddiscussedthestate ofthe art,contemporarychallenges,andfuture re-
searchin imaging,processing,analyzing,modeling, and rendering of real-world,
dynamicscenes.Theseminarwasorganizedinto11sessionsofpresentations,dis-
cussions, and special-topic meetings. The seminar brought together junior and
seniorresearchersfromcomputer vision,computer graphics,and imagecommu-
nication, both from academia and industry to address the challenges in compu-
tational video.
Forfivedays,workshopparticipantsdiscussedtheimpactofaswellastheop-
portunitiesarisingfromdigitalvideoacquisition,processing,representation,and
display. Over the course of the seminar, the participants addressed contempo-
rarychallengesindigitalTVandmovieproduction;pointedatnewopportunities
in an all-digital production pipeline; discussed novel ways to acquire; represent
andexperiencedynamiccontent;accruedawish-listforfuture videoequipment;
proposed new ways to interact with visual content; and debated possible future
mass-marketapplications for computational video.
Viable research areas in computational video identified during the seminar
included motion capture of faces, non-rigid surfaces, and entire performances;
reconstruction and modeling of non-rigid objects; acquisition of scene illumina-
tion, time-of-flight cameras;motion field and segmentation estimation for video
editing; as well as free-viewpoint navigation and video-based rendering. With
regardto technologicalchallenges,seminar participants agreedthat the “rolling
shutter” effect of CMOS-based video imagers currently poses a serious prob-
lem for existing computer vision algorithms. It is expected, however, that this
problem will be overcome by future video imaging technology. Another item on
the seminar participants’ wish list for future camera hardware concerned high
frame-rate acquisition to enable more robust motion field estimation or time-
multiplexed acquisition. Finally, it was expected that plenoptic cameraswill hit
VI Preface
the commercial market within the next few years, allowing for advanced post-
processingfeaturessuchasvariabledepth-of-field,stereopsis,ormotionparallax.
The papers presented in these post-workshop proceedings were carefully se-
lected througha blind peer-reviewprocess with three independent reviewersfor
each paper.
WearegratefultothepeopleatDagstuhlCastleforsupportingthisseminar.
We thank all participants for their talks and contributions to discussions and
all authors who contributed to this book. Moreover, we thank all reviewers for
their elaborate assessment and constructive criticism, which helped to further
improve the quality of the presented articles.
August 2011 Daniel Cremers
Marcus Magnor
Martin R. Oswald
Lihi Zelnik-Manor
Table of Contents
Video Processing and Computational Video
Towards Plenoptic Raumzeit Reconstruction......................... 1
Martin Eisemann, Felix Klose, and Marcus Magnor
Two Algorithms for Motion Estimation from Alternate Exposure
Images ......................................................... 25
Anita Sellent, Martin Eisemann, and Marcus Magnor
UnderstandingWhatWeCannotSee:AutomaticAnalysisof4DDigital
In-Line Holographic Microscopy Data............................... 52
Laura Leal-Taix´e, Matthias Heydt, Axel Rosenhahn, and
Bodo Rosenhahn
3D Reconstruction and Video-Based Rendering of Casually Captured
Videos.......................................................... 77
Aparna Taneja, Luca Ballan, Jens Puwein, Gabriel J. Brostow, and
Marc Pollefeys
Silhouette-Based Variational Methods for Single View
Reconstruction .................................................. 104
Eno T¨oppe, Martin R. Oswald, Daniel Cremers, and Carsten Rother
Single Image Blind Deconvolution with Higher-Order Texture
Statistics ....................................................... 124
Manuel Martinello and Paolo Favaro
Compressive Rendering of Multidimensional Scenes................... 152
Pradeep Sen, Soheil Darabi, and Lei Xiao
Efficient Rendering of Light Field Images ........................... 184
Daniel Jung and Reinhard Koch
Author Index.................................................. 213
Towards Plenoptic Raumzeit Reconstruction
Martin Eisemann, Felix Klose, and Marcus Magnor
Computer Graphics Lab, TU Braunschweig, Germany
Abstract. Thegoalofimage-basedrenderingistoevokeavisceralsense
ofpresenceinasceneusingonlyphotographsorvideos.Ahugevarietyof
differentapproacheshavebeendevelopedduringthelast decade.Exam-
iningtheunderlyingmodelswefindthreedifferentmaincategories:view
interpolation based on geometry proxies, pureimage interpolation tech-
niquesand complete sceneflow reconstruction. In thispaperwe present
threeapproachesforfree-viewpointvideo,oneforeachofthesecategories
anddiscusstheirindividualbenefitsanddrawbacks.Wehopethatstudy-
ingthedifferentapproacheswillhelpothersinmakingimportantdesign
decisions when planning a free-viewpoint videosystem.
Keywords: Free-Viewpoint Video, Image-Based Rendering, Dynamic
Scene-Reconstruction.
1 Introduction
As humans we perceive most of our surroundings through our eyes, and visual
stimuli affect all of our senses, drive emotion, arouse memories and much more.
That is one of the reasons why we like to look at pictures. A major revolution
occurred with the introduction of moving images, or videos. The dimension of
time was suddenly added, which gave incredible freedom to film- and movie
makers to tell their story to the audience.
With more powerful hardware, computation power and clever algorithms we
are now able to add a new dimension to videos, namely the third spatial di-
mension. This will give users or producers the possibility to change the camera
viewpointonthefly.Butthereisadifferencebetweenthespatialdimensionand
time. While free-viewpointvideo allowsto change the viewpoint notonly to po-
sitionscapturedbytheinputcamerasbutalsotoanyotherpositionin-between,
the dimension of time is usually only captured at discrete time steps, caused by
the recording framerate of the input cameras. For a complete scene flow repre-
sentation, not only space but also time needs to be reconstructed faithfully.
In this paper we present three different approaches for free-viewpoint video
and space-time interpolation. After reviewing related work in the next section
we will continue with our Floating Textures [1] in Section 3 as an example for
highqualityfree-viewpointvideowithdiscretetimesteps.Foreachdiscretetime
stepa geometryof the scene is reconstructedandtexturedby multiview projec-
tive texture mapping, as this process is error-pronewe present a warping-based
refinement to correct for the resulting artifacts. In Section 4 we will describe
D.Cremersetal.(Eds.):VideoProcessingandComputationalVideo,LNCS7082,pp.1–24,2011.
(cid:2)c Springer-VerlagBerlinHeidelberg2011
2 M. Eisemann, F. Klose, and M. Magnor
the transition from discrete to continuous space-time interpolation, by discard-
ing the geometry and working on images correspondences alone we can create
perceptually plausible image interpolations [2,3,4]. As purely image-based ap-
proaches place restrictions on the viewing position, we introduce an algorithm
towardscompletesceneflowreconstructioninSection5[5].Allthreeapproaches
havetheirbenefits anddrawbacksandthe choiceshouldalwaysbe basedonthe
requirements one has for his application.
2 Related Work
In a slightly simplified version,the plenoptic function P(x,y,z,θ,φ,t) describes
radianceasafunctionof3-Dpositioninspace(x,y,z),direction(θ,φ) andtime
t [6]. With sufficiently many input views a direct reconstructionof this function
ispossible.Initiallydevelopedforstaticsceneslightfieldrendering[7]ispossibly
the most puristic and closest variant to direct resampling. Light field rendering
can be directly extended to incorporate discrete [8,9] or even continuous time
steps[10].Tocoveralargerdegreeofviewinganglesatacceptableimagequality,
however, a large number of densely packed images are necessary [11,12,13]. By
employing a prefiltering step the number of necessary samples can be reduced,
but at the cost of more blurry output images [14,15,16].
Wider camera spacings require more sophisticated interpolation techniques.
One possibility is the incorporation of a geometry proxy. Given the input im-
ages a 3D representation for each discrete time step is reconstructed and used
for depth guided resampling of the plenoptic function [17,18,19,20,21,22]. For
restricted scene setups the incorporation of template models proves beneficial
[23,24,25].Onlyfewapproachesreconstructatemporallyconsistentmesh,which
allows for continuous time interpolation [26,27]. To deal with insufficient recon-
structionaccuracyAganjet al.[28]andTakaiet al.[29]deformtheinputimages
or both the mesh and the input images, respectively, to diminish rendering ar-
tifacts. Unfortunately none of these approaches allow for real-time rendering
without a time intensive preprocessing phase.
Insteadofreconstructinga geometryproxy,purely image-basedinterpolation
techniques rely only onimagecorrespondences.If complete correspondencesbe-
tween image pixels can be established, accurate image warping becomes possi-
ble [30].Mark et al. [31]follow the seminalapproachof Chen et al. [30] but also
handle occlusion and discontinuities during rendering.While useful to speed up
renderingperformance,theirapproachesareonlyapplicableto syntheticscenes.
Beieret al. [32]proposeto use a manuallyguidedline-basedwarpingmethod to
interpolate between two images, known from its use in Michael Jackson’s music
video”Black&White”.Aphysically-validviewsynthesisbyimageinterpolation
is proposed by Seitz et al. [33,34].
For very similar images, optical flow techniques have proven useful [35,36].
Highlypreciseapproachesexist,whichcanbecomputedatreal-timeor(almost)
interactive rates [37,38,39]. Einarsson et al. [40] created a complete acquisition
system, the so-called light stage 6, for acquiring and relighting human locomo-
tion.Due tothehighamountofimagesacquiredtheycoulddirectlyincorporate
Towards Plenoptic Raumzeit Reconstruction 3
optical flow techniques to create virtual camera views in a light field renderer,
by direct warping of the input images.
Correspondence estimation is only one part of an image-based renderer. The
imageinterpolationitselfisanothercriticalpart.Fitzgibbonetal.[41]useimage-
based priors, i.e., they enforce similarity to the input images, to remove any
ghosting artifacts. Drawbacks are very long computation times and the input
images must be relatively similar in order to achieve good results. Mahajan et
al. [42] proposed a method for plausible image interpolation that searches for
the optimal path for a pixel transitioning from one image to the other in the
gradientdomain.Aseachoutputpixelintheinterpolatedviewistakenfromonly
a single source image, ghosting or blurring artifacts are avoided, but if wrong
correspondences are estimated unaesthetic deformations might occur. Linz et
al. [43] extend the approach of Mahajan et al. [42] to space-time interpolation
withmulti-imageinterpolationbasedonGraph-cutsandsymmetricopticalflow.
In the unstructured video rendering by Ballan et al. [44] the static background
ofasceneisdirectlyreconstructed,whiletheactorintheforegroundisprojected
onto a billboard and the view switches at a specific point between the cameras
where the transition is least visible.
3 Floating Textures
Image-based rendering (IBR) systems using a geometry proxy have the benefit
of free camera movement for each reconstructed time step. The drawback is
that any reconstruction method with an insufficient number of input images is
imprecise. While this may be no big problem when looking at the mesh alone,
it will become rather obvious when the mesh is to be textured again. Therefore
the challengeisgeneratinga perceptuallyplausible renderingwith onlyasparse
setup of cameras and a possibly imperfect geometry proxy.
Commonly in IBR the full bidirectional reflectance distribution function,
i.e. how a point on a surface appears depending on the viewpoint and light-
ing, is approximated by projective texture mapping [45] and image blending.
Typically the blending factors are based on the angular deviation of the view
vector to capture view-dependent effects. If too few input images are available
or the geometry is too imprecise, ghosting artifacts will appear as the projected
textures will not match on the surface.
In this section we assume that a set of input images, the corresponding,pos-
sibly imprecise, calibrationdata and a geometry proxy are given. The task is to
findawaytotexturethisproxyagainwithoutnoticeableartifactsandshadowing
the imprecision of the underlying geometry.
Without occlusion, any novel viewpoint can, in theory, be rendered directly
from the input images by warping, i.e. by simply deforming the images, so that
the following property holds:
Ij =WI →I ◦Ii , (1)
i j
where WI →I ◦Ii warps an image Ii towards Ij according to the warp field
i j
WI →I .TheproblemofdeterminingthewarpfieldWI →I betweentwoimages
i j i j
4 M. Eisemann, F. Klose, and M. Magnor
!t
Fig.1. Rendering with Floating Textures [1]. The input photos are projected from
camera positions Ci onto the approximate geometry GA and onto the desired image
plane of viewpoint V. The resulting intermediate images Ivi exhibit mismatch which
is compensated by warping all Ivi based on the optical flow to obtain the final image
IvFloat.
Ii, Ij is a heavily researched area in computer graphics and vision. If pixel
distances betweencorrespondingimage features arenottoo large,algorithmsto
robustly estimate per-pixel optical flow are available [37,46]. The issue here is
that in most cases these distances will be too large.
Inordertorelaxthecorrespondencefindingproblem,theproblemcanliterally
be projected into another space, namely the output image domain. By first
projecting the photographs from cameras Ci onto the approximate geometry
surface GA and rendering the scene from the desired viewpoint Cv, creating
the intermediate images Iv, the correspondingimage features are broughtmuch
i
closer together than they have been in the originalinput images,Figure 1. This
opens up the possibility of using optical flow estimation to the intermediate
imagesIvi torobustlydeterminethepairwiseflowfieldsWIv→Iv.Tocompensate
i j
formorethantwoinputimages,alinearcombinationoftheflowfieldsaccording
to (3) can be applied to all intermediate images Iv, which can then be blended
i
togethertoobtainthefinalrenderingresultIv .Toreducecomputationalcost,
Float
instead of establishing for n input photos (n−1)n flow fields, it often suffices
to consider only the 3 closest input images to the current viewpoint. If more
than 3 input images are needed, the quadratic effort can be reduced to linear
complexity by using intermediate results.
Theprocessingstepsaresummarizedinthefollowingfunctionsandvisualized
in Figure 1:
(cid:2)n
IvFloat = (WIvi ◦Ivi)ωi (2)
i=1
(cid:2)n
WIv = ωjWIv→Iv (3)
i i j
j=1
