Chapter 1 of High Dynamic Range Video by Frédéric Dufaux, Patrick Le Callet, Rafal Mantiuk, Marta Mrak

59 Pages·1.642 MB·English and mathematics

by  Steve Mann and Mir Adnan Ali

CHAPTER
1
THE FUNDAMENTAL BASIS OF
HDR: COMPARAMETRIC
EQUATIONS
S.Mann ,†,‡,M.A.Ali
∗ ∗
UniversityofToronto,Toronto,ON,Canada
∗
RotmanSchoolofManagementCDL,Toronto,ON,Canada†
Meta,RedwoodCity,CA,UnitedStates‡
CHAPTER OUTLINE
1.1 IntroductiontoHighDynamicRangeImaging..................................................................... 2
1.1.1 TheFundamentalConceptofHDRSensingandMetasensing................................ 2
1.1.2 TheFundamentalPrincipleofHDR:DynamicRangeandDynamageRange............... 4
1.1.3 HDRImagingTechniques.......................................................................... 5
1.1.4 HDRFromMultipleExposures.................................................................... 5
1.2 HistoricalMotivationforHDRImaging............................................................................. 5
1.3 TheoryofHDRImaging................................................................................................ 6
1.3.1 TheWyckoffPrincipleandtheRangeofLight.................................................. 7
1.3.2 What’sGoodfortheDomainIsGoodfortheRange............................................ 8
1.3.3 Extending Dynamic Range and Improvement of Range Resolution by Combining
DifferentlyExposedPicturesoftheSameSubjectMatter .................................... 8
1.3.4 ThePhotoquantity,q................................................................................ 9
1.3.5 TheCameraasanArrayofLightMeters......................................................... 9
1.3.6 TheAccidentallyDiscoveredCompander........................................................ 10
1.3.7 WhyStockhamWasWrong......................................................................... 12
1.3.8 TheValueofDoingtheExactOppositeofWhatStockhamAdvocated...................... 13
1.3.9 UsingDifferentlyExposedPicturesoftheSameSubjectMattertoGetaBetterEstimate
ofq.................................................................................................... 14
1.3.10ExposureInterpolationandExtrapolation........................................................ 18
1.4 ComparametricImageProcessing:ComparingDifferentlyExposedImagesoftheSameSubjectMatter 18
1.4.1 MisconceptionsAboutGammaCorrection....................................................... 18
1.4.2 ComparametricPlotsandComparametricEquations.......................................... 20
1.4.3 ZetaCorrectionofImages.......................................................................... 21
1.4.4 TheAffineComparametricEquationandAffineCorrectionofImages....................... 22
1.4.5 ThePreferredCorrectionofImages............................................................... 24
1.4.6 Some Solutionsto Some Comparametric EquationsThat Are ParticularlyIllustrative
orUseful.............................................................................................. 25
1.4.7 PropertiesofComparametricEquations ......................................................... 25
1
HighDynamicRangeVideo.http://dx.doi.org/10.1016/B978-0-08-100412-8.00001-2
©2016ElsevierLtd.Allrightsreserved.
2 CHAPTER 1 THE FUNDAMENTALBASIS OF HDR
1.5 PracticalImplementations........................................................................................... 27
1.5.1 ComparingTwoImagesThatDifferOnlyinExposure.......................................... 27
1.5.2 JointHistogramsandComparagrams............................................................. 27
1.5.3 ComparametricRegressionandtheJointHistogram........................................... 28
1.5.4 ComparametricRegressiontoaStraightLine................................................... 29
1.5.5 ComparametricRegressiontothePreferredModel............................................. 31
1.6 ToneMappinginHDRSystems...................................................................................... 34
1.6.1 AnExtremeExampleWithSpatiotonalProcessingofPhotoquantities...................... 36
1.7 AnalyticalSolutionofComparametricEquations................................................................. 38
1.7.1 Overview.............................................................................................. 38
1.7.2 FormalSolutionbyScalingOperator............................................................. 38
1.7.3 SolutionbyOrdinaryDifferentialEquation...................................................... 39
1.8 CompositingasBayesianJointEstimation......................................................................... 40
1.8.1 ExampleJointEstimator ........................................................................... 44
1.8.2 DiscussionRegardingCompositingviatheCCRF............................................... 46
1.8.3 ReviewofAnalyticalComparametricEquations ................................................ 47
1.9 EfficientImplementationofHDRReconstruction................................................................. 48
1.9.1 Implementation...................................................................................... 50
1.9.2 CompressionPerformance ......................................................................... 53
1.9.3 Conclusion............................................................................................ 56
Acknowledgments........................................................................................................... 57
References.................................................................................................................... 57
1.1 INTRODUCTION TO HIGH DYNAMIC RANGE IMAGING
High dynamic range (HDR) imaging originated in the 1970s through digital eyeglass (DEG) and
wearable computing as a seeing aid and, more generally, wearable technology as a sensory aid,
which includes also HDR metasensing (the sensing of sensors and the sensing of their capacity to
sense). HDR video as a way of seeing included other senses as well, such as HDR radar for the
blind (Mann, 2001), and as a way of seeing radio waves with augmented reality overlays (Mann,
1992, 2001) (http://wearcam.org/swim/).Wearable computing gives rise to a rich sensory landscape
that includes video, audio, and radar, plus physiological signals such as electrocardiogram and
electroencephalogram,all of which made use of HDR signal capture and processing (Mann, 2001).
Thisworkispartofthegeneralfield ofresearchandpracticeknownas“sousveillance”(undersight),
definedas“wearingandimplantingvarioussensors,effectors,andmultimediacomputationinorderto
redefinepersonalspaceandmodifysensoryperceptioncomputationally”(Mann,2004;Shakhakarmi,
2014).
1.1.1 THE FUNDAMENTAL CONCEPT OF HDR SENSING AND METASENSING
We beginbyintroducingthe fundamentalconceptof HDR sensing, whichallows thedynamicrange
of a sensor to be increased toward its “dynamage range” — that is, for a sensor to capture the full
1.1 INTRODUCTION TO HIGH DYNAMIC RANGE IMAGING 3
FIGURE1.1
Left:World’searliestknownphotograph,takenin1826onaplatecoatedinbitumen(petroleumtar,theasphalt
commonlyusedasroofingmaterial).ImagefromWikimediaCommons.Center:Atypicalrooftopwithmodified
bitumenshingles.Bitumeniscommonlycoveredinstonesorgranulestoprotectitfromdamagebysunlight.
Nevertheless,thesouth-facingsideoftheroofshowsextensivedamageduetoexposuretosunlight.Right:
Closeupshowingsunlight-damagedsouthernexposure.
rangeofsignalsallthewaytoitslimit,asdefinedasthelimittowhatitcansensewithoutpermanent
damagetoit.
Photography has a long and interesting history. For many years, it has been known that objects
exposed to bright sunlight fade. Of particular note is the fact that bitumen (petroleum tar, asphalt,
pitch,orthelike),commonlyusedonrooftops,iseasilydamagedandbecomeshard,brittle,andfragile
withprolongedexposuretolight(Fig.1.1,centerandright).Theworld’sfirstknownphotographwas
captured,in1826,onaplatecoatedinbitumen(Fig.1.1,left).Subsequently,variousimprovementsto
photographyweremadetomakematerialsmoresensitivetolight,butthedynamicrangeofphotographs
wastypicallylessthanthatofhumanvision.Similarly,theinventionofmotionpicturesandoftelevision
gaverisetovideocapture,butalsowithsimilarproblemsregardingdynamicrange.
Early video cameras (Fig. 1.2, left) used image pickup tubes that could be easily damaged by
overexposure. As a result, the user must be careful with the f-stop on the camera lens, not to
open it too far, and many cameras used to have an iris control button to protect the camera from
damage:
Iriscontrolbutton:Afeaturethatclosesdowntheirisorapertureofthelenstoprotectthesensitive
videocameratubewhenthecameraisnotinoperation.Cameratubesthatareexposedtooverlybright
lightorsundevelopa‘burn’thatmaybecomepermanent”(JackandTsatsulin,2002).
Also, “some lenses even have a ‘C’ setting after the highest f-stop which means the lens is
completelyclosed,lettingnolightthroughatall”(InmanandSmith,2009).
Becauseearlyvideocameraswereeasilydamagedbyexcessivelightexposure,theuserhadtobe
carefulwiththef-stoponthecameralens,andopenituponlyenoughtogetaproperexposure.
HDR video was made possible by the invention of sensors (eg, camera image sensors) that can
be overexposed without permanent damage. Unlike old vidicon cameras, for which there was less
difference between the dynamic range and dynamage range, modern cameras have a much greater
dynamage range than their dynamic range. This allows them to produce images that are massively
4 CHAPTER 1 THE FUNDAMENTALBASIS OF HDR
FIGURE1.2
Left:Earlyvideocameras(televisioncameras),suchasthisoneatthe1936SummerOlympics,usedcamera
tubesthatcouldbepermanentlydamagedbyoverexposure.ImagefromWikimediaCommons.Centerand
right:Earlytelevisioncameralensestypicallyhadaniriswitha“C”setting,whichmeansthattheirisis
completelyclosed.Thiswastoprotectthesensorfrombeingdamagedbylight.Whenthecamerawasnotin
use,orwaspassingintoaregionofbrightlight,thelenscouldbeclosedtoprotectthesensorfromdamage.
Generally,thecameraoperatorhadtobeverycarefultoopenuptheirisonlyenoughforaproperexposure,but
notsofarastopermanentlydamagethecamerasensor.
overexposed, thus making it possible to capture images at an extreme range of exposures, and it is
theseextremeexposuresthatallowustoseeextremeshadowdetail(duetothemassivelyoverexposed
images) and extreme highlight detail (due to having one or more massively underexposed images
aswell).
HDR allows us to increase the dynamic range, ideally, all the way up to being equal to the
dynamagerange.
1.1.2 THE FUNDAMENTAL PRINCIPLE OF HDR: DYNAMIC RANGE
AND DYNAMAGE RANGE
HDRworkswheneverthedynamagerangeofasensorexceedsitsdynamicrange.
Dynamicrangeanddynamagerangearedefinedasfollows:
Dynamicrangeistheratiobetweenthelargestandsmallestnonnegativequantity,suchasmagnitude,
amplitude,energy,orthelike,ofsound,light,orthelike,forwhichasmallincrementaldifferencein
thequantitycanstillbesensed(ie,therangeoverwhichchangesinthequantityremaindiscernible)
(Mannetal.,2011,2012).
Dynamagerangeistheratiobetweenthelargestquantitythatwillnotdamageasensorordeviceor
receiver,andthesmallestnonnegativequantityforwhichchangesinthequantityremaindiscernible
(Mannetal.,2011,2012).
1.2 HISTORICALMOTIVATION FOR HDR IMAGING 5
1.1.3 HDR IMAGING TECHNIQUES
HDRimagingisthesetoftechniquesthatcomputationallyextendtheusualorstandarddynamicrange
ofasignal.HDRimaginghasariseninmultiplefields,suchascomputationalphotography,computer
graphics,andanimation.HDRsignalsmaybeproducedinseveralways:bythecombiningofmultiple
lowerdynamicrangesignalsforHDRreconstruction;synthetically,bysimulationorraytracing;orby
useofHDRsensorsfordataacquisition.
1.1.4 HDR FROM MULTIPLE EXPOSURES
Accordingto Robertsonet al. (2003),“thefirst reportof digitallycombiningmultiplepicturesofthe
samescenetoimprovedynamicrangeappearstobe(Mann,1993).”
HDRimagingbyreconstructionfrommultipleexposuresisdefinedasfollows:
DefinitionofHDRreconstruction:Theestimationofatleastonephotoquantityfromapluralityof
differentlyexposedimagesofthesamesceneorsubjectmatter(Mann,1993,2000,2001;Mannand
Picard,1995a;AliandMann,2012;Robertsonetal.,2003;Reinhardetal.,2005).
Specifically,HDRreconstructionreturnsanestimateofaphotoquantity(orsequenceofestimates
inthecaseofvideo),q(x,y)(anypossiblyspatiallyortemporallyvaryingqorq(x),q(x,y,z),q(x,y,t),
orq(x,y,z,t)),onthebasisofa pluralityofexposuresf f(kq(x,y)),atexposuresettingsk, where
i i i
=
thereisalsonoiseineachoftheseexposuresf,throughacameraresponsefunction,f,whichisoften
i
unknown (although it may also be known, or it may be linear, or it may be the identity map). The
exposuresettingsk mayalsobeunknown.
i
Aseparateoptionalstepoftonemappingthephotoquantigraph,q,maybetaken,ifdesired—for
example, to produce an output image that can be printed or displayed on low dynamic range (LDR)
output media. In situations where there is no need for a human-viewable HDR image (eg, HDR-
based input to a computer vision system such as the wearable face-recognizer Mann, 1996b), the
photoquantigraphmayhavedirectusewithouttheneedtoconvertittoanLDRimage.
Atypicalapproachtogenerateqfromf istotransformeachoftheinputimagesf toestimatesof
i i
that photoquantity,and then to combine the results with use of a weighted sum (Mann, 1993, 2000,
2001;MannandPicard,1995a;DebevecandMalik,1997;Robertsonetal.,2003).Otherapproaches
are probabilistic in nature, and typically use nonlinear optimization (Ali and Mann, 2012; Pal et al.,
2004).
1.2 HISTORICAL MOTIVATION FOR HDR IMAGING
HDR reconstruction from multiple exposures originated with author S. Mann (described as “the
father of the wearable computer,” IEEE International Solid-State Circuits Conference, February 7,
2000), through a childhood fascination with sensing and metasensing that led to the invention of
the DEG (Fig. 1.3). This includes the use of wearable sensors to process and mediate daily life,
from wearable technologiesas a photographicart form (Mann, 1985;Ryals, 1995),to gesture-based
augmented/augmediatedreality (AR) (Mann, 1997b) for the capture, enhancement,and rendering of
6 CHAPTER 1 THE FUNDAMENTALBASIS OF HDR
FIGURE1.3
AuthorS.MannwithDEGandaDEG-basedHDRweldinghelmet.Left:OriginalDEG,incomparisonwitha
morerecentcommercialproduct.Right:EarlyDEG-basedweldinghelmetprototype.TheEyeTapprinciplewas
usedtocaptureeyeward-boundlightandprocessitwithHDRvideooverlays,alongwithaugmediatedreality:
augmentedindarkareaswith3Dcomputer-generatedoverlaysthatadapttoappearwheretheyareleast
distracting,whilethevisionisdeliberatelydiminishedinbrightareasoftheelectricweldingarc.
everydayexperience,a generalizedform of self-sensing known as “sousveillance”(“undersight”),in
contrast to the more familiar and common practice of “surveillance” (“oversight”). Sousveillance,
as a field of research, has recently expanded greatly, and been given a variety of new names — for
example,“lifelogging,”“quantifiedself,”“self-quantifying,’“self-quantification,”“personalimaging,”
“personal informatics,” “personal sensing,” “self-tracking,” “self-analytics,” “autoveillance,” “self-
(sur/sous)veillance,”“bodyhacking,”“personalmediaanalytics,”and“personalinformatics.”Sousveil-
lancealsoincludesmetaveillance.Metaveillanceistheseeingofsight,visualizationofvision,sensing
ofsensors,andsensingtheircapacitytosense(Fig.1.4).AccordingtoNicholasNegroponte,Founder,
Director,andChairmanoftheMITMediaLab,“SteveMannistheperfectexampleofsomeone...who
persistedinhisvisionandendedupfoundinganewdiscipline.”FromBangorDailyNews—Sep.26,
1997;laterappearsinTorontoStar—Jul.8,2001.
1.3 THEORY OF HDR IMAGING
The theoryand practiceofquantigraphicimage processing,with comparametricequations,arose out
of the field of sousveillance (wearable computing,quantimetricself-sensing, etc.) within the context
of mediatedreality (Mann,1997a)and personalimaging(Mann,1997b).However,it has potentially
much more widespread applications in image processing than just the wearable photographic and
videographicvisionsystemsforwhichitwasdeveloped.Accordingly,ageneralformulationthatdoes
notnecessarily involvea wearable camerasystem will be given.This section followsveryclosely,if
notidentically,thatgiveninMann(2000)andthetextbookIntelligentImageProcessing(Mann,2001).
1.3 THEORYOF HDR IMAGING 7
(A) (B) (C) (D)
FIGURE1.4
Sousveillance:(A)SequentialWaveImprintingMachinedevelopedbyauthorS.Manninthe1970sandearly
1980sformakinginvisiblefieldssuchassoundwavesandradiowaves,visible,andalsotosensesensing
itself(Mann,1992).Inthisexample,atelevisionreceiver(“rabbitears”antenna)pickedupthevideosignal
fromawirelesssurveillancecameraandamplifiedthesignalwithenoughstrengthtodirectlydriveanarrayof
35electriclightbulbs.(B)Becauseofvideofeedback,thelampsilluminatedwhentheywerewithinthefieldof
viewofthesurveillancecamera.TheeffectwasanARdisplayrenderingthecamera’ssightfield(Mann,2014)
visibletoanyonewatchingthelightwandbeingwavedbackandforth(noneedforspecialeyewear).Aproblem
withmetasensingisthemassivedynamicrangepresentinthesightfield(notjustthecamera’sfield/angleof
view,butalsothewidevariationinvisualacuity,sorendered:morelightmeantmoresight,butthelight
saturatedandmadeitdifficulttoseedifferencesintheamountofsightpresentateachpointinspace).(C)To
overcomethislimitation,pseudocolormetasensingwasused.Heremultiple-exposurephotographsweretaken
throughvariouscoloredfilterswhileametasensorydevicewasmovedthroughthespacearoundthesensor
(cameraorhumaneye).Recentworkonveillametrics(JanzenandMann,2014)alsousespseudocolorto
renderHDRmetasensoryimagesfromalargecollectionofhighlyaccuratescientificmeasurementsofcamera
orhumansight.(D)AuthorS.Mann,in1996,withvideo(electrovisuogram),electrocardiogram,
electroencephalogram, respiration,skinconductivity,andvariousothersensorsthatprovideahighlycomplete
capture,processing,recording,andtransmissionofphysiologicalbodyfunctionplussurrounding
environmentalinformation.
FromMann,S.,2001.IntelligentImageProcessing.JohnWileyandSons,NewYork,p.384.
1.3.1 THE WYCKOFF PRINCIPLE AND THE RANGE OF LIGHT
The quantity of light falling on an image sensor array, or the like, is a real-valued function q(x,y)
of two realvariablesx and y. An image is typically a degradedmeasurementof this function,where
degredationsmaybedividedintotwocategories,thosethatactonthedomain(x,y)andthosethatact
ontherangeq.Sampling,aliasing,andblurringactonthedomain,whilenoise(includingquantization
noise)andthenonlinearresponsefunctionofthecameraactontherangeq.
Registering and combining multiple pictures of the same subject matter will often result in an
improvedimageofgreaterdefinition.Therearefourclassesofsuchimprovement:
1. increasedspatialresolution(domainresolution),
2. increasedspatialextent(domainextent),
3. increasedtonalfidelity(rangeresolution),and
4. increaseddynamicrange(rangeextent).
8 CHAPTER 1 THE FUNDAMENTALBASIS OF HDR
1.3.2 WHAT’S GOOD FOR THE DOMAIN IS GOOD FOR THE RANGE
The notionof producinga better picture by combiningmultiple inputpictureshas been well studied
with regard to the domain (x,y) of these pictures. Horn and Schunk (1981), for example, provided
a means of determining optical flow, and many researchers have used this result to spatially register
multipleimagesinordertoprovideasingleimageofincreasedspatialresolutionandincreasedspatial
extent. Subpixel registration methods such as those proposed by Irani and Peleg (1991) and Mann
andPicard(1994)attempttoincreasedomainresolution.Thesemethodsdependonaslight(subpixel)
shiftfromoneimagetothenext.Imagecompositing(mosaicing)methodssuchasthoseproposedby
Mann(1993),MannandPicard(1995c),andSzeliski(1996)attempttoincreasedomainextent.These
methodsdependonlargeshiftsfromoneimagetothenext.
Methods that are aimed at increasing domain resolution and domain extent tend to also improve
tonalfidelity,toalimitedextent,byvirtueofasignal-averagingandnoise-reducingeffect.However,we
willseeinwhatfollowsageneralizationoftheconceptofsignalaveragingcalled“quantigraphicsignal
averaging.”Thisgeneralizedsignalaveragingallowsimagesof differentexposureto be combinedto
furtherimproveontonalfidelity(rangeresolution),beyondimprovementspossiblebytraditionalsignal
averaging.Moreover,theproposedmethoddrasticallyincreasesdynamicrange(rangeextent).Justas
spatialshiftsinthedomain(x,y)improvetheimage,wewillalsoseehowexposureshifts(shiftsinthe
range,q)can,withtheproposedmethod,resultinevengreaterimprovementstotheimage.
1.3.3 EXTENDING DYNAMIC RANGE AND IMPROVEMENT OF RANGE RESOLUTION
BY COMBINING DIFFERENTLY EXPOSED PICTURES OF THE SAME SUBJECT MATTER
The principles of quantigraphic image processing and the notion of the use of differently exposed
pictures of the same subject matter to make a picture composite of extended dynamic range were
inspiredbythepioneeringworkofWyckoff(1962,1961),whoinventedso-called“extendedresponse
film.”
Mosteverydaysceneshaveafargreaterdynamicrangethancanberecordedonaphotographicfilm
orelectronicimagingapparatus.However,asetofpicturesthatareidenticalexceptfortheirexposure
collectivelyshowusmuchmoredynamicrangethananysinglepicturefromthatset,andalsoallowthe
camera’sresponsefunctiontobeestimated,towithinasingleconstantscalarunknown(Mann,1993,
1996a;MannandPicard,1995b).
Asetoffunctions
fi(x) f(kiq(x)), (1.1)
=
where k are scalar constants, is known as a Wyckoff set (Mann, 1993, 1996a). A Wyckoff set of
i
functionsf(x)describesaset ofimagesdifferingonlyin exposurewhenx (x,y)isthecontinuous
i
=
spatial coordinate of the focal plane of an electronic imaging array (or piece of film), q is the
quantity of light falling on the array (or film), and f is the unknownnonlinearityof the camera’s(or
combinedfilm’sandscanner’s)responsefunction.Generally,f isassumedtobeapointwisefunction
(eg,invarianttox).
1.3 THEORYOF HDR IMAGING 9
1.3.4 THE PHOTOQUANTITY, q
The quantity, q, in Eq. (1.1), is called the “photoquantigraphic quantity (Mann, 1998),” or just the
“photoquantity”(or“photoq”)forshort.Thisquantityisneitherradiometric(eg,neitherradiancenor
irradiance)norphotometric(eg,neitherluminancenorilluminance).Mostnotably,becausethecamera
will not necessarily have the same spectral response as the human eye, or, in particular, that of the
photopic spectral luminous efficiency function as determined by the CIE and standardized in 1924,
qisnotbrightness,lightness,luminance,orilluminance.Instead,quantigraphicimagingmeasuresthe
quantityoflightintegratedoverthespectralresponseoftheparticularcamerasystem,
q ∞qs(λ)s(λ)dλ, (1.2)
= 0
Z
where q (λ) is the quantity of light falling on the image sensor and s is the spectral sensitivity of an
s
elementofthe sensorarray.Itis assumedthatthespectralsensitivitydoesnotvaryacrossthe sensor
array.
1.3.5 THE CAMERA AS AN ARRAY OF LIGHT METERS
The quantity q reads in units that are quantifiable (eg, linearized or logarithmic), in much the same
way that a photographiclight meter measures in quantifiable (linear or logarithmic)units. However,
justasthephotographiclightmeterimpartstothemeasurementitsownspectralresponse(eg,a light
meter using a selenium cell will impartthe spectral response of selenium cells to the measurement),
quantigraphic imaging accepts that there will be a particular spectral response of the camera, which
willdefinethequantigraphicunitq.Eachcamerawilltypicallyhaveitsownquantigraphicunit.Inthis
way,thecameramayberegardedasanarrayoflightmeters,eachbeingresponsivetothequantigral:
q(x,y) ∞qss(x,y,λ)s(λ)dλ, (1.3)
= 0
Z
whereq isthespatiallyvaryingspectraldistributionoflightfallingontheimagesensor.
ss
Thus,varyingnumbersofphotonsoflesser orgreaterenergy(frequencytimesPlanck’sconstant)
are absorbed by a given element of the sensor array, and, over the temporal quantigration time of a
single framein the videosequence(orthe exposuretime of a still image)resultin the photoquantity
givenbyEq.(1.3).
In the case of a color camera, or other color processes, q(x,y) is simply a vector quantity. Color
images may arise from as few as two channels, as in the old bichromatic (orange and blue) motion
pictures, but more typically arise from three channels, or sometimes more as in the four-coloroffset
printing, or even the high-quality Hexachrome printing process. A typical color camera might, for
example,includethreechannels—forexample, q (x,y),q (x,y),q (x,y) —whereeachcomponent
r g b
[ ]
isderivedfromaseparatespectralsensitivityfunction.Alternatively,anotherspacesuchasYIQ,YUV,
or the like may be used, in which, for example, the Y (luminance) channel has full resolution and
the U and V channels have reduced (eg, half in each linear dimension giving rise to one quarter the
numberofpixels)spatialresolutionandreducedquantizationaldefinition.PartIIIofthisbookcovers
representationandcodingofHDRvideoingeneral.
In this chapter, the theory will be developed and explained for grayscale images, where it is
understood that most images are color images, for which the procedures are applied either to the
10 CHAPTER 1 THE FUNDAMENTALBASIS OF HDR
separate color channels or by way of a multichannel quantigrahic analysis. Thus, in both cases
(grayscaleand color)the continuousspectralinformationq (λ) is lost throughconversionto a single
s
number q or to typically three numbers, q , q , q . Although it is easiest to apply the theory in this
r g b
chaptertocolorsystemshavingdistinctspectralbands,thereisnoreasonwhyitcannotalsobeapplied
tomorecomplicatedpolychromatic,possiblytensor,quantigrals.
Ordinarily,camerasgiverisetonoise—forexample,thereisnoisefromthesensorelementsand
further noise within the camera (or equivalently noise due to film grain and subsequent scanning of
a film, etc.). Thus, a goal of quantigraphic imaging is to attempt to estimate the photoquantity q in
the presenceof noise. Because q (λ) is destroyed,the best we can do is to estimate q. Thus, q is the
s
fundamentalor“atomic”unitofquantigraphicimageprocessing.
1.3.6 THE ACCIDENTALLY DISCOVERED COMPANDER
In general, cameras do not provide an output that varies linearly with light input. Instead, most
cameras contain a dynamic range compressor, as illustrated in Fig. 1.5. Historically, the dynamic
range compressor in video cameras arose because it was found that televisions did not produce a
linearresponsetothevideosignal.Inparticular,itwasfoundthatearlycathoderayscreensprovided
a light output approximately equal to voltage raised to the exponent of 2.5. Rather than build a
circuit into every television to compensate for this nonlinearity,a partial compensation (exponentof
Every camera equiv.
Subject to ideal photoq. camera Degraded depiction
matter CAMERA with noise and distortion Storage, DISPLAY of subject matter
ns" nsor Compressor tprraoncsemssisinsigo,n e,tc. Expander Cathode ray tube
e e
L S
"
∼
q f f1 f−1 q^
Light
rays n n
q f
Sensor noise Image noise
FIGURE1.5
Typicalcameraanddisplay.Lightfromsubjectmatterpassesthroughalens(typicallyapproximatedwith
simplealgebraicprojectivegeometry,eg,anidealized“pinhole”)andisquantifiedinunits“q”byasensorarray
wherenoisen isalsoadded,toproduceanoutputwhichiscompressedindynamicrangebyatypically
q
unknownfunctionf.Furthernoisen isintroducedbythecameraelectronics,includingquantizationnoiseif
f
thecameraisadigitalcameraandcompressionnoiseifthecameraproducesacompressedoutputsuchasa
JPEGimage,givingrisetoanoutputimagef (x,y).Theapparatusthatconvertslightraysintof (x,y)is
1 1
labeled“CAMERA.”Theimagef istransmittedorrecordedandplayedbackintoadisplaysystem(labeled
1
“DISPLAY”),wherethedynamicrangeisexpandedagain.Mostcathoderaytubesexhibitanonlinearresponse
tovoltage,andthisnonlinearresponseistheexpander.Theblocklabeled“expander”isgenerallyasideeffect
ofthedisplay,andisnotusuallyaseparatedevice.Itisdepictedasaseparatedevicesimplyforclarity.Typical
printmediaalsoexhibitanonlinearresponsethatembodiesanimplicit“expander.”