Which tone-mapping operator is the best? A comparative study of perceptual quality

9.2 MB·

by  Xim Cerdá-Company

Which Tone-Mapping Operator Is the Best? A Comparative
Study of Perceptual Quality
´ ´
XIM CERDA-COMPANY, C. ALEJANDRO PARRAGA and XAVIER OTAZU
Computer Vision Center,
Computer Science Department,
Universitat Auto´noma de Barcelona,
6
Spain
1
0
2 January 19, 2016

n
a
J
Index terms— Tone-Mapping Operators, Psy- developments.
8
chophysics, High Dynamic Range, Visual Percep-
1
tion,HumanVisualSystem,Vision,ImageProcess-

1 Introduction
] ing, ANOVA, Euclidean Distance,Pairwise Com-
R
parison
G Inalmostallnaturalisticviewingsituations,weare
immersedinscenesthatcouldbedescribedasHigh
.
s Abstract Dynamic Range (HDR), in other words, the ener-
c
[ geticdifferencebetweenthebrightestandthedark-

Tone-mapping operators (TMO) are designed to est patch is much higher than the difference an
1
generate perceptually similar low-dynamic range imagingdevicecanfaithfullycapture. Forinstance,
v
0 images from high-dynamic range ones. We stud- the energy ratio between sunlight and starlight is
5 ied the performance of fifteen TMOs in two psy- approximately about 100,000,000:1 Ferwerda and
4 chophysicalexperimentswhereobserverscompared Luka (2009). If the Human Visual System (HVS)
4
the digitally generated tone-mapped images to was to linearly represent these extreme differences,
0
their corresponding physical scenes. All experi- it would require a much larger sensitivity range for
.
1
ments were performed in a controlled environment itsretinalsensors(rodsandcones)andneuralpath-
0
andthesetupsweredesignedtoemphasisedifferent ways than is achievable within the limitations of
6
1 image properties: in the first experiment we evalu- biological chemistry. Instead, millions of years of
: ated the local relationships among intensity-levels, evolutionhavesolvedthisproblembyadaptingthe
v
andinthesecondoneweevaluatedglobalvisualap- sensorial and neural machinery, allowing it to non-
i
X pearance among physical scenes and tone-mapped linearly convert the large natural intensity range
r images, which were presented side by side. We into a much smaller range of about 10,000:1 Rein-
a
ranked the TMOs according to how well they re- hard et al. (2005). This adaptation is possible be-
produce the results obtained in the physical scene. cause of the combined activity of the pupil, rods
Our results show that ranking position clearly de- and cones, horizontal cells, bipolar cells, amacrine
pends on the adopted evaluation criteria, which cells, ganglionar cells, interplexiform cells, the pri-
implies that, in general, these tone-mapping al- mary visual cortex, etc. Snowden et al. (2006).
gorithms consider either local or global image at- Thisdynamicrangepresentsanimportantprob-
tributes but rarely both. We conclude that a more lem for visual representation technologies mostly
thorough and standardised evaluation criteria are because common imaging devices (cameras and
needed to study all the characteristics of TMOs, monitors) are only able to obtain/display images
as there is ample room for improvement in future within a much smaller range of about 100:1 Rein-
1
Figure 1: General workflow of a tone-mapping process. Several low dynamic range (LDR) images,
with different exposure times covering different luminosity ranges of a physical scene, are captured by a
camera. From these images a high dynamic range (HDR) image is obtained, which can be processed by
a TMO in order to obtain a new LDR image that can be displayed on a standard LDR monitor.
hard et al. (2005) which can be increased up to stant some of the original image characteristics.
1000:1forspecializedHDRled-baseddisplaysRup- The performance of these TMOs depends on sev-
pertsberg et al. (2007). To solve this problem, eral factors including lighting and viewing condi-
an assortment of non-linear image processing tech- tions, aesthetic/realistic preferences, local/global
niques were defined to display HDR scenes in lim- assumptions, etc. and are usually evaluated us-
ited Low Dynamic Range (LDR) devices. To con- ing computational (Aydin et al. (2008); Yeganeh
struct the HDR image, we use many LDR images andWang(2012))andpsychophysical(Dragoetal.
ofthesamescenetakenatdifferentexposurevalues (2003a);Kuangetal.(2004);Yoshidaetal.(2005);
to capture a much larger dynamic range (see Fig- Ledda et al. (2005); Ashikhmin and Goyal (2006);
ure 1). This HDR image is generated by extract- Cad´ık et al. (2006); Yoshida et al. (2007); Kuang
ing from each LDR image the information corre- et al. (2007b,a); Akyu¨z et al. (2007); Cad´ık et al.
sponding to its region of interest (where it is nei- (2008)) methods. In this work we present a new
ther over- or under-exposed) and combining them. set of experiments and analysis to psychophysi-
Since this new HDR image cannot be displayed on cally evaluate the performance of 15 state-of-the-
a standard LDR monitor, an algorithmic solution art TMOs. Unlike previous studies, all the experi-
is needed to reduce its dynamic range to match ments were performed in a controlled environment
that of the monitor. A common solution is to use and tone-mapped images were presented side by
a Tone-Mapping Operator (TMO) to reduce the side with the physical scene. All tone-mapped im-
dynamic range while keeping approximately con- ages’ results were compared to physical scene’s re-
2
sults, so we obtained a set of rankings, according consistent with those in the overall rendering re-
to the different criteria, that determine which is sults, if not the same.
the best TMO at representing the original scene as
human observers perceive it.
2.1.2 Experiments with HDR scene as ref-
erence
2 State-of-the-Art Yoshidaetal. 2005conductedapsychophysicalex-
perimentbasedonadirectcomparisonbetweenthe
2.1 Previous TMO Psychophysical appearance of real-world scenes and TMO images
Studies of these scenes displayed on a LDR monitor. In
their experiment, they differentiate between global
Although the idea of using algorithms to match and local operators, and introduced, for the first
the brightness of real scenes to that of imaging time, the comparison between tone-mapped image
devices was not new Miller et al. (1984); Tumblin and real scene, selecting two different indoor ar-
andRushmeier(1993),TMOsdidnotbecomepop- chitectural scenes. Fourteen subjects were asked
ular until the early 2000s due to digital cameras to give ratings according to several criteria like re-
were very exclusive, so many TMOs were devel- alism (image naturalness in terms of reproducing
opedbetween1996and2008Ferwerdaetal.(1996); theoverallappearanceoftherealworldviews)and
Ashikhmin(2002);DurandandDorsey(2002);Fat- image appearance (brightness, contrast, detail re-
tal et al. (2002); Reinhard et al. (2002); Drago production in dark regions, and in bright ones).
et al. (2003b); Krawczyk et al. (2005); Li et al. They found that none of these image appearance
(2005); Reinhard and Devlin (2005); Kuang et al. attributeshadastrong influence ontheperception
(2007a);Mertensetal.(2007);Meylanetal.(2007); of naturalness by itself.
Kim and Kautz (2008); Ferradans et al. (2011); The same authors extended their previous work
Otazu(2012). Thus,uptonowmanydifferentpsy- with the aim of finding out which attributes
chophysical studies have been performed and they of image appearance accounted for the differ-
can be classified as follows: ences between tone-mapped images and the real
scene Yoshida et al. (2007). They observed a
clear distinction between global and local opera-
2.1.1 Experiments without HDR scene as
tors. However, they concluded again that none
reference
of the image attributes evaluated had a strong
Oneofthefirstpsychophysicalexperimentstoeval- influence on the perception of naturalness by it-
uate TMOs was performed by Drago et al. 2003a. selfwhichsuggestedthatnaturalnessdependsona
They compared the performance of 6 TMOs on combination of the other attributes with different
4 different (synthetic and photographic) scenes by weights.
asking subjects to make pairwise perceptual evalu- In an another work, Ashikhmin and Goyal 2006
ations and by rating stimuli with respect to three performed three different experiments. Subjects
attributes: apparentimagecontrast,apparentlevel rankeddifferenttone-mappedimagesdependingon
of detail, and apparent naturalness. Their results the asked question. In the first one, the authors
showed that preferred operators produced detailed asked which image they liked more without having
images with moderate contrast. the reference scene. In the second one, the authors
Kuang et al. 2004 performed pairwise compar- askedwhichimageseemedmorerealwithoutview-
isons on 8 different tone-mapping operators using ing the reference scene and, in the third one, they
10 different scenes and two conditions (color and askedwhichimagewastheclosesttotherealscene
grey-level) where subjects had to choose the pre- viewing the reference scene. They observed that
ferred image considering general rendering perfor- rankingsweretotallydifferentwhensubjectscould
mance (including tone compression performance, compare the tone-mapped image to the reference
color saturation, natural appearance, image con- scene.
trast and image sharpness). Their results showed In a subsequent study, Kuang et al. 2007b
thatthegrey-scaletone-mappingperformancesare performed three different experiments they named
3
preference evaluation, image-preference modelling in an adjoining room.
and accuracy evaluation. In the preference eval- While looking for a definition of an overall im-
uation experiment, pairwise comparisons between age quality measure, Cad´ık et al. 2006 studied the
tone-mapped images were performed. Here they relationships between some image attributes such
used only color images and the aim was to evalu- as brightness, contrast, reproduction of colors and
ate the general rendering performance by instruct- reproduction of details. They performed two psy-
ingobserverstoconsiderperceptualattributessuch chophysicalexperiments,using14TMO,inorderto
as overall impression on image contrast, colorful- proposeaschemeofrelationshipsbetweentheseat-
ness, image sharpness, and natural appearance. In tributes, being aware that some special attributes,
contrast, in the image-preference modelling exper- which were not evaluated (e.g. glare stimulation,
iment, they rated grey-scale images (which were visual acuity and artifacts), can influence their re-
grey-scaleversionsofthefirstexperimentcolorim- lationships. Inthefirstone,10subjectswereasked
ages). Here, observers considered perceptual at- toperformratingsusingfivecriteria: overallimage
tributes such as highlight details, shadow details, quality and the four basic attributes (brightness,
overall contrast, sharpness, colorfulness and ap- contrast, reproduction of detail and colors). These
pearance of artifacts, comparing the TMO’s visual evaluations were performed using a real scene as a
rendering”totheirinternalrepresentationofa‘per- reference(atypicalrealindoorHDRscene). Inthe
fect’ image in their minds” Kuang et al. (2007b). second experiment, subjects did not have access to
In the accuracy evaluation, both pairwise compar- therealsceneandtheyhadtorankimageprintouts
ison and rating techniques were used in order to according to the overall image quality and the four
evaluate the perceptual accuracy of the rendering basic attributes.
algorithms. ThepairwisecomparisonofTMOswas In a new study, Cad´ık et al. 2008 performed ex-
performed without viewing the real scene and sub- actlythesametypeofexperimentsaddingtwonew
jects were asked to compare the overall impression scenes, thatis, theyhadatotalofthreescenes, i.e.
on image contrast, colorfulness, image sharpness, a real indoor HDR scene, a HDR outdoor scene
and overall natural appearance. An additional rat- and a night urban HDR scene. In the first exper-
ing evaluation was performed using the real scenes iment, subjects were asked to rate overall image
set up in the adjoining room as references. Here, quality and the quality of reproduction of five at-
subjects had to rate image attributes like highlight tributes by comparing samples to the real scene.
contrast, shadow contrast, highlight colorfulness, These attributes were the same four basic ones of
shadow colorfulness, overall contrast and the over- their previous work and the lack of disturbing im-
allrenderingaccuracycomparingtotheoverallap- age artifacts (which was one of the non-evaluated
pearance of the real-world scenes. In both experi- special attributes in Cad´ık et al. (2006)). These
ments, observers did not have immediate access to experimentswereset-upinanuncontrollednatural
the real scene and had to rely on their memories environment, so subjects had to perform the ex-
(either short- or long-term) to perform the tasks. periments at the same time of the day as the HDR
In 2007, Kuang et al. presented the iCAM06 imagewasacquired. Inthesecondexperiment,sub-
operator Kuang et al. (2007a). To validate it, the jectshadnopossibilityofdirectlycomparingtothe
authorsperformedtwopsychophysicalexperiments real scene and had to rank the image printouts ac-
similartothepreviousonesinKuangetal.(2007b). cordingtotheoverallimagequality,andthequality
The first experiment was a pairwise comparison of basic attributes.
withoutviewingthereferencescene. Observershad
to choose the tone-mapped image that they pre-
2.1.3 Experiments using an HDR monitor
ferred based on overall impression on image qual-
ity(consideringcontrast,colorfulness,imagesharp- In 2005, Ledda et al. 2005 performed two different
ness, and overall natural appearance). In the sec- psychophysical experiments comparing 6 different
ond experiment, observers were also asked to eval- tone-mapping operators to linearly mapped HDR
uate overall rendering accuracy by comparing the scenes displayed on a HDR device. They used 23
overall appearance of the rendered images to their different color and grey-scale HDR scenes showing
corresponding real-world scenes, which were set up 3 different images per comparison: the HDR and
4
two tone-mapped images. In the first experiment, Otazu (2012). There is a third group of “hybrid”
subjectswereaskedtoselecttheTMOimagemore tone-mapping operators which could be global or
similar to the HDR reference by judging its global local depending on how they are run. One exam-
appearance. In the second one, they were asked to ple is Reinhard et al. (2002) and another one is
make their judgment based on reproduction detail. Ferradans et al. (2011), which is developed in two
Inalaterwork,Akyu¨zetal. 2007askedsubjects stages,thefirstglobalandthesecondlocal. Abrief
to rank six images (1 HDR image, 3 tone-mapped summary of the properties of each tone-mapping
images, 1 objectively good LDR exposure value operator used in our experiments is given in Ta-
and 1 subjectively good LDR exposure value) ble 1 (and in the text below). The first column
according to their subjective preferences. Sur- shows the names that we will use to refer to each
prisingly, they found that participants did not operator throughout this work as defined below.
systematicallypreferredtone-mappedHDRimages
TMO Global/Local HVS Luminance Color
over the best single LDR exposures.
Ashikhmin L (cid:88) (cid:88)
Drago G (cid:88)
As seen above, all previous studies have been Durand L (cid:88) (cid:88)
focused on subjective comparisons of global and Fattal L (cid:88)
Ferradans H (cid:88) (cid:88) (cid:88)
local image appearance attributes such as con- Ferwerda G (cid:88) (cid:88) (cid:88)
trast, colourfulness, sharpness, reproduction arti- iCAM06 L (cid:88) (cid:88) (cid:88)
facts, etc. either within TMOs or against the real KimKautz G (cid:88)
Krawczyk L (cid:88)
scene. Whilethisisnodoubtextremelyimportant,
Li L (cid:88)
we believe a good TMO is also likely to keep the
Mertens -
interrelations between objects attributes inside a Meylan L (cid:88) (cid:88) (cid:88)
scene the same as in the physical scene. As far as Otazu L (cid:88) (cid:88)
Reinhard G (cid:88)
we know, no study has been conducted to evaluate Reinhard-Devlin G (cid:88)
whether objects represented within a TMO image
maintain the same perceived visual differences as Table 1: Summary of used TMO’s characteristics.
the real scene, and this is the principal aim of our Second column shows whether the TMO is global
work. (G), local (L) or hybrid (H). Third column shows
whetheritisinspiredbytheHumanVisualSystem,
and following columns show whether it processes
2.2 Tone-Mapping Operators
luminace and color information.
Tone-mapping operators can be classified in two
groups: global and local processing. Global opera- - Ashikhmin (2002) (Ashikhmin). This local
torsperformthesamecomputationinallpixels,re- tone-mappingoperatorisinspiredbytheprocessing
gardlessofspatialposition,whichmakethemmore mechanisms present at the first stages of the Hu-
computationally efficient at the cost of losing con- man Visual System. Intensity range is compressed
trast and image detail. Some examples of global by a local luminance adaptation function and, in a
TMO are Drago et al. (2003b); Ferwerda et al. last step, detail information is added.
(1996); Kim and Kautz (2008); Reinhard and De- -Dragoetal.(2003b)(Drago). Thisglobaltone-
vlin (2005). On the other hand, local operators, mapping operator is based on luminance logarith-
which take into account neighbouring pixels, pro- mic compression that, depending on scene content,
duce images with more contrast and higher level uses a predetermined logarithmic basis to preserve
of detail, but they may show problems with halos contrast and details.
around high contrast edges. Local operators are - Durand and Dorsey (2002) (Durand). This lo-
inspired on the local adaptation process present cal tone-mapping operator decomposes the image
at the early processing stages of the human vi- in two layers: the base and the detail. The base
sual system. Some examples of local operators are layeriscompressedusingbilateralfiltering,andthe
Ashikhmin(2002);DurandandDorsey(2002);Fat- magnitudes of the detail layer are preserved.
tal et al. (2002); Krawczyk et al. (2005); Kuang - Fattal et al. (2002) (Fattal). This local tone-
etal.(2007a);Lietal.(2005);Meylanetal.(2007); mapping operator manipulates the gradient fields
5
of the luminance image. Its idea is to identify high algorithm is applied on the mosaic image captured
gradients in different scales and attenuate their by the sensors. In a second step, it introduces a
magnitudes, while maintaining their directions. variationofthecenter/surroundspatialopponency.
- Ferradans et al. (2011) (Ferradans). This -Otazu(2012)(Otazu). Thislocaltone-mapping
hybrid tone-mapping operator is divided in two operator is based on a multipurpose human colour
stages. Inthefirststage,itappliesaglobalmethod perception algorithm. It decomposes the intensity
that implements the visual adaptation, trying to channel in a multiresolution contrast decomposi-
mimic human cones’ saturation. In the second tion and applies a non-linear saturation model of
stage,itenhanceslocalcontrastusingavariational visual cortex neurons.
model inspired by color vision phenomenology. - Reinhard et al. (2002) (Reinhard). Hybrid
- Ferwerda et al. (1996) (Ferwerda). This global global-localtone-mappingoperator,thatcanbeex-
tone-mapping operator is based on computational ecuted as global or local. It performs a global scal-
model of visual adaptation that was adjusted to fit ingofthedynamicrangefollowedbyadodgingand
psychophysical results on threshold visibility, color burning (local) processes. In our work, this opera-
appearance, visual acuity, and sensitivity over the tor was run as global which is its default value in
time. the toolbox.
- Kuang et al. (2007a) (iCAM06). This local -ReinhardandDevlin(2005)(Reinhard-Devlin).
tone-mapping operator is based on the iCAM06 Thisglobaltone-mappingoperatorusesamodelof
colorappearancemodel,whichgivestheperceptual photoreceptors adaptation which can be automati-
attributesofeachpixel,likelightness,chromaticity, cally adjusted to the general light level.
hue, contrast and sharpness. It includes an inverse
model which considers viewing conditions to gen-
erate the result. 3 Methods
- Kim and Kautz (2008) (KimKautz). This
global tone-mapping operator is based on the as- 3.1 Laboratory Setup
sumption that human vision sensitivity is adapted
to the average log luminance of the scene and that Experiments were performed in a dark room, i.e.
it follows a Gaussian distribution. in a controlled environment, using a ViSaGe MKI
- Krawczyk et al. (2005) (Krawczyk). This local Stimulus Generator, and a calibrated Mitsubishi
tone-mappingoperatorisinspiredontheanchoring Diamond-Pro(cid:13)R2045u CRT monitor side-by-side
theory Gilchrist et al. (1999). It decomposes the with the handmade real HDR scene. Both mon-
image into patches of consistent luminance (frame- itor and real scene were setup so that objects in
works) and calculates, locally, the lightness values. bothscenessubtendedapproximatelythesamean-
- Li et al. (2005) (Li). This local tone-mapping gleandlookedsimilarlypositionedtotheobserver.
operator is based on multiscale image decomposi- We built three different HDR scenes, each in-
tion that uses a symmetrical analysis-synthesis fil- cluding a grey-level reference table and two solid
terbanktoreconstructthesignal,andapplieslocal objects (parallelepipeds). The reference table was
gaincontroltothesubbandstoreducethedynamic built by printing a series of 65 grey squares (2.8
range. x 2.2 cm) arranged in a flat 11x6 distribution
- Mertens et al. (2007) (Mertens). This tech- whose spectral reflectance increased monotonically
nique fuses original LDR images of different ex- from top to bottom, as measured by our PR-655
posure values (exposure fusion) to obtain the final SpectraScan(cid:13)RSpectroradiometer (see Figure 2).
“tone-mapped”image,whichavoidsthegeneration The parallelepipeds consisted of 3.6 x 3.6 x vari-
of an HDR image. Guided by simple quality mea- able length between 9.4 and 10 cm pieces of wood,
sureslikesaturationandcontrast,itselects“good” whose sides were covered with random samples of
pixelsofthesequenceandcombinesthemtocreate thesameprintedpaperasthereferencetables. The
the resulting image. rest of the scenes consisted of many plastic and
-Meylanetal.(2007)(Meylan). Thislocaltone- woodenobjectsofdifferentcoloursandshapes(see
mappingoperatorisderivedfromamodelofretinal Figure 3).
processing. In a first step, a basic tone-mapping Weplacedoneoftheparallelepipedsinthebright
6
(a)Scene1 (b)Scene2 (c)Scene3
Figure 3: To show the general appearance of the physical scenes, here we show a single LDR exposure
(choosen by simple visual inspection by the authors) from the set of LDR exposures used to create the
HDR images.
ones shown beside them. The experiments were
conducted in a controlled environment, where the
only light sources were the lamp illuminating the
real scene and the indirect light produced by the
CRT monitor. Since experiments were performed
in a dark room, reflections from all other objects
and the walls were minimised. The dynamic range
of the scenes were approximately 105 for scene 1
and 106 for scenes 2 and 3.
We know that an accurate representation of the
scene luminance distribution is not possible to
Figure 2: Luminance in cd/m2 of Refer- achievefromcameraimages,butmultipleexposure
ence Table Patches measured by our PR-655 values improve the image information contained in
SpectraScan(cid:13)RSpectroradiometer. We can observe the HDR image McCann and Rizzi (2012). Thus,
thattheirspectralreflectancedescreasedmonoton- a set of 25 photographs were taken at different ex-
ically from the 1st patch to the 65th. posurevalues(from15secto1/6000sec)usingthe
same aperture, focal distance, zoom settings and
visualfield. IndividualimageswerestoredinRAW
format and transformed into 16 bits sRGB (using
part of the scene and the other under the shade
thecameramanufacturer’ssoftware). HDRimages
cast by the reference table. This was done in order
were obtained using the HDR Toolbox for MAT-
to have some reference objects directly lit by the
LAB Banterle et al. (2011).
scene illumination and others in the dark. Two
surfacesofoneparallelepipedandthreeoftheother
one were visible from subjects’ location, resulting
3.2 Experiments
in 15 different grey visible surfaces in total. Scenes
showedawiderangeofcolorsandwereilluminated In order to compare TMO algorithms, we per-
withanincandescentlampof100Wwhosebulbwas formed two different experiments. The aim of the
painted blue to simulate D65 illumination. firstexperimentwastostudytheinternal(local)re-
We photographed the real scene using a cali- lationships among grey-levels in the tone-mapping
brated Sigma Foveon SD10 camera placed in the image and in the real scene. The aim of the sec-
exact same position as subjects’ head during the ond experiment was to rank TMOs according to
psychophysical experiment, and camera setup was how similar their results were perceived to the real
arranged so that the images presented later on the scene, using a global criterion. In both cases, we
monitor looked geometrically similar to the real obtained a ranking of the different TMOs.
7
All experiments started with a 1-minute sub- 3.2.2 Experiment 2: Global Criterion
ject adaptation to the ambient light and imple-
mentations of some of TMOs have been obtained This experiment consisted on a pairwise compari-
from HDR Toolbox for MATLAB Banterle et al. sonoftone-mappedimagesobtainedusingdifferent
(2011) (all except Ferradans, iCAM06, Li, Meylan TMOsinthepresenceoftheoriginalscene(sideby
and Otazu, which have been obtained from their side). After 1-minute adaptation in front of the
authors’ web page). In order to avoid to benefit physical scene, a pair of tone-mapped images of
any of these operators, we have run all of them thesamephysicalscenewasrandomlyselectedand
with their default parameters. In Ferradans’ case, presented sequentially to the observer on the CRT
we had to determine two different parameters and screen (besides the real scene). Objects in both
we chose the default values specified in their paper digitalandphysicalsceneslookedgeometricallythe
(ρ=0 and α−1 =3). same, subtending the same angle to the observer.
Subjects could toggle a gamepad button to select
whichimageofthetone-mappedpairwaspresented
3.2.1 Experiment 1: Local Criterion on the monitor (only one image was displayed at a
time)andtoselecttheimagethatwasmoresimilar
This experiment consists on two different tasks:
totherealscene. Asbefore,therewasnotimelimit
Task 1. Afteradaptation,subjectswereaskedto
to perform the comparisons, but subjects were ad-
match, in the real scenes (i.e. with monitor turned
vised to complete a trial in less than 30 seconds.
off), the brightnesses of the 5 parallelepipeds’ sur-
After an image was chosen, a grey background was
facestothebrightnessesofthereferencetableinthe
shownfortwoseconds,andadifferentrandompair
scene(seeFigure4a). Althoughtherewerenotime
was selected for the next trial. Every subject per-
constraintstoperformthetasks,theywereadvised
formed 105 comparisons per scene taking around
to take no more than 30 seconds per match.
25 minutes in total. There were three experimen-
Task 2. Here the real scene was not visible tal conditions, corresponding to the three different
and the observers just saw a digital tone-mapped physical scenes created (see Figure 3). Between
version of the real scene presented on the monitor. conditions, subjects were forced to go out of the
Their task was similar as in Part 1, except that laboratory to take a 5 to 10 minutes break while
the matchings were conducted entirely on the grey the physical scene was replaced.
patches shown on the screen (see Figure 4b).
Agroupof10peoplewithnormalorcorrected-to-
normalvision,7malesand3femalesrecruitedfrom
There were three conditions for the experiment, our lab academic and research community, com-
corresponding to the three different scenes created pleted this experiment. This group was comprised
(seeFigure3). Observersmatchedthe5surfacesin by people aged between 17 and 54. Seven of them
all15differenttone-mappedimagesandinthereal were naive to the aims of the experiment.
scene for each of the three scenes (conditions), so
they performed a total of 240 matchings. Match-
ings were conducted by writing results on a piece
4 Results
of paper. For every scene, all tone-mapped images
were shown in random order.
Task 1 was completed by a group of 12 ob- 4.1 Experiment 1: Local Criterion
servers with normal or corrected-to-normal vision,
recruited from our lab academic/research commu- In every scene evaluated in Section 3.2.1, we ob-
nity. This group (8 males and 4 females) was com- tained16setsofexperimentalvalues(therealscene
prised by people aged between 17 and 54. Nine in Task 1 plus 15 tone-mapped images in Task 2)
of them were completely naive to the aims of the for each of the 15 considered surfaces.
experiment. Task 2 was completed by 10 of the We performed two different analysis to evaluate
previous observers (8 males and 2 females). to what extend the local interrelations perceived
After all matches, we converted the results to by the observers in the tone-mapped versions cor-
cd/m2 using the measurements shown in Figure 2. responded to those perceived in the real scene.
8
(a)Task1 (b)Task2
Figure 4: In Experiment 1, observers performed two tasks. In Task 1 (Figure 4a), observers had to
matchthebrightnessesofthe5parallelepipeds’surfacestothebrightnessesof5patchesinthereference
table. In Task 2 (Figure 4b), observers had to perform the same task on the TMO image displayed on
the calibrated monitor. (Red arrows are randomly drawed just for illustrative purposes).
4.1.1 Analysis 1 ilar to each other. In each row, algorithms’ names
are arranged according to their output’s similar-
In the first analysis, we compared the perceived
ity to the real scene: the further from the “real
distancesfromeachtone-mappingalgorithmtothe
scene” label (in red), the more dissimilar the re-
real scene. The real scene and the TMOs were
sults. From this data, a ranking (Table 3) of tone-
defined in a 15-dimensional space, where each di-
mapping operators is obtained by computing how
mension corresponds to each evaluated surface. To
manytimesaTMOcanbeconsideredsimilartothe
obtain this 15-dimensional vector, for each TMO
real scene. Data obtained in this section is shown
and for the real scene, the psychophysical data for
in Appendix B.
all observers for each grey-level surface was aver-
aged. Then, in order to avoid the distortion and
other noise, we performed a Principal Component 4.2 Experiment 2: Global Criterion
Analysis (PCA) to reduce the dimensionality and
we kept the first 6 components (which represents Fromthepairwisecomparisonresults,wedefineda
95.40% of data information). A ranking was ob- preference matrix for each subject and each scene.
tained (see Table 2) by measuring the Euclidean We constructed a directed graph where the nodes
distance from each operator to the real scene in were the evaluated TMOs and the arrows pointed
this 6-dimensional space. from a preferred TMO to a non-preferred TMO,
e.g. iftheTMO ispreferredovertheTMO (tone-
i j
mapped image from TMO is more similar to the
4.1.2 Analysis 2 i
real scene than the one from TMO ), we drew an
j
In order to know if there are significant differ- arrow from nodei to nodej, for i(cid:54)=j.
ences between these sets of experimental values, From this graph, we were able to analyse intra-
we calculated 15 different ANOVAs (one per grey- subjectconsistencycoefficientζ foreachscene. The
level surface). In all cases, significant differences consistencycoefficientforeachsubjectandsceneis
(at p < 0.05) between the algorithms were found. defined by
For each ANOVA, a Fisher’s Least Significant Dif-
ference (FLSD) post-hoc test was computed (Fig-
n
ure 5) to analyze which TMO was different to the n(n−1)(2n−1) 1(cid:88)
d = − a2 ,
real scene. In this figure, rows correspond to grey- st 12 2 ist
i=1
level surfaces and for each one we show the sev- (1)
(cid:40)
eral groups obtained by FLSD. Every group shows 1− 24dst,if n is odd.
ζ = n3−n
the set of algorithms that could be considered sim- st 1− 24dst ,if n is even.
n3−4n
9
Figure 5: Interesting cases of Fisher’s Least Significant Difference Post-hoc Test carried out for the
different grey-level surfaces (from 6th to 10th). Each line corresponds to the set of TMOs that are not
significantly different at p<0.05. The real scene is indicated in red.
where s is the scene number (s ∈ [1,3]), t is the Case V Montage (2003)) and 95% confidence lim-
subject number (t ∈ [1,m]), n is the number of its. SinceSpearman’scorrelationbetweenrankings
evaluated TMOs, and a is the number of arrows of the three scenes are equal or higher than 0.90
i
which leave the node . The maximum ζ value is 1 (p < 0.05), we computed the mean value along all
i
(perfect consistency within-subject). the scenes (Table 5).
The consistency or the agreement between sub-
jects, i.e. inter-subject agreement, is measured by
5 Discussion
the Kendall Coefficient of Agreement Kendall and
Babington-Smith (1940). This measure is defined
FromSection3.2.1,weseeinTable2thatiCAM06
by
has the smallest local distance to the real scene.
2(cid:80) (cid:0)pij(cid:1) Similarly, in Table 3 iCAM06 is again the best al-
u = i(cid:54)=j 2 −1 (2) gorithm and can be considered similar to the real
s (cid:0)m(cid:1)(cid:0)n(cid:1)
2 2 sceneon14outof15cases,hencelocalrelationships
among grey-levels in iCAM06 tone-mapped image
wherep isthenumberoftimesTMO ispreferred
ij i
can be considered similar to the ones in the real
overTMO andmisthenumberofsubjects. Since
j
scene. Since iCAM06 is based on a color appear-
the number of subjects is even (m = 10), the pos-
ance model that estimates perceptual attributes
sible minimum value of u, given by Equation 2,
is u = − 1 and its possible maximum value is such as lightness, chromaticity, hue, contrast and
m−1 sharpness, it is expected for this method to be in
u=1.
line with observers’ perception.
In order to study if u values are significant, we
s
Spearman’s correlation between rankings of Ta-
used the chi-sqared test (χ2). The χ2 values are
s ble 2 and Table 3 shows a correlation of 0.62
defined by
(p<0.05), being some algorithms in very different
positions, where Ferradans is an interesting case.
n(n−1)(1+u (m−1))
χ2 = s (3) In Table 2 this TMO algorithm is in the 12th po-
s 2
sition, but in Table 3 it is in the 2nd one. In the
The number of degrees of freedom of the chi- first case (Euclidean distance), its distance to the
squared test is given by n(n−1). real scene is bigger than most of the other algo-
2
In Table 4, we show all statistical measures rithms and in the second case (ANOVA analysis),
for each scene, where we can see that intra- and it can be considered similar to the real scene in 11
inter-subject consistency values are very high and out of 15 cases. This is because there are a few in-
they are statistically significant. Then, in Fig- stances where Ferradans is very different from the
ure 6, we show the results of the overall paired real scene and these instances bias the Euclidean
comparison evaluations for every scene (obtained distance calculation. This can be seen in Figure 5,
from Thurstone’s Law of Comparative Judgment, whereFerradans’resultsarefarawayfromthereal
10