On the realistic validation of photometric redshifts, or why Teddy will never be Happy

5.5 MB·

by  R. Beck

MNRAS000,000–000(0000) Preprint21March2017 CompiledusingMNRASLATEXstylefilev3.0
On the realistic validation of photometric redshifts, or why
Teddy will never be Happy
R. Beck1(cid:63), C.-A. Lin2,3, E. E. O. Ishida4, F. Gieseke5, R. S. de Souza6,7,
M. V. Costa-Duarte7,8, M. W. Hattab9, A. Krone-Martins10,
for the COIN Collaboration
1Department of Physics of Complex Systems, E¨otv¨os Lora´nd University, Budapest 1117, Hungary
2Service d'Astrophysique, CEA Saclay, Orme des Merisiers, Bˆat 709, 91191 Gif-sur-Yvette, France
3Fenglin Veteran Hospital, 2 Zhongzheng Rd. Sec. 1, Fenglin Township, Hualien 97544, Taiwan
7 4Laboratoire de Physique Corpusculaire, Universit´e Clermont-Auvergne, 4 Avenue Blaise Pascal, 63178, Aubi`ere Cedex, France
1 5University of Copenhagen, Sigurdsgade 41, 2200 Copenhagen, Denmark
0 6MTA E¨otv¨os University, EIRSA“Lendulet”Astrophysics Research Group, Budapest 1117, Hungary
2 7Instituto de Astronomia, Geof´ısica e Ciˆencias Atmosf´ericas, Universidade de Sa˜o Paulo, R. do Mata˜o 1226, 05508-090, SP, Brazil
r 8Leiden Observatory, Leiden University, Niels Bohrweg 2, 2333 CA Leiden, The Netherlands.
a 9Center for Biomarker Research and Personalized Medicine, Virginia Commonwealth University, Richmond, VA, USA
M 10CENTRA/SIM, Faculdade de Ciˆencias, Universidade de Lisboa, Ed. C8, Campo Grande, 1749-016, Lisboa, Portugal

0
2
21March2017

]
A
ABSTRACT
G
Two of the main problems encountered in the development and accurate validation
.
h of photometric redshift (photo-z) techniques are the lack of spectroscopic coverage in
p feature space (e.g. colours and magnitudes) and the mismatch between photometric
- error distributions associated with the spectroscopic and photometric samples. Al-
o
thoughtheseissuesarewellknown,thereiscurrentlynostandardbenchmarkallowing
r
t aquantitativeanalysisoftheirimpactonthefinalphoto-z estimation.Inthiswork,we
s presenttwogalaxycatalogues,Teddy andHappy,builttoenableamoredemanding
a
[ and realistic test of photo-z methods. Using photometry from the Sloan Digital Sky
Survey and spectroscopy from a collection of sources, we constructed datasets which
3
mimic the biases between the underlying probability distribution of the real spec-
v
troscopic and photometric sample. We demonstrate the potential of these catalogues
8
by submitting them to the scrutiny of different photo-z methods, including machine
4
learning(ML)andtemplatefittingapproaches.Beyondtheexpectedbadresultsfrom
7
8 most ML algorithms for cases with missing coverage in feature space, we were able to
0 recognizethesuperiorityofglobalmodelsinthesamesituationandthegeneralfailure
. across all types of methods when incomplete coverage is convoluted with the presence
1
of photometric errors - a data situation which photo-z methods were not trained to
0
deal with up to now and which must be addressed by future large scale surveys. Our
7
1 catalogues represent the first controlled environment allowing a straightforward im-
: plementation of such tests. The data are publicly available within the COINtoolbox
v
(https://github.com/COINtoolbox/photoz catalogues).
i
X
Key words: galaxies: distances and redshifts – catalogues – methods: statistical –
r methods: data analysis – techniques: photometric.
a
1 INTRODUCTION comparedtotheirhigherresolutioncounterpart,thespectro-
scopictechnique(spec-z),photo-z measurementsaresubject
Photometric redshift (photo-z) estimation has become a
to higher uncertainty, they are also more efficient, cheaper,
widespreadandvitaltoolintheastronomicalfield.Although
andabletoprobemoredistantobjects(e.g.Hildebrandt,H.
etal.2008).Thesecharacteristicsmakethemmoresuitable
forsomeastrophysicalproblems.Anexampleoftheformeris
(cid:63) [email protected]
©0000TheAuthors
2 R. Beck et al.
the weak gravitational lensing (Abdalla et al. 2008), which its performance on the validation set. This approach ne-
measures the coherent galaxy shape distortion by gravita- glects to take into account the fact that the distribution
tional potentials, and is relatively less sensitive to the red- ofgalaxiesinthespaceofobservables(i.ephotometricmag-
shift measurement. The lensing signal is a convolution of nitudes/colours) is generally very different for the spectro-
the density contrast distribution with a broad kernel that scopicandthephotometricsamples,andeventheirrangein
has the effect to smooth the sensitivity of the signal to the the magnitude/colour space can differ. Moreover, given the
redshiftaccuracy.Therefore,inordertoobtainafastermea- quality requirements demanded for spectroscopic observa-
surementandtomaximizethedatavolume,photo-z iscom- tion, the photometric sample encloses a larger photometric
monly use in weak lensing studies. uncertainty which may break the direct relation between
However, with the arrival of the Stage-IV lensing sur- magnitude/colours and redshift, even when this relation is
veys (Natarajan et al. 2014), the goal on cosmological con- clearlydefinedinthespectroscopicsample.Insummary,the
straints becomes more ambitious, and consequently, the re- performancemetricsobtainedontheformerisnotrepresen-
quirements on photo-z precision and accuracy equally in- tative of results for the latter. In the worst-case scenario,
crease. For instance, for the Euclid1 mission of the Euro- theseissuesareignoredaltogether,asinthebenchmarkpa-
pean Space Agency (ESA), the initial requirements on the pers aforementioned. In other cases, error flags or feature
biasandthescatterineachredshiftbinwere0.002and0.05 selectionresultsareprovidedalongsidethephoto-z estima-
foratotalnumberof2billiongalaxies,respectively(Laureijs tion, notifying users when extrapolation is performed and
etal.2011).Spectroscopicfollow-upforsuchalargenumber resultsshouldnotbetrusted(Bresciaetal.2014;Becketal.
of objects is infeasible and such stringent requirements on 2016b; Stensbo-Smidt et al. 2017) - essentially cutting the
redshiftmeasurementsareextremelychallengingforcurrent coverage of the photometric sample.
photo-z methods. Apart from weak lensing, other applica- Intermediateapproaches,aimingatdealingwithatleast
tionsexistsuchaslargescalestructure(Malavasietal.2016) one of these problems have also been reported. In situa-
and gravitational waves (Antolini & Heyl 2016), which also tions where spectroscopic and photometric samples share
require improvements in the current techniques. the same coverage in magnitude/colour space, it is possi-
Inthe questfor aviablephoto-z alternative capableto ble to adapt the spectroscopic sample2 distribution in or-
handlethesizeandcomplexityofmodernastronomicalsur- der to get it closer to the photometric one (e.g. Lima et al.
veys,aplethoraofdifferentmethodshavebeenproposedand 2008; Sa´nchez et al. 2014; Kremer et al. 2015). However, in
tested.Thesearecommonlydividedintotwomainclasses:i) real-data scenarios, especially when upcoming surveys are
template fitting (e.g., Ben´ıtez 2000; Bolzonella et al. 2000; considered,eventhatassumptionwillnothold.Toobtaina
Csabaietal.2000;Ilbertetal.2006;Coeetal.2006;Bram- measure of photo-z performance that is realistic for the ac-
mer et al. 2008; Leistedt et al. 2016; Beck et al. 2016a) ii) tual use case, i.e. when the photometric sample has a much
empirical(e.g.Wadadekar2005;Borisetal.2007;Milesetal. widercoverageincolourspacethanthespectroscopicsample
2007;Budava´ri2009;Carlilesetal.2010;O’Milletal.2011; and,atthesametime,thereisacorrelationbetweencolours
Krone-Martins et al. 2014; Elliott et al. 2015; Hogan et al. andphotometricerrors,itiscrucialthatweevaluatephoto-z
2015; Cavuoti et al. 2015) and iii) hybrid techniques (e.g. methods in more realistic data situations.
Beck et al. 2016b). In this paper, we provide for the first time a complete
In template fitting techniques, a set of synthetic spec- benchmark template to allow a realistic evaluation of the
traisdeterminedfromsynthesisedstellarpopulationmodels performance of photometric redshift estimators. Using pho-
for a given set of metallicities, star formation histories and tometryfromtheSloan Digital Sky Survey DataRelease12
initialmassfunctions,amongotherproperties.Thephoto-z (SDSS-DR12, Alam et al. 2015) and spectroscopic redshift
iscalculatedbydeterminingthesyntheticphotometry(and measurements from a variety of different sources, we were
thusspectraltemplateandredshift)whichbestfitsthepho- abletoconstructvalidationsampleswhichfollowthecolour
tometric observations. Empirical techniques, on the other coverage and shape distribution from the original SDSS-
hand,usuallyrequireadatasetwithspectroscopicallymea- DR12 photometric sample. These enable an unprecedented
sured redshifts in order to train an algorithm which will realistic view into the accuracy of current photo-z meth-
subsequently be applied to a pure photometric sample. Hy- odsandprovideastartingpointtothedevelopmentofnew
brid methods represent a combination of the previous ones, techniques which take these issues into account.
using an empirical step to first determine the photometric The outline of this paper is as follow. In §2 we show
redshiftandatemplatefittingstepwherephysicalinforma- how we built our benchmark samples from a combination
tion provided by the templates can be used to evaluate the of spectroscopic and photometric data. In §3 we describe
accuracy of the photo-z determination. the methods we have used to access the impact of non-
There have been several notable publications that con- representativeness.In§4wecomparethesummarystatistics
trastedtheperformanceofempiricaland/orspectralmodel andperformanceofdifferentphoto-z estimators.Wepresent
fitting codes (e.g., Csabai et al. 2003; Hildebrandt et al. a discussion of our results in §5.
2010; Abdalla et al. 2011; Dahlen et al. 2013). However, Throughout the paper, we use SDSS modelMag broad-
whenphoto-z algorithmsareevaluatedintheliterature,the bandmagnitudesintheSDSSasinhmagnitudesystem(Lup-
available spectroscopic set is randomly split into training ton et al. 1999), which have been corrected for Galactic ex-
andvalidationsets.Roughly,thealgorithmisoptimizedus- tinction according to Schlegel et al. (1998).
ing the training set and its accuracy estimated based on
2 InmachinelearningjargonsuchmethodsareasubclassofDo-
1 http://sci.esa.int/euclid/ main Adaptation techniques.
MNRAS000,000–000(0000)
Realistic validation of photometric redshifts 3
Teddy A & B Teddy C Teddy D
F
D
P
15 18 21 0 2 4 0 1 2 0.2 0.7 1.2 0.0 0.3 0.6
r-band u g g r r i i z
− − − −
Figure 1.Distributionsforr-bandmagnitudeand4SDSScoloursfortheTeddycatalogue.
SampleA SampleB SampleC SampleD library (MacDonald & Bernstein 2010). Although these is-
Teddy 74309 74557 97980 75924 sues are widely known, at the moment there is no standard
Happy 74950 74900 60315 74642 ground to quantify the sensitivity of each photo-z method
inthepresenceofsuchgap.TheTeddycataloguepresented
Table 1. Numbers of galaxies contained in different samples of
theTeddy andHappy catalogues.Thetotalnumberofgalaxies here was designed to fulfil this role. It was completely built
in the SDSS-DR12 spectroscopic sample is 2,040,465, which we using SDSS-DR12 spectroscopic sample, which ensures the
extendedto2,209,299(seeSec.2.2). qualityofphotometricmeasurementsandallowustoisolate
the effect of colour-magnitude coverage from its correlation
with photometric errors.
2 THE CATALOGUES
For each of the four SDSS colours, we defined intervals
Toenablerealisticperformanceestimationforphoto-z meth- forming a 4-dimensional rectangular parallelepiped (1.0 <
odswepresenttwodatasetsbuilttomimicthemaincauses u−g < 2.2, 1.2 < g −r < 2.0, 0.1 < r −i < 0.8, and
of non-representativeness between spectroscopic (training) 0.2<i−z<0.5).Wethenselected≈150,000objectsfrom
andphotometric(test)samples:thedisparityincolour-space the SDSS-DR12 spectroscopic sample whose colours were
coverage (Teddy) and the differences between photomet- within this space and with an extra constraint on r-band
ricerrordistributions(Happy).Eachcatalogueiscomposed magnitude, r <21 - this is called the narrow set. All other
of 4 samples with known spec-z: one following the charac- objects were grouped into what we called the wide set.
teristics of the real spectroscopic sample, which should be From the narrow and wide sets, we constructed one
used for training/calibration purposes (A) and three hold- training sample (named A) and three test samples (named
ingincreasingdegreesofnon-representativenessofA,which B, C, and D) for comparing different scenarios. Sample A
should be used as test (or validation) sets (B, C and D). and B were created by splitting the narrow set into two
In what follows, we describe how the catalogues were con- comparable parts, sample C was constructed by truncating
structed and the main effects they allow us to probe. the wide set on the colour coverage of the narrow set and
finally,sampleDwasmadebysamplingrandomlyfromthe
wide set. The number of objects contained in each sample
2.1 Teddy: the effect of colour coverage
is shown in Tab. 1. In summary, sample A and B follow ex-
Thedisparityinfeature-spacedistributionbetweentraining actly the same feature space distribution, sample C has the
andtestsetshasbeenknowntoimpactclassificationandre- samecoverageofA,butadifferentdistribution,andsample
gressiontasksinmachinelearning(Quionero-Candelaetal. D has a wider coverage in colour-magnitude space.
2009).Specificallyinthephoto-z case,thistranslatesintoa The r-band magnitude and colour distributions for all
spectroscopic sample which fails to cover the entire domain samples in the Teddy catalogue are shown in Fig. 1. We
occupied by the photometric sample in colour-magnitude observe at ease the colour cut imposed on samples A, B,
parameter space. It has been already reported that such a and C. By construction, samples C and D follow the same
gap introduces significant biases in redshift estimation for distributioninthespaceregion,buttheirprobabilitydensity
empirical (e.g. MacDonald & Bernstein 2010) as well as for functions(PDFs)differduetomarginalization.Asignificant
template fitting techniques (e.g. Dahlen et al. 2013). number of galaxies with colours outside the 4-dimensional
In the context of empirical methods, in which there is parallelepiped contribute to the disparity between the two
no input information beyond magnitudes and colours from curves.
the spectroscopic sample, it is straightforward to expect WeconsiderAthespectroscopicsample(usedfortrain-
that results will not be reliable beyond the training do- ing) and B, C and D as distinct photometric samples (used
main. Machine learning methods only learn by example, for validation/test). These correspond to increasingly com-
thus their results should not be extrapolated. For template plex data situations. Training in A and testing in B rep-
fitting methods, the region of the parameter space not cov- resents the best case scenario where distributions are com-
ered by the spectroscopic sample can be directly related to pletely representative of each other - and thus must yield
fainter objects having active galactic nucleus (AGN) activ- thebestpossibleresults.TraininginAandtestinginCcor-
ity and metallicity levels not represented in the template responds to the situation of simply ignoring data outside
MNRAS000,000–000(0000)
4 R. Beck et al.
Happy A & B DR12 spec
F
D
P
Happy C Happy D DR12 photo
F
D
P
15 18 21 24 −2 0 2 4 0 1 2 3 −1 0 1 −1 0 1
r-band u g g r r i i z
− − − −
Figure 2.Distributionsformagnitudeinr-bandand4colorsfortheHappy cataloguecomparedwiththefullspectroscopic(red)and
photometric(purple)distributionsfromSDSS-DR12.
spectroscopic coverage - in this case methods like applying extend the DR12 spectroscopic set (S1) by cross-matching
weightsshouldprovideslightlybetterresultsthanthestan- SDSS photometric measurements with galaxies from other
dard approach. Training in A and testing in D corresponds spectroscopicsurveys.Thisapproachprovidesadeepersam-
toasituationwithincompletecoverage,wherepuremachine ple of spectroscopic galaxies, while also keeping the use of
learning methods are not expected to provide good results. actual SDSS photometry and its inherent systematics.
It is important to emphasize that the colour and mag- We followed the Bayesian cross-matching methodology
nitude distributions in Teddy are not realistic, but they described in Budava´ri & Szalay (2008), only accepting rel-
do provide a test bench to probe the robustness of photo- atively secure matches, with a Bayes factor above 10,000
z methods against feature distribution shape and coverage. (Eq. 16 in Budava´ri & Szalay 2008). The following surveys
Foranidealphoto-z estimator,resultsfromallthreeconfig- were used in the match:
urations should be equivalent.
• 2dF (Colless et al. 2001, 2003, 770 matches),
• 6dF (Jones et al. 2004, 2009, 765 matches),
• DEEP2 (Davis et al. 2003; Newman et al. 2013, 7456
2.2 Happy: the effect of photometric errors
matches),
Once we have a data set which enables probing the robust- • GAMA (Driver et al. 2011; Baldry et al. 2014, 53373
ness of photo-z methods in the case of incomplete colour matches),
and r-magnitude coverage, we approach other important • PRIMUS (Coil et al. 2011; Cool et al. 2013, 32459
differencesbetweenspectroscopicandphotometricsamples: matches),
the presence of photometric errors and their correlation • VIPERS (Garilli et al. 2014; Guzzo et al. 2014, 18967
withcolourcoverage.InconstructingTeddy,werearranged matches),
data from the original SDSS-DR12 spectroscopic data set. • VVDS (Le F`evre et al. 2004; Garilli et al. 2008, 8381
Thus,allitssamplessharethesamespectroscopicdataqual- matches),
ity level. This is not what happens in the real situation, • WiggleZ(Drinkwateretal.2010;Parkinsonetal.2012,
wherephotometricobservationsarestatisticallyfainterand 43874 matches), and
of poorer quality than the spectroscopic ones. • zCOSMOS (Lilly et al. 2007, 2009, 2789 matches).
The Happy catalogue was created to allow a clear as-
sessmentoftheimpactofphotometricerrorsonphoto-z esti- RefertoBecketal.(2016b)foradditionaldetailsregarding
mation,whileatthesametimemorecloselyresemblingthe the cross-match – the same data were used here, with the
colour space differences between the original SDSS-DR12 importantdistinctionthatphotometriccolouranderrorcuts
spectroscopic and photometric data sets. were not applied.
Our first goal was to reproduce the colour-magnitude This procedure enabled us to find 168,834 matches,
space distribution of the SDSS-DR12 photometric sample, which extended the total number of galaxies with spectro-
only with objects with measured spectroscopic redshifts, so scopicredshiftsto2,209,299,andalsoextendedthecolour-
thatthephoto-z methodscouldbeproperlyevaluated.How- magnitude space coverage of the sample such that the pa-
ever,astheDR12spectroscopicsetdoesnotcontainobjects rameter range of the SDSS photometric set was covered.
withmoreextremecoloursandfaintermagnitudes,itisnot In order to build, from the new extended spectro-
an adequate source of example galaxies. Thus, we chose to scopicsample(E1),asubsetwhichfollowsthesamecolour–
MNRAS000,000–000(0000)
Realistic validation of photometric redshifts 5
magnitude distribution as the original SDSS-DR12 photo- rors the equivalent scenario of traditional photo-z valida-
metric set, we randomly selected 75,000 objects from the tion exercises. Happy C illustrates a photometric sample
SDSS-DR12 photometric sample (S2)3 and performed a thathasbeencuttoconformtothetrainingsample,witha
nearestneighbour(NN)searchinE1(incolour/r-magnitude largerproportionofobjectshavinghighphotometricerrors.
space). Happy D serves as a complete photometric sample, with
ForeachobjectinS2wesearchforits1st NNtoinclude both a wider parameter coverage and higher measurements
intoournewset,HappyD.Toavoidduplicateentries,ifthe errors.Thus,HappyCandDmustonlybeusedfortesting,
givenNNwasalreadyincluded,weselectthenextclosestNN representingincreasingdegreesofcomplexityandsimilarity
(2nd, 3rd, etc.) which was not already in Happy D. with the real photometric situation.
Then,wesimilarlyconstructedtwonewsubsetstorep-
resenttheDR12spectroscopicsample,HappyAandHappy
B. The former will act as a training set/spectroscopic sam-
3 PHOTOMETRIC REDSHIFT ESTIMATION
ple, while the latter will be a test set that has the same
distribution of photometric properties as the training set. Aimingatquantifyingtheimpactoftheabovediscussedef-
Thus,werandomlyselected2×75,000objectsfromS1,and fectsintraditionalphoto-z estimationmethods,weselected
searchedfortheirnearestneighbourinE1usingthemethod afewexamplestoillustratehowthecolour–magnitudecov-
outline above, again avoiding any duplicates within Happy erageanditscorrelationswithphotometricredshiftaccuracy
sets. canbequantifiedinfuturediscussionsonphoto-z methods.
Finally, to create an intermediate sample that is be-
tween the photometric error properties of S1 and S2, we
decidedtoperformaphotometricerrorcut.Ourgoalwasto 3.1 Empirical methods
reproducethesamerangeofphotometricerrorsasinS1,but
Inwhatfollows,weintroduceaselectionofempiricalphoto-
with a distribution that resembles S2, being more weighted
z estimation techniques that were chosen to represent this
towardshighererrors.Thus,thecutwaschosentobeatthe
class of methods.
98th percentile(todiscardoutliers)ofthephotometricerror
distribution of S1 for each observed feature. We randomly
selected150,000objectsfromS2,searchedfortheirnearest
3.1.1 Artificial neural network (ANNz)
neighboursinE1followingthesameprocedure,andapplied
the error cut. This yielded the set Happy C, composed of ANNz (Collister & Lahav 2004) implements a particular
≈60,000galaxies.WenotethatcontrarytoallotherHappy type of artificial neural networks known as multi-layer per-
setpairings,HappyCandHappyDwereallowedtooverlap ceptron,whichisformedbyasetoflayers,eachoneofthem
to avoid excessively selecting from the less populated faint populated by a number of nodes. The first layer receives
end of E1. the observed magnitudes, or colours, the final layer returns
Cutting the photometric error range of Happy C to theestimatedphoto-z valuesandtheintermediatelayersare
match that of Happy A and B had an important side ef- considered hidden - since they can contain any number of
fect: the magnitude and colour ranges (note: the range, not nodes.Allnodesinagivenlayerholdanactivationfunction
the shape of the distribution) were also essentially cut to and are connected to all the nodes in adjacent layers, with
the limits of Happy A and B. This shows that the effects eachconnectioncorrespondingtoaweightparameterwi,j4.
of photometric error and magnitude/colour coverage are in Given a training set with measured magnitudes and
fact very much intertwined, one cannot modify one with- spectroscopic redshifts, the network is trained by determin-
out affecting the other. There are two feasible explanations ing the set weight parameters values, www, which minimizes
for why the colour ranges shrink because of the error cut. the cost function
First, this observation could indicate that the wider colour
N
distributions in the photometric sample are mainly caused E =(cid:88)[z (www,m¯)−z ]2, (1)
p spec
by the higher photometric errors smearing the distribution, i=1
not by containing physically different extreme galaxies that
wherem¯ isthesetofobservedmagnitudes,z istheoutput
p
are missingfromthespectroscopicsample. Second,itcould
given by the last layer and z the spectroscopic redshift
spec
be a consequence of galaxy types with extreme colours be-
(Abdalla et al. 2011).
ing significantly more likely to have high measurement er-
The neural network used in this study is configured to
rors, therefore these would be preferentially eliminated by
have two intermediate layers, resulting in four layers in to-
the cut, and also wouldn’t be present in the spectroscopic
tal. The first layer receives five input features: the r-band
sample. Figure 2 shows the r-magnitude and colour PDFs
magnitude and four colors, normalized by their respective
for all samples in Happy, and the number of objects con-
mean and standard deviation. The two intermediate layers
tained in each set is shown in Tab. 1.
contain each ten nodes. The final layer outputs one node
Following the same reasoning as presented in the pre-
for the photo-z prediction, so that the whole network has a
vious section, we built Happy A to act as a spectroscopic
5-10-10-1 structure.
sample and, as such, should be used for training. Happy In what follows, the network was trained in Teddy A
B is completely representative of Happy A and so mir- andHappyAandsubsequentlyappliedtotheothersamples
3 Tobeprecise,theobjectswereselectedfroma2millionelement 4 Theindexesindicatethetwonodesconnectedbythisparame-
randomsub-sampleofS2. ter.
MNRAS000,000–000(0000)
6 R. Beck et al.
in each catalogue. We did not consider measurement errors to estimate f directly without imposing any specific func-
in this work. tional formula. In fact, f(x) = (cid:80)∞ α φ (x) where φ ’s
k=1 k k k
are known basis functions. Restricting the upper bound of
k to a reasonable number K, f(x) ≈ (cid:80)K α φ (x). Then
3.1.2 Local linear regression (LLR) k=1 k k
one can estimate α’s as usual using parametric regression
Thelocallinearregressionmethodfindsthek-nearestneigh- methods. Examples of basis functions include cubic spline
boursof the testgalaxyincolour space fromatrainingset, basis,B-splines,Haarwaveletbasisfunctionsandradialba-
andperformsahyperplanefitontheseneighbours.Thisway, sis functions.
afunctionalformisfittedthatalsofollowsthelocaltrends, Thisset-updoesnotimposeanyassumptiononf.Even
andthereforecanbequiteflexible.Theimplementationused with a modest number of predictors, this attractive prop-
hereisthesameasinBecketal.(2016b),refertothatpaper ertyrequiresestimatingahugenumberofparameters.Thus
formoredetails.WenotethatwhileBecketal.(2016b)per- f cannot be estimated properly due to the curse of dimen-
formed a template fitting step after the photo-z estimation sionality.Totacklethisproblem,Hastie&Tibshirani(1990)
itself,inthispaperwedonotutilizetheextraphysicalinfor- suggests to fit generalized additive models (GAMs).
mation, therefore that step was omitted entirely. Thus, the GAMsassumethatf(x)=f1(x1)+f2(x2)+...+fp(xp).
method used here can be categorized as purely empirical. Ifeachfj demandsmcomponentsthenthetotalnumberof
The number of nearest neighbours used in Beck et al. parameters is p×m which is reasonable even for moder-
(2016b) is k = 100, but since we are also testing extrapo- ate size studies. Penalized least squares (e.g. Ruppert et al.
lation capabilities, that number was increased to k = 1000 2003) is a powerful approach to fit GAMs. Traditionally,
here. The increase does not noticeably affect estimation re- GAMsarefittedusingbackfittingalgorithm(Hastie&Tib-
sultswhenwithinthecoverageofthetrainingset,buttaking shirani 1990). If each fj estimated by fˆj then f(x) can
a larger colour space region into account better determines be estimated by fˆ(x) = (cid:80)pj=1fˆj(xj). A prediction at x0
the functional form, and significantly reduces scatter when is obtained by fˆ(x ) = (cid:80)p fˆ(x ). Additional details
0 j=1 j 0j
extrapolating. Of course, there is a trade-off in computa- on fitting models and conducting inferential statistics using
tional performance when using more neighbours. GAMScanbefoundin(Hastie&Tibshirani1990;Ruppert
et al. 2003; Wood 2006). We should note that the GAM
methodology herein developed is a more general extension
3.1.3 Generalized additive model (GAM)
ofthecosmophotoz(Elliottetal.2015)package,whofirst
Generalized linear models (GLMs), as its name suggests, introduced the use of GLMs for redshift estimation.
assume - through a link function - a linear relationship be-
tween the response variable y and set of predictors x. The
distribution of y is a member of the exponential family and
the link function is monotonic and differentiable. However,
3.1.4 Random forest
GLMs also allow different relationships between the mean
and the variance. For example, in linear models (LMs) the Randomforestsdepictensemblesofindividualclassification
response mean is independent of the variance and given or regression trees, which are fit given the available train-
simply by E(y) = xTβ. For Poisson models with log link ing data. Each tree is built from top to bottom, where the
logE(y) = xTβ, the mean is equal to the variance. In this rootcorrespondstoallconsideredtraininginstancesandthe
context,β isavectorofunobservableregressioncoefficients leavestosubsetsofinstances.Theinternalnodesofeachtree
to be estimated from the data using Maximum Likelihood areusuallysplitrecursively(intotwo)basedoncertainsplit-
(ML) methods. ting criteria. The overall recursive process stops as soon as
Let βˆ be the ML estimate of β. A prediction at a new some stopping criterion is fulfilled per leaf node. In case of
point x is obtained by xTβˆ for LMs and by exp(xTβˆ) for fully grown trees, each leaf corresponds to single pattern or
0 0 0
Poisson models. GLMs also enclose logistic and gamma re- toagroupof“pure”patterns(i.e.,agroupofpatternshaving
gression, among many others. For the following discussion all the same label).
we will only consider LMs but the same reasoning can be The original way of constructing random forests is to
followed for any member of GLMs. For a comprehensive consider, for each individual tree, a subset of the train-
treatment on LMs see Isobe et al. (1990); Kutner (2005); ingpatterns,calledbootstrapsample (Breiman2001).These
Dobson(2010);Christensen(2011);Myersetal.(2012)and bootstrapsamplesaredrawnuniformlyatrandom(withre-
forGLMsseeNelder&Wedderburn(1972);Dobson(2010); placement) to generate slightly different training sets and,
Myersetal.(2012).ForGLMsapplicationsinastronomythe hence, slightly different individual trees. Another way is to
readerisrefereedtoAndreon&Hurn(e.g.2010);DeSouza considerslightlydifferentsplitsateachinternalnodeby,e.g.,
etal.(e.g.2015a,b);Elliottetal.(e.g.2015);DeSouzaetal. consideringdifferentfeaturedimensionsorrandomsplitting
(e.g. 2016). thresholds.Theoverallperformanceofsuchaforestoftrees
It is understood that the exact functional relationship is usually much better than the one of the individual trees
between E(y) and x is unknown. In other words, E(y) = (due to the variance reduction that stems from combining
f(x) for unknown f. LMs assume that f can be approxi- the predictions made by the trees).
mated by xTβ. Despite its simplicity, LMs have been suc- The splitting processes taking place at the internal
cessfullyappliedinmanyareas.However,ithasbeenfound nodesisbasedonmeasuringthegainin“purity”,whichcan
thatthelinearityassumptioncanberestrictiveandtoosim- be,inturn,measuredviadifferentscores.Typicalones,mea-
pletoaccountfornon-trivialrelationshipsinthedata.Non- suring how impure a set of patterns associated with a node
parametric regression relaxes this assumption and allows is, are the mean squared error (MSE) for regression prob-
MNRAS000,000–000(0000)
Realistic validation of photometric redshifts 7
lems or the Gini index for classification problems. We refer The default implementation, used in this work, con-
to Breiman (2001) for a detailed description. structs a minimum representative template set of 5 spec-
Inthiswork,weconsiderrandomforestregressorsaswe tra derived from the application of a non-negative matrix
are interested in real-valued redshift estimates. While var- factorization (NMF - Blanton & Roweis (2007)) algorithm
ious parameters can be set for random forests, the perfor- to the set of 485 synthetic spectra provided by Bruzual &
mances of the induced models are often very similar among Charlot(2003).NMFcanbeconsideredakindof“principal
eachotheraslongasreasonableparameterassignmentsare component”derivation, with the additional constraint that
chosen. In the remainder of this work, we consider the fol- the linear combination coefficients need to be non-negative.
lowing setup: For all random forest models, 500 individual Results are thus more interpretable than the ones derived
fully-grown trees are fitted given bootstrap samples, where from a standard principal component analysis (Ishida & de
√
dfeaturesaretestedperinternalnodesplitusingtheMSE Souza 2011, 2013; Jolliffe 2013; De Souza et al. 2014).
as impurity measure. Althoughthetemplatesusedinthismethoddoprovide
a larger wavelength coverage, it is important to emphasize
that the semi-analytical models themselves are constructed
3.2 Template fitting methods fromdetailedvisualandNIRobservationsofnearbyobjects.
Thus,theaccuracyofthesemodelsinpredictingthespectral
Inthissection,wecontinueontooutliningthedetailsofthe
behaviour of high redshift galaxies is also limited. As the
spectraltemplatefittingmethodsthatwereanalysedinthis
authorspointedoutthemselves,thediscrepancyinUVand
paper.
NIRfluxesbetweenobservedspectraandtheonechosenby
EAZYislargerintherest-frameUVandNIRwavelengths.
Consequently, its results will also be subjected to the lack
3.2.1 Bayesian Photometric Redshifts (BPZ)
of representativeness discussed above.
BPZ5 applies Bayesian inference to the problem of photo-
metric redshift estimation (Ben´ıtez 2000). In this context,
the probability of a galaxy with measured colour and mag- 3.2.3 Photo-z-SQL
nitudes, {C,mmm}, to have a redshift z can be described as
Photo-z-SQL7 is a recent Bayesian template fitting photo-
z implementation in C# that can be integrated into a
p(z|C,mmm)∝p(z|mmm)p(C|z), (2) databaserunningMicrosoftSQLServer(Becketal.2016a).
Thecodeisfairlyflexibleinthechoiceoftemplatesandpri-
wherep(C|z)istheprobabilityoftheobservedcoloursgiven
orsandthuscaneasilyadoptsuccessfulapproachesfoundin
agalaxyatredshiftz(likelihood)andp(z|mmm)istheexpected
the literature. Moreover, it features an iterative photomet-
redshiftdistributionforgalaxieswithmeasuredmagnitudes
ric zero-point calibration to optionally take into account a
mmm (prior). This description assumes that the likelihood de-
spectroscopic training set.
pends only on the measured magnitudes and morphological
We used the stand-alone version of the code, searching
galaxytype(Ben´ıtez2000).Thefeaturewhichdifferentiates
for the maximum probability photo-z value using Bayesian
this approach from the others which came before it is the
estimation. We present results for two configurations, both
introduction of the prior. It improves over the simple tem-
computed on a redshift grid with a linear step size of 0.01
platefittingχ2minimizationbyallowingtheintroductionof
between z=0 and z=1.
informationaboutthegalaxymorphologicaltypeandhelps
Thefirstconfiguration,whichwerefertoasSQLBPZ,
avoidingunrealisticredshiftestimationsbyusingsimpleas-
uses the BPZ spectral template set of Coe et al. (2006),
sumptionsastherangeofredshiftexpectedforaparticular
and the prior of Ben´ıtez (2000) that had been derived from
survey.
HDF-N data. It also utilizes an empirical filter zero point
In this work we present results using the default set of
calibration based on the training sets, and adds a separate
8 spectral energy distributions (SEDs) based on Coleman
photometricerrortermof0.02magstoaccountfortemplate
et al. (1980) and Kinney et al. (1996) and two extra inter-
mismatch. The two major differences between this and the
polated ones between each pair (default option). The filters
earlier BPZ approach (§3.2.1) are the differing calibration,
zero-points were calibrated using samples A.
and the larger number of interpolated templates (ten be-
tween each pair).
The second configuration – denoted by SQL LP – uses
3.2.2 EAZY the Le PHARE spectral templates of Ilbert et al. (2009) in
Easy and Accurate Redshifts from Yale6 (EAZY)minimizes conjunction with a flat prior. In this case, zero point cali-
bration was not used, and the extra error term was chosen
thedependenceonspectroscopicallymeasuredspectrabyre-
to be only 0.01 mags due to the larger and more detailed
lying on synthetic SEDs from semi-analytical models. This
setoftemplates(641forSQLLP,asopposedto71forSQL
set,despitenotenclosingallpossiblegalaxytypesanddust
BPZ).
models,providesfurthercompletenesstoUVandNIRwave-
These configurations were selected to optimize results
lengths than SEDs built from spectroscopic observations
on the Teddy B and Happy B samples, which correspond
(Brammer et al. 2008).
totheusualcaseofonlyhavingvalidationinformationbased
5 http://www.stsci.edu/∼dcoe/BPZ/
6 http://www.astro.yale.edu/eazy/ 7 https://github.com/beckrob/Photo-z-SQL
MNRAS000,000–000(0000)
8 R. Beck et al.
onaspectroscopicset.Inthosesamples,theSQLLPconfig- photo-z predictions for most galaxies with r−i > 0.7 are
urationdidnotbenefitfromeitherthecalibrationorapply- located around z ≈0.46 and predictions for those with
photo
ingtheHDF-Nprior,whiletheSQLBPZcasewasimproved 0.6<r−i<0.7arefoundaroundz ≈0.36.Therefore,
photo
by both. a local minimum in the galaxy population at r−i ≈ 0.7
Forreference,theSQLBPZandSQLLPconfigurations would yield a deficit of predictions around z ≈ 0.41,
photo
correspond to the notation BPZ HDF Err ZP and LP Flat which corresponds to the“neck”we observe below the ap-
Err, respectively, in Beck et al. (2016a). parent horizontal feature in Fig. 3. We conclude that the
r−i distributions of sets B and C are responsible for this
result.
4 RESULTS Results for Teddy D are shown on the right panels of
Fig.3andtheleftpanelofTab.2.Comparedtothetwopre-
4.1 Diagnostics of estimators
viouscases,thestd,MAD,andoutlierratearesignificantly
Following earlier works that compare photo-z methods largerforallmethods.Asexpected,thisconfirmsthatifwe
(Hildebrandtetal.2010;Dahlenetal.2013),weselectedfour donotaccountforthedifferenceofcolourcoveragebetween
summarystatisticstoquantifythephoto-z estimationqual- spec-z and photo-z, the assumed error on photo-z will be
ityofthevariousmethodstestedhere.Weconsiderthenor- underestimated.
malizedredshifterror,∆z =(z −z )/(1+z ),
norm spec photo spec Apartfromthisgeneraloutcomeacrossempiricalmeth-
and from the distribution of ∆znorm, we compute its mean ods, two distinct behaviours have been observed on set D.
(which is also the average bias), standard deviation (std),
The z -z scatter from GAM and LLR stay close to
spec photo
median absolute deviation (MAD), and outlier rate. Out-
thediagonal,whileANNzandrandomforestshowtwohor-
liers are defined by |∆z | > 0.15. The median absolute
norm izontal features, resulting in wrong predictions for galaxies
deviation MAD=median(|∆z |) is computed with out-
norm with a true redshift at z < 0.2 or z > 0.45. Essentially, in
liersincluded;andthemean∆z andstandarddeviation
norm the latter case z values are truncated at the end of the
photo
σ(∆z )arecomputedwiththeoutliersremovedfromthe
norm trainingsetcoverage.Thiseffecthasalsobeenillustratedin
samples.
Fig. 4. ANNz clearly has strong z -dependent bias, while
spec
its overall bias is rather small due to positively and nega-
tively biased regions cancelling out. In contrast, LLR has
4.2 Results from Teddy
much smaller redshift-dependent bias, but the overall bias
In order to quantify the impact of the lack of r- is higher. A similar comparison between random forest and
magnitude/colour coverage between spectroscopic and pho- GAM presents almost the same picture, with the exception
tometric samples, we applied the photo-z methods de- that GAM also has low overall bias.
scribed above to the Teddy catalogue. Methods were
This result is a consequence of intrinsic differences be-
trained/calibrated on Teddy A and tested on Teddy B,
tween the methods we tested. GAM is a method that fits a
C and D to represent increasing levels of mismatch.
rather general set of smooth functions to the training data.
There is a“global”relationship between covariates and the
response variable, which is why we will refer to GAM and
4.2.1 Empirical methods
similar methods as“global”methods. Of course, the func-
Figure 3 shows the photo-z estimations versus their spec-z tioncanbeevaluatedevenwhenthereisnocoverageofthe
values.Eachrowrepresentsoneofthefourmachine-learning trainingset,whichiswhy“global”methodsareexpectedto
methods described in §3.1. Teddy B, represented on the perform well when extrapolating, and indeed that is what
left panels of Fig. 3, yields very satisfactory results in gen- we observe on Teddy D.
eral.Thisobservationisnotsurprisingsincethistestingset
On the other hand, methods that are strictly depend
shares,byconstruction,thesamefeaturespacepropertiesof
on the examples present in the training set and do not at-
thetrainingset,Teddy A.Thediagnosticsofresults,given
tempt to extrapolate its features, e.g. nearest neighbours
by the left panel of Tab. 2, show that the absolute value of
and random forests, are not expected to be able to perform
thenormalizedbiasdoesnotexceed7×10−4 andtheoutlier
well beyond the boundaries of the training set – hence the
rate0.2%forallfourmethods.Wealsoseethat,duetothe
observed truncation on set D. For such models, the max-
color cut, the spec-z of most galaxies of set B is between
imum redshift values that can be predicted is determined
0.15 and 0.6.
bytheredshiftsgiveninthetrainingdata.Neuralnetworks
Results for Teddy C are shown on the middle panels
could, in principle, fit an arbitrary“global”functional for-
of Fig. 3. While set C shares the same color coverage as set
mula, but in practice we observe the same behaviour with
A, the distribution differs. This disparity of color distribu-
ANNzaswiththerandomforest,i.e.dependencedominated
tionhasaminorimpactonthephoto-zscatter.Asshownby
bycolour-magnitudespaceneighbours.Thereforewewillre-
Tab. 2, the std and the outlier rate of Teddy C is slightly
fer to random forest, ANNz and similar methods as“local”
higher than that of Teddy B, whereas the mean and the
methods.
MADarenotaffectedbyasignificantchange.Readersmay
notice a horizontal feature privileging a prediction around LLRisaninterestinghybridoftheprevioustwoclasses:
z ≈ 0.45 for both sets B and C. This feature can be it is based on nearest neighbours, thus it should be“local”,
photo
explained by the lack of objects in the r −i distribution but it also fits a linear functional formula that can be used
around0.7.Fig.1showsthatr−ipeaksat0.6and0.8.By toextrapolate.Indeed,withenoughneighbours(k=1000),
examining the colour of these objects, we discovered that LLR does extrapolate reasonably well on Teddy D.
MNRAS000,000–000(0000)
Realistic validation of photometric redshifts 9
ANNz
1.0
0.8
z
-
oto 0.6
h
P
0.4
0.2
Teddy B Teddy C Teddy D
0.0
GAM
1.0
0.8
z
-
oto 0.6
h
P
0.4
0.2
Teddy B Teddy C Teddy D
0.0
LLR
1.0
0.8
z
-
oto 0.6
h
P
0.4
0.2
Teddy B Teddy C Teddy D
0.0
Random forest
1.0
0.8
z
-
oto 0.6
h
P
0.4
0.2
Teddy B Teddy C Teddy D
0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Spec-z Spec-z Spec-z
Figure 3. Results on three testing sets of the Teddy catalogue (columns) obtained from four empirical photo-z methods (lines). The
colourgradientshowsthelogarithmicdensity.Thedashedlinesdefinetheperfectprediction(center)andthelimitsforbeingconsidered
outliers.NumericalresultsareshowninTab.2-leftpanel.
MNRAS000,000–000(0000)
10 R. Beck et al.
Diagnostics Diagnostics
Method Set Mean Std MAD Outlierrate Method Set Mean Std MAD Outlierrate
(×10−2) (×10−2) (×10−2) (%) (×10−2) (×10−2) (×10−2) (%)
B 0.03 2.35 1.16 0.18 B 3.51 3.07 3.63 0.49
ANNz C -0.01 2.45 1.15 0.26 BPZ C 3.48 3.30 3.69 0.58
D -0.08 5.67 3.61 3.09 D 2.61 4.73 3.66 3.60
B 0.05 2.62 1.34 0.11 B -2.99 3.82 3.05 2.71
GAM C 0.06 2.79 1.38 0.18 EAZY C -3.71 4.07 3.57 4.03
D -0.06 3.93 2.23 2.28 D -3.64 4.62 4.34 6.58
B 0.07 2.35 1.14 0.19 B 2.13 3.34 2.28 0.43
LLR C 0.05 2.44 1.14 0.28 SQLBPZ C 1.68 3.40 2.00 0.64
D 1.76 4.08 2.46 3.80 D 0.94 4.06 2.41 2.01
B 0.03 2.38 1.18 0.17 B -0.45 3.40 1.95 0.53
Random
C -0.01 2.49 1.17 0.26 SQLLP C -0.7 3.61 2.14 0.70
forest
D 0.16 6.85 5.24 6.70 D -0.48 4.19 2.74 3.29
Table 2.ResultsfortheTeddycatalogue.
ter: ≈0.035 for BPZ and EAZY, and ≈0.02 for SQL BPZ.
SQLLPhasthelowestbias,≈0.005.Onthesesamples,the
machinelearningmethodshaveaclearedgeinperformance.
A visual inspection of Fig. 5 reveals that the high bias
(andoutlierrate)ofEAZYisduetoasharpturnawayfrom
thediagonalabovez =0.4.Incontrast,forthetwoBPZ
spec
template methods, the bias is caused by an upward shift of
the entire population.
OnthesampleTeddy D,whichisthecaseillustrating
extrapolation outside the coverage of the training set, we
can again observe a high bias around ≈ 0.03 for the BPZ
and EAZY methods, with outlier rates of 3.6%, 6.6%, and
stdrisingto≈0.046.However,inthecaseofSQLBPZand
SQLLP,theoverallbiasremainsbelow0.01,withascatter
of ≈0.04 and 2−3% outliers.
For the extrapolating case of Teddy D, which is the
worst-case scenario in this catalogue, we compare the nu-
merical diagnostics for both empirical and template fitting
methodsinFig.6.Inthisfigure,eachphoto-z methodisrep-
resented by a different symbol and each panel corresponds
to a different diagnostic. The black star in the origin of
eachpanelrepresentsthebest-casescenario,whereamethod
would lie. Thus, the closer a symbol is from the black star
the better the corresponding method performed according
Figure4.Violinplotofthenormalizedphoto-z estimationerror
on Teddy set D, for the ANNz and LLR methods. The redshift to a given diagnostic. In this visualization, it is possible to
binshaveawidthof0.1. notethattemplatefittingmethodsoutperform“local”,non-
extrapolating empirical methods on this sample, and are
evencomparableto“global”methodsaslongastheiroverall
4.2.2 Template fitting methods biascanbemanagede.g.withpropercalibration.However,
GAM still has a slight edge over the best-performing tem-
The photo-z estimationresults onthe catalogue Teddy for
plate fitting methods in this test, SQL LP and SQL BPZ.
the four template fitting methods (introduced in §3.2) are
shown in Fig. 5, with each row of scatterplots correspond-
ing to a method. Each column represents a subsample of
4.3 Results from Happy
Teddy: in order, sets B, C and D. Teddy A was used as
thecalibrationsampleformethodswhereitwasapplicable. Although Teddy is a good starting point to probe the bias
Numerical diagnostics are presented in the right panel of between photometric and spectroscopic samples, it is still
Tab. 2. simplerthantherealscenario.Itwasintendedtoisolatethe
TheresultsdonotchangesignificantlybetweenTeddy effect of gap in feature space coverage, but also it was built
BandCforanyofthefourmethods,onlybecomingslightly entirely from the SDSS spectroscopic sample, which means
worse for the latter. The std and MAD values are all in the thatallitsobjectsfulfilthesamespectroscopicdataquality
neighbourhood of 0.03, and the outlier fraction is ≈ 0.5% requirements.Inwhatfollowsweshallrelaxthisassumption
(withtheexceptionofEAZY,where3−4%).However,most and quantify the performance of photo-z estimators in a
ofthebiasvaluesarerelativelyhigh,comparabletothescat- harder and more realistic scenario. We now present results
MNRAS000,000–000(0000)