Forecast Verification: A Practitioner's Guide in Atmospheric Science

247 Pages·2003·2.53 MB·English

by  Ian T. Jolliffe, David B. Stephenson

Forecast Verification
Forecast Verification
A Practitioner’s Guide in
Atmospheric Science
Editedby
IAN T. JOLLIFFE
UniversityofAberdeen
and
DAVID B. STEPHENSON
UniversityofReading
Copyright(cid:1)2003John Wiley&SonsLtd,TheAtrium,Southern Gate,Chichester,
West SussexPO198SQ,England
Telephone((cid:1)44)1243779777
e-mail(for ordersandcustomerserviceenquiries):[email protected]
Visit our HomePageonwww.wileyeurope.comorwww.wiley.com
AllRightsReserved.Nopart ofthispublicationmaybereproduced,storedin a retrieval
systemortransmitted in anyformorbyanymeans,electronic,mechanical,photocopying,
recording,scanningor otherwise,except underthetermsoftheCopyright,Designsand
PatentsAct 1988orunderthetermsofalicenceissued bytheCopyright LicensingAgency
Ltd,90TottenhamCourtRoad,LondonW1T4LP,UK,withoutthepermissioninwritingof
thePublisher.RequeststothePublishershouldbeaddressedtothePermissionsDepartment,
John Wiley&SonsLtd,TheAtrium,Southern Gate,Chichester,West SussexPO198SQ,
England,[email protected],orfaxedto((cid:1)44)1243770620.
Thispublication isdesignedto provideaccurateandauthoritativeinformation inregard to
thesubjectmattercovered.ItissoldontheunderstandingthatthePublisherisnotengagedin
renderingprofessionalservices.Ifprofessionaladviceorother expert assistanceisrequired,
theservicesofacompetent professionalshould besought.
O t her W iley E dit orial O ffices
John Wiley&SonsInc.,111River Street,Hoboken,NJ07030,USA
Jossey-Bass,989Market Street,San Francisco,CA94103-1741,USA
Wiley-VCH VerlagGmbH,Boschstr.12,D-69469Weinheim,Germany
John Wiley&SonsAustralia Ltd,33ParkRoad,Milton,Queensland 4064,Australia
John Wiley&Sons(Asia)PteLtd,2ClementiLoop#02-01,JinXingDistripark,
Singapore129809
JohnWiley&SonsCanadaLtd,22WorcesterRoad,Etobicoke,Ontario,CanadaM9W1L1
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsin
print maynot beavailablein electronicbooks.
Library ofCongressCataloging-in-PublicationData
Forecast verification:apractitioner’sguidein atmosphericscience/edited byIan T.
JolliffeandDavidB.Stephenson.
p.cm.
Includesbibliographicalreferencesand index.
ISBN 0-471-49759-2(alk.paper)
1.Weatherforecasting–Statisticalmethods–Evaluation.I.Jolliffe,I.T.II.Stephenson,
David B.
QC996.5.F6772003
551.63–dc21 2002192424
BritishLibrary CataloguinginPublicationData
Acataloguerecord for thisbookisavailablefromtheBritish Library
ISBN 0-471-49759-2
Typesetin 10.5/13pt TimesNewRoman byKolamInformation ServicesPvt.Ltd,
Pondicherry,India
Printed andboundin Great Britain byAntonyRoweLtd,Chippenham,Wiltshire
Thisbookisprintedonacid-freepaperresponsiblymanufacturedfromsustainableforestryin
which at leasttwo treesareplantedfor each oneused forpaper production.
Contents
List of contributors ix
Preface xi
Chapter1 Introduction 1
I an T . J olliffe and D avid B . S t ephenson
1.1 A BriefHistoryand Current Practice 1
1.1.1 History 1
1.1.2 Current Practice 3
1.2 R ea sons for F oreca st Vecrai ftiion and its Benetfsi 4
1.3 T ypes of F oreca st s a nd Vcear itfiion D a ta 6
1.4 Scores, Skill, and Value 7
1. 4. 1 Sk ill Sco r es 8
1. 4. 2 A r ct iiaf il Sk ill 9
1. 4. 3 St a t ist ica l Sciga nn ifcie 10
1.4.4 ValueAdded 11
1.5 Data Qualityand Other PracticalConsiderations 11
Chapter2 BasicConcepts 13
JacquelineM. Potts
2.1 Introduction 13
2.2 TypesofPredictand 13
2.3 ExploratoryMethods 14
2.4 NumericalDescriptiveMeasures 19
2.5 Probability, Random Variablesand Expectations 23
2.6 Joint, Marginal, and ConditionalDistributions 24
2.7 Accuracy, Association and Skill 26
2.8 PropertiesofScoringRules 27
2. 9 Ver icfai tion as a R egression Problem 27
2.10 The M urphy–Winkler F ramework 29
2.11 D imensionalit y of t he Vecriaf ition Problem 36
Chapter3 BinaryEvents 37
IanB. Mason
3.1 Introduction 37
3. 2 Ver icfai tion M easures 39
3.2.1 SomeBasicDescriptiveStatistics 41
3.2.2 PerformanceMeasures 45
3. 3 Ver icfai tion of Binary F oreca st s: Theoretica l
Considerations 56
vi Contents
3.3.1 A G eneral F ramework for Vifeicra tion:
TheDistributions-oriented Approach 56
3.3.2 PerformanceMeasuresin TermsofFactorisations
oftheJoint Distribution 58
3. 3. 3 M et a vecra itf iio n : C r it er ia f o r S cr een in g P er f o r m a n ce
Measures 60
3.3.4 OptimalThresholdProbabilities 63
3 . 3 . 5 S a m p lin g U n cer t a in t yd ean nc ed InCt eor vna flis for
PerformanceMeasures 64
3.4 SignalDetection Theoryand TheROC 66
3.4.1 TheSignalDetection Model 67
3.4.2 TheRelativeOperatingCharacteristic 68
3. 4. 3 Ver ciafi tion M easures on R OC Axes 71
3. 4. 4 Ver ciafi tion M easures F rom Signal D etect ion T heory 73
Chapter4 CategoricalEvents 77
Robert E. Livezey
4.1 Introduction 77
4.2 T he Contingency T able: N otation, Dn ietfiions
and MeasuresofAccuracy 79
4. 2. 1 N o t a t io n a nn dit ioDn se fi 79
4.2.2 MeasuresofAccuracy 81
4.3 SkillScores 82
4.3.1 DesirableAttributes 82
4.3.2 Gandinand MurphyEquitableScores 84
4.3.3 GerrityEquitableScores 88
4.3.4 LEPSCAT 91
4.3.5 SummaryRemarksonScores 92
4.4 SamplingVariabilityoftheContingency
Tableand SkillScores 93
Chapter5 ContinuousVariables 97
MichelDe´que´
5.1 Introduction 97
5.2 Forecast Examples 97
5.3 First-order Moments 99
5.3.1 Bias 99
5.3.2 MeanAbsoluteError 101
5.3.3 Bia s Correct ion a nd Arctiifai l Sk ill 101
5.3.4 MeanAbsoluteErrorand Skill 102
5.4 Second and Higher-order Moments 103
5.4.1 MeanSquaredError 103
5.4.2 MSE SkillScore 104
5.4.3 MSE ofScaledForecasts 105
5.4.4 Correlation 106
5.4.5 AnExample:Testingthe‘Limit ofPredictability’ 110
5.4.6 Rank Correlations 110
5.4.7 ComparisonofMomentsoftheMarginal
Distributions 113
Contents vii
5.5 ScoresBased on CumulativeFrequency 115
5.5.1 Linear Error in ProbabilitySpace 115
5.5.2 Quantile–QuantilePlots(q–qPlots) 116
5.5.3 ConditionalQuantilePlots 116
5.6 ConcludingRemarks 119
Chapter6 VerificationofSpatialFields 121
WasylDrosdowsky andHuqiangZhang
6.1 Introduction:TypesofFieldsand Forecasts 121
6.2 TemporalAveraging 124
6.3 SpatialAveraging 126
6.3.1 MeasuresCommonlyUsed intheSpatialDomain 126
6.3.2 MapTypingandAnalogueSelection 131
6.3.3 Accountingfor SpatialCorrelation 132
6.4 Assessment ofModelForecastsin the
Spatio-temporalDomain 132
6.4.1 PrincipalComponent Analysis(EOF Analysis) 132
6.4.2 CombiningPredictabilitywith Model
F o r eca st Vecr ai tf iio n 133
6.4.3 SignalDetectionAnalysis 134
6. 5 Ver icfai tion of Spatia l R ainfall F oreca st s 135
Chapter7 ProbabilityandEnsembleForecasts 137
ZoltanToth, OlivierTalagrand,Guillem Candille
and Yuejian Zhu
7.1 Introduction 137
7.2 Main AttributesofProbabilisticForecasts 138
7.3 ProbabilityForecastsofBinaryEvents 142
7.3.1 TheReliabilityCurve 143
7.3.2 TheBrierScore 145
7. 3. 3 Ver ciafi t io n B a sed o n D ecisio n
ProbabilityThresholds 149
7.4 ProbabilityForecastsofMoreThan
Two Categories 151
7.4.1 Vector Generalization oftheBrier Score 151
7.4.2 Information Content asaMeasureofResolution 152
7.5 ProbabilityForecastsofContinuousVariables 154
7.5.1 TheDiscreteRankedProbabilityScore 154
7.5.2 TheContinuousRankedProbabilityScore 155
7.6 SummaryStatisticsfor EnsembleForecasts 155
7.6.1 EnsembleMeanErrorand Spread 156
7.6.2 EqualLikelihood FrequencyPlot 157
7.6.3 AnalysisRank Histogram 159
7.6.4 MultivariateStatistics 159
7.6.5 TimeConsistencyHistogram 161
7.7 LimitationsofProbabilityand Ensemble
F o r eca st Vecr ai fti io n 162
7.8 ConcludingRemarks 162
viii Contents
Chapter8 EconomicValueandSkill 165
DavidS. Richardson
8.1 Introduction 165
8.2 TheCost/LossRatio Decision Model 166
8.2.1 Valueofa DeterministicBinaryForecast System 168
8.2.2 ProbabilityForecasts 172
8.2.3 ComparisonofDeterministicandProbabilistic
BinaryForecasts 175
8.3 TheRelationship Between Valueand theROC 176
8.4 OverallValueand theBrier SkillScore 180
8.5 Skill, Value, and EnsembleSize 183
8.6 Summary 186
Chapter9 ForecastVerification:Past, PresentandFuture 189
D avid B . S t ephenson and I an T . J olliffe
9.1 Introduction 189
9.2 ReviewofKeyConcepts 189
9.3 Forecast Evaluation in Other Disciplines 192
9.3.1 Statistics 192
9.3.2 Financeand Economics 194
9.3.3 Environmentaland EarthSciences 196
9.3.4 Medicaland ClinicalStudies 197
9.4 FutureDirections 198
Glossary 203
References 215
AuthorIndex 227
Subject Index 231
List of Contributors
G. Candille LaboratoiredeMe´te´orologieDynamique, EcoleNormale
Supe´rieure, 24RueLhomond, F 75231Pariscedex05,
France. [email protected]
M. De´que´ Meteo-FranceCNRM/GMGEC/EAC, 42Avenue
Coriolis, 31057Toulousecedex01, France.
[email protected]
W. Drosdowsky BureauofMeteorologyResearchCentre,BMRC,POBox
1289K, Melbourne3001, Australia. w.drosdowsky@bom.
gov.au
I . T . J o lli ffe D ep a r t m en t o f M a t h em a t ica l S cien ces, U n iv er s it y o f
Aberdeen, King’sCollege, Aberdeen AB243UE, UK.
[email protected]
R.E. Livezey W/OS4, ClimateServicesDivision, Room 13228, SSMC2,
1325East West Highway, Silver Spring, MD 20910–3283,
USA. [email protected]
I.B. M a son Canberra M eteorological Ocfefi, P O Bo x 797, C a n b er r a ,
ACT 2601, Australia. [email protected]
J.M. Potts Biomathematicsand StatisticsScotland, TheMacaulay
Institute, Craigiebuckler, Aberdeen AB158QH, UK.
[email protected]
D. Richardson MeteorologicalOffice,LondonRoad,Bracknell,Reading,
RG122SZ, UK.
[email protected]
D.B. Stephenson Department ofMeteorology, UniversityofReading,
EarleyGatePO Box243,ReadingRG66BB, UK.
[email protected]
O. Talagrand LaboratoiredeMe´te´orologieDynamique, EcoleNormale
Supe´rieure, 24RueLhomond, F 75231Pariscedex05,
France. [email protected]
Z. Toth NOAAatNationalCentersforEnvironmentalPrediction,
5200Auth Rd., Room 207, Camp Springs, MD 20746,
USA. [email protected]
H. Zhang Bureau ofMeteorologyResearch Centre, BMRC, PO
Box1289K, Melbourne3001, Australia. h.zhang@bom.
gov.au
x List of Contributors
Y. Zhu NOAAatNationalCentersforEnvironmentalPrediction,
5200AuthRd.,Room207,CampSprings,MD20746,USA.
[email protected]