Table Of ContentOXFORD STATISTICAL SCIENCE SERIES
SERIES EDITORS
A. C. ATKINSON London School of Economics
D. A. PIERCE Radiation Effects Research Foundation, Hiroshima
M. J. SCHERVISH Carnegie Mellon University
D. M. TITTERINGTON University of Glasgow
R. J. CARROLL Texas A & M University
D. J. HAND Imperial College
R. L. SMITH University of North Carolina
OXFORD STATISTICAL SCIENCE SERIES
1. A. C. Atkinson: Plots, transformations, and regression
2. M. Stone: Coordinate-free multivariate statistics
3. W. J. Krzanowski: Principles of multivariate analysis: a user's perspective
4. M. Aitkin, D. Anderson, B. Francis, and J. Hinde: Statistical modelling in GLIM
5. Peter J. Diggle: Time series: a bio statistical introduction
6. Howell Tong: Non-linear time series: a dynamical system approach
7. V. P. Godambe: Estimating functions
8. A. C. Atkinson and A. N. Donev: Optimum experimental designs
9. U. N. Bhat and I. V. Basawa: Queuing and related models
10. J. K. Lindsey: Models for repeated measurements
11. N. T. Longford: Random coefficient models
12. P. J. Brown: Measurement, regression, and calibration
13. Peter J. Diggle, Kung-Yee Liang, and Scott L. Zeger: Analysis of longitudinal data
14. J. I. Ansell and M. J. Phillips: Practical methods for reliability data analysis
15. J. K. Lindsey: Modelling frequency and count data
16. J. L. Jensen: Saddlepoint approximations
17. Steffen L. Lauritzen: Graphical models
18. A. W. Bowman and A. Azzalini: Applied smoothing methods for data analysis
19. J. K. Lindsey: Models for repeated measurements Second edition
20. Michael Evans and Tim Swartz: Approximating integrals via Monte Carlo and
deterministic methods
21. D. F. Andrews and J. E. Stafford: Symbolic computation for statistical inference
22. T. A. Severini: Likelihood methods in statistics
23. W. J. Krzanowski: Principles of multivariate analysis: a user's perspective Updated
edition
24. J. Durbin and S. J. Koopman: Time series analysis by state space models
25. Peter J. Diggle, Patrick Heagerty, Kung-Yee Liang, and Scott L. Zeger: Analysis of
longitudinal data Second edition
26. J. K. Lindsey: Nonlinear models in medical statistics
27. Peter J. Green, Nils L. Hjort, and Sylvia Richardson: Highly structured stochastic
systems
28. Margaret S. Pepe: The statistical evaluation of medical tests for classification and
prediction
The Statistical Evaluation
of Medical Tests
for Classification and Prediction
Margaret Sullivan Pepe
Professor of Bio statistics, University of Washington
and Fred Hutchinson Cancer Research Center
OXFORD
UNIVERSITY PRESS
This book has been printed digitally and produced in a standard specification
in order to ensure its continuing availability
OXFORD
UNIVERSITY PRESS
Great Clarendon Street, Oxford 0X2 6DP
Oxford University Press is a department of the University of Oxford.
It furthers the University's objective of excellence in research, scholarship,
and education by publishing worldwide in
Oxford New York
Auckland Cape Town Dar es Salaam Hong Kong Karachi
Kuala Lumpur Madrid Melbourne Mexico City Nairobi
New Delhi Shanghai Taipei Toronto
With offices in
Argentina Austria Brazil Chile Czech Republic France Greece
Guatemala Hungary Italy Japan South Korea Poland Portugal
Singapore Switzerland Thailand Turkey Ukraine Vietnam
Oxford is a registered trade mark of Oxford University Press
in the UK and in certain other countries
Published in the United States
by Oxford University Press Inc., New York
© Oxford University Press, 2003
The moral rights of the author have been asserted
Database right Oxford University Press (maker)
Reprinted 2011
All rights reserved. No part of this publication may be reproduced,
stored in a retrieval system, or transmitted, in any form or by any means,
without the prior permission in writing of Oxford University Press,
or as expressly permitted by law, or under terms agreed with the appropriate
reprographics rights organization. Enquiries concerning reproduction
outside the scope of the above should be sent to the Rights Department,
Oxford University Press, at the address above
You must not circulate this book in any other binding or cover
And you must impose this same condition on any acquirer
ISBN 978-0-19-850984-4
Printed and bound by CPI Group (UK) Ltd, Croydon, CRO 4YY
PREFACE
The purpose of this book is to provide a systematic framework for the statistical
theory and practice of research studies that seek to evaluate clinical tests used in
the practice of medicine. My hope is that it will be found useful by both practicing
biostatisticians and more academic research biostatisticians. In addition, it could
be used as a textbook for a graduate level course in medical statistics.
While my background is set in mainstream biostatistical research (survival
analysis, clinical trials, measurement error, missing data and so forth), I was first
introduced to the problem of diagnostic test evaluation in 1995 when collaborat
ing on a hearing screening study. To my surprise, the statistical concepts that the
audiology researchers were using were only vaguely familiar to me. They were
not part of standard biostatistical training. Moreover, the methods available were
fairly rudimentary, lacking, for example, flexibility to adjust for covariates or to
handle clustered data. Since then my involvement with several other projects
has led me to conclude that these statistical concepts should be applied more
widely. At the same time, a more rigorous and expansive development of statis
tical methodology in this field has been taking place. This book should serve as
both a reference for the statistician applying these techniques in practice and as
a survey of the current state of statistical research for the academic interested
in developing new methodology in this field.
There is a growing need for the rigorous scientific evaluation of tests that
purport to identify disease or other conditions. The pace at which new medical
technologies are being developed to detect disease, to predict its course and to
evaluate response to therapy, is truly amazing. Proteomics, genomics, cell flow
cytometry and imaging modalities are among the array of new tools available.
In addition, improvements to existing technologies are continually being made.
My recent experience with the National Cancer Institute-sponsored Early Detec
tion Research Network (EDRN), which seeks to develop biomarkers for cancer,
has greatly reinforced the perspective that rigorous scientific studies must be
conducted to determine the accuracy and value of these biotechnologies. The
statistical methods described in this book should be useful for the design and
evaluation of such studies.
Much of the literature on evaluating biomedical tests appears in substantive
journals, particularly in radiology journals. Methods have not been developed to
the same extent or with the same rigor as they have for therapeutic research and
for epidemiologic research. There is a recent trend towards articles appearing in
mainstream biostatistical journals. I hope that this book will serve in some ca
pacity to stimulate further development of statistical methods and to place them
firmly within the realm of mainstream biostatistical research. Although most of
the book is written with the practicing statistician as the intended audience,
v
VI PREFACE
some sections are written specifically for the academic research biostatistician.
In particular, the comments on needed research that appear in the remarks at
the end of each chapter may be of interest.
My challenge and delight in writing has been to give a coherent structure
to the existing statistical methodology for evaluating tests and biomarkers. The
book of course reflects my understanding and perspectives on the field. Although
I have tried hard to provide a broad, balanced exposition, it is colored by my
experience with applications and by my research interests. Others would see the
field somewhat differently and provide more or less emphasis to various topics.
I hope that the book will provide a point of departure for productive discourse
in the field.
Writing this book has been both an enormously satisfying and exhausting
process. I could not have done it without a great deal of help. First, those who
typed the manuscript at various stages, especially Noelle Noble, who took on the
task with no prior knowledge of WI$L and amazed us with her skills obtained
over a short period of time. Others who worked on the book are Sandy Walbrek,
Pete Mesling and Molly Jackson. I have been fortunate to work with such a
talented and efficient group of people over the years.
Gary Longton, a superb statistician and colleague, not only produced anal
yses and figures for the book, but in doing so helped clarify my thinking on
numerous points. Our datasets and programs, written in STATA, are posted at
the website http://www.fhcrc.org/labs/pepe/book.
My initial work on the book was supported by the University of Washington
Genentech Professorship (1999-2001). I am grateful to the faculty at the UW,
especially Tom Fleming and Patrick Heagerty, who encouraged me to undertake
the project and who provided space and funding. Later work was supported
by the EDRN. I admire the foresight of our funding agency who saw the need
for statistical development in this field. I appreciate input from colleagues at
the EDRN and at the General Clinical Research Center at the University of
Washington.
Graduate students at the University of Washington were key to my complet
ing the book. They provided detailed critical feedback on the chapters, worked
through illustrations and ran simulation studies. Equally important, their inter
est in the topic and high expectations of me fueled my work to the end. I thank
especially Todd Alonzo (now at the University of Southern California), Chaya
Moskowitz (now at Memorial Sloan Kettering), Lori Dodd (now at the National
Cancer Institute), San-San Ou, Zheng Zhang, Elizabeth Rosenthal, Yingye Zheng
and Kyeong Kim. A number of people that I consider experts in the field were
kind enough to review the book for me. Although remaining weaknesses are en
tirely my doing, I want to thank Ziding Feng, Les Irwig, Larry Kessler, Charles
Metz, Carolyn Rutter, David Shapiro, Jeremy Taylor, Gerald van Belle, Stephen
Walter, Noel Weiss, Stuart Baker and Ray Carroll for providing me with valuable
feedback.
I am grateful for the educational and professional opportunities afforded me
PREFACE vii
that provided me with the background to write this book. I cannot thank all of
my teachers and mentors by name here, but I do want to thank Ross Prentice in
particular, my boss at the Fred Hutchinson Cancer Research Center. His support
at the early stages of my career was very important. I thank my parents, of course.
Their personal support remains as important as ever.
Finally, I want to thank my husband Mike who, while extremely supportive
of this endeavor, has always helped me maintain it in a healthy perspective. His
love, his companionship, his sense of fun and his educated interest in the book
lead me to dedicate it to him.
University of Washington and M. S. P.
Fred Hutchinson Cancer Research Center
July 2002
The author is donating royalties from the sale of this book to the charity
Doctors Without Borders
This page intentionally left blank
CONTENTS
Notation xv
1 Introduction 1
1.1 The medical test 1
1.1.1 Tests, classification and the broader context 1
1.1.2 Disease screening versus diagnosis 2
1.1.3 Criteria for a useful medical test 2
1.2 Elements of study design 3
1.2.1 Scale for the test result 4
1.2.2 Selection of study subjects 4
1.2.3 Comparing tests 5
1.2.4 Test integrity 5
1.2.5 Sources of bias 6
1.3 Examples and datasets 8
1.3.1 Overview 8
1.3.2 The CASS dataset 8
1.3.3 Pancreatic cancer serum biomarkers study 10
1.3.4 Hepatitis metastasis ultrasound study 10
1.3.5 CARET PSA biomarker study 10
1.3.6 Ovarian cancer gene expression study 11
1.3.7 Neonatal audiology data 11
1.3.8 St Louis prostate cancer screening study 11
1.4 Topics and organization 11
1.5 Exercises 12
2 Measures of accuracy for binary tests 14
2.1 Measures of accuracy 14
2.1.1 Notation 14
2.1.2 Disease-specific classification probabilities 14
2.1.3 Predictive values 16
2.1.4 Diagnostic likelihood ratios 17
2.2 Estimating accuracy with data 21
2.2.1 Data from a cohort study 21
2.2.2 Proportions: (FPF, TPF) and (PPV, NPV) 22
2.2.3 Ratios of proportions: DLRs 24
2.2.4 Estimation from a case-control study 25
2.2.5 Merits of case-control versus cohort studies 26
2.3 Quantifying the relative accuracy of tests 27
2.3.1 Comparing classification probabilities 28
2.3.2 Comparing predictive values 29
ix
