Table Of ContentSpringer Series in Statistics
Advisors:
P. Bickel, P. Diggle, S. Fienberg, U. Gather,
I. Olkin, S. Zeger
Phillip Good
Permutation, Parametric
and Bootstrap Tests
of Hypotheses
Third Edition
With 22 Illustrations
PhillipGood
205W.UticaAvenue
HuntingtonBeach,CA92648
USA
[email protected]
LibraryofCongressCataloging-in-PublicationData
Good,PhillipI.
Permutation,ParametricandBootstrapTestsofHypotheses
PhillipGood.—3rded.
p.cm.—(Springerseriesinstatistics)
Includesbibliographicalreferencesandindex.
ISBN0-387-20279-X(hardcover:alk.paper)
1. Statisticalhypothesistesting. 2. Resampling(Statistics) I. Title. II. Series.
QA277.G6432004
519.5′6—dc22 2004050436
ISBN0-387-20279-X Printedonacid-freepaper.
©1994,2000,2005SpringerScience+BusinessMedia,Inc.
All rights reserved. This work may not be translated or copied in whole or in part without the
writtenpermission ofthepublisher (SpringerScience+BusinessMedia, Inc.,233Spring St.,New
York,NY10013,USA),exceptforbriefexcerptsinconnectionwithreviewsorscholarlyanalysis.
Useinconnectionwithanyformofinformationstorageandretrieval,electronicadaptation,com-
putersoftware,orbysimilarordissimilarmethodologynowknownorhereafterdevelopedisfor-
bidden.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if
theyarenotidentifiedassuch,isnottobetakenasanexpressionofopinionastowhetherornot
theyaresubjecttoproprietaryrights.
PrintedintheUnitedStatesofAmerica. (MVY)
9 8 7 6 5 4 3 2 1 SPIN10952368
springeronline.com
Preface to the Third Edition
This text is intended to provide a strong theoretical background in testing
hypotheses and decision theory for those who will be practicing in the real
worldorwhowillbeparticipatinginthetrainingofreal-worldstatisticiansand
biostatisticians. In previous editions of this text, my rhetoric was somewhat
tentative. I was saying, in effect, “Gee guys, permutation methods provide
a practical real-world alternative to asymptotic parametric approximations.
Why not give them a try?” But today, the theory, the software, and the
hardware have come together. Distribution-free permutation procedures are
the primary method for testing hypotheses. Parametric procedures and the
bootstrap are to be reserved for the few situations in which they may be
applicable. Four factors have forced this change:
1. Desire by workers in applied fields to use the most powerful statistic for
their applications. Such workers may not be aware of the fundamental
lemma of Neyman and Pearson, but they know that the statistic they
wanttouse—acomplexscoreoraratioofscores,doesnothaveanalready
well-tabulated distribution.
2. Pressure from regulatory agencies for the use of methods that yield exact
significance levels, not approximations.
3. Agrowingrecognitionthatmostreal-worlddataaredrawnfrommixtures
of populations.
4. A growing recognition that missing data is inevitable, balanced designs
the exception.
Thus, it seems natural that the theory of testing hypothesis and the more
general decision theory in which it is embedded should be introduced via
the permutation tests. On the other hand, certain relatively robust paramet-
ric tests such as Student’s t continue to play an essential role in statistical
practice.
As the present edition is intended to replace rather than supplement exist-
ing graduate level texts on testing hypotheses and decision theory, it includes
“prelims” — 2004/10/12 — page v — #5
vi Preface to the Third Edition
materialonparametricmethodsaswellasthepermutationtestsandtheboot-
strap.Therevisedandexpandedtextincludesmanymorereal-worldillustra-
tions from economics, geology, law, and clinical trials. Also included in this
new edition are a chapter on multifactor designs expounding on the theory of
synchronouspermutationsdevelopedbyFortunatoPesarinandhiscolleagues
and sections on sequential analysis and adaptive treatment allocation.
Algebra and an understanding of discrete probability will take the reader
throughallbuttheappendix,whichutilizesprobabilitymeasuresinitsproofs.
A one-semester graduate course would take the student through Chapters
1–3 and any portions of Chapters 4, 5, and Appendix that seem appropriate.
The second semester would take the student through Chapters 6 and 7, and
whatever portions of the remaining chapters seem germane to the instructor
and students’ interests.
An appendix utilizing measure theory has been provided for the benefit of
the reader and instructor who may wish to have a mathematically rigorous
foundation for the theory of testing hypotheses for continuous as well as dis-
crete variables. For example, Section 2 of the appendix extends the proof of
the Fundamental Lemma in Chapter 3 to the continuous case.
Thenumberofexerciseshasbeengreatlyincreasedfrompreviouseditions.
Exercisesrangefromtheconcept-driven,designedtodevelopthestudent’ssta-
tisticalintuitioninpracticalsettings,tothehighlymathematical.Instructors
arefreetopickandchooseinaccordancewiththemathematicalandpractical
sophistication of their classes and the objectives of their courses.
To ensure greater comprehension of fundamental concepts, many essen-
tial results are now presented in the form of exercises. Although the primary
motivation for this change came from instructors, feedback from the autodi-
dacthaspersuadedusthatfullunderstandingcanonlybegainedfromactual
usage.
Hopefully,thiseditionreflectsthelessonsI’velearnedfromaseriesofinter-
activeon-linecoursesofferedthroughstatistics.com.Immediatefeedbackfrom
mystudentshasforcedmetorevisethetextagainandagain.ThelateJoseph
Hodges once said, “The ideal mathematics lecture would be entirely free of
symbols.” The current text avoids symbols to the degree I am capable of
doingso,theiroccasionaluseaweakness.Asymptoticresultsareavoided;the
emphasis in this strongly theoretical work is on the practical.
If you find portions of this text particularly easy to understand the credit
goestoCliffLunneborgforhisinsightfulreviewoftheentiretext,toFortunato
PesarinforhismanycontributionstoChapter9,andtoNormanMarshallfor
his comments on Chapter 8.
September 2004 Phillip Good
Huntington Beach, California
brother [email protected]
“prelims” — 2004/10/12 — page vi — #6
Preface to the Second Edition
In 1982, I published several issues of a samdizat scholarly journal called Ran-
domization with the aid of an 8-bit, 1-MH personal computer with 48K
of memory (upgraded to 64K later that year) and floppy disks that held
400Kbytes. A decade later, working on the first edition of this text, I used a
16-bit, 33-MH computer with 1Mb of memory and a 20-Mb hard disk. This
preface to the second edition comes to you via a 32-bit, 300-MH computer
with 64-Mb memory and a 4-Gb hard disk. And, yes, I paid a tenth of what
I paid for my first computer.
This relationship between low-cost readily available computing power and
the rising popularity of permutation tests is no coincidence. Simply put, it is
faster today to compute an exact p-value than to look up an approximation
in a table of the not-quite-appropriate statistic. As a result, more and more
researchers are using Permutation Tests to analyze their data.
Of course, some of the increased usage has also come about through the
increased availability of and improvements in off-the-shelf software, as can be
seen in the revisions in this edition to Chapter 12 (Publishing Your Results)
and Chapter 13 (Increasing Computation Efficiency).
These improvements helped persuade me it was the time to publish a first
course in statistics based entirely on resampling methods (an idea first pro-
posedbythelateF.N.David).Asaresult,PermutationTests hasbecometwo
texts: one, Resampling Methods, designed as a first course, and this second
editionaimedatupperdivisiongraduatestudentsandpractitionerswhomay
already be familiar with the application of other statistical procedures. The
popularquestionsectionattheendofeachchapternowcontainsanumberof
thesis-levelquestions,whichmayormaynotbesolvableintheirpresentform.
While the wide applicability of permutation tests continues to be emphasized
here, their limitations are also revealed. Examples include expanded sections
oncomparingvariances(Chapter3,TestingHypotheses),testinginteractions
inbalanceddesigns(Chapter4,ExperimentalDesign),andmultipleregression
(Chapter 7, Dependence).
“prelims” — 2004/10/12 — page vii — #7
viii Preface to the Second Edition
Sections on sequential analysis (Chapter 4) and comparing spatial distri-
butions (Chapter 8) are also new. Recent major advances in the analysis of
multiple dependent tests are recorded in Chapter 5 on multivariate analysis.
Mythankstothemanyindividualswhopreviewedchaptersforthisedition,
including,inalphabeticalorder,BrianCade,MikeErnst,BarbaraHeller,John
Kimmel, Patrick Onghena, Fortunato Pesarin, and John Thaden.
April 2000 Phillip Good
Huntington Beach, California
“prelims” — 2004/10/12 — page viii — #8
Preface to the First Edition
Permutation tests permit us to choose the test statistic best suited to the
task at hand. This freedom of choice opens up a thousand practical applica-
tions, including many which are beyond the reach of conventional paramet-
ric statistics. Flexible, robust in the face of missing data and violations of
assumptions, the permutation test is among the most powerful of statistical
procedures. Through sample size reduction, permutation tests can reduce the
costs of experiments and surveys.
This text on the application of permutation tests in biology, medicine,
science, and engineering may be used as a step-by-step self-guiding reference
manual by research workers and as an intermediate text for undergraduates
and graduates in statistics and the applied sciences with a first course in
statistics and probability under their belts.
Research workers in the applied sciences are advised to read through
Chapters 1 and 2 once quickly before proceeding to Chapters 3 through 8,
whichcovertheprincipalapplicationstheyarelikelytoencounterinpractice.
Chapter 9 is a must for the practitioner, with advice for coping with real-
life emergencies such as missing or censored data, after-the-fact covariates,
and outliers.
Chapter 10 uses practical applications in archeology, biology, climatology,
education, and social science to show the research worker how to develop
new permutation statistics to meet the needs of specific applications. The
practitioner will find Chapter 10 a source of inspiration as well as a practical
guide to the development of new and novel statistics.
The expert system in Chapter 11 will guide you to the correct statistic for
your application. Chapter 12, more “must” reading, provides practical advice
onexperimentaldesignandshowshowtodocumenttheresultsofpermutation
tests for publication.
Chapter13describestechniquesforreducingcomputationtime;aguideto
off-the-shelf statistical software is provided in an appendix.
“prelims” — 2004/10/12 — page ix — #9
x Preface to the First Edition
The sequence of recommended readings is somewhat different for the stu-
dentandwilldependonwhetherheorsheisstudyingthepermutationtestsby
themselvesor aspart of alarger courseon resamplingmethodsencompassing
both the permutation test and the bootstrap resampling method.
This book can replace a senior-level text on testing hypotheses. I have
also found it of value in introducing students who are primarily mathemati-
cianstotheapplicationswhichmakestatisticsauniquemathematicalscience.
Chapters 1, 2, and 14 provide a comprehensive introduction to the theory.
Despiteitsplacementinthelatterpartofthetext,Chapter14,onthetheory
of permutation tests, is self-standing. Chapter 3 on applications also deserves
a careful reading. Here in detail are the basic testing situations and the basic
tests to be applied to them. Chapters 4, 5, and 6 may be used to supple-
ment Chapter 3, time permitting (the first part of Chapter 6 describing the
Fisher exact test is a must). Rather than skipping from section to section,
it might be best for the student to consider one of these latter chapters in
depth—supplementing his or her study with original research articles.
My own preference is to parallel discussions of permutation methods with
discussion of a second resampling method, the bootstrap. Again, Chapters 1,
2, and 3—supplemented with portions of Chapter 14—are musts. Chapter 7,
on tests of dependence, is a natural sequel. Students in statistical computing
also are asked to program and test at least one of the advanced algorithms in
Chapter 12.
Forthereader’sconveniencethebibliographyisdividedintofourparts:the
first consists of 34 seminal articles; the second of two dozen background arti-
cles referred to in the text that are not directly concerned with permutation
methods; the third of 111 articles on increasing computational efficiency; and
a fourth, principal bibliography of 574 articles and books on the theory and
application of permutation techniques.
Exercises are included at the end of each chapter to enhance and reinforce
yourunderstanding.Butthebestexerciseofallistosubstituteyourowndata
for the examples in the text.
My thanks to Symantek, TSSI, and Perceptronics without whose Grand-
View(cid:1)r outliner,Exact(cid:1)r equationgenerator,andEinsteinWriter(cid:1)r wordpro-
cessor this text would not have been possible.
I am deeply indebted to Mike Chernick for our frequent conversations and
hismanyinvaluableinsights,toMikeErnst,AlanForsythe,KarimHiriji,John
Ludbrook,RezaModarres,andWilliamSchucanyforreadingandcommenting
on portions of this compuscript and to my instructors at Berkeley, including
E. Fix, J. Hodges, E. Lehmann, and J. Neyman.
February 1994 Phillip Good
Huntington Beach, California
“prelims” — 2004/10/12 — page x — #10
Contents
Prefaceto the Third Edition................................... v
Prefaceto the Second Edition ................................. vii
Prefaceto the First Edition.................................... ix
1 A Wide Range of Applications............................. 1
1.1 Basic Concepts........................................ 1
1.1.1 Stochastic Phenomena ......................... 1
1.1.2 Distribution Functions ......................... 3
1.1.3 Hypotheses................................... 3
1.2 Applications .......................................... 5
1.3 Testing a Hypothesis................................... 7
1.3.1 Five Steps to a Test ........................... 8
1.3.2 Analyze the Experiment ....................... 8
1.3.3 Choose a Test Statistic ........................ 8
1.3.4 Compute the Test Statistic ..................... 9
1.3.5 Determine the Frequency Distribution of
the Test Statistic.............................. 9
1.3.6 Make a Decision .............................. 10
1.3.7 Variations on a Theme......................... 10
1.4 A Brief History of Statistics in Decision-Making........... 10
1.5 Exercises ............................................. 12
2 Optimal Procedures ....................................... 13
2.1 Defining Optimal...................................... 13
2.1.1 Trustworthy .................................. 13
2.1.2 Two Types of Error ........................... 14
2.1.3 Losses and Risk............................... 16
2.1.4 Significance Level and Power ................... 17
“prelims” — 2004/10/12 — page xi — #11
