Table Of ContentWhat Counts as
Credible Evidence
in APPLIED
RESEARCH
and
EVALUATION
?
PRACTICE
Tothoseengagedinthestruggletohelpcreateabetterfuture,
includingthosewhoseeffortsinvolvetheuseofappliedresearch
andevaluation
What Counts as
Credible Evidence
in APPLIED
RESEARCH
and
EVALUATION
?
PRACTICE
Stewart I. Donaldson
Claremont Graduate University
Christina A. Christie
Claremont Graduate University
Melvin M. Mark
The Pennsylvania State University
Editors
Copyright©2009bySagePublications,Inc.
Allrightsreserved.Nopartofthisbookmaybereproducedorutilizedinanyform
orbyanymeans,electronicormechanical,includingphotocopying,recording,orby
any information storage and retrieval system, without permission in writing from
thepublisher.
Forinformation:
SAGEPublications,Inc. SAGEPublicationsIndiaPvt.Ltd.
2455TellerRoad B1/I1MohanCooperative
ThousandOaks, IndustrialArea
California91320 MathuraRoad,NewDelhi110044
E-mail:[email protected] India
SAGEPublicationsLtd. SAGEPublications
1Oliver’sYard Asia-PacificPte.Ltd.
55CityRoad 33PekinStreet#02-01
LondonEC1Y1SP FarEastSquare
UnitedKingdom Singapore048763
PrintedintheUnitedStatesofAmerica
LibraryofCongressCataloging-in-PublicationData
Whatcountsascredibleevidenceinappliedresearchandevaluationpractice?/
StewartI.Donaldson,ChristinaA.Christie,[and]MelvinM.Mark,editors.
p.cm.
Includesbibliographicalreferencesandindex.
ISBN978-1-4129-5707-6(pbk.:acid-freepaper)
1. Research—Evaluation. 2. Research—Socialaspects. 3. Socialsciences—
Methodology. 4. Evaluationresearch(Socialactionprograms) 5. Educational
accountability. I. Donaldson,StewartI.(StewartIan) II. Christie,ChristinaA.
III. Mark,MelvinM.
Q180.55.E9W532009
001.4′2—dc22 2008017667
Thisbookisprintedonacid-freepaper.
08 09 10 11 12 10 9 8 7 6 5 4 3 2 1
AcquisitionsEditor: VickiKnight
EditorialAssistant: LaurenHabib
ProductionEditor: CatherineM.Chilton
CopyEditor: TeresaHerlinger
Typesetter: C&MDigitals(P)Ltd.
Proofreader: AnnetteR.VanDeusen
Indexer: KathleenPaparchontis
CoverDesigner: CandiceHarman
MarketingManager: StephanieAdams
Contents
Preface vii
Acknowledgments viii
About the Editors x
About the Contributors xii
Part I. Introduction 1
1. In Search of the Blueprint for an Evidence-Based
Global Society 2
Stewart I. Donaldson
2. Social Inquiry Paradigms as a Frame for the
Debate on Credible Evidence 19
Christina A. Christie and Dreolin Fleischer
Part II. Experimental Approaches as the
Route to Credible Evidence 31
3. When Getting It Right Matters: The Case for
High-Quality Policy and Program Impact Evaluations 32
Gary T. Henry
4. Randomized Controlled Trials: A Gold Standard
With Feet of Clay? 51
Leonard Bickman and Stephanie M. Reich
5. What Is Credible Evidence in Education? The Role of
the What Works Clearinghouse in Informing the Process 78
Russell Gersten and John Hitchcock
6. Evaluation Methods for Producing Actionable Evidence:
Contextual Influences on Adequacy and Appropriateness
of Method Choice 96
George Julnes and Debra Rog
Part III. Nonexperimental Approaches
for Building Credible Evidence 133
7. Demythologizing Causation and Evidence 134
Michael Scriven
8. Evidence as “Proof” and Evidence as “Inkling” 153
Jennifer C. Greene
9. Reasoning With Rigor and Probity: Ethical Premises
for Credible Evidence 168
Sharon F. Rallis
10. Seeing Is Believing: The Credibility of Image-Based
Research and Evaluation 181
Sandra Mathison
11. Toward a Practical Theory of Evidence for Evaluation 197
Thomas A. Schwandt
Part IV. Conclusions 213
12. Credible Evidence: Changing the Terms of the Debate 214
Melvin M. Mark
Epilogue: A Practitioner’s Guide for Gathering Credible
Evidence in the Evidenced-Based Global Society 239
Stewart I. Donaldson
Author Index 252
Subject Index 258
Preface
This volume addresses one of the most important and contentious issues
challengingappliedresearchandevaluationpracticetoday:whatconsti-
tutescredibleevidence.Wefearthathistoryhasbeenrepeatingitselfoflate,
with the earlier quantitative–qualitative paradigm war appearing again in
varied guises. These include public debates and serious professional argu-
mentsaboutfederalresearchandevaluationdesignpriorities,aswellasstate-
mentsfromseveralprofessionalassociationsregardingtheroleofrandomized
controlled trials (RCTs) and the existence of “gold standards” for evidence.
Muchhumancapitalandmorethanonefriendshiphavebeenlostinheated
disagreements about what counts as credible and actionable evidence. Lost
too,wefear,havebeenpotentiallyproductiveconversationsaboutthevaried
ways in which applied research and evaluation can contribute to action and
understandingsthatcanassistinaddressingimportantsocialneeds.
Onegoalofthisbookwastoaddressthesethornyissuesinamoreposi-
tive and productive manner. We hope that collectively the chapters in this
book provide more light than heat regarding the fundamental challenge
of providing credible evidence, a challenge that faces contemporary applied
researchersandevaluatorsinprofessionalpracticeonaday-to-daybasis.
Inanefforttoaccomplishthisgoal,adiverseandinternationallyrenowned
cast of applied researchers and evaluators were invited to participate in the
ClaremontGraduateUniversityStaufferSymposiumfocusedonthequestion
of “What Counts as Credible Evidence in Applied Research and Evaluation
Practice?” This illuminating and action-packed day in Claremont was expe-
riencedbymorethan200participantsfromawidevarietyofbackgrounds—
evaluators,researchers,privateconsultants,students,faculty,andprofessionals
frommanyfields.Thepresentersallsharedtheirlatestthinkingabouthowto
answerthequestion,andlatereachpreparedamoredetailedchaptertohelp
illuminate from their vantage point the key challenges about the nature of
evidenceinappliedresearchandevaluationpractice.
vii
viii——WhatCountsasCredibleEvidence?
The result of these efforts is this volume. It explores a broad array of
issues that address the fundamental challenges of designing and executing
high-quality applied research and evaluation projects. Few works have
attempted to sort out these issues in a way that directly informs contempo-
raryappliedresearch,evaluation,andevidence-basedpractice.Wehopethat
readers will garner a new and clear understanding of the philosophical,
theoretical, methodological, political, ethical, and pragmatic dimensions of
gathering credible and actionable evidence to answer fundamental research
and evaluation questions across diverse disciplinary boundaries and real-
worldcontexts.Amongthekeyfeaturesofthisvolumearethatit
Offers authoritative statements about what counts as credible evidence in applied
researchandevaluation,providedbyleadingscholarsacrosssuchfieldsaseducation,
psychology,healthandhumanservices,publicpolicy,andpublicadministration
Covers experimental and nonexperimental methods, with experts in their use
criticallyappraisingthecredibilityofevidencefromthesevariedapproaches
Provides summaries of strengths and weaknesses across the varied approaches
togatheringcredibleevidence
Containsdiversedefinitionsofevidence,sothatstudentsandotherreaderscan
betterunderstandandevaluatethelandscapeofthishighlydebatedtopic
Includesafullchapteronthepracticalimplicationsofthediscussionsabouthow
best to gather credible evidence in modern applied research, evaluation, and
evidence-basedpractice
Insum,then,wehopethisvolumewillbecomeavaluableresourceforprac-
ticingevaluators;appliedresearchers;studentspreparingforappliedresearch
and evaluation careers; scholars and teachers of evaluation and applied
researchmethods;andotherprofessionalsinterestedinhowbesttostudyand
evaluate programs, policies, organizations, and other initiatives designed to
improvesomeaspectofthehumanconditionandsocietalwell-being.
Acknowledgments
Theeditorsofthisvolumewouldliketoexpresstheirdeepgratitudetoour
editorialassistant,ShabnamOzlati,forherwonderfuldedicationandexcel-
lent work on the production of this manuscript. Special thanks to the coor-
dinator of the Claremont Graduate University Stauffer Symposium, Paul
Thomas, and his fantastic group of graduate student volunteers for making
the Stauffer Symposium a smashing success. We are very thankful to the
Preface——ix
chapter authors for sharing with us their best thinking and advice about
whatcountsascredibleevidence,andforgoingtheextramiletomaketheir
chaptersengagingandwidelyaccessible.Wesincerelyappreciatethehelpful
reviews and suggestions for improvement provided by the reviewers, Larry
A. Braskamp, Loyola University, Chicago; John C. Ory, University of
Illinois,MelvinHall,NorthernArizonaUniversity;MichaelSchooley,Centers
for Disease Control and Prevention; and the Sage editorial staff: Vicki
Knight,SeniorAcquisitionsEditor;SeanConnelly,AssociateEditor;Lauren
Habib, Editorial Assistant; Stephanie Adams, Marketing Manager;
CatherineChilton,ProductionEditor;TeresaHerlinger,CopyEditor.
Finally,thisprojectwouldnothavebeenpossiblewithoutthegenerousgift
provided by the John Stauffer Charitable Trust to the School of Behavioral
and Organizational Sciences, Claremont Graduate University. We are abun-
dantlygratefulforthisessentialfinancialsupport.
StewartI.Donaldson
ChristinaA.Christie
Claremont,CA
MelvinM.Mark
StateCollege,PA
