How Do We Know If a Program Made a Difference?

346 Pages·2014·9.39 MB·English

How Do We Know If a Program
Made a Difference?
A Guide to Statistical Methods for Program
Impact Evaluation
Peter M. Lance • David K. Guilkey • Aiko Hattori • Gustavo Angeles

How do we know if a program made a
difference?
A guide to statistical methods for program
impact evaluation
Peter M. Lance
David K. Guilkey
Aiko Hattori
Gustavo Angeles
Suggested citation:
Lance, P., D. Guilkey, A. Hattori and G. Angeles. (2014). How do we know if a program made
a difference? A guide to statistical methods for program impact evaluation. Chapel Hill, North
Carolina: MEASURE Evaluation.
ISBN: 978-0-692-23861-5
Prepared by MEASURE Evaluation, University of North Carolina at Chapel Hill.
MEASURE Evaluation is funded through the U.S. Agency for International Development (USAID)
under the terms of cooperative agreement GHA-A-00-08-00003-00, which is implemented by the
CarolinaPopulationCenterattheUniversityofNorthCarolinaatChapelHill,withFuturesGroup,
ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University. The
views expressed in this publication do not necessarily reflect the views of USAID or the United
States government. MS-14-87 (May 2014).
Cover image File:Snow-cholera-map-1.jpg retrieved from Wikimedia Commons is licensed under
PD-old-70.
i
Asking for Forgiveness
The Universe is change. Life is opinion.
-Marcus Aurelius
There is a long-running, vigorous and evolving methodological debate about the appropriate or
optimal way to evaluate the impact of a program. This manual strives not to convince readers
of the merits of one particular alternative or another, but instead simply to present the various
options in as approachable a fashion as possible and then let them decide for themselves where
they stand. In other words, it strives to be impartial. However, that is probably an impossible
standard. We cannot help but have our own views, which probably subtly intrude on the manual.
To those who disagree with us we ask for their pardon but remind them that this shortcoming was
probably inevitable.
Writing this manual has required review of a large number of contributions to the literature on
impact evaluation methods, as well as many actual impact evaluations. Many of these are cited in
this manual, which has now been through hundreds of revisions. In the course of those revisions,
with the attendant re-arrangement, deletion and insertion, it is possible that some citations were
accidentally dropped and others have become un-moored from the train of thought for which they
were originally introduced. We offer unqualified apology for all such instances.
That said, updated versions of this manual will be released from time to time. We invite those
with comments or criticism of the substance of the manual to submit their thoughts so that they
can inform future versions. Errors and omissions in citations that are communicated to us will be
corrected immediately.
ii
Contents
Acknowledgements vii
Prologue ix
1 Introduction 1
2 The Program Impact Evaluation Challenge 5
2.1 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 The Estimation Challenge: Basic Ideas . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 The Estimation Challenges: Some Common Estimators . . . . . . . . . . . . . . . . 16
2.3.1 Simple Comparison of Mean Outcomes Y Between Participants and Non-
Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.3 Some Specific Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4 Other Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.1 Causality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.4.2 Representative Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.3 Observer and Hawthorne Effects . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.4.4 Impact Evaluation Based on Pilot or Trial Programs . . . . . . . . . . . . . . 35
2.4.5 Other programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4.6 The Challenge of Defining the Counterfactual . . . . . . . . . . . . . . . . . . 37
2.4.7 Levels of Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3 Randomization 39
3.1 Randomization: The Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.2 Experimental Evaluations: Some Specific Examples . . . . . . . . . . . . . . . . . . . 42
3.2.1 John Snow and the Causes of Cholera . . . . . . . . . . . . . . . . . . . . . . 42
3.2.2 The RAND and Oregon Health Insurance Experiments . . . . . . . . . . . . 46
3.2.3 PROGRESA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.2.4 The Work and Iron Status Evaluation (WISE) . . . . . . . . . . . . . . . . . 50
3.2.5 Poverty Action Lab Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.2.6 Methodological Benchmark Studies . . . . . . . . . . . . . . . . . . . . . . . 52
3.2.7 Negative Income Tax Experiments . . . . . . . . . . . . . . . . . . . . . . . . 53
3.2.8 Unintended Random Experiments . . . . . . . . . . . . . . . . . . . . . . . . 54
3.3 The Case for Randomization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.3.1 Quasi-Experimental Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.3.2 LaLonde’s Critique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.4 Randomization and Its Discontents . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
iii
iv CONTENTS
3.5 Estimation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.6 Some Closing Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4 Selection on Observables 67
4.1 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.1.1 Regression Basics: The Simple Case . . . . . . . . . . . . . . . . . . . . . . . 69
4.1.2 Multiple Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.1.3 “Bad” Controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.1.4 The Program Participation Decision . . . . . . . . . . . . . . . . . . . . . . . 108
4.1.5 Standard Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.1.6 Regression and Randomized Data . . . . . . . . . . . . . . . . . . . . . . . . 121
4.2 Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
4.2.1 Matching: The Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.2.2 Regression as Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.2.3 The Failure of Common Support . . . . . . . . . . . . . . . . . . . . . . . . . 135
4.2.4 The Propensity Score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.2.5 Other Propensity Score Strategies . . . . . . . . . . . . . . . . . . . . . . . . 141
4.2.6 An Empirical Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
4.3 Some Closing Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
5 Within Estimators 149
5.1 Classic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
5.1.1 The First Differences Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . 149
5.1.2 Linear Fixed Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
5.1.3 Measurement Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
5.1.4 Nonlinear Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
5.1.5 Hausman-Type Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
5.2 The Difference-in-Differences Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
5.2.1 The Basic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
5.2.2 Extensions and Complications . . . . . . . . . . . . . . . . . . . . . . . . . . 196
5.2.3 Experimental Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
5.3 Some Closing Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
6 Instrumental Variables 202
6.1 Instrumental Variables Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
6.1.1 The Classic Linear Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
6.1.2 Limited Dependent Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
6.1.3 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
6.1.4 Natural and Purposeful Social Experiments . . . . . . . . . . . . . . . . . . . 281
6.2 Local Average Treatment Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
6.3 Regression Discontinuity Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
6.4 Some Closing Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
7 Final Thoughts 315
7.1 A Brief Recap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
7.1.1 Randomized Control Trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
7.1.2 Selection on Observables Models . . . . . . . . . . . . . . . . . . . . . . . . . 316
7.1.3 Within Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
CONTENTS v
7.1.4 Instrumental Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
7.2 The Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
7.3 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
7.4 Impact Evaluation Meets Philosophy . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
References 320
vi CONTENTS