Table Of ContentRunning Randomized Evaluations
Running Randomized Evaluations
A PRACTICAL GUIDE
RACHEL KUDZAI
GLENNERSTER and TAKAVARASHA
PRINCETON UNIVERSITY PRESS
Princeton and Oxford
Copyright © 2013 by Princeton University Press
Published by Princeton University Press, 41 William Street, Princeton, New Jersey 08540
In the United Kingdom: Princeton University Press, 6 Oxford Street, Woodstock, Oxfordshire
OX20 1TW
press.princeton.edu
Cover design by Leah E. Horgan
All Rights Reserved
Library of Congress Cataloging-in-Publication Data
Glennerster, Rachel.
Running randomized evaluations : a practical guide / Rachel Glennerster
and Kudzai Takavarasha.
pages cm
Includes bibliographical references and index.
ISBN 978-0-691-15924-9 (hardcover : alk. paper) — ISBN 978-0-691-15927-0
(pbk. : alk. paper)
1. Evaluation research (Social action programs) 2. Social sciences—Research.
I. Takavarasha, Kudzai, 1973– II. Title.
H61.G5544 2013
001.4′34—dc23 2013014882
British Library Cataloging-in-Publication Data is available
Th is book has been composed in Minion Pro with ITC Franklin Gothic Display by
Princeton Editorial Associates Inc., Scottsdale, Arizona
Printed on acid-free paper. ∞
Printed in the United States of America
10 9 8 7 6 5 4 3 2 1
CONTENTS
Preface vii
Abbreviations and Acronyms ix
1 The Experimental Approach 1
2 Why Randomize? 24
3 Asking the Right Questions 66
4 Randomizing 98
5 Outcomes and Instruments 180
6 Statistical Power 241
7 Threats 298
8 Analysis 324
9 Drawing Policy Lessons 386
Appendix 421
Glossary 443
Index 453
PREFACE
Over the past 10 years, there has been a dramatic increase in the quan-
tity and quality of rigorous impact evaluations that have tested the ef-
fectiveness of specifi c approaches to reducing poverty from Britain to
Burkina Faso. Randomized impact evaluations have enjoyed a partic-
ularly dramatic increase in popularity. Twenty years ago, few random-
ized evaluations were carried out by governments (outside the United
States), international agencies, or nongovernmental organizations—
except as a way to test the eff ectiveness of medicines or vaccines. Today
hundreds of randomized evaluations are being undertaken by all sorts
of diff erent organizations and on virtually every subject of impor-
tance to policy, including education, health, governance, the environ-
ment, and empowerment.
Th ese randomized impact evaluations are improving our under-
standing of the fundamental processes that underlie the persistence of
poverty and the pathways out of it. Each evaluation answers some
questions and raises new ones, the next building on the last, succes-
sively adding to our knowledge of what works, knowledge we need to
design better programs and more eff ectively fi ght poverty.
We have written this book for those who want to help generate a
rigorous evidence base on ways to reduce poverty wherever it exists in
the world by running (or taking part in) rigorous impact evaluations.
Running a randomized evaluation requires making hundreds of prac-
tical decisions and constant trade-off s, especially in resource-poor set-
tings where the best-laid plans can be quickly overtaken by events.
Much of this book draws on the practical experience of the Abdul
Latif Jameel Poverty Action Lab (J-PAL), a research network based at
vii
the Massachusetts Institute of Technology with affi liated research cen-
ters and researchers at universities around the world.
Th e camaraderie within J-PAL means that every publication, con-
ference, or talk has many contributors, including those who never
attract the limelight. So many people have provided input and advice
on this book that it is almost disingenuous to isolate just two authors.
Pascaline Dupas and Michael Kremer have answered hundreds of our
questions and commented on numerous draft s over the several years
it has taken to write this book. Th ey are also responsible for getting us
into the randomization business in the fi rst place. Marc Shotland gave
valuable input and advice on virtually every chapter and is the creator
of the online tools that accompany this book. Alison Cappellieri cre-
ated the appendix, glossary, and bibliography and copyedited the
entire book. Claire Walsh fact checked all mentions of existing stud-
ies. Leah Horgan designed the fi gures and the cover, and John Floretta
gave detailed input on Chapter 5. Ben Feigenberg and Ben Marx
checked the technical precision of our nontechnical language, espe-
cially in Chapters 4, 6, and 8, and Ben Marx generated all the power
graphs. Catlin Tulloch and Anna Yalouris helped with the examples on
cost-eff ectiveness. Michael Kremer and Ben Olken patiently helped us
work through some of the tougher econometric issues. We are deeply
grateful for the detailed comments of many anonymous referees and
are particularly grateful to Guido Imbens and Jörn-Steff en Pischke for
their very thorough and thoughtful reviews, which led to a much im-
proved book. Finally, without Mary Ann Bates this book would never
have seen the light of day. Her fresh eyes, good judgment, creative re-
writes, and merciless blue pencil whipped the book into shape.
None of the work described in this book would have been possible
without the dedication of all the implementing organizations that
opened their programs up to evaluation. Th ey spent countless hours
discussing what should be tested and how, fi guring out how to bring
an element of randomization into their programs, and advising on
how diff erent outcomes could be measured. Th ey also demonstrated
considerable courage, knowing that an evaluation might fi nd that
their programs did not work. Th e nongovernmental organizations and
their leaders who were among the pioneers of this work in developing
countries deserve special mention: Chip Bury at International Child
Support Africa, Neelima Khetan at Seva Mandir, and Madav Chavan
and Rukmini Banerji at Pratham.
viii / PREFACE
ABBREVIATIONS AND ACRONYMS
3ie International Initiative for Impact Evaluation
AHTS Agricultural Household Tracking Survey
BMI body mass index
CCT conditional cash transfer
CDD community-driven development
CDF cumulative distribution function
DIME Development Impact Evaluation
ETP Extra Teacher Program
FICA Federal Insurance Contributions Act
FMB First Macro Bank
FWER family-wise error rate
IAT implicit association test
ICS International Child Support
IID independent and identically distributed
IRB institutional review board
ITN insecticide-treated (bed) net
ITT intention-to-treat
J-PAL Abdul Latif Jameel Poverty Action Lab
LLIN long-lasting insecticide-treated net
MDE minimum detectable eff ect
MFI microfi nance institutions
MIT Massachusetts Institute of Technology
NERICA New Rice for Africa
NGO nongovernmental organization
NONIE Network of Networks on Impact Evaluation
ix
