The Handbook of Research Synthesis and Meta-Analysis

610 Pages·2009·5.558 MB·English

by  Harris Cooper, Larry V. Hedges, Jeffrey C. Valentine

HANDBOOK OF RESEARCH SYNTHESIS AND META-ANALYSIS
2ND EDITION

HANDBOOK
THE
RESEARCH
OF
SYNTHESIS
META-ANALYSIS
AND
2ND EDITION
Edited by
HARRIS COOPER, LARRY V. HEDGES,
AND JEFFREY C. VALENTINE
RUSSELL SAGE FOUNDATION NEW YORK
THE RUSSELL SAGE FOUNDATION
The Russell Sage Foundation, one of the oldest of America’s general purpose foundations, was
established in 1907 by Mrs. Margaret Olivia Sage for “the improvement of social and living con-
ditions in the United States.” The Foundation seeks to fulfi ll this mandate by fostering the devel-
opment and dissemination of knowledge about the country’s political, social, and economic prob-
lems. While the Foundation endeavors to assure the accuracy and objectivity of each book it
publishes, the conclusions and interpretations in Russell Sage Foundation publications are those
of the authors and not of the Foundation, its Trustees, or its staff. Publication by Russell Sage,
therefore, does not imply Foundation endorsement.
BOARD OF TRUSTEES
Mary C. Waters, Chair
Kenneth D. Brody Kathleen Hall Jamieson Nancy Rosenblum
W. Bowman Cutter, III Melvin J. Konner Shelley E. Taylor
Christopher Edley Jr. Alan B. Krueger Richard H. Thaler
John A. Ferejohn Cora B. Marrett Eric Wanner
Larry V. Hedges
Library of Congress Cataloging-in-Publication Data
The handbook of research synthesis and meta-analysis / Harris Cooper, Larry V. Hedges,
Jeffrey C. Valentine, editors. — 2nd ed.
p. cm.
Rev. ed. of: The handbook of research synthesis. c1994.
Includes bibliographical references and index.
ISBN 978-0-87154-163-5
1. Research—Methodology—Handbooks, manuals, etc. 2. Information storage and
retrieval systems—Research—Handbooks, manuals, etc. 3. Research—Statistical methods—
Handbooks, manuals, etc. I. Cooper, Harris M. II. Hedges, Larry V. III. Valentine, Jeff C.
IV. Handbook of research synthesis.
Q180.55.M4H35 2009
001.4(cid:1)2—dc22 2008046477
Copyright © 2009 by Russell Sage Foundation. All rights reserved. Printed in the United States of
America. No part of this publication may be reproduced, stored in a retrieval system, or transmit-
ted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise,
without the prior written permission of the publisher.
Reproduction by the United States Government in whole or in part is permitted for any purpose.
The paper used in this publication meets the minimum requirements of American National Stan-
dard for Information Sciences—Permanence of Paper for Printed Library Materials. ANSI
Z39.48-1992.
RUSSELL SAGE FOUNDATION
112 East 64th Street, New York, New York 10065
10 9 8 7 6 5 4 3 2 1
To the authors of the Handbook chapters,
for their remarkable cooperation and commitment
to the advance of science and society

CONTENTS
Foreword xi
About the Authors xv
PART I INTRODUCTION 1
1. Research Synthesis as a Scientifi c Process 3
Harris Cooper and Larry V. Hedges
PART II FORMULATING A PROBLEM 17
2. Hypotheses and Problems in Research Synthesis 19
Harris Cooper
3. Statistical Considerations 37
Larry V. Hedges
PART III SEARCHING THE LITERATURE 49
4. Scientifi c Communication and Literature Retrieval 51
Howard D. White
5. Using Reference Databases 73
Jeffrey G. Reed and Pam M. Baxter
6. Grey Literature 103
Hannah R. Rothstein and Sally Hopewell
PART IV CODING THE LITERATURE 127
7. Judging the Quality of Primary Research 129
Jeffrey C. Valentine
vii
viii CONTENTS
8. Identifying Interesting Variables and Analysis Opportunities 147
Mark W. Lipsey
9. Systematic Coding 159
David B. Wilson
10. Evaluating Coding Decisions 177
Robert G. Orwin and Jack L. Vevea
PART V STATISTICALLY DESCRIBING STUDY OUTCOMES 205
11. Vote-Counting Procedures in Meta-Analysis 207
Brad J. Bushman and Morgan C. Wang
12. Effect Sizes for Continuous Data 221
Michael Borenstein
13. Effect Sizes for Dichotomous Data 237
Joseph L. Fleiss and Jesse A. Berlin
PART VI STATISTICALLY COMBINING EFFECT SIZES 255
14. Combining Estimates of Effect Size 257
William R. Shadish and C. Keith Haddock
15. Analyzing Effect Sizes: Fixed-Effects Models 279
Spyros Konstantopoulos and Larry V. Hedges
16. Analyzing Effect Sizes: Random-Effects Models 295
Stephen W. Raudenbush
17. Correcting for the Distorting Effects of Study Artifacts in
Meta-Analysis 317
Frank L. Schmidt, Huy Le, and In-Sue Oh
PART VII SPECIAL STATISTICAL ISSUES AND PROBLEMS 335
18. Effect Sizes in Nested Designs 337
Larry V. Hedges
19. Stochastically Dependent Effect Sizes 357
Leon J. Gleser and Ingram Olkin
20. Model-Based Meta-Analysis 377
Betsy Jane Becker
PART VIII DATA INTERPRETATION 397
21. Handling Missing Data 399
Therese D. Pigott
22. Sensitivity Analysis and Diagnostics 417
Joel B. Greenhouse and Satish Iyengar
23. Publication Bias 435
Alex J. Sutton
CONTENTS iX
PART IX TYING RESEARCH SYNTHESIS TO SUBSTANTIVE ISSUES 453
24. Advantages of Certainty and Uncertainty 455
Wendy Wood and Alice H. Eagly
25. Research Synthesis and Public Policy 473
David S. Cordray and Paul Morphy
PART X REPORTING THE RESULTS 495
26. Visual and Narrative Interpretation 497
Geoffrey D. Borman and Jeffrey A. Grigg
27. Reporting Format 521
Mike Clarke
PART XI SUMMARY 535
28. Threats to the Validity of Generalized Inferences 537
Georg E. Matt and Thomas D. Cook
29. Potentials and Limitations 561
Harris Cooper and Larry V. Hedges
Glossary 573
Appendix A 585
Appendix B 591
Index 595