Expert political judgment: how good is it? How can we know?

340 Pages·2006·0.996 MB·English

by  Tetlock, Philip Eyrikson

Expert Political Judgment
This page intentionally left blank
Expert
Political Judgment
how good is it?
how can we know?
Philip E. Tetlock
princeton university press
princeton and oxford
Copyright © 2005 by Princeton University Press
Published by Princeton University Press, 41 William Street, Princeton,
New Jersey 08540
In the United Kingdom: Princeton University Press, 3 Market Place,
Woodstock, Oxfordshire OX20 1SY
All Rights Reserved
Sixth printing, and first paperback printing, 2006
Paperback ISBN-13: 978-0-691-12871-9
Paperback ISBN-10: 0-691-12871-5
THELIBRARYOFCONGRESSHASCATALOGEDTHECLOTHEDITIONOFTHISBOOKASFOLLOWS
Tetlock, Philip.
Expert political judgment : how good is it? how can we know? / Philip E. Tetlock.
p. cm.
Includes bibliographical references and index.
ISBN-13:978-0-691-12302-8 (alk. paper)
ISBN-10:0-691-12302-0 (alk. paper)
1. Political psychology. 2. Ideology. I. Title.
JA74.5.T38 2005
320'.01'9—dc22 2004061694
British Library Cataloging-in-Publication Data is available
This book has been composed in Sabon
Printed on acid-free paper. ∞
pup.princeton.edu
Printed in the United States of America
10 9 8 7 6
To Jenny, Paul, and Barb
This page intentionally left blank
Contents
Acknowledgments ix
Preface xi
Chapter1
Quantifying the Unquantifiable 1
Chapter2
The Ego-deflating Challenge of Radical Skepticism 25
Chapter3
Knowing the Limits of One’s Knowledge: Foxes Have
Better Calibration and Discrimination Scores than Hedgehogs 67
Chapter4
Honoring Reputational Bets: Foxes Are Better
Bayesians than Hedgehogs 121
Chapter5
Contemplating Counterfactuals: Foxes Are More Willing
than Hedgehogs to Entertain Self-subversive Scenarios 144
Chapter6
The Hedgehogs Strike Back 164
Chapter7
Are We Open-minded Enough to Acknowledge the Limits
of Open-mindedness? 189
Chapter8
Exploring the Limits on Objectivity and Accountability 216
Methodological Appendix 239
Technical Appendix
Phillip Rescober and Philip E. Tetlock 273
Index 313
This page intentionally left blank
Acknowledgments
The juryis out on just how much bad judgment I showed by undertaking
the good-judgment project. The project dates back to the year I gained
tenure and lost my generic excuse for postponing projects that I knew
were worth doing, worthier than anything I was doing back then, but also
knew would take a long time to come to fruition. As I write twenty years
later, the data are still trickling in and the project now threatens to outlast
not just my career but me. Some long-term forecasts that experts offered
will not come due until 2026. But most of the data are tabulated, some
surprising patterns have emerged, and I see no reason for delaying the
write-up into my retirement.
Of course, a project of this duration requires the cooperation of many
people over many years. My greatest collective debt is to the thoughtful
professionals who patiently worked through the often tedious batteries
of questions on what could have been, what is, and what might yet be. I
told them at the outset that I did not intend to write a book that named
names, or that, by exploiting hindsight bias, incited readers to glorify
those who got it right or ridicule those who got it wrong. I promised
strict confidentiality. The book that would emerge from this effort would
be variable-centered, not person-centered. The focus would be on the
links between how people think and what they get right or wrong, at
various junctures, in a kaleidoscopically shifting world. I realize that the
resulting cognitive portrait of expert political judgment is not altogether
flattering, but I hope that research participants, even the “hedgehogs”
among them, do not feel shabbily treated. I level no charges of judgmen-
tal flaws that do not also apply to me.
Another great debt is to the many colleagues who offered method-
ological and theoretical advice that saved me from making an even big-
ger fool of myself than I may have already done. Barbara Mellers, Paul
Tetlock, and Phillip Rescober offered invaluable guidance on how to de-
sign measures of forecasting skill that were sensitive to the variety of in-
genious objections that forecasters raised when either high probability
events failed to materialize or low probability events did materialize.
And colleagues from several disciplines—including psychology, political
science, economics, history, and the hybrid field of intelligence analysis—
made suggestions at various junctures in this long journey that, in my
opinion at least, improved the final product. I cannot remember the
source of every insightful observation at every stage of this project, but