Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics

287 Pages·2014·2.82 MB·English

by  Gary Smith

DID YOU KNOW that building a quarry in your backyard can increase your
property value? Or that America’s unemployment rate is actually zero? Or that
drinking a full pot of coffee every morning adds years to your life, but drinking
two cups a day increases the risk of cancer?
If you believe the above, then Professor Gary Smith has a World Cup–
predicting octopus he’d like to show you. The sad truth is that “facts” like these
are routinely presented with a straight face by credentialed academics and
backed up with reams of raw data. In Standard Deviations, Smith skillfully
unpacks the various ways we are duped by data every day. He deftly
demonstrates how a straightforward set of findings can be teased and
manipulated to reflect whatever the researcher wants to see.
Lying with statistics is a time-honored con, and in this age of Big Data even
the most accredited findings can be suspect. Blending the keen statistical eye of
Nate Silver with the probing insights of Daniel Kahneman and Dan Ariely,
Smith demystifies the math behind the dismal science, making it easy to spot
flaws all around and find the truth hidden in plain sight.
M STANDARD DEVIATIONS
ORE ADVANCE PRAISE FOR
“Gary Smith’s Standard Deviations is both a statement of principles for doing
statistical inference correctly and a practical guide for interpreting the
(supposedly) data-based inferences other people have drawn. Cleverly written
and engaging to read, the book is full of concrete examples that make clear not
just what Smith is saying but why it matters. Readers will discover that lots of
what they thought they’d learned is wrong, and they’ll understand why.”
—BENJAMIN M. FRIEDMAN
William Joseph Maier Professor
of Political Economy, Harvard University “Standard Deviations shows in
compelling fashion why humans are so susceptible to the misuse of statistical
evidence and why this matters. I know of no other book that explains important
concepts such as selection bias in such an entertaining and memorable manner.”
—RICHARD J. MURNANE
Thompson Professor of Education and Society,
Harvard Graduate School of Education “We all learn in school that there are
three kinds of lies: lies, damn lies, and statistics. Gary Smith’s new book imparts
true substance to this point by setting forth myriad examples of how and why
statistics and data-crunching at large are susceptible to corruption. The great risk
today is that the young will forget that deductive logic is vastly more powerful
than inductive logic.”
—HORACE “WOODY” BROCK
President, Strategic Economic Decisions, Inc.
“Statistical reasoning is the most used and abused form of rhetoric in the field of
finance. Standard Deviations is an approachable and effective means to arm
oneself against the onslaught statistical hyperbole in our modern age. Professor
Smith has done us all a tremendous service.”
—BRYAN WHITE
Managing Director, BlackRock, Inc.
“It’s entertaining, it’s gossipy, it’s insightful—and it’s destined to be a classic.
Based on a lifetime of experience unraveling the methodical blunders that
remain all too frequent, this book communicates Gary Smith’s wisdom about
how not to do a data analysis. Smith’s engaging rendering of countless painful
mistakes will help readers avoid the pitfalls far better than merely mastering
theorems.”
—EDWARD E. LEAMER
Distinguished Professor and Chauncey J. Medberry
Chair in Management, UCLA “Standard Deviations will teach you how not to be
deceived by lies masquerading as statistics. Written in an entertaining style with
contemporary examples, this book should appeal to everyone, whether interested
in marriages or mortgages, the wealth of your family, or the health of the
economy. This should be required reading for everyone living in this age of (too
much?) information.”
—ARTHUR BENJAMIN
Professor of Mathematics, Harvey Mudd College
and author of Secrets of Mental Math
“One of those rare books that make people better for having read it.”
—JAY CORDES
Senior Manager, RookMedia.net
“Most of the authoritative, sciencey-sounding claims we’re fed by the media are
polluted by distortions, biases, and plain old errors. In Standard Deviations,
Gary Smith sets the record straight.”
—DAVID H. FREEDMAN
Author of Wrong: Why Experts Keep Failing Us—and
How to Know When Not to Trust Them
Copyright
This edition first published in hardcover in the United States and the United
Kingdom in 2014 by Overlook Duckworth, Peter Mayer Publishers, Inc.
New York
141 Wooster Street
New York, NY 10012
www.overlookpress.com
For bulk and special sales, please contact [email protected],
or write us at the above address.
London
30 Calvin Street
London E1 6NW
[email protected]
www.ducknet.co.uk
Copyright © 2014 by Gary Smith All rights reserved. No part of this publication
may be reproduced or transmitted in any form or by any means, electronic or
mechanical, including photocopy, recording, or any information storage and
retrieval system now known or to be invented, without permission in writing
from the publisher, except by a reviewer who wishes to quote brief passages in
connection with a review written for inclusion in a magazine, newspaper, or
broadcast.
ISBN: 978-1-4683-1068-9
To my wife Margaret and my children Josh,
Jo, Chaska, Cory, Cameron, and Claire
CONTENTS
MORE ADVANCE PRAISE FOR STANDARD DEVIATIONS
COPYRIGHT
DEDICATION
INTRODUCTION
1. Patterns, Patterns, Patterns
2. Garbage In, Gospel Out
3. Apples and Prunes
4. Oops!
5. Graphical Gaffes
6. Common Nonsense
7. Confound It!
8. When You’re Hot, You’re Not
9. Regression
10. Even Steven
11. The Texas Sharpshooter
12. The Ultimate Procrastination
13. Serious Omissions
14. Flimsy Theories and Rotten Data
15. Don’t Confuse Me With Facts
16. Data Without Theory
17. Betting the Bank
18. Theory Without Data
19. When to Be Persuaded and When to Be Skeptical
SOURCES
INDEX
INTRODUCTION
W
E LIVE IN THE AGE OF BIG DATA. THE POTENT COMBINATION of fast computers
and worldwide connectivity is continually praised—even worshipped. Over and
over, we are told that government, business, finance, medicine, law, and our
daily lives are being revolutionized by a newfound ability to sift through reams
of data and discover the truth. We can make wise decisions because powerful
computers have looked at the data and seen the light.
Maybe. Or maybe not. Sometimes these omnipresent data and magnificent
computers lead to some pretty outlandish discoveries. Case in point, serious
people have seriously claimed that:
• Messy rooms make people racist.
• Unborn chicken embryos can influence computer random-event
generators.
• When the ratio of government debt to GDP goes above 90 percent,
nations nearly always slip into recession.
• As much as 50 percent of the drop in the crime rate in the United States
over the past twenty years is because of legalized abortion.
• Drinking two cups of coffee a day substantially increases the risk of
pancreatic cancer.
• The most successful companies tend to become less successful, while the
least successful companies tend to become more successful, so that soon
all will be mediocre.
• Athletes who appear on the cover of Sports Illustrated or Madden NFL
are jinxed in that they are likely to be less successful or injured.
• Living near power lines causes cancer in children.
• Humans have the power to postpone death until after important
ceremonial occasions.
• Asian Americans are more susceptible to heart attacks on the fourth day
of the month.
• People live three to five years longer if they have positive initials, like
ACE.
• Baseball players whose first names began with the letter D die, on
average, two years younger than players whose first names began with
the letters E through Z.
• The terminally ill can be cured by positive mental energy sent from
thousands of miles away.
• When an NFC team wins the Super Bowl, the stock market almost
always goes up.
• You can beat the stock market by buying the Dow Jones stock with the
highest dividend yield and the second lowest price per share.
These claims—and hundreds more like them—appear in newspapers and
magazines every day even though they are surely false. In today’s Information
Age, our beliefs and actions are guided by torrents of meaningless data. It is not
hard to see why we repeatedly draw false inferences and make bad decisions.
Even if we are reasonably well informed, we are not always alert to the ways in
which data are biased or irrelevant, or to the ways in which scientific research is
flawed or misleading. We tend to assume that computers are infallible—that no
matter what kind of garbage we put in, computers will spit out gospel. It happens
not just to laymen in their daily lives, but in serious research by diligent
professionals. We see it in the popular press, on television, on the Internet, in
political campaigns, in academic journals, in business meetings, in courtrooms,
and, of course, in government hearings.
Decades ago, when data were scarce and computers nonexistent, researchers
worked hard to gather good data and thought carefully before spending hours,
even days, on painstaking calculations. Now with data so plentiful, researchers
often spend too little time distinguishing between good data and rubbish,
between sound analysis and junk science. And, worst of all, we are too quick to
assume that churning through mountains of data can’t ever go wrong. We rush to
make decisions based on the balderdash these machines dish out—to increase
taxes in the midst of a recession, to trust our life savings to financial quants who
impress us because we don’t understand them, to base business decisions on the
latest management fad, to endanger our health with medical quackery, and—
worst of all—to give up coffee.
Ronald Coase cynically observed that, “If you torture the data long enough, it
will confess.” Standard Deviations is an exploration of dozens of examples of
tortuous assertions that, with even a moment’s reflection, don’t pass the smell
test. Sometimes, the unscrupulous deliberately try to mislead us. Other times, the
well-intentioned are blissfully unaware of the mischief they are committing. My
intention in writing this book is to help protect us from errors—both external and