Table Of ContentOUP CORRECTED PROOF – FINAL, 22/07/20, SPi
The Phantom Pattern Problem
OUP CORRECTED PROOF – FINAL, 22/07/20, SPi
THE PHANTOM
PATTERN PROBLEM
The Mirage of Big Data
Gary Smith and Jay CordeS
1
OUP CORRECTED PROOF – FINAL, 22/07/20, SPi
1
Great Clarendon Street, Oxford, OX2 6DP,
United Kingdom
Oxford University Press is a department of the University of Oxford.
It furthers the University’s objective of excellence in research, scholarship,
and education by publishing worldwide. Oxford is a registered trade mark of
Oxford University Press in the UK and in certain other countries
© Gary Smith and Jay Cordes 2020
The moral rights of the authors have been asserted
First Edition published in 2020
Impression: 1
All rights reserved. No part of this publication may be reproduced, stored in
a retrieval system, or transmitted, in any form or by any means, without the
prior permission in writing of Oxford University Press, or as expressly permitted
by law, by licence or under terms agreed with the appropriate reprographics
rights organization. Enquiries concerning reproduction outside the scope of the
above should be sent to the Rights Department, Oxford University Press, at the
address above
You must not circulate this work in any other form
and you must impose this same condition on any acquirer
Published in the United States of America by Oxford University Press
198 Madison Avenue, New York, NY 10016, United States of America
British Library Cataloguing in Publication Data
Data available
Library of Congress Control Number: 2020930015
ISBN 978–0–19–886416–5
Printed and bound by
CPI Group (UK) Ltd, Croydon, CR0 4YY
Links to third party websites are provided by Oxford in good faith and
for information only. Oxford disclaims any responsibility for the materials
contained in any third party website referenced in this work.
OUP CORRECTED PROOF – FINAL, 22/07/20, SPi
In his earlier life as the very successful author of statistics textbooks, Gary Smith had a
knack for creating creative applications that helped students learn important statistical
concepts in a fun and intuitive way. In this collaboration with Jay Cordes, Smith takes
this same approach mainstream with entertaining example after entertaining example
highlighting their central point that not all patterns are meaningful. Smith and Cordes
argue that the solution to the dilemma is not more data, but rather more intelligent
theorizing about how the world works. Readers should heed their warning—and, those
who don’t should not be surprised if they make an appearance in the next Smith and
Cordes book as a cautionary tale!
Shawn Bushway, Senior Policy Researcher
Behavioral and Policy Sciences Department, RAND Corporation
A nice little antidote to big claims about big benefits of Big Data.
Marc Abrahams, Editor of the Annals of Improbable Research
founder of the Ig Nobel Prize ceremony
It’s refreshing to see a book on paleo that’s about distance running, pattern recognition,
and jokes, rather than scarfing steaks, pumping iron, and violence. But, as Gary Smith
and Jay Cordes explain and demonstrate, pattern recognition can lead to superficially
appealing but ultimately misleading conclusions.
Andrew Gelman, Professor of Statistics and Computer Science
Columbia University
Gary and Jay hit the ball out of the park with “The Phantom Pattern Problem: The
Mirage of Big Data.” Full of fun stories and spurious correlations and patterns, the book
excels at its aim: Explaining the hazards of big data, how many can easily be fooled by
putting too much trust in blind statistics, as well as highlighting many pitfalls such as
overfitting, data mining with out-of-sample data, over-reliance on backtesting, and
“Hypothesizing after the Results are Known,” or HARKing. The text is a home run on
the importance of building models guided by human expertise, the critical process of
theory before data, and is a welcome addition to any reader’s library.
Brian Nelson, CFA, President
Investment Research, Valuentum Securities, Inc.
The legendary economist Ronald Coase once famously said, ‘If you torture the data long
enough, it will confess.’ As Smith and Cordes demonstrate in spades, the era of Big Data
has only exacerbated Coase’s assertion. Packed with great examples and solid research,
“The Phantom Pattern Problem” is a cri de coeur to those who believe in the unassailable
power of data.
Phil Simon
Award winning author of Too Big to Ignore: The Business Case for Big Data
Using easily understood examples from sports, the stock market, economics, medical
testing, and gambling, Smith and Cordes illustrate how data analytics and big data can
be seductively misleading. I learned a lot.
Robert J. Marks II, Ph.D.
Distinguished Professor of Electrical & Computer Engineering, Baylor University
Director, The Walter Bradley Center for Natural & Artificial Intelligence
OUP CORRECTED PROOF – FINAL, 22/07/20, SPi
This book is dedicated to our families.
OUP CORRECTED PROOF – FINAL, 22/07/20, SPi
INTRODUCTION
Surely You Jest
In October 2001, Apple’s Steve Jobs unveiled the iPod, a revolutionary
hand-held music player with a built-in hard drive: “To have your
whole CD library with you at all times is a quantum leap when it
comes to music. You can fit your whole music library in your pocket.”
Despite the $399 price tag, sales were phenomenal, hitting thirty-nine
million in 2006, before being eclipsed by the 2007 introduction of the
iPhone.
Figure I.1 shows that the explosion of iPod sales in 2005 and 2006
coincided with an increase in the number of murders in the United States.
Were people killing each other in order to get their hands on an iPod?
Were iPod listeners driven insane by the incessant music and then
murdering friends and strangers?
When we showed Figure I.1 to a friend, her immediate reaction was,
“Surely you jest.” We are jesting, but there is a reason why we jest. In 2007,
the Urban Institute, a highly regarded Washington think-tank, released a
research report on the increase in murders in 2006 and 2007:
The rise in violent offending and the explosion in the sales of iPods and other
portable media devices is more than coincidental. We propose that, over the past
two years, America may have experienced an iCrime wave.
Unlike us, they were not jesting.
We have all been warned over and over that correlation is not c ausation,
but too often, we ignore the warnings. We have inherited from our distant
ancestors an often-irresistible desire to seek patterns and succumb to their
allure. We laugh at some obviously nutty correlations; for example, the
number of lawyers in Nevada is statistically correlated with the number
SURELY YOU JEST | 1
OUP CORRECTED PROOF – FINAL, 22/07/20, SPi
50 18
40
ns ds
o n
milli 30 iPods ousa
Pod Sales, 20 murders 17 Murders, th
i
10
0 16
2004 2005 2006
Figure I.1 iPod sales and murders.
of people who died after tripping over their own two feet. Yet other
correlations, like iPod sales and murders, have a seductive appeal. If
esteemed researchers at the Urban Institute can be seduced by fanciful
correlations, so can any of us.
Thousands of people didn’t kill each other so that they could steal their
iPods, and thousands of iPod listeners weren’t driven murderously insane.
Murders and iPod sales both happened to increase in 2005 and 2006, as
did many other things. The serendipitous correlation between murders
and iPod sales did not last long. Murders dropped in 2007 and have fallen
since then, even though iPod sales continued to grow for a few more years
until they were dwarfed by iPhone sales.
The correlation between murders and iPod sales is particularly
laughable since it is based on a mere three years of data. Anything that
increased (or decreased) steadily during this two-year span will be highly
correlated with murders—for example, ice cream sales in the U.S.
Figure I.2 shows that the correlation between ice cream sales and
murders is as good as the correlation between iPod sales and murders.
Were people killing each other in order to snatch iPods or ice cream
cones? Almost surely neither, and yet, two experienced researchers at the
Urban Institute assumed that correlation was causation.
2 | THE PHANTOM PATTERN PROBLEM
OUP CORRECTED PROOF – FINAL, 22/07/20, SPi
1365 18
ns
o
gall 1355 ds
ns of ice cream usan
am, millio 1345 murders 17 urders, tho
Cre M
e 1335
c
I
1325 16
2004 2005 2006
Figure I.2 Ice cream sales and murders.
How, in this modern era of big data and powerful computers, can
experts be so foolish? Ironically, big data and powerful computers are part
of the problem.
We have all been bred to be fooled—to be attracted to shiny patterns
and glittery correlations. Big data and powerful computers feed this
addiction because they make it so easy to find such baubles—and they
also ensure that most of what we find is as worthless as the fanciful claim
that iPods increased the murder rate.
It is up to us to resist the allure, to not be fooled by phantom patterns.
SURELY YOU JEST | 3
OUP CORRECTED PROOF – FINAL, 22/07/20, SPi
CHAPTER 1
Survival of the Sweaty
Pattern Processors
Compared to other animals, humans are not particularly strong
or powerful. We don’t have sharp teeth, claws, or beaks. We don’t
have sledgehammer horns, tusks, or tails. We don’t have body
armor. We are not great swimmers or sprinters. How did our distant
ancestors not only survive, but become masters of the planet?
Two things powered our ascent: cooling efficiently and recognizing
patterns.
Survival of the Sweatiest
Our first great advantage is that we are born to run. Our prehistoric
ancestors walked and ran everywhere, chasing things to eat and running
away from beasts that wanted to eat them. Humans are not fast, but we do
have exceptional endurance. The early humans who could run down their
prey by wearing them out ate plenty of meat, which made them stronger
and more likely to mate and pass along their genes to their children.
Through countless generations of chasing, feasting, and mating, humans
who couldn’t keep up were weeded out of the gene pool.
Hunting based on endurance running is called persistence hunting—
pursuing prey until they cannot run any more. Part of our success is due
to how our bodies evolved two million years ago in ways that facilitate
endurance running, long before our more recent ancestors developed
arrows, spears, and other projectile weapons. Among the adaptations
identified by Harvard Anthropology Professor Daniel Lieberman:
SURVIVAL OF THE SWEATY PATTERN PROCESSORS | 5
