Data-ism: The Revolution Transforming Decision Making, Consumer Behavior, and Almost Everything Else

194 Pages·2015·1.05 MB·English

by  Steve Lohr

DEDICATION
TO TERRY AND NIKKI, FOR TIME
SERVED, AND ALL THE LAUGHS
ALONG THE WAY
ACKNOWLEDGMENTS
A
book may be written by a single person, but books are always collaborations.
An editor nudges things along, a publisher makes a bet, and the adventure
begins. There is an original idea, but it gets refined along the way, modified,
honed and crafted by reporting, thinking and guidance.
The germ of this book really came in January 2012, with a call from a senior
editor at the New York Times and former editor of the Times Magazine, Gerry
Marzorati. At the time, he was helping guide the recently revamped Sunday
Review section, and Gerry observed that I had written a number of articles about
the field that was being called big data, and that the subject was getting more
attention all the time, including being prominently on the agenda of the World
Economic Forum annual meeting in Davos that month. We talked for a while,
and he asked if there was a long overview piece to be done that explained the
phenomenon for a general audience. I replied that I thought there was. The result
was a lead article for Sunday Review that ran in early February 2012 under the
headline “The Age of Big Data.”
The Sunday Review piece generated a lot of interest, Web traffic, and reader
comments. I hadn’t thought of a book, but my agent, Max Brockman of
Brockman Inc., did, and encouraged me, and offered sound advice. Hollis
Heimbouch, publisher of the HarperBusiness imprint of HarperCollins, was the
editor who made the bet that what is a rich subject could be made into a story—
and that I could do it. Hollis is everything you want in an editor—by turns
enthusiastic, patient, and insistent, someone who listens to your ideas and brings
plenty of incisive ones of her own. She’s also a very sure hands-on text editor in
every way, from shaping the structure of a book to cutting. At HarperCollins, I’d
also like to thank Eric Meyers, an associate editor and a deft practitioner of the
dark arts of manuscript wrangling.
At the Times, several editors have—and still do—encourage my reporting in
the field of data science, and its implications. They include Larry Ingrassia,
Dean Murphy, Damon Darlin, Suzanne Spector, Joseph Plambeck, James
Kerstetter, Quentin Hardy, David Gillen, Lon Teter, Thomas Kuntz, and David
Corcoran. Conversations with Natasha Singer have sharpened my thinking on
the subject of data and privacy. Then there is John Markoff, a science reporter
these days; he and I have labored agreeably and often together in the data-laden
precincts of business and science coverage for years.
While I was on leave from the Times, Mark Hansen generously offered me
office space to work in and travel from and an appointment as a research fellow
at the Brown Institute for Media Innovation. Mark is a director of the Brown
Institute, a collaboration between the Columbia Graduate School of Journalism
and the Stanford School of Engineering. Mark and Bernd Girod, the Stanford
director of the institute, are leading an innovative effort to use data tools in
storytelling and public policy. At Columbia, my workspace was in the career
services office, and my thanks to the professionals there for kindly making the
accommodation: Julie Hartenstein, Gina Boubion, Anusha Shrivastava, Izabela
Rutkowski, and, earlier, Jacqueline DeLaFuente.
This is a book of narrative nonfiction. So I am especially thankful to the
people who agreed to be interviewed, and showed the patience for follow-up and
fact-checking inquiries. At the top of the list is Jeffrey Hammerbacher, who gave
the most in time, thoughts, and candor. He not only went through many hours of
interviews but also offered introductions to his parents, his wife, classmates from
high school and college, and friends and work colleagues, past and present. All
with no strings attached.
For other parts of the book, there were people I spent whole days with, such
as Timothy Buchman at the Emory medical center in Atlanta and Nick
Dokoozlian in the grape vineyards of central California and Michael Haydock in
suburban Minneapolis.
Many others were interviewed for this book. They include Sam Adams,
Brooke Barrett, Richard Berner, Patrick Bosworth, Thomas Botts, Erik
Brynjolfsson, John Calkins, Murray Campbell, Dennis Charney, Herbert Chase,
Jeffrey Chester, Sharath Cholleti, Adam D’Angelo, Arne Duncan, Sue Duncan,
Tony Fadell, Edward Felten, David Ferrucci, Rachana Shah Fischer, Brian
Gehlich, Jim Goodnight, Nagui Halim, Hendrik Hamann, Glenn Hammerbacher,
Lenore Hammerbacher, Danny Hillis, Jeffrey Immelt, Jon Iwata, James Kalina,
Kaan Katircioglu, Gary King, Jon Kleinberg, Martin Kohn, Randy Komisar,
Patricia Kovatch, Edward Lazowska, and Michael Linderman.
They also include Mark Malhotra, James Manyika, Yoky Matsuoka, Andrew
McAfee, David McQueeney, Douglas Merrill, Tom Mitchell, Craig Mundie,
Arvind Narayanan, Tim O’Donnell, Sharoda Paul, Alex Pentland, Claudia
Perlich, Stephanie Pieroni, William Pulleyblank, Tara Richardson, Virginia
Rometty, Itamar Rosenn, Jeff Rothschild, Marc Rotenberg, Alex Rubinsteyn,
William Ruh, Eric Schadt, Benjamin Scheuer, Neal Sidhwaney, Larry Smarr,
Andrew Smeall, Jim Spohrer, Halle Tecco, Menka Uttamchandani, Veronica
Vargas, Anil Varma, David Vivero, David Vladeck, Donald Walker, Danny
Weitzner, and Michelle Zhou.
At IBM, I owe particular thanks to Jon Iwata and Ed Barbini for clearing the
way for a broad reporting swath with IBM scientists and executives, and to
Vineeta Durani and Angela Lee Sullivan for tirelessly making the interviews
happen.
Finally, thanks to Terry and Nikki for understanding about work and the
missed vacations, and to Nickie Trucco for my missing the Christmas holidays
in 2013. But then, as I told them at the time, this book was a good problem to
have.
CONTENTS
Dedication
Acknowledgments
1 How Big Is Big Data?
2 Potential. Potential. Potential.
3 Bet the Company
4 Sight and Insight
5 The Rise of the Data Scientist
6 Data Storytelling: Correlation and Context
7 Data Gets Physical
8 The Yin and Yang of Behavior and Data
9 The Long Game
10 The Prying Eyes of Big Data
11 The Future: Data Capitalism
Notes
Index
About the Author
Also by Steve Lohr
Credits
Copyright
About the Publisher
1
HOW BIG IS BIG DATA?
J
ust outside Memphis, an industrial symphony of machines and humans
shuttles goods to and fro, their carefully orchestrated movements and identifying
marks tracked by bar-code scanners and chips emitting radio waves. Mechanical
arms snatch the plastic shrink-wrapped bundles off a conveyor belt, as forklifts
ferry the packages onto trucks for long-distance travel. Flesh-and-blood humans
guide and monitor the flow of goods and drive the forklifts and trucks.
McKesson, which distributes about a third of all of the pharmaceutical
products in America, runs this sprawling showcase of efficiency. Its buildings
span the equivalent of more than eight football fields, forming the hub of
McKesson’s national distribution network—a feat of logistics that sends goods
to 26,000 customer locations, from neighborhood pharmacies to Walmart. The
main cargo is drugs, roughly 240 million pills a day. The pharmaceutical
distribution business is one of high volumes and razor-thin profit margins. So,
understandably, efficiency has been all but a religion for McKesson for decades.
Yet in the last few years, McKesson has taken a striking step further by
cutting the inventory flowing through its network at any given time by $1
billion. The payoff came from insights gleaned from harvesting all the product,
location, and transport data, from scanners and sensors, and then mining that
data with clever software to identify potential time-saving and cost-cutting
opportunities. The technology-enhanced view of the business was a
breakthrough that Donald Walker, a senior McKesson executive, calls “making
the invisible visible.”
In Atlanta, I stand outside one of the glassed-in rooms in the fifth-floor
intensive care unit at the Emory University Hospital. Inside, a dense thicket of
electronic devices, a veritable forest of medical computing, crowds the room: a
respirator, a kidney machine, infusion machines pumping antibiotics and
painkilling opiates, and gadgets monitoring heart rate, breathing, blood pressure,
oxygen saturation, and other vital signs. Nearly every machine has its own
computer monitor, each emitting an electronic cacophony of beeps and alerts. I
count a dozen screens, larger flat panels and smaller ones, smartphone-sized.
A typical twenty-bed intensive care unit generates an estimated 160,000 data
points a second. Amid all that data, informed and distracted by it, doctors and
nurses make decisions at a rapid clip, about 100 decisions a day per patient,
according to research at Emory. Or more than 9.3 million decisions about care
during a year in an ICU. So there is ample room for error. The overwhelmed
people need help. And Emory is one of a handful of medical research centers
that is working to transform critical care with data, both in adult and neonatal
intensive care wards. The data streams from the medical devices monitoring
patients are parsed by software that has been trained to spot early warning
signals that a patient’s condition is worsening.
Digesting vast amounts of data and spotting seemingly subtle patterns is
where computers and software algorithms excel, more so than humans. Dr.
Timothy Buchman heads up such an effort at Emory. A surgeon, scientist, and
experienced pilot, Buchman uses a flight analogy to explain his goal. GPS
(Global Positioning System) location data on planes is translated to screen
images that show air-traffic controllers when a flight is going astray—“off
trajectory,” as he puts it—well before a plane crashes. Buchman wants the same
sort of early warning system for patients whose pattern of vital signs is off
trajectory, before they crash, in medical terms. “That’s where big data is taking
us,” he says.
T
he age of big data is coming of age, moving well beyond Internet incubators
in Silicon Valley, such as Google and Facebook. It began in the digital-only
world of bits, and is rapidly marching into the physical world of atoms, into the
mainstream. The McKesson distribution center and the Emory intensive care
unit show the way—big data saving money and saving lives. Indeed, the long
view of the technology is that it will become a layer of data-driven artificial
intelligence that resides on top of both the digital and the physical realms.
Today, we’re seeing the early steps toward that vision. Big-data technology is
ushering in a revolution in measurement that promises to be the basis for the
next wave of efficiency and innovation across the economy. But more than