Table Of ContentWHO’S
DRIVING
INNOVATION?
NEW TECHNOLOGIES AND
THE COLLABORATIVE STATE
PALGR AVE POLICY ESSENTIAL S
Jack Stilgoe
Who’s Driving Innovation?
Jack Stilgoe
Who’s Driving
Innovation?
New Technologies and the Collaborative State
Jack Stilgoe
Department of Science
and Technology Studies
University College London
London, UK
ISBN 978-3-030-32319-6 ISBN 978-3-030-32320-2 (eBook)
https://doi.org/10.1007/978-3-030-32320-2
© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer
Nature Switzerland AG 2020
This work is subject to copyright. All rights are solely and exclusively licensed by the
Publisher, whether the whole or part of the material is concerned, specifically the rights
of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction
on microfilms or in any other physical way, and transmission or information storage and
retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are
exempt from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and
information in this book are believed to be true and accurate at the date of publication.
Neither the publisher nor the authors or the editors give a warranty, expressed or implied,
with respect to the material contained herein or for any errors or omissions that may have
been made. The publisher remains neutral with regard to jurisdictional claims in published
maps and institutional affiliations.
Cover credit: eStudio Calamar, Sant Feliu de Guíxols, Spain
This Palgrave Macmillan imprint is published by the registered company Springer Nature
Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
A
cknowledgements
The ideas that led to this book began to form during a summer at the
Centre for the Study of the Sciences and the Humanities in Bergen.
Thank you to Matthias Kaiser and his colleagues for having me to stay.
Research for the book has been supported by the Alan Turing Institute,
the European Commission, as part of the SCALINGS project (Horizon
2020 grant 788359), and the Economic and Social Research Council,
who fund the Driverless Futures? project (grant ES/S001832/1). I was
lucky to have conversations and comments on drafts from a number of
experts, including Jamie Bartlett, Peter Bradwell, Tom Cohen, Diane
Coyle, Stanley Flink, Milos Mladenovic, Saba Mirza, Melanie Smallman,
Langdon Winner and three anonymous reviewers. Finally, thanks are
due to my colleagues and students at University College London’s
Department of Science and Technology Studies.
v
c
ontents
1 Who Killed Elaine Herzberg? 1
2 Innovation Is Not Self-Driving 7
3 The Politics of Tech 21
4 In Dreams Begins Responsibility 39
5 The Collaborative State 55
References 67
Index 73
vii
CHAPTER1
Who Killed Elaine Herzberg?
Elaine Herzberg did not know that she was part of an experiment. She
was walking her bicycle across the road at 10 p.m. on a dark desert night
in Tempe, Arizona. Having crossed three lanes of a four-lane highway,
Herzberg was run down by a Volvo SUV travelling at 38 miles per hour.
Shewaspronounceddeadat10:30p.m.
Thenextday,theofficerinchargeoftheinvestigationrushedtoblame
the pedestrian. Police Chief Sylvia Moir told a local newspaper, ‘It’s very
clear it would have been difficult to avoid this collision… she came from
the shadows right into the roadway… the driver said it was like a flash.’1
Accordingtotherulesoftheroad,Herzbergshouldnothavebeenthere.
Hadshebeenatthecrosswalkjustdowntheroad,thingswouldprobably
haveturnedoutdifferently.
Rafaela Vasquez was behind the wheel of the Volvo, but she wasn’t
driving.Thecar,operatedbyUber,wasin‘autonomous’mode.Vasquez’s
jobwastomonitorthecomputerthatwasdoingthedrivingandtakeoverif
anythingwentwrong.Afewdaysafterthecrash,thepolicereleasedavideo
from a camera on the rear-view mirror. It showed Vasquez looking down
at her knees in the seconds before the crash and for almost a third of the
21-minutejourneythatleduptoit.Datatakenfromherphonesuggested
thatshehadbeenwatchinganepisodeof‘TheVoice’ratherthantheroad.
Embarrassinglyforthepolicechief,hercolleagues’investigationcalculated
that,hadVasquezbeenlookingattheroad,shewouldhaveseenHerzberg
andbeenabletostopmorethan40feetbeforeimpact.2
©TheAuthor(s)2020 1
J.Stilgoe,Who’sDrivingInnovation?,
https://doi.org/10.1007/978-3-030-32320-2_1
2 J.STILGOE
Driversandpedestriansmakemistakesallthetime.Aregularlyrepeated
statistic is that more than 90% of crashes are caused by human error. The
TempePolicereportconcludedthatthecrashhadbeencausedbyhuman
frailtiesonbothsides:Herzbergshouldnothavebeenintheroad;Vasquez
forherpartshouldhaveseenthepedestrian,sheshouldhavetakencontrol
ofthecarandsheshouldhavebeenpayingattentiontoherjob.Inthecrash
investigation business, these factors are known as ‘proximate causes’. But
if we focus only on proximate causes, we fail to learn from the novelty of
thesituation.Herzbergwasthefirstpedestriantobekilledbyaself-driving
car.TheUbercrashwasnotjustacaseofhumanerror.Itwasalsoafailure
oftechnology.
Here was a car on a public road in which the driving had been dele-
gatedtoacomputer.Athingthathadveryrecentlyseemedimpossiblehad
become,onthestreetsofArizona,mundane—somundanethattheperson
who was supposed to be monitoring the system had, in effect, switched
off.3 Thecar’ssensors—360-degreeradar,short-andlong-rangecameras,
alidarlaserscannerontheroofandaGPSsystem—weresupposedtopro-
vide superhuman awareness of the surroundings. The car’s software was
designedtointerpretthisinformationbasedonthousandsofhoursofsim-
ilarexperiences,identifyingobjects,predictingwhattheyweregoingtodo
next and plotting a safe route. This was artificial intelligence in the wild:
notplayingchessortranslatingtextbutsteeringtwotonnesofmetal.
When high-profile transport disasters happen in the US, the National
Transportation Safety Board is called in. The NTSB are less interested in
blamethaninlearningfrommistakestomakethingssafer.Theirinvestiga-
tionsarepartofthereasonwhyairtravelissoastonishinglysafe.In2017,
forthefirsttime,awholeyearpassedinwhichnotasinglepersondiedin
acommercialpassengerjetcrash.Ifself-drivingcarsaregoingtobeassafe
as aeroplanes, regulators need to listen to the NTSB. The Board’s report
ontheUbercrashconcludedthatthecar’ssensorshaddetectedanobject
intheroadsixsecondsbeforethecrash,butthesoftware‘didnotinclude
a consideration for jaywalking pedestrians’.4 The AI could not work out
what Herzberg was and the car continued on its path. A second before
thecarhitHerzberg,thedrivertookthewheelbutswervedonlyslightly.
Vasquezonlyappliedthebrakesafterthecrash.
In addition to the proximate causes, Elaine Herzberg’s death was the
resultofasetofmoredistantchoicesabouttechnologyandhowitshould
bedeveloped.Claimingthattheywereinaraceagainstothermanufactur-
ers,Uberchosetotesttheirsystemquicklyandcheaply.Otherself-driving
1 WHOKILLEDELAINEHERZBERG? 3
carcompaniesputtwoormorequalifiedengineersineachoftheirtestvehi-
cles. Vasquez was alone and she was no test pilot. The only qualification
sheneededbeforestartingworkwasadrivinglicence.
Uber’s strategy filtered all the way down into its cars’ software, which
was much less intelligent than the company’s hype had implied. As the
company’s engineers worked out how to make sense of the information
coming from the car’s sensors, they balanced the risk of a false positive
(detectingathingthatisn’treallythere)againsttheriskofafalsenegative
(failingtoreacttoanobjectthatturnsouttobedangerous).Afterearlier
testsofself-drivingcarsinwhichsoftwareoverreactedtothingslikesteam,
plasticbagsandshadowsontheroads,engineersretunedtheirsystems.The
misidentification of Elaine Herzberg was partly the result of a conscious
choice about how safe the technology needed to be in order to be safe
enough. One engineer at Uber later told a journalist that the company
had ‘refused to take responsibility. They blamed it on the homeless lady
[Herzberg], the Latina with a criminal record driving the car [Vasquez],
eventhoughweallknewPerception[Uber’ssoftware]wasbroken.’5
The companies that had built the hardware also blamed Uber. The
president of Velodyne, the manufacturer of the car’s main sensors, told
Bloomberg, ‘Certainly, our lidar is capable of clearly imaging Elaine and
her bicycle in this situation. However, our lidar doesn’t make the deci-
sion to put on the brakes or get out of her way.’6 Volvo made clear that
they had nothing to do with the experiment. They provided the body of
thecar,notitsbrain.Anautomaticbrakingsystemthatwasbuiltintothe
Volvo—using well-established technology—would almost certainly have
saved Herzberg’s life, but this had been switched off by Uber engineers,
whoweretestingtheirowntechnologyanddidn’twantinterferencefrom
anothersystem.
We don’t know what Elaine Herzberg was thinking when she set off
across the road. Nor do we know exactly what the car was thinking.
Machinesmakedecisionsdifferentlyfromhumansandthedecisionsmade
bymachinelearningsystemsareofteninscrutable.However,theevidence
fromthecrashpointstoarecklessapproachtothedevelopmentofanew
technology.Ubershoulderedsomeoftheblame,agreeinganout-of-court
settlement with the victim’s family and changing their approach to safety.
But to point the finger only at the company would be to ignore the con-
text. Roads are dangerous places, particularly in the US and particularly
forpedestrians.Acenturyofdecisionsbypolicymakersandcarmakershas
producedasystemthatgivespowerandfreedomtodrivers.Tempe,partof
4 J.STILGOE
the sprawling metropolitan area of Phoenix, is car-friendly. The roads are
wideandneatandtheweatherisgood.Itisideallysuitedtotestingaself-
drivingcar.Forapedestrian,theplaceanditsinfrastructurecanfeelhostile.
Official statistics bear this out. In 2017, Arizona was the most dangerous
stateforpedestriansintheUS.7
Members of Herzberg’s family sued the state government on the
groundsthat,first,thestreetswereunsafeforpedestriansand,second,pol-
icymakerswerecomplicitinUber’sexperiments.Inadditiontotheclimate
and the tidiness of the roads, Uber had been attracted to Tempe by the
governorofArizona,DougDucey.Thecompanyhadstartedtheirtesting
inSanFrancisco,neartheirheadquarters.Butwhenoneoftheirself-driving
carsranaredlight,CaliforniaregulatorstoldUberthattheyneededa$150
permit.UberobjectedandDuceyseizedhisopportunity.WiththeGover-
nor’sblessing,thecompanyhadalreadybeentestinginsecretonthestreets
of Phoenix. Ducey could now go public and claim that he had tempted a
bigtechcompanyawayfromSiliconValley.Hetweeted‘Thisiswhatover-
regulationlookslike#ditchcalifornia’and‘HereinAZweWELCOMEthis
kind of technology & innovation! #ditchcalifornia #AZmeansBIZ’. With
almost no oversight, Uber moved their experiments to Arizona in 2016.
WhenHerzbergwaskilledlessthan18monthslater,Ducey’senthusiasm
collapsedandUberwerethrownoutoftheirnewlaboratory.
When technologies fail, it is often hard to find the person responsible
and easy for those involved to blame others or claim it was a freak occur-
rence. It’s a symptom of a wider problem, which is that we aren’t clear
who is in control of the development of new technologies. When tech-
nological dreams meet the real world, the results are often disappointing
and occasionally messy. Policymakers are seduced by the promise of new
technologies, which arrive without instructions for how they should be
governed.Itisalltoocommonforregulationtobeanafterthought.Inthe
world of aviation, it’s called a tombstone mentality: defects are noticed,
lessons are learned and rules are written in grim hindsight. In Arizona,
policymakers allowed a private experiment to take place in public, with
citizens as unwitting participants. It ended badly for everyone involved.
Tragediesareopportunitiesforlearning,opportunitiestochallengeclaims
madeabouttechnologyandopportunitiestothinkaboutalternatives.We
shouldaskifatechnologyissafeenough,butthisalsomeansasking,Safe
enough for what? Why are self-driving cars being developed? Where are
theytakingus?Aspoliticianscompeteintheirenthusiasmforinnovation,
