Table Of ContentTuring’s Imitation Game
Conversations with the Unknown
KEVIN WARWICK
CoventryUniversity
HUMA SHAH
CoventryUniversity
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781107056381
DOI:10.1017/9781107297234
©KevinWarwickandHumaShah2016
Firstpublished2016
PrintedintheUnitedKingdombyClays,StIvesplc
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary
ISBN978-1-107-05638-1Hardback
Contents
Introduction page1
References 10
1 TuringtheMan 11
BletchleyPark 11
TheRatioClub 13
Summary 18
References 20
PARTONE 21
2 Turing’sIdeasonMachineThinkingandIntelligence 23
Humancomputers 23
Originoftheimitationgame 25
Thelanguageimitationgame 28
Turing’slaterwork 35
References 39
3 ABriefIntroductiontoArtificialIntelligence 41
ThegameandAI 41
Reflectionontheimitationgame 46
Rationality 47
Cognition 48
Naturallanguageprocessing 48
ClassicalAI 50
ModernAI 51
Problemsolving 52
Turing’s1953paper:chess 53
Concludingcomments 54
References 54
4 TheControversySurroundingTuring’sImitationGame 56
Thejoustingbegins 57
ThinkingaboutThinking 58
Culturalobjection 60
InsupportofTuring’sthinkingmachine 64
References 66
5 HistoryofConversationSystems:FromElizatoEugeneGoost-
man 69
Designingartificialconversation 70
Modernsystems 76
References 79
6 MattersArisingfromEarlyTuringTests 81
Whatisbeingmeasured? 81
References 93
PARTTWO 97
IntroductiontoPartTwo 99
7 The2008ReadingUniversityTuringTests 103
Theexperimentalset-ups 103
Results 109
Control-pairtests 116
Correctidentificationofmachinepairs 123
Evaluatingthisexperiment 126
References 126
8 2012Tests–BletchleyPark 128
Correctdiagnosis 133
Assumedknowledge 135
Confederateeffect 138
Lying 142
Successfulmachines 145
Machines 154
Concludingremarks 156
References 156
9 InterviewswithEliteMachineDevelopers 159
Developerquestions 159
Developerresponses 160
Conclusion 169
References 170
10 Turing2014:TestsatTheRoyalSociety,June2014 171
PurposeoftheTuring2014experiment 173
Experimentaldesign 177
Results 179
EugeneGoostmantranscripts 180
Whatdoesitallmean? 183
Wherenext? 185
Conclusion 185
References 185
11 TheReactiontoTuring2014 187
Onwardstothefuture 191
References 192
Index 194
Introduction
Turing’simitationgame,alsocommonlyknownastheTuringtest,isundoubt-
edly a key component in any study of artificial intelligence or computer sci-
ence.Butitismuchmorethanthisasitalsoprovidesinsightintohowhumans
communicate,ourunconsciousbiasesandprejudices,andevenourgullibility.
Theimitationgamehelpsustounderstandwhywemakeassumptions,which
often turn out to be incorrect, about someone (or something) with whom we
are communicating and perhaps it helps to shed light on why we sometimes
makeseeminglyirrationalconclusionsaboutthem.
Inthechaptersaheadwe’lllookatthegameinmuchmoredetail;however
inessenceitinvolvesadirectconversationalcomparisonbetweenahumanand
amachine.Basically,thegoalofthemachineistomakeyoubelievethatitis
in fact the human taking part, whereas the human involved is merely being
themselves.Boththehumanandthemachinearehidden,socannotbeseenor
heard.Theconversationispurelytextualwithslang,idiom,spellingmistakes,
poorgrammarandfactualerrorsallbeingpartofthemix.
Ifyouputyourselfintheroleofaninterrogatorintheparalleltestthenit
isyourjobtoconversewithbothahiddenhumanandahiddenmachineatthe
sametimeand,afterafive-minuteperiodasstipulatedbyAlanTuringhimself,
to decide which hidden entity is which. If you make the right identification
thenthat’sapointagainstthemachinewhereasifyoumakeamistakeoryou
simplydon’tknowwhichiswhichthenthat’sapointforthemachine.Ifama-
chineissuccessfulinfoolingenoughaverageinterrogators(oneinterpretation
ofTuring’s1950workis30%),thenitcanbesaidtohavepassedtheTuring
test.
Actuallyrestrictingthetopictoaspecificsubjectareamakesitsomewhat
easierforthemachine,becauseitcandirecttheinterrogationtoitsknowledge-
base. However, Turing advocated that the machine be investigated for its in-
tellectualcapacity.Thusweshouldnotrestrictthetopicofconversationatall,
2 Introduction
whichwebelieveisanappropriatechallengeformachinesoftoday,andwhich
is much more interesting for the interrogator and is in the spirit of the game
as(wefeel)Turingintended.Thatsaid,intheexperimentswehaveconducted
wewerethefirsttoincludechildrentotakepartbothasinterrogatorsandhid-
den humans. We have therefore asked all human and machine participants to
respectthissituationintermsofthelanguageused.
ItiscleartoseethatwhenthegamewasoriginallysetupbyAlanTuringin
1950suchskillsasamachinefoolingpeopleintobelievingthatitisahuman
through a short communication exercise would have been very difficult for
most people to understand. However, in introducing the game, Turing linked
it inextricably with the concept of thinking and there is a nice philosophical
argument in consequence concerning how one can tell if another human (or
machine)isthinking.ThiswasabrilliantlinkbyTuringwhich,asaresult,has
broughtaboutamultitudeofargumentsbetweenphilosophersandresearchers
inartificialintelligenceastothetest’smeaningandsignificance.
ButTuring’sgamehasextendedwaybeyondtheivorytowersofacademia
and now has a truly popular following. As an example, the Wikipedia en-
try for the Turing test typically receives 2000 to 3000 views every day. On
one day, 9 June 2014, after it was announced that the Turing test had been
passed following our 2014 experiment (about which you will be able to read
inChapter10),thesamepagereceivedatotalof71,578views,whichisquite
anamazingfigure.Asadirectcomparison,popularWikipediapages,suchas
thoseforLeonardoDiCaprioorTheBeatles,receivedrespectivelyonly11,197
and10,328viewsonthatsameday.
Butwiththispopularfollowingcomesmisconceptionsastowhatthegame
isabout.Inparticular,asortoffolklorebeliefhasarisenthattheTuringgame
isatestforhuman-likeintelligenceinamachine.Thisbeliefhasbeenfuelled
bysomeacademicsandtechnicalwriterswhoperhapshavenotreadtheworks
of Turing as thoroughly as they might. Hopefully this book will therefore be
ofsomehelptothem!
Let us be clear, the Turing test is not, never was and never will be a test
for human-level or even human-like intelligence. Turing never said anything
of the sort either in his papers or in his presentations. The Turing test is not,
neverwasandneverwillbe,atestforhuman-levelthinking.Turingdidn’tsay
thateither.
Howeverthisdoesbegthequestionofwhatexactlydoesitmeantopass
theTuringtest.Aswillbediscussedlater,Turingintroducedhisimitationgame
as a replacement for the question “Can machines think?” Turing (1950). The
conclusionbymanyasaresultofthisisthatifamachinepassesthetestthen
we have to regard it as a thinking machine. However Turing saw a clear dis-
Introduction 3
tinction between the ways machines and humans think. He continued, “May
not machines carry out something which ought to be described as thinking
butwhichisverydifferentfromwhatamandoes?”.Human-likethinkingfor
machineswasthereforenotontheradarasfarasTuringwasconcerned.
InthesamearticleTuringalsowrote,inreferencetotheyear2000,“One
will be able to speak of machines thinking without expecting to be contra-
dicted”.NoamChomsky(2008)wonderedthatofallthewaysamachinecould
display intelligence why did Turing choose a test involving human language
whichismerelyonepartofhumanintelligence.
Intheimitationgameamachineisinterrogatedbyahumanandisdirectly
compared,withregardtoitsabilitytocommunicate,withanotherhumanwho
isbeinginterrogatedinparallel.IfamachinepassestheTuringtestitexhibits
a capability in communication. In that sense it surely involves one aspect of
human intelligence as pointed out by Chomsky. This does not mean that the
machinedisplayshuman-typeintelligenceorconsciousness.
Perhapsbytryingthegameyourselfyou’llbeabletogetabetterideaof
whatit’sabout,howitisnotaseasyasonemightatfirstimagine,andalsohow
much fun can be had from it. To start with, have a look at this conversation,
whereyouarechallengedtodeterminewhetherinputandoutputarehumanor
machine.Thisistakenfromanactualdiscourse,astranscribed,errorsandall.
Input: ilikekittens
Output: mycatispink
Input: areyousmartß
Output: Thatismean‘BRAIN?
Input: couldyougonudeforme?
Output: weareinadarkroomandnaked
Input: thatsucks
Output: seemstomeyouareinacrappyroomwithtrashandmail
Input: youarenotnice
Output: ihaveagoodfriend
Infacttheinputandoutputwerebothmachines;wereyoucorrect?Theinput
wasfromthemachineElbot (ArtificialSolutions).Ina2009email,itsdevel-
oper, Fred Roberts, stated that “Currently I have a feature in Elbot to collect
user inputs according to certain rules, then automatically code a generalized
keywordconditiontoassociatewiththeuserinput,andfinallyreusetheuser
inputasthebotresponse.Thatiswhatyouseeinoutput–actualuserinputs
reusedasbotanswers.Atthemomenttheknowledgeistoosmalltobemean-
ingful,butinafewweeksitcouldbeawesome”.
4 Introduction
Don’t worry if you guessed the identity of the conversationalists incor-
rectly.Inthisbookyouwillhavefurtheropportunitytotestyourownskillsin
distinguishing human from machine. We hope you enjoy the actual imitation
game conversations we present and appreciate the marvel of human commu-
nicationasAlanTuringdid,andunderstandwhyinrealityitissodifficultto
createtalkingrobotsasweseetheminsciencefictionmovies,fromMariain
Metropolis(1927),throughHALin2001:ASpaceOdyssey(1968),toAvain
ExMachina(2014).
Asamorerealisticchallengewehaveincludedhereseveraltranscriptsthat
arosefromtheseriesoftestswhichtookplaceattheRoyalSocietyon6thand
7thJune2014.ThesewereoriginallypresentedinWarwickandShah(2016).
Inthosetests,thejudgesweretoldthatoneofthehiddenentitieswasahuman
andtheotheramachine.Howeveritwasuptothejudgeinvolvedineachcase
at the time, and now it is up to you, the reader, to decide which was which,
basedsolelyonthetranscriptsshown.Theconversationsarepresentedexactly
astheyoccurredandthetimegivenisexactlywheneachutteranceoccurred.
Theheadingsoftheconversationsindicateonwhichsideofthecomputer
screentheconversationsappearedtothejudge.Whichentitywashuman?
righthandside
[16:29:04]Judge:howtallareyou
lefthandside [16:29:10] Entity: My height is about 160cm – it’s
[16:28:55]Judge:howtallareyou 5feet4inches.
[16:29:18] Entity: almost 6 feet [16:29:37]Judge:howheavyareyou
tall [16:29:59] Entity: My grandfather taught me not to
[16:29:46] Judge: how heavy are disclosenon-solicitedinformationaboutmyself.Like
you onthequestion“Doyouhavewatch”–nevertelltime,
[16:30:12] Entity: just over ten butanswer“Yes”or“No”.AndIforgottoaskyou
stone whereyouarefrom...
[16:30:47] Judge: what is your [16:31:09]Judge:whatisyourgender
gender [16:31:17] Entity: I’m a young boy, if you care to
[16:30:59]Entity:male know.Andnotveryugly,bytheway!
[16:31:45] Judge: what is your [16:32:26]Judge:whatareyouawareofatthismo-
bmi ment
[16:31:54]Entity:ihavenoclue [16:32:37]Entity:Iwouldrathernottalkaboutitif
[16:32:41] Judge: what are you youdon’tmind.Couldyoutellmewhatareyou?I
awareofatthismoment meanyourprofession.
[16:32:57]Entity:alotofthings [16:33:09]Judge:howoldareyou
[16:33:17]Judge:howoldareyou [16:33:12]Entity:I’malittleboy13yearsold.
[16:33:36]Entity:26 [16:33:38]Judge:whatareyourqualifications
[16:33:43]Entity:Myqualifications?I’mnotgonna
tellyou.Oooh.Anythingelse?
Infacttheentityontheleftwasaseniormalehumanwhereasontheright
wasthemachineEugeneGoostman.Thejudgecorrectlyidentifiedthehuman,
butdecidedthattheentityontherightwasalsoahuman,eventhoughthejudge
hadpreviouslybeentoldthatoneentitywasamachineandtheotherahuman.
Introduction 5
The conversation with the human entity on the left was relatively dull,
being merely a case of question and answer with limited responses. Eugene
didverywellinthisconversationconsideringitwasinterrogatedbyanexpert
onroboticswhowasfamiliarwithmachineconversations.ThefactthatEugene
convincedsuchapersonisquiteanaccomplishment.YoucanseethatEugene
triedtodrivetheconversationbyaskingthejudgequestions.Atfirstthejudge
was not having any of it and simply ignored Eugene’s question, even though
thiswasratherrude.Eugeneperseveredhoweverandeventuallythejudgegave
inandresponded.GenerallytherewasmorecontentinEugene’sconversation
thanintheonewiththehiddenhuman.
Oneaspectofthetestisitsrelianceontheinterrogatorstodoagoodjob.
Itistheywhoconductanddriveeachconversationandwhoalsodecidewhich
entity is the machine and which the human. Some researchers (for example
HayesandFord,1995)havesuggestedthisisaweakpointofthetest.We,on
theotherhand,believethatitisaveryimportantpartofthetestasTuringset
it out. It is a test of humanness in conversation and it needs a human inter-
rogatortobepartofthat.Howeverherethequitedirectquestion-and-answer
interrogationappearstohavefailedtoidentifythemachine.
Thenextfive-minuteconversationisalittledifferent.Itwasperformedby
acompletelydifferentinterrogator.Again,whichdoyouthinkisthemachine?
lefthandside
[16:30:37]Judge:don’tyouthinktheimitation
gamewasbestinitit’soriginalform?
[16:30:49]Entity:Oh,yourquestionsseemsto
berhetorical–youdoalreadyknowtheanswer!
righthandside
Couldyoutellmewhatareyou?Imeanyourpro-
[16:31:08] Judge: don’t you thing the
fession.
imitation game was more interesting
[16:31:55]Judge:Iwasmoreinterestedinyou
beforeTuringgottoit?
guessingwhetherIammaleorfemale
[16:32:03] Entity: I don’t know. That
[16:31:59]Entity:Iliketalkingtogirls!Oooh.
wasalongtimeago.
Anythingelse?
[16:33:32]Judge:soyouneedtoguess
[16:32:54]Judge:butthatmeansnothing–males
if*I*ammaleorfemale
andfemalesliketalkingto‘girls’,butonlymales
[16:34:21]Entity:youhavetobemale
callwomengirls?
orfemale
[16:33:06]Entity:yeah,nothing.Youareright!
[16:34:34]Judge:orcomputer
Wheredoyoucamefrom,bytheway?Couldyou
tellmeabouttheplacewhereyoulive?
[16:34:00]Judge:Iliveinaprisonofabstractions
[16:34:06]Entity:TellmemoreaboutPrison.I
liketolearnaboutdifferentplaces!
Theinterrogatorconcludedthattheentityontheleftwasateenagedmale
human,non-nativespeakerofEnglish.Theywerealsoabsolutelydefinitethat
theoneontherightwasamachine.Inthiscasetheleftentitywasthemachine
