The Hanson-Yudkowsky AI-Foom Debate

741 Pages·2013·2.8 MB·English

The Hanson-Yudkowsky
AI-Foom Debate
Robin Hanson and Eliezer Yudkowsky
The Hanson-Yudkowsky
AI-Foom Debate

The Hanson-Yudkowsky
AI-Foom Debate
RobinHansonandEliezerYudkowsky
MIRI
RobinHansonisanassociateprofessorofeconomicsat
GeorgeMasonUniversityandaresearchassociateatthe
FutureofHumanityInstituteofOxfordUniversity.
EliezerYudkowskyisaResearchFellowattheMachine
IntelligenceResearchInstituteandistheforemost
researcheronFriendlyAIandrecursiveself-improvement.
Publishedin2013bythe
MachineIntelligenceResearchInstitute,
Berkeley94704
UnitedStatesofAmerica
intelligence.org
ReleasedundertheCreativeCommons
Attribution-NonCommercial-ShareAlike3.0Unported
license.
CCBY-NC-SA3.0
isbn-10:1939311039
isbn-13:978-1-939311-03-0
(pdf)
TheMachineIntelligenceResearchInstitutegratefully
acknowledgeseachoftheauthorsfortheirideasand
contributionstowardthisimportanttopic.Specialthanks
toCarlShulmanandJamesMillerfortheirguestpostsin
thedebate.
Allchaptersandcommentsarewrittenbyandcopyright
theirrespectiveauthors.BookcovercreatedbyWeni
PratiwiandAlexVermeer.
iv
Contents
Contents v
Foreword viii
I Prologue
1 FundUberTool?—RobinHanson 3
2 EngelbartasUberTool?—RobinHanson 6
3 FriendlyTeams—RobinHanson 9
4 FriendlinessFactors—RobinHanson 12
5 TheWeakInsideView—EliezerYudkowsky 18
6 SettingtheStage—RobinHanson 25
7 TheFirstWorldTakeover—EliezerYudkowsky 29
8 Abstraction,NotAnalogy—RobinHanson 38
9 WhenceYourAbstractions?—EliezerYudkowsky 43
II MainSequence
10 AIGoFoom—RobinHanson 51
11 OptimizationandtheIntelligenceExplosion—Eliezer
Yudkowsky 55
12 Eliezer’sMeta-levelDeterminism—RobinHanson 65
v
Contents
13 ObservingOptimization—EliezerYudkowsky 72
14 Life’sStoryContinues—EliezerYudkowsky 81
15 EmulationsGoFoom—RobinHanson 88
16 BrainEmulationandHardTakeoff—CarlShulman 107
17 BillionDollarBots—JamesMiller 117
18 SurprisedbyBrains—EliezerYudkowsky 120
19 “Evicting”BrainEmulations—CarlShulman 130
20 Cascades,Cycles,Insight...—EliezerYudkowsky 135
21 WhenLifeIsCheap,DeathIsCheap—RobinHanson 146
22 ...Recursion,Magic—EliezerYudkowsky 152
23 Abstract/DistantFutureBias—RobinHanson 161
24 Engelbart:InsufficientlyRecursive—EliezerYudkowsky 166
25 TotalNanoDomination—EliezerYudkowsky 175
26 DreamsofAutarky—RobinHanson 189
27 TotalTechWars—RobinHanson 196
28 SingletonsRuleOK—EliezerYudkowsky 204
29 StuckInThroat—RobinHanson 216
30 DisappointmentintheFuture—EliezerYudkowsky 219
31 IHeartCyc—RobinHanson 225
32 IstheCity-ularityNear?—RobinHanson 231
33 RecursiveSelf-Improvement—EliezerYudkowsky 235
34 WhitherManufacturing?—RobinHanson 253
35 HardTakeoff—EliezerYudkowsky 258
36 TestNear,ApplyFar—RobinHanson 270
37 PermittedPossibilitiesandLocality—EliezerYudkowsky273
38 UnderconstrainedAbstractions—EliezerYudkowsky 287
39 BewareHockey-StickPlans—RobinHanson 298
40 EvolvedDesires—RobinHanson 303
41 SustainedStrongRecursion—EliezerYudkowsky 314
42 FriendlyProjectsvs.Products—RobinHanson 329
43 IsThatYourTrueRejection?—EliezerYudkowsky 333
vi
RobinHansonandEliezerYudkowsky
44 SharedAIWins—RobinHanson 339
45 ArtificialMysteriousIntelligence—EliezerYudkowsky 344
46 WrappingUp—RobinHanson 352
47 TrueSourcesofDisagreement—EliezerYudkowsky 359
48 TheBadGuyBias—RobinHanson 371
49 Disjunctions,Antipredictions,Etc.—EliezerYudkowsky 374
50 AreAIsHomoEconomicus?—RobinHanson 381
51 TwoVisionsOfHeritage—RobinHanson 385
52 TheMechanicsofDisagreement—EliezerYudkowsky 390
III Conclusion
53 WhatCoreArgument?—RobinHanson 397
54 WhatIThink,IfNotWhy—EliezerYudkowsky 402
55 NotTakingOvertheWorld—EliezerYudkowsky 409
IV Postscript
56 WeAgree:GetFroze—RobinHanson 419
57 YouOnlyLiveTwice—EliezerYudkowsky 423
58 Hanson-Yudkowsky Jane Street Debate 2011—Robin
HansonandEliezerYudkowsky 431
59 DebatingYudkowsky—RobinHanson 493
60 FoomDebate,Again—RobinHanson 499
61 AI-FoomDebateSummary—KajSotala 505
62 IntelligenceExplosionMicroeconomics—EliezerYud-
kowsky 562
Bibliography 715
vii
Foreword
Inlate2008,economistRobinHansonandAItheoristEliezerYud-
kowskyconductedanonlinedebateaboutthefutureofartificialintel-
ligence,andinparticularaboutwhethergenerallyintelligentAIswill
beabletoimprovetheirowncapabilitiesveryquickly(a.k.a.“foom”).
James Miller and Carl Shulman also contributed guest posts to the
debate.
Theoriginaldebatetookplaceinalongseriesofblogposts,which
are collected here. This book also includes a transcript of a 2011
in-person debate between Hanson and Yudkowsky on this subject,
a summary of the debate written by Kaj Sotala, and a 2013 techni-
calreportonAItakeoffdynamics(“intelligenceexplosionmicroeco-
nomics”)writtenbyYudkowsky.
Commentsfromtheauthorsareincludedattheendofeachchap-
ter,alongwithalinktotheoriginalpost. Thecuriousreaderisen-
couraged to use these links to view the original posts and all com-
viii
RobinHansonandEliezerYudkowsky
ments. This book contains minor updates, corrections, and addi-
tionalcitations.
ix