Rationality: From AI to Zombies

1580 Pages·2015·9.05 MB·English

by  Eliezer Yudkowsky

Rationality
From AI to Zombies

Eliezer Yudkowsky
Eliezer Yudkowsky is a Research Fellow at the Machine
Intelligence Research Institute.
Written by Eliezer Yudkowsky
Published in 2015
Machine Intelligence Research Institute
Berkeley 94704
United States of America
intelligence.org
Released under the Creative Commons Attribution-
NonCommercial-ShareAlike 3.0 Unported license.
CC BY-NC-SA 3.0
-10: 1-939311-15-2
ISBN
-13: 978-1-939311-15-3
ISBN
( )
EPUB
The Machine Intelligence Research Institute gratefully
acknowledges the generous support of all those involved in the
publication of this book, and the donors who support MIRI’s work
in general.
Cover by Tim Woolley.
Contents
Preface
Biases: An Introduction
Book I: Map and Territory
A. Predictably Wrong
1. What Do I Mean By “Rationality”?
2. Feeling Rational
3. Why Truth? And . . .
4. . . . What’s a Bias, Again?
5. Availability
6. Burdensome Details
7. Planning Fallacy
8. Illusion of Transparency: Why No One Understands You
9. Expecting Short Inferential Distances
10. The Lens That Sees Its Own Flaws
B. Fake Beliefs
11. Making Beliefs Pay Rent (in Anticipated Experiences)
12. A Fable of Science and Politics
13. Belief in Belief
14. Bayesian Judo
15. Pretending to be Wise
16. Religion’s Claim to be Non-Disprovable
17. Professing and Cheering
18. Belief as Attire
19. Applause Lights
C. Noticing Confusion
20. Focus Your Uncertainty
21. What Is Evidence?
22. Scientific Evidence, Legal Evidence, Rational Evidence
23. How Much Evidence Does It Take?
24. Einstein’s Arrogance
25. Occam’s Razor
26. Your Strength as a Rationalist
27. Absence of Evidence Is Evidence of Absence
28. Conservation of Expected Evidence
29. Hindsight Devalues Science
D. Mysterious Answers
30. Fake Explanations
31. Guessing the Teacher’s Password
32. Science as Attire
33. Fake Causality
34. Semantic Stopsigns
35. Mysterious Answers to Mysterious Questions
36. The Futility of Emergence
37. Say Not “Complexity”
38. Positive Bias: Look into the Dark
39. Lawful Uncertainty
40. My Wild and Reckless Youth
41. Failing to Learn from History
42. Making History Available
43. Explain/Worship/Ignore?
44. “Science” as Curiosity-Stopper
45. Truly Part of You
Interlude: The Simple Truth
Book II: How to Actually Change Your Mind
Rationality: An Introduction
E. Overly Convenient Excuses
46. The Proper Use of Humility
47. The Third Alternative
48. Lotteries: A Waste of Hope
49. New Improved Lottery
50. But There’s Still a Chance, Right?
51. The Fallacy of Gray
52. Absolute Authority
53. How to Convince Me That 2 + 2 = 3
54. Infinite Certainty
55. 0 And 1 Are Not Probabilities
56. Your Rationality Is My Business
F. Politics and Rationality
57. Politics is the Mind-Killer
58. Policy Debates Should Not Appear One-Sided
59. The Scales of Justice, the Notebook of Rationality
60. Correspondence Bias
61. Are Your Enemies Innately Evil?
62. Reversed Stupidity Is Not Intelligence
63. Argument Screens Off Authority
64. Hug the Query
65. Rationality and the English Language
66. Human Evil and Muddled Thinking
G. Against Rationalization
67. Knowing About Biases Can Hurt People
68. Update Yourself Incrementally
69. One Argument Against An Army
70. The Bottom Line
71. What Evidence Filtered Evidence?
72. Rationalization
73. A Rational Argument
74. Avoiding Your Belief’s Real Weak Points
75. Motivated Stopping and Motivated Continuation
76. Fake Justification
77. Is That Your True Rejection?
78. Entangled Truths, Contagious Lies
79. Of Lies and Black Swan Blowups
80. Dark Side Epistemology
H. Against Doublethink
81. Singlethink
82. Doublethink (Choosing to be Biased)
83. No, Really, I’ve Deceived Myself
84. Belief in Self-Deception
85. Moore’s Paradox
86. Don’t Believe You’ll Self-Deceive
I. Seeing with Fresh Eyes
87. Anchoring and Adjustment
88. Priming and Contamination
89. Do We Believe Everything We’re Told?
90. Cached Thoughts
91. The “Outside the Box” Box
92. Original Seeing
93. Stranger than History
94. The Logical Fallacy of Generalization from Fictional Evidence
95. The Virtue of Narrowness
96. How to Seem (and Be) Deep
97. We Change Our Minds Less Often Than We Think
98. Hold Off On Proposing Solutions
99. The Genetic Fallacy
J. Death Spirals
100. The Affect Heuristic
101. Evaluability (and Cheap Holiday Shopping)
102. Unbounded Scales, Huge Jury Awards, and Futurism
103. The Halo Effect
104. Superhero Bias
105. Mere Messiahs
106. Affective Death Spirals
107. Resist the Happy Death Spiral
108. Uncritical Supercriticality
109. Evaporative Cooling of Group Beliefs
110. When None Dare Urge Restraint
111. The Robbers Cave Experiment
112. Every Cause Wants to Be a Cult
113. Guardians of the Truth
114. Guardians of the Gene Pool
115. Guardians of Ayn Rand
116. Two Cult Koans
117. Asch’s Conformity Experiment
118. On Expressing Your Concerns
119. Lonely Dissent
120. Cultish Countercultishness
K. Letting Go
121. The Importance of Saying “Oops”
122. The Crackpot Offer
123. Just Lose Hope Already
124. The Proper Use of Doubt
125. You Can Face Reality
126. The Meditation on Curiosity
127. No One Can Exempt You From Rationality’s Laws
128. Leave a Line of Retreat
129. Crisis of Faith
130. The Ritual
Book III: The Machine in the Ghost
Minds: An Introduction
Interlude: The Power of Intelligence
L. The Simple Math of Evolution
131. An Alien God
132. The Wonder of Evolution
133. Evolutions Are Stupid (But Work Anyway)
134. No Evolutions for Corporations or Nanodevices
135. Evolving to Extinction
136. The Tragedy of Group Selectionism
137. Fake Optimization Criteria
138. Adaptation-Executers, Not Fitness-Maximizers
139. Evolutionary Psychology
140. An Especially Elegant Evolutionary Psychology Experiment
141. Superstimuli and the Collapse of Western Civilization
142. Thou Art Godshatter