Table Of Contenti
titlepage
ii
copyrightpage
Tomyfamily,andtoJackie.
iv
λ
Preface
ThisbookisintendedforanyonewhowantstobecomeabetterLispprogrammer.
ItassumessomefamiliaritywithLisp,butnotnecessarilyextensiveprogramming
experience. Thefirst few chapterscontaina fair amountof review. I hopethat
thesesectionswillbeinterestingtomoreexperiencedLispprogrammersaswell,
becausetheypresentfamiliarsubjectsinanewlight.
It’sdifficulttoconveytheessenceofaprogramminglanguageinonesentence,
butJohnFoderarohascomeclose:
Lispisaprogrammableprogramminglanguage.
There is more to Lisp than this, but the ability to bend Lisp to one’s will is a
largepartofwhatdistinguishesaLispexpertfromanovice. Aswellaswriting
theirprogramsdowntowardthelanguage,experiencedLispprogrammersbuild
thelanguageuptowardtheirprograms. Thisbookteacheshowtoprograminthe
bottom-upstyleforwhichLispisinherentlywell-suited.
Bottom-up Design
Bottom-updesignisbecomingmoreimportantassoftwaregrowsincomplexity.
Programstodaymay haveto meet specificationswhich are extremelycomplex,
orevenopen-ended. Undersuchcircumstances,thetraditionaltop-downmethod
sometimes breaksdown. In its place therehas evolveda style of programming
v
vi PREFACE
quite differentfromwhat is currentlytaughtin most computerscience courses:
a bottom-upstyle in which a programis written as a series of layers, each one
actingasasortofprogramminglanguagefortheoneabove. XWindowsandTEX
areexamplesofprogramswritteninthisstyle.
Thethemeofthisbookistwofold: thatLispisanaturallanguageforprograms
written in the bottom-upstyle, and that the bottom-upstyle is a natural way to
writeLisp programs. OnLisp will thusbeofinterestto twoclasses ofreaders.
For people interested in writing extensibleprograms, this bookwill show what
youcandoifyouhavetherightlanguage. ForLispprogrammers,thisbookoffers
apracticalexplanationofhowtouseLisptoitsbestadvantage.
The title is intended to stress the importanceof bottom-upprogrammingin
Lisp. Instead of just writing your program in Lisp, you can write your own
languageonLisp,andwriteyourprograminthat.
It is possible to write programs bottom-upin any language, but Lisp is the
mostnaturalvehicleforthisstyleofprogramming. InLisp,bottom-updesignis
not a special techniquereservedfor unusuallylargeor difficult programs. Any
substantialprogramwillbewrittenpartlyinthisstyle. Lispwasmeantfromthe
start tobean extensiblelanguage. Thelanguageitself is mostlya collectionof
Lispfunctions,nodifferentfromtheonesyoudefineyourself. What’smore,Lisp
functionscan be expressedas lists, which are Lisp datastructures. This means
youcanwriteLispfunctionswhichgenerateLispcode.
AgoodLispprogrammermustknowhowtotakeadvantageofthispossibility.
Theusualwaytodosoisbydefiningakindofoperatorcalleda macro. Mastering
macros is one of the most importantsteps in movingfrom writing correctLisp
programsto writing beautiful ones. IntroductoryLisp books have room for no
morethanaquickoverviewofmacros:anexplanationofwhatmacrosare,together
withafewexampleswhichhintatthestrangeandwonderfulthingsyoucando
withthem. Thosestrangeandwonderfulthingswillreceivespecialattentionhere.
Oneoftheaimsofthisbookistocollectinoneplaceallthatpeoplehavetillnow
hadtolearnfromexperienceaboutmacros.
Understandably, introductory Lisp books do not emphasize the differences
between Lisp and other languages. They have to get their message across to
studentswhohave,forthemostpart,beenschooledtothinkofprogramsinPascal
terms. It would only confusematters to explain that, while defun looks like a
proceduredefinition,itisactuallyaprogram-writingprogramthatgeneratescode
whichbuildsafunctionalobjectandindexesitunderthesymbolgivenasthefirst
argument.
OneofthepurposesofthisbookistoexplainwhatmakesLispdifferentfrom
otherlanguages. WhenIbegan,Iknewthat,allotherthingsbeingequal,Iwould
muchratherwriteprogramsinLispthaninCorPascalorFortran. Iknewalsothat
thiswasnotmerelyaquestionoftaste. ButIrealizedthatifIwasactuallygoing
PREFACE vii
toclaimthatLispwasinsomewaysabetterlanguage,Ihadbetterbepreparedto
explainwhy.
WhensomeoneaskedLouisArmstrongwhatjazzwas,hereplied“Ifyouhave
toaskwhatjazzis,you’llneverknow.” Buthedidanswerthequestioninaway:
heshowedpeoplewhatjazzwas. That’sonewaytoexplainthepowerofLisp—to
demonstratetechniquesthatwouldbedifficultorimpossibleinotherlanguages.
Mostbooksonprogramming—evenbooksonLispprogramming—dealwiththe
kindsofprogramsyoucouldwriteinanylanguage. OnLisp dealsmostlywith
the kinds of programs you could only write in Lisp. Extensibility, bottom-up
programming, interactive development, source code transformation, embedded
languages—thisiswhereLispshowstoadvantage.
Inprinciple,ofcourse,anyTuring-equivalentprogramminglanguagecando
the same things as anyother. But that kindof poweris not what programming
languages are about. In principle, anything you can do with a programming
languageyoucandowithaTuringmachine;inpractice,programmingaTuring
machineisnotworththetrouble.
So when I say that this book is about how to do things that are impossible
inotherlanguages,Idon’tmean“impossible”inthemathematicalsense,butin
the sense that matters for programminglanguages. That is, if you had to write
someoftheprogramsinthisbookinC,youmightaswelldoitbywritingaLisp
compilerinCfirst. EmbeddingProloginC,forexample—canyouimaginethe
amountofworkthatwouldtake? Chapter24showshowtodoitin180linesof
Lisp.
IhopedtodomorethansimplydemonstratethepowerofLisp,though. Ialso
wantedtoexplainwhyLispisdifferent. Thisturnsouttobeasubtlequestion—too
subtle to be answered with phrases like “symbolic computation.” What I have
learnedsofar,IhavetriedtoexplainasclearlyasIcan.
PlanoftheBook
Sincefunctionsarethe foundationofLispprograms,thebookbeginswith sev-
eral chapters on functions. Chapter 2 explains what Lisp functions are and the
possibilities they offer. Chapter 3 then discusses the advantages of functional
programming,thedominantstyleinLispprograms. Chapter4showshowtouse
functionstoextendLisp. ThenChapter5suggeststhenewkindsofabstractions
wecandefinewithfunctionsthatreturnotherfunctions. Finally,Chapter6shows
howtousefunctionsinplaceoftraditionaldatastructures.
Theremainderofthebookdealsmorewith macrosthanfunctions. Macros
receivemoreattentionpartlybecausethereismoretosayaboutthem,andpartly
becausetheyhavenottillnowbeenadequatelydescribedinprint. Chapters7–10
viii PREFACE
formacompletetutorialonmacrotechnique. Bytheendofityouwillknowmost
ofwhatanexperiencedLispprogrammerknowsaboutmacros: howtheywork;
howtodefine,test,anddebugthem;whentousemacrosandwhennot;themajor
typesofmacros;howtowriteprogramswhichgeneratemacroexpansions;how
macrostylediffersfromLispstyleingeneral;andhowtodetectandcureeachof
theuniqueproblemsthatafflictmacros.
Followingthis tutorial, Chapters11–18showsomeofthe powerfulabstrac-
tions you can build with macros. Chapter 11 shows how to write the classic
macros—thosewhichcreatecontext,orimplementloopsorconditionals. Chap-
ter12explainstheroleofmacrosinoperationsongeneralizedvariables. Chap-
ter13showshowmacroscanmakeprogramsrunfasterbyshiftingcomputation
to compile-time. Chapter 14 introduces anaphoricmacros, which allow you to
usepronounsinyourprograms. Chapter15showshowmacrosprovideamore
convenient interface to the function-buildersdefined in Chapter 5. Chapter 16
showshowtousemacro-definingmacrostomakeLispwriteyourprogramsfor
you. Chapter17discussesread-macros,andChapter18,macrosfordestructuring.
With Chapter 19 begins the fourth part of the book, devoted to embedded
languages. Chapter 19 introduces the subject by showing the same program, a
program to answer queries on a database, implemented first by an interpreter
and then as a true embedded language. Chapter 20 shows how to introduce
intoCommonLispprogramsthenotionofacontinuation,anobjectrepresenting
the remainder of a computation. Continuations are a very powerful tool, and
canbe usedto implementbothmultipleprocessesandnondeterministicchoice.
Embeddingthese controlstructures in Lisp is discussed in Chapters 21 and22,
respectively. Nondeterminism, which allows you to write programs as if they
hadforesight,soundslikeanabstractionofunusualpower. Chapters23and24
presenttwoembeddedlanguageswhichshowthatnondeterminismlivesuptoits
promise: acompleteATNparserandanembeddedPrologwhichcombinedtotal
◦ about200linesofcode.
Thefactthattheseprogramsareshortmeansnothinginitself.Ifyouresortedto
writingincomprehensiblecode,there’snotellingwhatyoucoulddoin200lines.
Thepointis,theseprogramsarenotshortbecausetheydependonprogramming
tricks,butbecausethey’rewrittenusingLispthewayit’smeanttobeused. The
pointofChapters23and24is nothowtoimplementATNs inonepageofcode
orPrologintwo,buttoshowthattheseprograms,whengiventheirmostnatural
Lispimplementation,simplyarethatshort. Theembeddedlanguagesinthelatter
chaptersprovideaproofbyexampleofthetwinpointswithwhichIbegan: that
Lispis a naturallanguageforbottom-updesign,andthatbottom-updesignis a
naturalwaytouseLisp.
Thebookconcludeswith a discussionof object-orientedprogramming,and
particularly CLOS, the Common Lisp Object System. By saving this topic till
PREFACE ix
last, we see more clearly the way in which object-oriented programmingis an
extensionofideasalreadypresentinLisp. Itisoneofthemanyabstractionsthat
canbebuiltonLisp.
Achapter’sworthofnotesbeginsonpage387. Thenotescontainreferences,
additionaloralternativecode,ordescriptionsofaspectsofLispnotdirectlyrelated
tothepointathand. Notesareindicatedbyasmallcircleintheoutsidemargin,
likethis. ThereisalsoanAppendix(page381)onpackages. ◦
JustasatourofNewYorkcouldbeatourofmostoftheworld’scultures,a
studyofLispastheprogrammableprogramminglanguagedrawsinmostofLisp
technique. MostofthetechniquesdescribedherearegenerallyknownintheLisp
community,butmanyhavenottillnowbeenwrittendownanywhere. Andsome
issues,suchastheproperroleofmacrosorthenatureofvariablecapture,areonly
vaguelyunderstoodevenbymanyexperiencedLispprogrammers.
Examples
Lispisafamilyoflanguages. SinceCommonLisppromisestoremainawidely
useddialect,mostoftheexamplesinthisbookareinCommonLisp. Thelanguage
wasoriginallydefinedin1984bythepublicationofGuySteele’sCommonLisp:
theLanguage(CLTL1). Thisdefinitionwassupersededin1990bythepublication
ofthesecondedition(CLTL2),whichwillinturnyieldplacetotheforthcoming ◦
ANSIstandard.
Thisbookcontainshundredsofexamples,rangingfromsingleexpressionsto
aworkingPrologimplementation. Thecodeinthisbookhas,whereverpossible,
beenwrittentoworkinanyversionofCommonLisp. Thosefewexampleswhich
needfeaturesnotfoundinCLTL1implementationsareexplicitlyidentifiedinthe
text. Later chapters contain some examples in Scheme. These too are clearly
identified.
ThecodeisavailablebyanonymousFTPfromendor.harvard.edu,where
it’s in the directory pub/onlisp. Questions and comments can be sent to
[email protected].
Acknowledgements
WhilewritingthisbookIhavebeenparticularlythankfulforthehelpofRobert
Morris. IwenttohimconstantlyforadviceandwasalwaysgladIdid. Several
oftheexamplesinthisbookarederivedfromcodeheoriginallywrote,including
the version of for on page 127, the version of aand on page 191, match on
page239,thebreadth-firsttrue-chooseonpage304,andtheProloginterpreter
x PREFACE
inSection24.2. Infact,thewholebookreflects(sometimes,indeed,transcribes)
conversationsI’vehadwithRobertduringthepastsevenyears. (Thanks,rtm!)
IwouldalsoliketogivespecialthankstoDavidMoon,whoreadlargeparts
ofthemanuscriptwithgreatcare,andgavemeveryusefulcomments. Chapter12
wascompletelyrewrittenathissuggestion,andtheexampleofvariablecapture
onpage119isonethatheprovided.
Iwas fortunatetohaveDavidTouretzkyandSkonaBrittainas thetechnical
reviewersforthebook. Severalsectionswereaddedorrewrittenattheirsugges-
tion. Thealternativetruenondeterministicchoiceoperatoronpage397isbased
onasuggestionbyDavidToureztky.
Severalotherpeopleconsentedtoreadallorpartofthemanuscript,including
Tom Cheatham, Richard Draves (who also rewrote alambda and propmacro
back in 1985), John Foderaro, David Hendler, George Luger, Robert Muller,
MarkNitzberg,andGuySteele.
I’mgratefultoProfessorCheatham,andHarvardgenerally,forprovidingthe
facilitiesusedtowritethisbook. ThanksalsotothestaffatAikenLab,including
TonyHartman,JanuszJuda,HarryBochner,andJoanneKlys.
Thepeopleat PrenticeHall did agreat job. I feelfortunateto haveworked
with Alan Apt, a good editor and a good guy. Thanks also to Mona Pompili,
ShirleyMichaels,andShirleyMcGuirefortheirorganizationandgoodhumor.
TheincomparableGinoLeeoftheBowandArrowPress,Cambridge,didthe
cover. Thetreeonthecoveralludesspecificallytothepointmadeonpage27.
ThisbookwastypesetusingLATEX,alanguagewrittenbyLeslieLamportatop
DonaldKnuth’s TEX, with additionalmacrosby L. A. Carr, Van Jacobson, and
Guy Steele. The diagrams were done with Idraw, by John Vlissides and Scott
Stanton. The whole was previewedwith Ghostview, by Tim Theisen, which is
builtonGhostscript,byL.PeterDeutsch. GaryBisbeeofChironInc.produced
thecamera-readycopy.
I owe thanks to many others, including Paul Becker, Phil Chapnick, Alice
Hartley, Glenn Holloway, Meichun Hsu, Krzysztof Lenk, Arman Maghbouleh,
HowardMullings,NancyParmet,RobertPenny,GarySabot,PatrickSlaney,Steve
Strassman,DaveWatkins,theWeickers,andBillWoods.
Mostofall,I’dliketothankmyparents,fortheirexampleandencouragement;
andJackie,whotaughtmewhatImighthavelearnedifIhadlistenedtothem.
Ihopereadingthisbookwillbefun. OfallthelanguagesIknow,IlikeLisp
the best, simply because it’s the most beautiful. This book is about Lisp at its
lispiest. Ihadfunwritingit,andIhopethatcomesthroughinthetext.
PaulGraham
