Table Of ContentContents
1. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2. AnIntroduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1 TheEvolutionofPython . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Python2vs Python3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 ThePythonProgrammingLanguage . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. AVeryShortTutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.1 UsingPython . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.2 PythonStatements,LineStructureandIndentation . . . . . . . . . . . . . . . . . 6
3.3 Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.4 FlowControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.5 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.6 DataStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.7 Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.8 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.9 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.10 InputandOutput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.11 GettingHelp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4. Intermezzo:Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.1 NamesandBinding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2 CodeBlocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3 Name-spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4 Scopes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.5 eval() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.6 exec() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5. Objects201 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.1 Strong andWeak ObjectReferences . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.2 TheTypeHierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
NoneType . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
NotImplementedType . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
EllipsisType . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
CONTENTS
Numeric Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
SequenceType . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
CallableTypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
CustomType . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
ModuleType . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
File/IOTypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Built-inTypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6. ObjectOrientedProgramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.1 TheMechanicsofClassDefinitions . . . . . . . . . . . . . . . . . . . . . . . . . 50
ClassObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
InstanceObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
MethodObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.2 CustomizingUser-definedTypes . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
SpecialmethodsforTypeEmulation. . . . . . . . . . . . . . . . . . . . . . . . . 58
SpecialMethodsforcomparingobjects . . . . . . . . . . . . . . . . . . . . . . . 62
SpecialMethodsandAttributesforMiscellaneousCustomizations . . . . . . . . 63
6.3 AVectorclass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.4 Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Thesuperkeyword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
MultipleInheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6.5 StaticandClassMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
StaticMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
ClassMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.6 DescriptorsandProperties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
EnterPythonDescriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
ClassProperties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.7 AbstractBaseClasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
7. TheFunction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
7.1 FunctionDefinitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
7.2 FunctionsareObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7.3 Functionsaredescriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
7.4 CallingFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
UnpackingFunctionArgument . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
*and**FunctionParameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
7.5 NestedfunctionsandClosures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
7.6 AByteofFunctionalProgramming . . . . . . . . . . . . . . . . . . . . . . . . . 95
TheBasics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Comprehensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Functools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
CONTENTS
SequencesandFunctionalProgramming . . . . . . . . . . . . . . . . . . . . . . 102
8. IteratorsandGenerators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
8.1 Iterators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
TheItertoolsModule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
8.2 Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
GeneratorFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
GeneratorExpressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
TheBeautyofGeneratorsandIterators . . . . . . . . . . . . . . . . . . . . . . . 112
8.3 FromGeneratorsToCoroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
SimulatingMultitaskingwithCoroutines . . . . . . . . . . . . . . . . . . . . . . 115
8.4 Theyield fromkeyword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
8.5 AGameofLife . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
9. MetaProgrammingandCo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
9.1 Decorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
FunctionDecorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
DecoratorsinPython . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
PassingArgumentsToDecoratedFunctions . . . . . . . . . . . . . . . . . . . . 126
DecoratorFunctionswithArguments . . . . . . . . . . . . . . . . . . . . . . . . 126
Functools.wrap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
ClassDecorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
9.2 DecoratorRecipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
9.3 Metaclasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
MetaclassesinAction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Overriding__new__vs__init__inCustomMetaclasses . . . . . . . . . . . . . . 142
9.4 ContextManagers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
TheContextlibmodule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
10. ModulesAndPackages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
10.1 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
ReloadingModules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
10.2 HowareModulesfound? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
10.3 Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
RegularPackages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
NamespacePackages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
10.4 TheImportSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
TheImportSearchProcess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
WhyYouProbablyShouldNotReloadModules… . . . . . . . . . . . . . . . . . 157
10.5 DistributingPythonPrograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
11. InspectingObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
11.1 Handlingsourcecode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
11.2 InspectingClassesandFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
CONTENTS
11.3 InteractingwithInterpreterStacks . . . . . . . . . . . . . . . . . . . . . . . . . . 167
12. TheZenofPython… . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
1. Acknowledgements
I would love to take the opportunity to thank all who have reviewed and spotted issues in the
manuscript. This includes but is not limited to Ngozi Nwosu for taking the time out to review the
whole manual and point out a whole load of grammatical errors, Olivia Enewally, Roman Turna
andAbhenNgforpointingoutsomefactualandgrammaticalerrors.Awholelotofotherpeopleon
ReddithavepointedouterrorsandtothosepeopleIamreallygrateful.
Withouttheinputofallthesepeople,thismanuscriptwouldbeworthlessthanitcurrentlyis.Thank
youall!
2. An Introduction
The Python Programming language has been around for quite a while. Development work was
started on the first version of Python by Guido Van Rossum in 1989. Since then, it has grown to
become a highly loved and revered language that has been used and continues to be used in a host
ofdifferentapplicationtypes.
The Python interpreter and the extensive standard library that come with the interpreter are
available for free in source or binary form for all major platforms from the Python Web site¹. This
site also contains distributions of and pointers to many free third party Python modules, programs
andtools,andadditionaldocumentation.
ThePythoninterpretercaneasilybeextendedwithnewfunctionsanddatatypesimplementedinC,
C++oranyotherlanguagethatiscallablefromC.Pythonisalsosuitableasanextensionlanguage
forcustomisableapplications.Oneofthemostnotablefeatureofpythonistheeasyandwhite-space
awaresyntax.
ThisbookisintendedasaconciseintermediateleveltreatiseonthePythonprogramminglanguage.
There is a need for this due to the lack of availability of materials for python programmers at this
level. The material contained in this book is targeted at the programmer that has been through a
beginner level introduction to the Python programming language or that has some experience in
a different object oriented programming language such as Java and wants to gain a more in-depth
understanding of the Python programming language in a holistic manner. It is not intended as an
introductorytutorialforbeginnersalthoughprogrammerswithsomeexperienceinotherlanguages
mayfindtheveryshorttutorialincludedinstructive.
The book covers only a handful of topics but tries to provide a holistic and in-depth coverage
of these topics. It starts with a short tutorial introduction to get the reader up to speed with the
basicsofPython;experiencedprogrammersfromotherobjectorientedlanguagessuchasJavamay
find that this is all the introduction to Python that they need. This is followed by a discussion of
the Python object model then it moves on to discussing object oriented programming in Python.
With a firm understanding of the Python object model, it goes ahead to discuss functions and
functional programming. This is followed by a discussion of meta-programming techniques and
theirapplications.Theremainingchapterscovergenerators,acomplexbutveryinterestingtopicin
Python,modulesandpackaging,andpythonruntimeservices.Inbetween,intermezzosareusedto
discusstopicsareworthknowingbecauseoftheaddedunderstandingtheyprovide.
I hope the content of the book achieves the purpose for the writing of this book. I welcome all
feedbackreadersmayhaveandactivelyencouragereaderstoprovidesuchfeedback.
¹https://www.python.org/
AnIntroduction 3
2.1 The Evolution of Python
In December 1989, Guido Van Rossum started work on a language that he christened Python.
Guido Van Rossum had been part of the team that worked on the ABC programming language
aspartoftheAmoebaoperatingsystemsinthe1980satCWI(CentrumWiskunde&Informatica)in
AmsterdamandalthoughhelikedtheABCprogramminglanguage,hewasfrustratedbyanumber
of features or lack of thereof. Guido wanted a high level programming language that would speed
up the development of utilities for the the Amoeba project and ABC was not the answer. The ABC
programming language would however play a very influential role in the development of python
as Guido took parts he liked from the language and provided solutions for aspects of the ABC
programminglanguagethathefoundfrustrating.
GuidoreleasedthefirstversionofthePythonprogramminglanguageinFebruary1991.Thisrelease
wasobjectoriented,hadamodulesystem,includedexceptionhandling,functions,andthecoredata
typesof list,dict,strandothers.Pythonversion1.0wasreleasedinJanuary1994andthisrelease
includedfunctionalprogrammingconstructssuchaslambda,map,filterandreduce.
Python1.2wasthelastversionreleasedwhileGuidowasstillatCWI.In1995,VanRossumcontinued
hisworkonPythonattheCorporationforNationalResearchInitiatives(CNRI)inReston,Virginia
wherehereleasedseveralversionsofthelanguagewithindirectfundingsupportfromDARPRA.
Byversion1.4,PythonhadacquiredseveralnewfeaturesincludingtheModula-3inspiredkeyword
arguments and built-in support for complex numbers. It also included a basic form of data hiding
by name mangling. Python 1.5 was released on December 31, 1997 while python 1.6 followed on
September5,2000.
Python 2.0 was released on October 16, 2000 and it introduced list comprehensions, a feature
borrowed from the functional programming languages SETL and Haskell as well as a garbage
collectionsystemcapableofcollectingreferencecycles.
Python 2.2 was the first major update to the Python type system. This update saw the unification
ofPython’sin-builttypesanduserdefinedclasseswritteninPythonintoonehierarchy.Thissingle
unificationmadePython’sobjectmodelpurelyandconsistentlyobjectoriented.Thisupdatetothe
class system of Python added a number of features that improved the programming experience.
Theseincluded:
1. Theabilitytosubclassinbuilttypessuchasdictsandlists.
2. Theadditionofstaticandclassmethods
3. Theadditionofpropertiesdefinedbygetandsetmethods.
4. Anupdatetometa-classes,__new__()andsuper()methods,andMROalgorithms.
ThenextmajormilestonewasPython3releasedonDecember2,2008.Thiswasdesignedtorectify
certainfundamentaldesignflawsinthelanguagethatcouldnotbeimplementedwhilemaintaining
fullbackwardscompatibilitywiththe2.xseries.
AnIntroduction 4
2.2 Python 2 vs Python 3
Perhaps the most visible and disruptive change to the Python ecosystem has been the introduction
ofPython3.ThemajorchangesintroducedintoPython3includethefollowing:
1. print()isnowafunction
2. Some well known python APIs such as range(), dict.keys(), dict.values() return views
anditeratorsratherthanlistsimprovingefficiencywhentheyareused.
3. The rules for ordering comparisons have been simplified. For example, a heterogeneous list
cannotbesorted,becausealltheelementsofalistmustbecomparabletoeachother.
4. Theintegertypeshavebeenwhittleddowntoonlyone,i.e.int.longisalsoanint.
5. The division of two integers returns a float instead of an integer.// can be used to return an
integerwhendivisiontakesplace.
6. All texts are now unicode but encoded unicode text is represented as binary data and any
attempttomixtextanddatawillresultinanexception.Thisbreaksbackwardscompatibility
withpython2.xversions.
7. Python 3 also saw the introduction of some new syntax such as function annotations, the
nonlocalstatement,extendediterableunpacking,setliterals,dictionarycomprehensionsetc.
8. Python 3 also saw update to some syntax such as that of exception handling, meta-class
specification,listcomprehensionsetc.
The full details on changes from python 2 to python 3 can be viewed on the python website². The
restofthebookwillassumestheuseofPython3.4.
2.3 The Python Programming Language
ThePythonprogramminglanguagereferstothelanguageasdocumentedinthelanguagereference³.
There is no official language specification but the language reference provides enough details to
guideanyoneimplementingthePythonprogramminglanguage.TheimplementationofthePython
programming language available on the Python website is an implementation written in C and
commonlyreferredtoasCPython.Thisisnormallyusedasthereferenceimplementation.However,
there are other Python implementations in different languages. Popular among these are PyPy:
python implemented in python and Jython: python implemented in Java. For the rest of this book,
thereferenceCPython versionthatisfreelydistributedthroughthePythonwebsite⁴isused.
²https://docs.python.org/3.0/whatsnew/3.0.html
³https://docs.python.org/3.4/reference/
⁴https://python.org
3. A Very Short Tutorial
This short tutorial introduces the reader informally to the basic concepts and features of the
Python programming language. It is not intended to be a comprehensive introductory tutorial to
programming for a complete novice but rather assumes the reader has had previous experience
programmingwithanobjectorientedprogramminglanguage.
3.1 Using Python
PythonisinstalledbydefaultonmostUnix-basedsystemsincludingtheMacOSandvariousLinux-
based distributions. To check if python is installed, open the command-line and type python. If
not installed then python can be installed by visiting the python language website¹ and following
instructionsonhowtoinstallpythonforthegivenplatform.
The Python Interpreter
The python interpreter can be invoked by opening the command-line and typing in python. In the
casethatithasbeeninstalledbutthecommandcannotbefoundthenthefullpathtotheinstallation
should be used or added to the path. Invoking the interpreter using python brings up the python
interactivesessionwithanREPLprompt.Theprimaryprompt,>>>,signalsausertoenterstatements
whilethesecondaryprompt,...,signalsacontinuationlineasshowninthefollowingexample.
>>> def hello():
... print("Hello world")
...
>>>
Ausercantypeinpythonstatementsattheinterpreterpromptandgetinstantfeedback.Forexample,
we can evaluate expressions at the REPL and get values for such expressions as in the following
example.
¹https://www.python.org/
