Table Of ContentCopyright © 2016 by Dr. Martin Jones
All rights reserved. This book or any portion thereof may not be
reproduced or used in any manner whatsoever without the express
written permission of the publisher except for the use of brief
quotations in a book review.
ISBN-13: 978-1539103035
ISBN-10: 153910303X
http://pythonforbiologists.com
Set in PT Serif and Source Code Pro
About the author
Martin started his programming career by learning Perl during the
course of his PhD in evolutionary biology, and started teaching other
people to program soon after. Since then he has taught introductory
programming to hundreds of biologists, from undergraduates to PIs,
and has maintained a philosophy that programming courses must be
friendly, approachable, and practical.
In his academic career, Martin mixed research and teaching at the
University of Edinburgh, culminating in a two year stint as Lecturer
in Bioinformatics. He now runs programming courses for biological
researchers as a full time freelancer.
You can get in touch with Martin at
martin@pythonforbiologists.com
Martin's other works include Python for Biologists, Advanced Python
for Biologists and Python for complete beginners.
Table of Contents
1: Introduction 1
About this book » 1
Typography and formatting » 5
Getting in touch » 8
2: Environments for developing in Python 10
Introduction » 10
What you actually write code in » 10
Integrated development environments » 14
Managing Python and packages » 19
Working in a debugger » 25
Alternatives to pdb » 57
Recap » 59
3: Organizing and sharing code 61
Introduction » 61
Working with modules » 63
Distributing code » 80
Recap » 86
4: Testing your code 88
Introduction » 88
Testing with assert » 89
Automated testing with nose » 105
Types of assertions » 119
Designing for testing » 127
Special types of testing » 131
Recap » 137
5: Performance 139
Introduction » 139
Benchmarking » 140
Benchmarking memory » 147
Profiling » 160
Guidelines for performance optimization » 179
Specific tips for improving performance » 181
Further topics in performance » 194
Recap » 197
6: Building user interfaces 198
Why do we need a user interface? » 198
A simple program » 200
A simple interactive interface » 203
A simple command line interface » 207
Command line interfaces with argparse » 216
Alternatives to argparse » 241
Configuration files » 243
Web interfaces » 250
Graphical user interfaces » 257
Logging » 269
Logging messages » 273
Logging to a file » 276
Recap » 283
Chapter 1:Introduction
1: Introduction
About this book
Biologists who have just finished learning the basics of Python often ask
me the question: “What should I learn next?” One valid answer is to dive
into the features of the Python language that might be considered more
advanced – things like comprehensions, custom classes, and exception
handling1.
Another, slightly less obvious answer is to learn about the tools and
techniques that exist in Python to make the process of developing software
easier. These tend not to be core language features, but rather a collection
of libraries, programs and approaches that solve problems that commonly
occur when building scientific software. Questions like:
• How can I write code in less time?
• How do I share my code with other people?
• How do I make my code run faster?
• How do I make sure there aren't any errors in my code?
• How do I make my programs easier to use?
cannot be answered simply by pointing out different features of the Python
language. Rather, they have to be answered by referring to the ecosystem
of libraries and tools that have accumulated around Python.
These five questions form the basis of the five main chapters in this book.
To learn how to write code more efficiently we will look at Python's choice
1 My second book, Advanced Python for Biologists, represents my attempt to guide the biological
programmer through those parts of the language.
1
Chapter 1:Introduction
of development environments. To learn how to share code, we'll investigate
Python's packaging and distribution system. To make code faster and more
reliable, we'll learn about profiling and automated testing. And to make our
code usable by other people, we'll need to build user interfaces.
These five disparate topics all have three things in common. Firstly, they
are not specific to Python: users of all programming languages struggle
with similar problems, and often the Python solution draws inspiration
from other programming communities.
Secondly, they are all large and complex – each chapter could easily form
the basis for an entire book. However, they are also topics where a little bit
of knowledge is extremely useful. It's a common rule of thumb in
programming (and elsewhere) that 20% of the features of a given tool will
allow you to do 80% of what you need1. It's my hope that this book will
teach you the 20% of each topic – performance, distribution, testing, etc. –
that you will use most often, and hence be more useful than a stack of five
more specialized books.
Finally, all of the topics in this book involve not just new package and
function names, but new ideas. To discuss performance we need to learn
about epochs and big O notation; to understand testing we need to learn
about regressions and fixtures; and to talk about sharing code we need to
learn about dependencies and namespaces. You'll find that we spend a fair
amount of time in each chapter looking at the basic ideas behind each
topic, which will make life much easier when you eventually need to search
for a feature or tool that isn't explicitly covered here.
1 This idea is called the Pareto Principle and is found to apply to many fields.
2
Chapter 1:Introduction
How to use this book
Programming books generally fall into two categories; reference-type
books, which are designed for looking up specific bits of information, and
tutorial-type books, which are designed to be read cover-to-cover. This
book is a bit of both: each chapter is intended to be read from start to
finish, but the chapters themselves are relatively independent.
Within each chapter, we generally start off with a simple example and go
into progressively more complex territory, so later sections of a given
chapter won't make sense if you haven't read the earlier sections. This is
particularly true of the chapters on building user interfaces and automated
testing, both of which involve building up complex examples step by step.
Where material from one chapter is relevant to another, I've tried to
include references in footnotes.
Because the chapters are about very different topics, they differ quite a lot
in both structure and content. Some chapters – like the one on building
user interfaces– have a lot of code examples, while some – like the one on
environments – have very few. Chapters which discuss a lot of command
line tools naturally have a lot of examples of command line output; the
chapter on performance is one such example.
Many of the techniques we'll discuss – notably things like automated
testing, CPU profiling, and logging – are most useful when applied to larger
programs. This presents a dilemma when it comes to writing examples for
them: examples need to be small (in order to be easily understood, and to
fit on the page) but small programs are the very ones for which the
techniques are least useful! To counter this problem, I will ask you to use
your imagination: when we are looking at small example programs,
remember that they are standing in for the much large programs that we
will write in the real world.
3
Chapter 1:Introduction
Because we will spend a lot of time in this book discussing various modules
from the Python standard library, I have given a lot of web addresses
(mostly in footnotes) to various parts of the documentation. The links all
go to the Python 2 documentation, but you can easily get to the
corresponding page for other versions by using the drop down menu in the
top left of the page.
Example files
You can download a copy of the example files from here:
http://pythonforbiologists.com/index.php/exercise-files/
Save and extract the files into a folder wherever you like. The example files
are separated by chapter. It should be possible to run most of them without
modification. However, some require you to install modules that are not
part of the Python standard library, and some (particularly those in the
user interface folder) expect to be run with command line arguments. If in
doubt, refer to the text.
For readers of my previous books
Anyone who has read Python for Biologists or Advanced Python for Biologists
will notice a couple of ways in which this book is different. Most notably, I
have not included exercises at the end of each chapter as I have in previous
books. This is simply because the material doesn't lend itself to small, self-
contained programming exercises. Unlike in previous books, where each
chapter is devoted to a specific part of the language, in this book each
chapter is more concerned with giving an overview of a particular aspect of
software development. If it's not clear at the end of a given chapter how
you can apply the material to your own work, it's probably because you
haven't encountered that particular problem yet.
4
Chapter 1:Introduction
Readers of my previous books may recognize some of the example
programs; in this book I've often used existing examples as starting points
for discussion. Rest assured that if you notice slight differences in
solutions to familiar problems in this book, it's because I've changed the
code to better illustrate the ideas we are exploring.
Working on the command line
In this book we will be looking not just at the Python language itself, but
also at other tools that help us to write Python more effectively. Many of
these tools run on the command line, so in order to demonstrate them
we'll have to look at command lines and command line output. In the
chapter on performance we will use command line tools to benchmark and
profile our code; in the chapter on testing we will use command line tools
to run our tests, and in the chapter on user interfaces we will build a
command line interface for a program we have written.
If you're not used to running programs on the command line then these
examples may be difficult to interpret at first, but hopefully they will seem
more familiar the more you read. It's certainly worth getting comfortable
with using the command line as it's the only way to run many of the tools
we want to use when developing in Python.
Typography and formatting
A book like this has lots of special types of text – we'll need to look at
examples of Python code and output, the contents of files, and technical
terms. Take a minute to note the typographic conventions we'll be using.
In the main text of this book, bold type is used to emphasize important
points and italics for technical terms and filenames. Where code is mixed
in with normal text it's written in a monospaced font like this with
5
Chapter 1:Introduction
a grey background. Occasionally there are footnotes1 to provide additional
information that is interesting to know but not crucial to understanding, or
to give links to web pages.
Example Python code is highlighted with a solid border above and below,
and the name of the matching example file is written just underneath the
example to the right:
Some example code goes here
example.py
Not every bit of code has a matching example file – much of the time we'll
be building up a Python program bit by bit, in which case there will be a
single example file containing the finished version of the program. The
example files are in separate folders, one for each chapter, to make them
easy to find.
Sometimes it's useful to refer to a specific line of code inside an example.
For this, we'll use numbered circles like this❶:
a line of example code
another line of example code
this is the important line
❶
here is another line
Example output (i.e. what we see on the screen when we run the code) is
highlighted with a dotted border:
Some output goes here
1 Like this.
6
