Table Of ContentPython 3 Combat Guide
A comprehensive manual about tricks, good practices and coding
style for hands on Software Developer Managers and Developers
By Carles Mateo
Creation: v. 0.1 2014-01-01 10:10 Catalonia Time
Last Updated: v.0.99 2021-06-18 23:00 Ireland Time
Python Combat Guide 1 / 365 Carles Mateo
Dedication
I want to dedicate this book to my mother, who raised two children alone, in a hostile and
macho world.
She is specially responsible of my success for believing in me, and for buying me a MSX
computer when I was 5 years old, as I mentioned that I found my passion.
Thanks
I want to thank all my colleagues in MPSTOR (Sanmina’s Software Factory in Ireland for the
Hardware division), the first company I worked for in Ireland, for making me feel like we were
a family, and being always collaborative at the pair with being great Engineers. You’re real
friends guys!.
I want to thank specially to my former Manager and Engineering Director, Seán Whelan,
from whom I learned so much and who shared with me so many interesting conversations
the Fridays, in those coffees in Mr Cotton1, and which I considered so valuable that I use to
say that was my real bonus (and not the bonus from my salary).
Seán supported me with my DRAID project for rebuilding RAID drives very quickly, and
really helped me also to deal with bureaucracy and to deal with all the diatribes or making a
Software product that runs in the Hardware of a multinational company, so contacting the
right Hardware departments in Colorado for reporting Firmware bugs I found, coordinating
the Engineering Testing, reporting to CEO, conducting EVT2 across multiple Teams, etc…
I was also a bit ashamed of my “robot Terminator” English, and he was the first person to tell
me that my English was really good, only my accent was to be polished. After, many kind
Irish people told me that I have a really beautiful exotic accent.
I also want to thank to all the members of my Team in Sanmina and the ones that
contributed to the DRAID project, too many to mention, but in special the ones I hired:
• Kevin Maher, for creating MPMGR, the Python tool to manage DRAID and ZFS
• Dlo Bagari, for bringing that extra, bringing in scientific tools, and making the C
changes required to DRAID and ZFS code, in order to have the protection for the
Data I wanted when a drive dies suddenly
• Síle Neylon, for catching impossible bugs that nobody else will
• Franklyn Aghedo, for conducting Engineering Tests in a very professional and
automated way
Also to my loving colleagues of Blizzard, which make me feel like home, and make me enjoy
so much by sharing their knowledge with me and being real friends.
And to my Manager, Bianca, who not only does not cut my wings, but encourages me to go
1 Mr Cotton is a local business in Cork, which delivers a small ban, the size of an Smart car, with an
Italian professional coffee machine, and parks it in Business Parks at certain hours and sells coffee.
2 Engineering Validation Tests.
Python Combat Guide 2 / 365 Carles Mateo
for more, to reach higher limits, to solve the unsolvable, and to keep learning at my natural
brutal speed, and also for offering brilliant advice, sharing arcane knowledge, and offering
me his common sense too where my Engineer mind would go binary.
I would not be fair if I don’t mention all my colleagues at GNOC and OpsCenter.
We are a Super Team!.
I don’t mention all the names for security concerns.
I’ll only mention the ones that reviewed this book before becoming public:
Michela, Sid, Gunnar, Miranda, Nathaniel, Thomas and my best friend Edu H.
Thanks.
Also a memory for my long-term Engineer friends: J.A. Garrido, A. Andreu, Edu H., Cos, Edu
G., Maure, Dotas, Güifi, E. Gambin, Alessio Mavica, Xavi...
To the people of Ireland, and this country, that are really polite, friendly and caring, and
made me feel welcomed and loved and being a citizen of a good country.
Finally I want to thank the guys at my local pub, as otherwise they will not allow me in next
time. ;) Just kidding. When I arrived to the country I was alone, and all of you made me feel
welcome, and called me by my name and understood that I was Catalan, not Spanish.
In fact my first Irish friend in Ireland, Áinee, grabbed my phone and added herself to my
Facebook and told me -Now we’re friends!- :) Many more, followed, like Declan. :)
There I met many nice friends at the Quiz, like Pádraig.
Thanks guys!. :)
Go raibh míle maith agat!3
My neighbors Carmel, Mary, Margaret, Robert, that gave me chocolate bunnies for Eastern,
my first umbrella, medicines, friendship and so much caring, when I was far from my family.
As we are in covid madness I want to mention some of my favorite friends that keep me
loved and happy: Lau, Alex, Gema and Clara, Meri and Sònia, Sandra, Cris, Edu H., Cos,
Paquita, Kika, Ricky, Nessa, Deb, Lluís Urpinell, Lluís (cousin), ChocoLauri, Antonio A., Edu
G., Helga Pose, George (Jawji Milla), Andrea Lc, Jordi Soler… the list is infinite. Thanks!.
I really can’t mention everybody that has been nice to me, or that has helped me in one or
another way. Thank you all!.
3 In Irish, means: Thanks a million
Python Combat Guide 3 / 365 Carles Mateo
Contact me
If you want to contact me you can use the section in LeanPub.
If you prefer to reach me in other way, you can contact me through my LinkedIn profile:
https://www.linkedin.com/in/carlesmateo/
You can suggest new sections for the book, or problems you don’t know how to solve, report
typos...
You can follow me in my blog: https://blog.carlesmateo.com
If you’re not satisfied
If you’re not satisfied with this book, I’ve set the option in LeanPub so you can return it
during the first 45 days.
I think this is the only fair way to proceed.
Python Combat Guide 4 / 365 Carles Mateo
Index
Dedication..............................................................................................................................2
Thanks...................................................................................................................................2
Contact me.............................................................................................................................4
If you’re not satisfied...............................................................................................................4
About this Combat guide......................................................................................................12
About the author...................................................................................................................12
Disclaimer.............................................................................................................................12
About coronavirus.................................................................................................................13
For the very beginners..........................................................................................................13
Hacker Rank....................................................................................................................14
About Floating point / decimals and Locale (international users).....................................15
Why Python versus Bash (reasons for SRE)........................................................................16
OOP vs Procedural code (SRE)...........................................................................................16
A practical sample of OOP for the SRE world: SSH Server Automation......................18
SSH Server improved with Paramiko SSH Libraries....................................................22
OOP small explanations recap....................................................................................23
Why not Python 2.7?............................................................................................................24
Few differences between Python 2 and 3........................................................................24
print()...............................................................................................................................25
Use a printing method instead of calling directly print()....................................................26
Use of raw_input in classes.............................................................................................26
Floats...............................................................................................................................26
Floats in Databases and Currencies................................................................................27
Things I don’t like from Python.............................................................................................30
Tools....................................................................................................................................31
IDE: PyCharm Professional.............................................................................................31
CVS: GIT.........................................................................................................................31
Unit Testing: PyTest.........................................................................................................31
Google Docs and google drive.........................................................................................32
Directories used...................................................................................................................32
File names............................................................................................................................32
File names for Classes....................................................................................................33
Classes inside files, one file per class..............................................................................33
File extensions.................................................................................................................34
File paths.........................................................................................................................36
Working with GIT..................................................................................................................37
Gitlab...............................................................................................................................37
Commit messages...........................................................................................................37
Commit messages for Gitlab.......................................................................................38
Branches..........................................................................................................................40
Branching Strategy......................................................................................................41
Git flow........................................................................................................................43
GitHub flow..................................................................................................................45
Production branch with GitLab flow.............................................................................46
Python Combat Guide 5 / 365 Carles Mateo
Environment branches with GitLab flow......................................................................47
Release branches with GitLab flow.............................................................................48
Merge/Pull requests....................................................................................................49
Feature branches........................................................................................................50
Closing or Linking issues using comments in GitLab...................................................50
Removing feature branches after merge.....................................................................51
Have always backups from Master..............................................................................51
Git rebase....................................................................................................................51
Git submodules................................................................................................................52
Fetch................................................................................................................................54
Pull from the repo.............................................................................................................54
Pushing to the repo..........................................................................................................54
Keep an eye to new changes in commits through the Log section...................................55
Never ever submit to the repo local changes you want to test.........................................55
Never commit Debug Lines..............................................................................................55
Development Environment...................................................................................................56
Use Linux.........................................................................................................................56
Coding..................................................................................................................................58
General Style...................................................................................................................58
Structure......................................................................................................................58
Comments...................................................................................................................59
Use @TODO...............................................................................................................59
PEP-8..........................................................................................................................59
MT Notation.................................................................................................................60
Short Code in Methods is preferred.............................................................................62
About ' or " (apostrophe or double quote)........................................................................62
Return..............................................................................................................................63
Error code in return..........................................................................................................64
Predictability in return: Return always the same..............................................................64
Files.................................................................................................................................64
Operators.........................................................................................................................65
Tabs or spaces................................................................................................................65
Space to breath................................................................................................................66
Too much compressed code............................................................................................66
Use parentheses, Luke....................................................................................................68
Use Parenthesis on if evaluation as much as possible................................................69
Short-circuiting behavior in Chained Comparison........................................................70
Avoid using short-circuiting with methods....................................................................70
Variables and Names.......................................................................................................73
Do not use Global variables.............................................................................................73
Class variables are Static or not, depending on the context.............................................74
Declare variables on top of the Class and Method...........................................................77
Declaring variables on top in Methods and Functions.................................................77
Declaring variables on top in Classes..........................................................................78
Do not modify arrays and variables passed by reference (unless you want to do that)....81
How variables are passed to a method/function?.............................................................82
String, our immutable friend........................................................................................82
Mutable and Immutable variables....................................................................................83
Python Combat Guide 6 / 365 Carles Mateo
Class Names...................................................................................................................84
Class Constants...............................................................................................................84
Comparing with constants................................................................................................85
Properties........................................................................................................................85
Properties and variables defined on top of the Method....................................................85
Functions and Methods....................................................................................................86
Do not print inside classes, use the capacity of Python to return more than a parameter
instead.........................................................................................................................86
Warning: Do not repeat Method of Function names....................................................87
Function/Method Parameters with a default value (Keywords)....................................87
Keyword Argument for a Method/Function..................................................................88
Type Hinting and Annotations.....................................................................................91
Lists, Arrays, Dicts, Tuple................................................................................................93
Mixed power of Lists, Dicts..........................................................................................93
Dynamic grow..............................................................................................................94
“in” very important check to do....................................................................................94
Convenience of having a method for checking “in”......................................................95
Default dict, to save memory.......................................................................................96
Sort a List by key, example.........................................................................................97
Arrays and index -1..........................................................................................................97
Expect the unexpected (Corner cases)............................................................................98
The Happy Path..........................................................................................................98
Use getters and setters....................................................................................................99
Use update_ for methods that update Objects, not get_................................................100
Jumps of line \n to the end.............................................................................................101
If / else...........................................................................................................................102
Else in loops..............................................................................................................103
Careful String evaluations.........................................................................................103
Calling variables by name in Methods or Functions.......................................................104
If for text properties........................................................................................................104
Avoid repeating code.....................................................................................................105
Comments.....................................................................................................................106
Python doc.....................................................................................................................106
Document......................................................................................................................106
No dead code.................................................................................................................107
We avoid Magic Methods...............................................................................................107
The power of range........................................................................................................108
Avoid using yield, avoid using lambdas..........................................................................109
Avoid using list comprehensions....................................................................................110
Avoid using Regular Expressions, unless you know what you are doing.......................110
Creating Web Applications / Web Services with Flask...................................................111
Hello World Application.............................................................................................111
Another simple Web Flask Application......................................................................114
A more cool Flask Application...................................................................................115
For using in Production..............................................................................................119
Files for holding Data Structures and Databases...........................................................120
Collation....................................................................................................................120
Data types in the Database.......................................................................................120
Python Combat Guide 7 / 365 Carles Mateo
Data serialization...........................................................................................................121
Some tricks to Scale Production....................................................................................124
A real case of Serialization in the world of the videogames.......................................124
Adding compression..................................................................................................124
Database Sharding....................................................................................................125
Advice: Do not allow Marketing to query Production Servers....................................127
Use of the Slaves for reading....................................................................................128
Load Balancing..........................................................................................................128
Redis cache...............................................................................................................128
Cache on local webservers.......................................................................................128
NGINX, CDNs...........................................................................................................129
Python Virtual Environments..........................................................................................129
Creating a Virtual Environment in Python 3...............................................................129
Creating a Virtual Environment in Python 2...............................................................131
Activating the Virtual Environment.............................................................................131
Deactivating the Virtual Environment.........................................................................131
Knowing what Python Packages are installed................................................................132
requirements.txt file........................................................................................................133
Python Threads..............................................................................................................134
Prefixes: MT Notation....................................................................................................135
Expect the Unexpected..................................................................................................137
Corner cases and assumptions......................................................................................139
Do no escape to shell to run a program, if you can do it from Python............................139
Escaping to shell, when really needed...........................................................................140
An example of things that could be wrong.................................................................140
Try to have Common Libraries for the Common Tasks..................................................144
OOP...................................................................................................................................145
Initialising the variables in the Constructor.....................................................................148
Use init_variables()........................................................................................................149
Bug fixing............................................................................................................................151
Focus, make sure you don’t do mistakes.......................................................................151
Better Quality than Quantity...........................................................................................151
Cross Testing.................................................................................................................151
Check the Logs..............................................................................................................152
Methodology.......................................................................................................................155
Methodology serves a purpose......................................................................................155
We are modern heroes..............................................................................................155
Deploying to Production.................................................................................................155
Create a Deploy or Launch or Change Request Document in the issue tracker........155
Deploying to Production using immutable Docker Containers and/or Microservices. 156
Requirements.txt.......................................................................................................158
In Dockerfile do not use FROM latest........................................................................160
Running in Production....................................................................................................160
Check for Log Errors regularly in Production Servers................................................160
Profiling..........................................................................................................................160
Forensics / Postmortem analysis...................................................................................160
Do not use Database Stored Procedures.......................................................................160
Webservices and API are good, microservices are not always the wisest choice..........161
Python Combat Guide 8 / 365 Carles Mateo
Blue green Deployment vs Canary, and Rolling Upgrade..............................................162
Automated Code Testing Quick Overview..........................................................................165
Unit Testing Framework.................................................................................................165
py.test versus unittest?..............................................................................................165
Tests are not optional....................................................................................................165
Run the Unit Tests.........................................................................................................166
Run tests frequently..................................................................................................166
Do not reuse other Tests...........................................................................................166
Importance of running good Tests (Unit Testing Tests)..................................................166
Problems from Bash invoking parameters with quotes or without..............................167
Emails in the Tests.........................................................................................................168
Testing Infrastructure as Code...........................................................................................169
Managing the Team............................................................................................................170
Managing takes time......................................................................................................170
Synchronizing the knowledge of the Team.....................................................................170
Demos and Share the Pain............................................................................................170
Be always fair.................................................................................................................171
Allow to learn and creativity...........................................................................................172
New members join the Team.........................................................................................173
Kick Off Meeting........................................................................................................173
Concentrated advice for my Team.................................................................................173
Methodology for the Development, Presentations, Documents and Demos.......................174
Prepare Demos: name Objects and Variables...............................................................174
Prepare Demos: prepare it in advance and test that everything works...........................174
Take care of the details..................................................................................................174
We don’t avoid confronting problems.............................................................................175
We document and write Functional Analysis..................................................................175
Checking the Functional Analysis before starting programming.....................................176
We reply the emails.......................................................................................................176
We do Stand up Meetings..............................................................................................176
Exercises for Candidates / New Recruitments....................................................................176
Tools to discuss remotely (screener).............................................................................177
First, the FooBar exercise..............................................................................................177
Second, Power of 2 exercise.........................................................................................177
What we take in consideration from the candidate?.......................................................177
Being nice......................................................................................................................178
Fair Questions for Developers.......................................................................................178
Working with pytest Unit Testing........................................................................................179
Installing py.test in Ubuntu.............................................................................................179
Installing py.test for Python2.....................................................................................179
Installing py.test for Python3.....................................................................................180
Running the Tests in a Virtual Machine like VirtualBox..................................................180
Running the Tests in a Docker container.......................................................................180
Location of the code samples and tests.........................................................................181
Writing the first test........................................................................................................181
Using classes for testing................................................................................................182
Testing a real class...................................................................................................183
Extending our first test...............................................................................................185
Python Combat Guide 9 / 365 Carles Mateo
Extending a bit more and getting errors.....................................................................186
Fixing our Test using TDD.............................................................................................189
Advantages and disadvantages of having everything in one single method...................190
Reusability.....................................................................................................................193
Adding improvements discussed....................................................................................193
Creating unit testing for DiskUtils...................................................................................195
Do not reuse variables in Unit Testing Code..................................................................197
Adding pytest to PyCharm Community Edition...............................................................197
Code Coverage..............................................................................................................200
Adding Dependency Injection........................................................................................203
Stubs and Mocks and Monkeypatching..........................................................................208
Putting all together.........................................................................................................213
Exercises...................................................................................................................221
Refactors.......................................................................................................................221
How to do a Refactor of legacy code.........................................................................222
Fixtures..........................................................................................................................223
Fixture to patch time.sleep........................................................................................223
Pytest config file conftest.py...........................................................................................226
Testing Exceptions.........................................................................................................228
Catching an expected Exception...............................................................................228
Code Coverage on Exceptions..................................................................................228
Test Helper for OS calls testing.................................................................................229
Testing STDOUT, print, and Fixture capsys...................................................................234
A Refactor from CSV reader spaghetti code to OOP with Unit Testing..........................235
CSV format................................................................................................................235
CSV reader code (spaghetti code)............................................................................236
Problems with this code spaghetti-code2.py..............................................................236
Addressing some errors more. spaghetti_code3.py...................................................239
Do not use dash in file names...................................................................................239
Use if __name__ == "__main__":..............................................................................239
Magic Attributes.........................................................................................................240
Extracting the functionality. spaghetti_code4.py........................................................241
Unit Testing for spaghetti_code4.py test_spaghetti_code4.py...................................243
Converting spaghetti_code4.py to OOP code............................................................246
The Unit Testing for the OOP code...........................................................................248
Converting a simple script to list drive to a well tested program with Unit Testing.....252
When to use Data Objects and when to use just Arrays or Dictionaries?..................290
Testing your code as Superuser and regular user.........................................................291
Running your Docker Container as non-root, regular user.............................................292
Test as root and non-root..........................................................................................292
Running Tests in actual Hardware?...........................................................................292
Tricks and Appendix...........................................................................................................293
A mix: Unit Testing on actual Hardware.........................................................................293
Creating PIP Packages..................................................................................................294
Convert a Python program to a Cron.............................................................................299
Using Crontab...........................................................................................................303
Using Python Scheduler Class and Docker...............................................................304
Catching CTRL + C inside your Python programs.....................................................306
Python Combat Guide 10 / 365 Carles Mateo
