Python for Secret Agents

269 Pages·2015·1.53 MB·English

by  Steven Lott

Python for Secret Agents Second Edition
Table of Contents
Python for Secret Agents Second Edition
Credits
About the Author
About the Reviewer
www.PacktPub.com
Support files, eBooks, discount offers, and more
Why subscribe?
Free access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
1. New Missions – New Tools
Background briefing on tools
Doing a Python upgrade
Preliminary mission to upgrade pip
Background briefing: review of the Python language
Using variables to save results
Using the sequence collections: strings
Using other common sequences: tuples and lists
Using the dictionary mapping
Comparing data and using the logic operators
Using some simple statements
Using compound statements for conditions: if
Using compound statements for repetition: for and while
Defining functions
Creating script files
Mission One – upgrade Beautiful Soup
Getting an HTML page
Navigating the HTML structure
Doing other upgrades
Mission to expand our toolkit
Scraping data from PDF files
Sidebar on the ply package
Building our own gadgets
Getting the Arduino IDE
Getting a Python serial interface
Summary
2. Tracks, Trails, and Logs
Background briefing – web servers and logs
Understanding the variety of formats
Getting a web server log
Writing a regular expression for parsing
Introducing some regular expression rules and patterns
Finding a pattern in a file
Using regular expression suffix operators
Capturing characters by name
Looking at the CLF
Reading and understanding the raw data
Reading a gzip compressed file
Reading remote files
Studying a log in more detail
What are they downloading?
Trails of activity
Who is this person?
Using Python to run other programs
Processing whois queries
Breaking a request into stanzas and lines
Alternate stanza-finding algorithm
Making bulk requests
Getting logs from a server with ftplib
Building a more complete solution
Summary
3. Following the Social Network
Background briefing – images and social media
Accessing web services with urllib or http.client
Who’s doing the talking?
Starting with someone we know
Finding our followers
What do they seem to be talking about?
What are they posting?
Deep Under Cover – NLTK and language analysis
Summary
4. Dredging up History
Background briefing–Portable Document Format
Extracting PDF content
Using generator expressions
Writing generator functions
Filtering bad data
Writing a context manager
Writing a PDF parser resource manager
Extending the resource manager
Getting text data from a document
Displaying blocks of text
Understanding tables and complex layouts
Writing a content filter
Filtering the page iterator
Exposing the grid
Making some text block recognition tweaks
Emitting CSV output
Summary
5. Data Collection Gadgets
Background briefing: Arduino basics
Organizing a shopping list
Getting it right the first time
Starting with the digital output pins
Designing an external LED
Assembling a working prototype
Mastering the Arduino programming language
Using the arithmetic and comparison operators
Using common processing statements
Hacking and the edit, download, test and break cycle
Seeing a better blinking light
Simple Arduino sensor data feed
Collecting analog data
Collecting bulk data with the Arduino
Controlling data collection
Data modeling and analysis with Python
Collecting data from the serial port
Formatting the collected data
Crunching the numbers
Creating a linear model
Reducing noise with a simple filter
Solving problems adding an audible alarm
Summary
Index

Python for Secret Agents Second Edition