Approaching (almost) any machine learning problem by Abhishek Thakur

301 Pages·2020·11.87 MB·English

Approaching (Almost) Any Machine Learning Problem

Approaching
(Almost) Any
Machine
Learning
Problem

ISBN: 978-82-692115-2-8
1
Approaching (Almost) Any Machine Learning Problem

It would not have been possible for me to write this book without the
support of my family and friends. I would also like to thank the reviewers
who selflessly devoted their time in reviewing this book (names in
alphabetical order).

Aakash Nain
Aditya Soni
Andreas Müller
Andrey Lukyanenko
Ayon Roy
Bojan Tunguz
Gilberto Titericz Jr.
Konrad Banachewicz
Luca Massaron
Nabajeet Barman
Parul Pandey
Ram Ramrakhya
Sanyam Bhutani
Sudalai Rajkumar
Tanishq Abraham
Walter Reade
Yuval Reina

I hope I did not miss anyone.

2
Approaching (Almost) Any Machine Learning Problem
Before you start, there are a few things that you must be aware of while going
through this book.

This is not a traditional book.

The book expects you to have basic knowledge of machine learning and deep
learning.

Important terms are bold.

Variable names and function/class names are italic.

═════════════════════════════════════════════════════════════════════════
All the code is between these two lines
═════════════════════════════════════════════════════════════════════════

Most of the times, the output is provided right after the code blocks.

Figures are locally defined. For example, figure 1 is the first figure

Code is very important in this book and there is a lot of it. You must go through
the code carefully and implement it on your own if you want to understand what’s
going on.

Comments in Python begin with a hash (#). All the code in this book is explained
line-by-line only using comments. Thus, these comments must not be ignored.

Bash commands start with $ or ❯.

If you find a pirated copy of this book (print or e-book or pdf), contact me directly
with the details so that I can take necessary actions.

If you didn’t code, you didn’t learn.
3
Approaching (Almost) Any Machine Learning Problem

Table of Contents
Setting up your working environment ..................................................... 5
Supervised vs unsupervised learning ....................................................... 7
Cross-validation ................................................................................... 14
Evaluation metrics ................................................................................ 30
Arranging machine learning projects ................................................... 73
Approaching categorical variables ........................................................ 85
Feature engineering ........................................................................... 142
Feature selection ................................................................................ 155
Hyperparameter optimization ............................................................. 167
Approaching image classification & segmentation .............................. 185
Approaching text classification/regression .......................................... 225
Approaching ensembling and stacking ............................................... 272
Approaching reproducible code & model serving ................................ 283

4
Approaching (Almost) Any Machine Learning Problem
Setting up your working environment

Before we begin with coding, it’s essential to get everything set-up on your
machine. Throughout this book, we will be using Ubuntu 18.04 and Python 3.7.6.
If you are a Windows user, you can install Ubuntu in multiple ways. On a virtual
machine, for example, Virtual Box which is provided by Oracle and is free software.
Alongside Windows as a dual boot system. I prefer dual boot as it is native. If you
are not an Ubuntu user, you might face problems with some of the bash scripts in
this book. To circumvent that you can install Ubuntu in a VM or go for Linux shell

on Windows.

Setting up Python on any machine is quite easy with Anaconda. I particularly like
Miniconda, which is a minimal installer for conda. It is available for Linux, OSX
and Windows. Since Python 2 support ended at the end of 2019, we will be using
the Python 3 distribution. You should keep in mind that miniconda does not come
with all the packages as regular Anaconda. We will, thus, be installing packages as
we go. Installing miniconda is quite easy.

The first thing that you need to do is download Miniconda3 to your system.

$ cd ~/Downloads
$ wget https://repo.anaconda.com/miniconda/...

where the URL after wget command is the URL from miniconda3 webpage. For
64-bit Linux systems, the URL at the time of writing this book was:

https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

Once you have downloaded miniconda3, you can run the following command:

$ sh Miniconda3-latest-Linux-x86_64.sh

Next, please read and follow the instructions on your screen. If you installed
everything correctly, you should be able to start the conda environment by typing
conda init the terminal. We will create a conda environment that we will be using
throughout this book. To create a conda environment, you can type:

$ conda create -n environment_name python=3.7.6
5
Approaching (Almost) Any Machine Learning Problem
This command will create a conda environment named environment_name which
can be activated using:

$ conda activate environment_name

And we are all set-up with the environment. Now it’s time to install some packages
that we would be using. A package can be installed in two different ways when you
are in a conda environment. You can either install the package from conda
repository or the official PyPi repository.

$ conda/pip install package_name

Note: It might be possible that some packages are not available in the conda repo.
Thus, installing using pip would be the most preferred way in this book. I have
already created a list of packages used while writing this book which is saved in the
environment.yml. You can find it in extra material available in my GitHub
repository. You can create the environment using the following command:

$ conda env create -f environment.yml

This command will create an environment called ml. To activate this environment
and start using it, you should run:

$ conda activate ml

And we are all set and ready to do some applied machine learning!

Always remember to be in the “ml” environment when coding along with this book.

Let’s start with our real first chapter now.

6
Approaching (Almost) Any Machine Learning Problem
Supervisedvsunsupervised learning

When dealing with machine learning problems, there are generally two types of
data (and machine learning models):
• Supervised data: always has one or multiple targets associated with it.
• Unsupervised data: does not have any target variable.

A supervised problem is considerably easier to tackle than an unsupervised one. A
problem in which we are required to predict a value is known as a supervised
problem. For example, if the problem is to predict house prices given historical
house prices, with features like presence of a hospital, school or supermarket,
distance to nearest public transport, etc. is a supervised problem. Similarly, when
we are provided with images of cats and dogs, and we know beforehand which ones
are cats and which ones are dogs, and if the task is to create a model which predicts
whether a provided image is of a cat or a dog, the problem is considered to be
supervised.

Figure 1: A supervised dataset.

As we see in figure 1, every row of the data is associated with a target or label. The
columns are different features and rows represent different data points which are
usually called samples. The example shows ten samples with ten features and a
target variable which can be either a number or a category. If the target is
categorical, the problem becomes a classification problem. And if the target is a real
7
Approaching (Almost) Any Machine Learning Problem
number, the problem is defined as a regression problem. Thus, supervised problems
can be divided into two sub-classes:

• Classification: predicting a category, e.g. dog or cat.
• Regression: predicting a value, e.g. house prices.

It must be noted that sometimes we might use regression in a classification setting
depending on the metric used for evaluation. But we will come to that later.

Another type of machine learning problem is the unsupervised type. Unsupervised
datasets do not have a target associated with them and in general, are more
challenging to deal with when compared to supervised problems.

Let’s say you work in a financial firm which deals with credit card transactions.
There is a lot of data that comes in every second. The only problem is that it is
difficult to find humans who will mark each and every transaction either as a valid
or genuine transaction or a fraud. When we do not have any information about a
transaction being fraud or genuine, the problem becomes an unsupervised problem.
To tackle these kinds of problems we have to think about how many clusters can
data be divided into. Clustering is one of the approaches that you can use for
problems like this, but it must be noted that there are several other approaches
available that can be applied to unsupervised problems. For a fraud detection
problem, we can say that data can be divided into two classes (fraud or genuine).

When we know the number of clusters, we can use a clustering algorithm for
unsupervised problems. In figure 2, the data is assumed to have two classes, dark
colour represents fraud, and light colour represents genuine transactions. These
classes, however, are not known to us before the clustering approach. After a
clustering algorithm is applied, we should be able to distinguish between the two
assumed targets. To make sense of unsupervised problems, we can also use
numerous decomposition techniques such as Principal Component Analysis
(PCA), t-distributed Stochastic Neighbour Embedding (t-SNE) etc.

Supervised problems are easier to tackle in the sense that they can be evaluated
easily. We will read more about evaluation techniques in the following chapters.
However, it is challenging to assess the results of unsupervised algorithms and a lot
of human interference or heuristics are required. In this book, we will majorly be
focusing on supervised data and models, but it does not mean that we will be
ignoring the unsupervised data problems.
8
Approaching (Almost) Any Machine Learning Problem

Figure 2: An unsupervised dataset.

Most of the time, when people start with data science or machine learning, they
begin with very well-known datasets, for example, Titanic dataset, or Iris dataset
which are supervised problems. In the Titanic dataset, you have to predict the
survival of people aboard Titanic based on factors like their ticket class, gender,
age, etc. Similarly, in the iris dataset, you have to predict the species of flower based
on factors like sepal width, petal length, sepal length and petal width.

Unsupervised datasets may include datasets for customer segmentation. For
example, you have data for the customers visiting your e-commerce website or the
data for customers visiting a store or a mall, and you would like to segment them or
cluster them in different categories. Another example of unsupervised datasets may
include things like credit card fraud detection or just clustering several images.

Most of the time, it’s also possible to convert a supervised dataset to unsupervised
to see how they look like when plotted.

For example, let’s take a look at the dataset in figure 3. Figure 3 shows MNIST
dataset which is a very popular dataset of handwritten digits, and it is a supervised
problem in which you are given the images of the numbers and the correct label
associated with them. You have to build a model that can identify which digit is it
when provided only with the image.

This dataset can easily be converted to an unsupervised setting for basic
visualization.
9