Machine Learning: Theory and Practice

299 Pages·2022·8.832 MB·English

by  Jugal Kalita

Machine Learning
Machine Learning: Theory and Practice provides an introduction to the most
popular methods in machine learning. The book covers regression including reg-
ularization, tree-based methods including Random Forests and Boosted Trees,
Artificial Neural Networks including Convolutional Neural Networks (CNNs),
reinforcement learning, and unsupervised learning focused on clustering. Topics
are introduced in a conceptual manner along with necessary mathematical de-
tails. The explanations are lucid, illustrated with figures and examples. For each
machine learning method discussed, the book presents appropriate libraries in
the R programming language along with programming examples.
Features:
•  Provides an easy-to-read presentation of commonly used machine learning al-
gorithms in a manner suitable for advanced undergraduate or beginning grad-
uate students, and mathematically and/or programming-oriented individuals
who want to learn machine learning on their own.
•  Covers mathematical details of the machine learning algorithms discussed to
ensure firm understanding, enabling further exploration.
•  Presents worked out suitable programming examples, thus ensuring concep-
tual, theoretical and practical understanding of the machine learning methods.
This book is aimed primarily at introducing essential topics in Machine Learning
to advanced undergraduates and beginning graduate students. The number of
topics has been kept deliberately small so that it can all be covered in a semester
or a quarter. The topics are covered in depth, within limits of what can be taught
in a short period of time. Thus, the book can provide foundations that will em-
power a student to read advanced books and research papers.

Machine Learning
Theory and Practice
Jugal Kalita
First edition published 2023
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
4 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
CRC Press is an imprint of Taylor & Francis Group, LLC
© 2023 Jugal Kalita
Reasonable efforts have been made to publish reliable data and information, but the author
and publisher cannot assume responsibility for the validity of all materials or the conse-
quences of their use. The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to copyright holders if
permission to publish in this form has not been obtained. If any copyright material has not
been acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted,
reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other
means, now known or hereafter invented, including photocopying, microfilming, and re-
cording, or in any information storage or retrieval system, without written permission
from the publishers.
For permission to photocopy or use material electronically from this work, access
www.copyright.com or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood
Drive, Danvers, MA 01923, 978-750-8400. For works that are not available on CCC please contact
[email protected]
Trademark notice: Product or corporate names may be trademarks or registered trade-
marks and are used only for identification and explanation without intent to infringe.
Library of Congress Cataloging‑in‑Publication Data
[Insert LoC Data here when available]
ISBN: 978-0-367-43354-3 (hbk)
ISBN: 978-0-367-43352-9 (pbk)
ISBN: 978-1-003-00261-1 (ebk)
DOI: 10.1201/9781003002611
Typeset in CMR10 font
by KnowledgeWorks Global Ltd.
Publisher’s note: This book has been prepared from camera-ready copy provided by the
authors.
Access the Support Material: www.routledge.com/ 9780367433543
To my mother Nirala
and my father Benudhar

Contents
Preface xiii
About the Author xv
1 Introduction 1
1.1 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Types of Machine Learning . . . . . . . . . . . . . . . . . . . 4
1.3.1 Supervised Learning . . . . . . . . . . . . . . . . . . . 4
1.3.1.1 Classification . . . . . . . . . . . . . . . . . . 4
1.3.1.2 Regression . . . . . . . . . . . . . . . . . . . 5
1.4 Unsupervised Machine Learning or Clustering . . . . . . . . 6
1.5 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . 7
1.6 Organization of the Book . . . . . . . . . . . . . . . . . . . . 8
1.7 Programming Language Used . . . . . . . . . . . . . . . . . . 10
1.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Regression 11
2.1 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.1 Linear Least Squares Regression . . . . . . . . . . . . 14
2.2 Evaluating Regression . . . . . . . . . . . . . . . . . . . . . . 19
2.2.1 Coefficient of Determination R2 . . . . . . . . . . . . . 19
2.2.2 Adjusted R2 . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.3 F-Statistic . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.3.1 Is the Model Statistically Significant? F-Test 23
2.2.4 Running the Linear Regression in R . . . . . . . . . . 25
2.2.5 The Role of Optimization in Regression . . . . . . . . 26
2.3 Multi-Dimensional Linear Regression . . . . . . . . . . . . . 27
2.3.1 Multi-dimensional Linear Least Squares Regression
Using R . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4 Polynomial Regression . . . . . . . . . . . . . . . . . . . . . 30
2.4.1 Polynomial Regression Using R . . . . . . . . . . . . . 33
2.5 Overfitting in Regression . . . . . . . . . . . . . . . . . . . . 39
2.6 Reducing Overfitting in Regression: Regularization . . . . . . 39
2.6.1 Ridge Regression . . . . . . . . . . . . . . . . . . . . . 40
2.6.2 Lasso Regression . . . . . . . . . . . . . . . . . . . . . 46
vii
viii Contents
2.6.3 Elastic Net Regression . . . . . . . . . . . . . . . . . . 47
2.7 Regression in Matrix Form . . . . . . . . . . . . . . . . . . . 48
2.7.1 Deriving LSRL Formula using Geometry and Linear
Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.7.2 Deriving LSLR Formula Using Matrix Calculus . . . . 53
2.8 Conclusions and Further Reading . . . . . . . . . . . . . . . 56
3 Tree-Based Classification and Regression 59
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.2 Inductive Bias . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.3 Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.3.1 General Tree Building Procedure . . . . . . . . . . . . 65
3.3.2 Splitting a Node . . . . . . . . . . . . . . . . . . . . . 67
3.3.2.1 Gini Index . . . . . . . . . . . . . . . . . . . 67
3.3.2.2 Entropy . . . . . . . . . . . . . . . . . . . . . 74
3.4 Simple Decision Trees in R . . . . . . . . . . . . . . . . . . . 77
3.4.1 Simple Decision Trees Using tree Library with Gini
Index and Discrete Features . . . . . . . . . . . . . . . 77
3.4.2 Simple Decision Trees Using tree Library with Gini
Index and Numeric Features. . . . . . . . . . . . . . . 79
3.4.3 Simple Decision Trees Using tree Library with Gini
Index and Mixed Features . . . . . . . . . . . . . . . . 81
3.5 Overfitting of Trees . . . . . . . . . . . . . . . . . . . . . . . 82
3.5.1 Pruning Trees in R . . . . . . . . . . . . . . . . . . . . 83
3.5.1.1 Cross-validation for Best Pruned Tree . . . . 84
3.5.2 Converting Trees to Rules . . . . . . . . . . . . . . . . 87
3.5.2.1 Building Rules in R . . . . . . . . . . . . . . 88
3.6 Evaluation of Classification . . . . . . . . . . . . . . . . . . . 90
3.6.1 Training and Testing Protocols . . . . . . . . . . . . . 90
3.6.2 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . 92
3.6.2.1 Basic Terminologies . . . . . . . . . . . . . . 92
3.6.2.2 Metrics for Binary Classification . . . . . . . 93
3.6.2.3 Evaluation of Binary Classification Using R . 94
3.6.2.4 Metrics for Multi-class Classification . . . . . 97
3.7 Ensemble Classification and Random Forests . . . . . . . . . 101
3.7.1 Bootstrapping and Bagging (Bootstrapped
Aggregation) . . . . . . . . . . . . . . . . . . . . . . . 101
3.7.2 Feature Sampling and Random Forests . . . . . . . . . 104
3.7.2.1 Do Random Forests Converge? . . . . . . . . 105
3.7.3 Random Forests in R . . . . . . . . . . . . . . . . . . 106
3.8 Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
3.8.1 Boosting on Loss: AdaBoost. . . . . . . . . . . . . . . 113
3.8.1.1 Multi-Class AdaBoost . . . . . . . . . . . . . 117
3.8.1.2 AdaBoost in R . . . . . . . . . . . . . . . . . 118
3.8.2 Regression Using Trees. . . . . . . . . . . . . . . . . . 123
Contents ix
3.8.2.1 Regression Using Single Trees . . . . . . . . 123
3.8.2.2 Regression Using Random Forests . . . . . . 124
3.8.2.3 Regression Using Loss-Boosted AdaBoost
Trees . . . . . . . . . . . . . . . . . . . . . . 124
3.8.2.4 Regression in R Using Trees . . . . . . . . . 124
3.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
4 Artificial Neural Networks 129
4.1 Biological Inspiration . . . . . . . . . . . . . . . . . . . . . . 129
4.2 An Artificial Neuron . . . . . . . . . . . . . . . . . . . . . . . 131
4.2.1 Activation Functions . . . . . . . . . . . . . . . . . . . 132
4.2.1.1 Sigmoid or Logistic Activation . . . . . . . . 132
4.2.1.2 tanh Activation Function . . . . . . . . . . . 134
4.2.1.3 Rectified Linear Unit . . . . . . . . . . . . . 136
4.3 Simple Feed-Forward Neural Network Architectures . . . . . 137
4.4 Training and Testing a Neural Network . . . . . . . . . . . . 140
4.4.1 Datasets Used in This Chapter . . . . . . . . . . . . . 140
4.4.2 Training and Testing a Neural Network in R Using
Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
4.4.2.1 Loading the MNIST Dataset . . . . . . . . . 143
4.4.2.2 Reshaping the Input . . . . . . . . . . . . . . 144
4.4.2.3 Architecture and Learning . . . . . . . . . . 145
4.4.2.4 Compiling and Running Experiments . . . . 146
4.4.2.5 The Entire Program . . . . . . . . . . . . . . 147
4.4.2.6 Observations about Empirical ANN Learning 149
4.5 Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . . 154
4.5.1 Backpropagation at a Conceptual Level . . . . . . . . 154
4.5.2 Mathematics of Backpropagation . . . . . . . . . . . . 155
4.5.2.1 Computing ∂E When j Is an Output Node 158
∂wij
4.5.2.2 Computing ∂E When j Is a Hidden Layer
∂wij
Node . . . . . . . . . . . . . . . . . . . . . . 161
4.5.2.3 Backpropagation with Different Activation
Functions . . . . . . . . . . . . . . . . . . . . 162
4.6 Loss Functions in Neural Networks . . . . . . . . . . . . . . . 165
4.7 Convolutional Neural Networks . . . . . . . . . . . . . . . . . 168
4.7.1 Convolutions . . . . . . . . . . . . . . . . . . . . . . . 169
4.7.1.1 Linear Convolutions . . . . . . . . . . . . . . 169
4.7.1.2 Two-Dimensional Convolutions . . . . . . . . 170
4.7.1.3 Three-Dimensional Convolutions . . . . . . . 172
4.7.2 Convolutional Neural Networks in Practice . . . . . . 172
4.7.2.1 Convolutional Layer and Activation . . . . . 174
4.7.2.2 CNNs Using Keras and R . . . . . . . . . . . 174
4.7.3 Pooling Layers in CNNs . . . . . . . . . . . . . . . . . 178
4.7.3.1 CNN using Pooling in R . . . . . . . . . . . 179
4.7.4 Regularization in CNNs . . . . . . . . . . . . . . . . . 180