FashionPolice:Application of Convolutional Neural Networks to Single-Step Apparel Recognition

125 Pages·2017·38.45 MB·English

by  Walid M. Marfouk

FashionPolice:
Application of Convolutional Neural Networks
to Single-Step Apparel Recognition on Social
Media in Scarce Training Data Contexts
Walid M. Marfouk
Advisor: Professor Alain L. Kornhauser
Submitted in partial fulfillment
of the requirements for the degree of
Bachelor Of Science And Engineering
Department of Operations Research and Financial Engineering
Princeton University
Thesis
June 2017
I hereby declare that I am the sole author of this thesis.
I authorize Princeton University to lend this thesis to other institutions or individuals for the
purpose of scholarly research.
Walid Marfouk
I further authorize Princeton University to reproduce this thesis by photocopying or by other
means, in total or in part, at the request of other institutions or individuals for the purpose of
scholarly research.
Walid Marfouk
Abstract
Over the past fifteen year, the field of object recognition has evolved from the limited invariant
alignmentmethodsofthe1960’sand1970’s, toneuralnetworkbasedvisiontechniquesnowadays.
Today’s state of the art algorithms perform simple recognition tasks, such as the MNIST hand-
written digit classification challenge, with virtually perfect success rates–they are better than
humans. Some other areas in computer vision lag behind, however.
This thesis addresses the challenge of recognizing fashion apparel as it appears in social media
images. Its goal is to build software that automatically detects and keeps track of a brand’s prod-
ucts in images posted on social media through a single-step recognition process, implemented
by a convolutional neural network able to adapt quickly, and inexpensively, to new collections.
Social media in general, and bloggers in particular, are crucial drivers of consumption in fashion:
in 2016, nearly one in two online shoppers declared being influenced by posts on social media
before making purchases, while only one in three declared being influenced by ads [61]. This
statistic is made all the more relevant by the $51.5 billion spent on online purchases of apparel
and accessories in the U.S. in 2015 (the biggest share of all online spending that year) [42].
This paper shifts away from the widely adopted two-step “detection and classification” method,
and bypasses the often used descriptive label-assigning classification paradigm and its associated
nearest-neighbor recommendation. Instead, the focus is turned to recognizing each item in a
brand’s collection as an instance of a single class. Implementing techniques in transfer learning,
residual learning, optimal learning acceleration, and training data transformation, the Fashion-
Police model developed in this thesis achieves above-baseline performance while maintaining
computational load at a minimum.
This is a business-oriented project; commercial applications could enable improvements to online
demand modeling, online supply chain control, and provide additional marketing insights.
ii
To my family for their unconditional support
Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
1 Introduction 1
1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Project Specifications & Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Theoretical Framework and Literature Review 9
2.1 Computer Vision & Object Recognition . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.1 Retrospective Survey of Object Recognition Techniques . . . . . . . . . . . 10
2.1.2 Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.1.3 Convolutional Neural Networks: ANN’s applied to Computer Vision . . . . 36
2.2 Advanced Machine Learning Techniques . . . . . . . . . . . . . . . . . . . . . . . . 46
2.2.1 Transfer Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.2.2 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.2.3 Residual Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.3 Fashion Apparel in Computer Vision . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.3.1 Detection & Classification: a two-step approach. . . . . . . . . . . . . . . . 56
2.3.2 Item Recommendation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.3.3 Training Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
iii
3 Project Overview & Development 61
3.1 Data Specifications & Transformations . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.1.1 Binary Classification & Datasets . . . . . . . . . . . . . . . . . . . . . . . . 62
3.1.2 Data Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.1.3 Data Augmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.1.4 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.1.5 Technical Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.2 TensorFlow Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.3 Model Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.3.1 Inception-v4 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.3.2 FashionPolice Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4 Conclusion 97
A Performance of Intermediary Models 99
B Inception-v4 Architecture 102
C Sample Real-World Test Images 106
Bibliography 110
iv
List of Tables
3.1 Summary of experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.2 Summary of data augmentation parameters . . . . . . . . . . . . . . . . . . . . . . 86
3.3 ‘Real-world’ mini-test results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
v
List of Figures
1.1 Instagram profile of Kim Kardashian . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Error example on the ThreadGenius homepage . . . . . . . . . . . . . . . . . . . . 4
1.3 ILSVRC-2010 test images and their corresponding classes . . . . . . . . . . . . . . 5
1.4 Louis Vuitton Alma PM handbag . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Louis Vuitton Bridle Foulard Print Down Jacket . . . . . . . . . . . . . . . . . . . 8
2.1 Roberts’ 1961 ‘blocks world’ object recognition process . . . . . . . . . . . . . . . . 12
2.2 3-D ‘blocks world’ object recognition process . . . . . . . . . . . . . . . . . . . . . 13
2.3 Application of Binford’s generalized cylinders method . . . . . . . . . . . . . . . . 14
2.4 Model-based vision system for industrial parts. . . . . . . . . . . . . . . . . . . . . 15
2.5 Moment invariants classification of Roman alphabet letters . . . . . . . . . . . . . 19
2.6 Acquired hypersurfaces in a 3-D restriction of the eigenspace . . . . . . . . . . . . 21
2.7 Image hyperpoint in a 3-D restriction of the eigenspace . . . . . . . . . . . . . . . 21
2.8 Biological neuron schematic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.9 TLU diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.10 Perceptron diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.11 Linear separability of a training sets’ output classes. . . . . . . . . . . . . . . . . . 27
2.12 Mark I Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.13 Impact of learning rates in gradient descent . . . . . . . . . . . . . . . . . . . . . . 32
2.14 Multi-layered artificial neural network architecture . . . . . . . . . . . . . . . . . . 33
2.15 ANN input layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.16 Local receptive field connection with one neuron in the following layer . . . . . . . 38
2.17 Filter stride length impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
vi
2.18 Feature map structure of hidden layers . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.19 MAX-pool operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.19 Neural dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.20 Modular building block in residual learning . . . . . . . . . . . . . . . . . . . . . . 55
2.21 Detection & classification in images: a two-step approach . . . . . . . . . . . . . . 58
2.22 Detection: overload of bounding boxes . . . . . . . . . . . . . . . . . . . . . . . . . 58
2.23 Model accuracy as a function of training set size . . . . . . . . . . . . . . . . . . . 60
3.1 Tuniques from the Louis Vuitton Fall/Winter 16-17 dataset . . . . . . . . . . . . . 62
3.2 Sample negative images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.3 Sample extracted from Kim Kardashian’s Instagram profile . . . . . . . . . . . . . 64
3.4 Parsing process applied to Figure 3.3.c . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.5 Precision, recall, and accuracy as performance metrics . . . . . . . . . . . . . . . . 71
3.6 Multiple GPU tower structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.7 State of the art neural architectures’ main characteristics . . . . . . . . . . . . . . 74
3.8 Simplified Inception module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.9 Filter concatenation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.10 Dimensionality reduction through 1 1 convolutions . . . . . . . . . . . . . . . . . 77
⇥
3.11 Inception module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.12 High-level Inception-v4 architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.13 Base model accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.14 Base model architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.15 Base model loss function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.16 Automated data pre-processing in base model . . . . . . . . . . . . . . . . . . . . . 84
3.17 FashionPolice model architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.18 Detail of FashionPolice architecture: Block 1 . . . . . . . . . . . . . . . . . . . . . 88
3.19 Detail of FashionPolice architecture: Blocks 2 & 3 . . . . . . . . . . . . . . . . . . 89
3.20 Detail of FashionPolice architecture: Reduction Blocks & Stem . . . . . . . . . . . 90
3.21 Training data transformation & augmentation . . . . . . . . . . . . . . . . . . . . . 93
3.22 FashionPolice engine accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
vii
A.1 Base model performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
A.2 Step 1 model performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
A.3 Step 2 model performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
A.4 Step 3 model performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
A.5 FashionPolice engine performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
B.1 Inception-v4 architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
B.2 Inception-v4: Inception Block A . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
B.3 Inception-v4: Inception Block B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
B.4 Inception-v4: Inception Block C . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
B.5 Inception-v4: Inception Reduction Module A . . . . . . . . . . . . . . . . . . . . . 104
B.6 Inception-v4: Inception Reduction Module B . . . . . . . . . . . . . . . . . . . . . 105
C.1 Real-world test: raw positive images . . . . . . . . . . . . . . . . . . . . . . . . . . 106
C.2 Real-world test: parsing of positive images . . . . . . . . . . . . . . . . . . . . . . . 107
C.3 Real-world test: processed positive images . . . . . . . . . . . . . . . . . . . . . . . 107
C.4 Real-world test: raw negative images . . . . . . . . . . . . . . . . . . . . . . . . . . 108
C.5 Real-world test: parsing of negative images . . . . . . . . . . . . . . . . . . . . . . 108
C.6 Real-world test: processed negative images . . . . . . . . . . . . . . . . . . . . . . 109
viii