Deep Binary Representation Learning for Single/Cross-Modal Data Retrieval

181 Pages·2017·10 MB·English

Deep Binary Representation
Learning for Single/Cross-Modal
Data Retrieval
Yuming Shen
School of Computing Sciences
University of East Anglia
A thesis submitted for the degree of
Doctor of Philosophy
January 2018
c This copy of the thesis has been supplied on condition that anyone who consults it is
(cid:13)
understood to recognise that its copyright rests with the author and that no quotation
from the thesis, nor any information derived therefrom, may be published without the
authors prior written consent.
Declaration
Parts of this thesis have been taken from certain published/to be pub-
lished conference/journal papers. All of these papers were written pri-
marily or partially by me, Yuming Shen, during and as a result of my
PhD research. These papers are listed below.
Y. Shen, L. Liu, L. Shao and J. Song. Deep Binaries: Encoding
•
Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval.
In IEEE International Conference on Computer Vision (ICCV),
2017
Y. Shen, L. Liu and L. Shao. Unsupervised Deep Generative
•
Hashing. In British Machine Vision Conference (BMVC), 2017
L.Liu, F.Shen, Y. Shen, X.LiuandL.Shao. DeepSketchHash-
•
ing: FastFree-HandSketch-BasedImageRetrieval. InIEEE Con-
ference on Computer Vision and Pattern Recognition (CVPR),
2017
Y.Shen,L.ZhangandL.Shao. Semi-SupervisedVision-Language
•
Mapping via Variational Learning. In IEEE International Con-
ference on Robotics and Automation (ICRA), 2017
Acknowledgements
I would like to extend my sincerest thanks to the following, who have
all helped in the completion of this thesis.
First of all, many thanks to my family for their unconditional support
during my PhD studies.
I would like to express my sincere appreciation to my supervisor Prof.
Ling Shao for providing me with the opportunity of pursuing this PhD
degree. We are allowed to have unprecedented research freedom under
his supervision, and he is always willing to offer valuable advice and
thorough suggestions on our research works.
I would like to thank Prof. Andy Day for his academic support.
Iwouldliketothankthecurrentandformerresearchstudents, Postdoc
researchers and visiting scholars in our lab, including Dr. Li Liu, Dr.
Yang Long, Dr. Mengyang Yu, Dr. Heng Liu, Dr. Haofeng Zhang,
Dr. Xiaoming Liu, Yi Zhou, Shidong Wang, Bingzhang Hu and Daniel
Organisciak, for their helpful discussion and research ideas.
I would also like to thank my co-authors, Prof. Fumin Shen and Prof.
Xianglong Liu, who have improved my research outcome from many
aspects.
Abstract
Datasimilaritysearchiswidelyregardedasaclassictopicintherealms
ofcomputervision, machinelearninganddatamining. Providingacer-
tain query, the retrieval model sorts out the related candidates in the
database according to their similarities, where representation learning
methods and nearest-neighbour search apply. As matching data fea-
tures in Hamming space is computationally cheaper than in Euclidean
space, learning to hash and binary representations are generally appre-
ciated in modern retrieval models. Recent research seeks solutions in
deep learning to formulate the hash functions, showing great potential
in retrieval performance. In this thesis, we gradually extend our re-
search topics and contributions from unsupervised single-modal deep
hashing to supervised cross-modal hashing finally zero-shot hashing
problems, addressing the following challenges in deep hashing.
First of all, existing unsupervised deep hashing works are still not at-
taining leading retrieval performance compared with the shallow ones.
To improve this, a novel unsupervised single-modal hashing model
is proposed in this thesis, named Deep Variational Binaries (DVB).
We introduce the popular conditional variational auto-encoders to for-
mulate the encoding function. By minimizing the reconstruction er-
ror of the latent variables, the proposed model produces compact bi-
narycodeswithouttrainingsupervision. Experimentsonbenchmarked
datasets show that our model outperform existing unsupervised hash-
ing methods.
The second problem is that current cross-modal hashing methods only
consider holistic image representations and fail to model descriptive
sentences, which is inappropriate to handle the rich semantics of in-
formative cross-modal data for quality textual-visual search tasks. To
handlethisproblem, weproposeasuperviseddeepcross-modalhashing
modelcalledTextual-VisualDeepBinaries(TVDB).Region-basedneu-
ral networks and recurrent neural networks are involved in the image
encoding network in order to make effective use of visual information,
while the text encoder is built using a convolutional neural network.
We additionally introduce an efficient in-batch optimization routine to
train the network parameters. The proposed mode successfully out-
performs state-of-the-art methods on large-scale datasets.
Finally, existing hashing models fail when the categories of query data
have never been seen during training. This scenario is further extended
into a novel zero-shot cross-modal hashing task in this thesis, and a
Zero-shot Sketch-Image Hashing (ZSIH) scheme is then proposed with
graph convolution and stochastic neurons. Experiments show that the
proposed ZSIH model significantly outperforms existing hashing algo-
rithms in the zero-shot retrieval task.
Experiments suggest our proposed and novel hashing methods outper-
form state-of-the-art researches in single-modal and cross-modal data
retrieval.
Contents
1 Introduction 1
1.1 Research Background . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Summary of Remaining Chapters . . . . . . . . . . . . . . . . . . . 4
2 Literature Review 6
2.1 Single-Modal Hashing . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 Unsupervised Hashing . . . . . . . . . . . . . . . . . . . . . 7
2.1.2 Supervised Hashing . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Cross-Modal Hashing . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1 Unsupervised Cross-Modal Hashing . . . . . . . . . . . . . . 10
2.2.2 Supervised Cross-Modal Hashing . . . . . . . . . . . . . . . 12
2.3 Deep Learning and Neural Networks . . . . . . . . . . . . . . . . . 13
2.3.1 Convolutional Neural Networks for Images and Texts . . . . 14
2.3.2 Hashing with Deep Neural Networks . . . . . . . . . . . . . 15
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3 Unsupervised Deep Hashing for Image Retrieval 18
3.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . 18
3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2.1 Shallow Unsupervised Hashing . . . . . . . . . . . . . . . . . 21
3.2.2 Deep Unsupervised Hashing . . . . . . . . . . . . . . . . . . 22
3.2.3 Auto-Encoding Variational Networks . . . . . . . . . . . . . 23
3.3 Deep Variational Binaries . . . . . . . . . . . . . . . . . . . . . . . 24
i
3.3.1 The Variational Model . . . . . . . . . . . . . . . . . . . . . 24
3.3.2 Exploiting Intrinsic Data Structure. . . . . . . . . . . . . . . 28
3.3.3 Hashing Objectives . . . . . . . . . . . . . . . . . . . . . . . 29
3.3.3.1 Quantization Penalty . . . . . . . . . . . . . . . . . 29
3.3.3.2 Bit Decorrelation . . . . . . . . . . . . . . . . . . . 30
3.3.3.3 In-Batch Similarity . . . . . . . . . . . . . . . . . . 30
3.3.4 Network Setup . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.3.5 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.3.6 Out-of-Sample Extension . . . . . . . . . . . . . . . . . . . . 33
3.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.4.1 Implementation Details . . . . . . . . . . . . . . . . . . . . . 34
3.4.2 Experimental Settings . . . . . . . . . . . . . . . . . . . . . 36
3.4.2.1 CIFAR-10 [84] . . . . . . . . . . . . . . . . . . . . 36
3.4.2.2 SUN-397 [187] . . . . . . . . . . . . . . . . . . . . 37
3.4.2.3 NUS-WIDE [22] . . . . . . . . . . . . . . . . . . . 37
3.4.3 Comparison with State-of-the-Art Methods . . . . . . . . . . 37
3.4.3.1 Baselines . . . . . . . . . . . . . . . . . . . . . . . 37
3.4.3.2 Quantitative Results with Deep Features . . . . . . 38
3.4.3.3 Quantitative Results with Hand-Crafted Features . 40
3.4.3.4 Efficiency . . . . . . . . . . . . . . . . . . . . . . . 40
3.4.4 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.4.4.1 Baselines . . . . . . . . . . . . . . . . . . . . . . . 41
3.4.4.2 Results and Analysis . . . . . . . . . . . . . . . . . 42
3.4.5 Hyper-Parameters . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4.5.1 Size of Latent Space . . . . . . . . . . . . . . . . . 44
3.4.5.2 Number of Clustering Centres . . . . . . . . . . . . 44
3.4.6 Qualitative results . . . . . . . . . . . . . . . . . . . . . . . 44
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4 Supervised Deep Hashing for Image-Sentence Cross Retrieval 47
4.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . 47
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2.1 Neural Networks for Image and Text . . . . . . . . . . . . . 51
ii
4.2.2 Real-Valued Image-Text Embedding . . . . . . . . . . . . . 52
4.2.3 Binary Image-Text Embedding . . . . . . . . . . . . . . . . 53
4.3 Deep Encoding Networks for TVDB . . . . . . . . . . . . . . . . . . 54
4.3.1 Image Binary Encoding Network . . . . . . . . . . . . . . . 56
4.3.1.1 Salient Semantic Region Selection . . . . . . . . . . 57
4.3.1.2 Regional Representation and Augmentation . . . . 57
4.3.1.3 Recurrent Network for Encoding . . . . . . . . . . 57
4.3.2 Sentence Binary Encoding Network . . . . . . . . . . . . . . 58
4.4 Stochastic Batch-Wise Code Learning . . . . . . . . . . . . . . . . . 59
4.4.1 Batch-wise alternating optimization . . . . . . . . . . . . . . 59
4.4.2 Stochastic-batched training procedure. . . . . . . . . . . . . 62
4.4.3 Out-of-Sample Extension . . . . . . . . . . . . . . . . . . . . 63
4.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.5.1 Implementation Details . . . . . . . . . . . . . . . . . . . . . 63
4.5.2 Experimental settings . . . . . . . . . . . . . . . . . . . . . . 66
4.5.2.1 Microsoft COCO [102] . . . . . . . . . . . . . . . . 66
4.5.2.2 IAPR TC-12 [45] . . . . . . . . . . . . . . . . . . . 66
4.5.2.3 INRIA Web Queries [83] . . . . . . . . . . . . . . . 69
4.5.3 Comparison with Existing Methods . . . . . . . . . . . . . . 69
4.5.3.1 Baselines . . . . . . . . . . . . . . . . . . . . . . . 71
4.5.3.2 Results and Analysis . . . . . . . . . . . . . . . . . 71
4.5.4 Ablation Study of Deep Encoding Network . . . . . . . . . . 74
4.5.4.1 Baselines . . . . . . . . . . . . . . . . . . . . . . . 74
4.5.4.2 Results and Analysis . . . . . . . . . . . . . . . . . 74
4.5.5 Training Efficiency and Encoding Time . . . . . . . . . . . . 75
4.5.5.1 Baselines . . . . . . . . . . . . . . . . . . . . . . . 76
4.5.5.2 Results and Analysis . . . . . . . . . . . . . . . . . 76
4.5.6 Intuitive Results . . . . . . . . . . . . . . . . . . . . . . . . 78
4.5.6.1 Positive Examples . . . . . . . . . . . . . . . . . . 79
4.5.6.2 Failure Cases . . . . . . . . . . . . . . . . . . . . . 79
4.5.7 Hyper-Parameters . . . . . . . . . . . . . . . . . . . . . . . . 80
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
iii
5 Deep Hashing for Zero-Shot Image-Sketch Retrieval 82
5.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . 82
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.3 The Proposed ZSIH Model . . . . . . . . . . . . . . . . . . . . . . . 87
5.3.1 Network Overview . . . . . . . . . . . . . . . . . . . . . . . 89
5.3.1.1 Image/Sketch Encoding Networks . . . . . . . . . . 89
5.3.1.2 Multi-Modal Network as Code Learner . . . . . . . 90
5.3.2 Fusing Sketch and Image with Kronecker Layer . . . . . . . 91
5.3.3 Semantic-Relation-EnhancedHiddenRepresentationwithGraph
Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.3.4 Stochastic Neurons and Decoding Network . . . . . . . . . . 93
5.3.5 Learning Objective and Optimization . . . . . . . . . . . . . 95
5.3.6 Out-of-Sample Extension . . . . . . . . . . . . . . . . . . . . 96
5.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.4.1 Implementation Details . . . . . . . . . . . . . . . . . . . . . 97
5.4.2 Zero-shot experimental settings . . . . . . . . . . . . . . . . 97
5.4.2.1 Sketchy Dataset [152] (Extended) . . . . . . . . . . 98
5.4.2.2 TU-Berlin Dataset [32] (Extended) . . . . . . . . . 98
5.4.3 Comparison with Existing Methods . . . . . . . . . . . . . . 98
5.4.3.1 Cross-Modal Hashing Baselines . . . . . . . . . . . 101
5.4.3.2 Zero-Shot Baselines . . . . . . . . . . . . . . . . . 101
5.4.3.3 Sketch-Image Mapping Baselines . . . . . . . . . . 101
5.4.3.4 Results and Analysis . . . . . . . . . . . . . . . . . 103
5.4.4 Conventional Fully-Supervised Category-Level SBIR . . . . . 104
5.4.5 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.4.5.1 Baselines . . . . . . . . . . . . . . . . . . . . . . . 105
5.4.5.2 Results and Analysis . . . . . . . . . . . . . . . . . 106
5.4.6 Training Efficiency and Overfitting . . . . . . . . . . . . . . 107
5.4.6.1 Overfitting . . . . . . . . . . . . . . . . . . . . . . 107
5.4.6.2 Training Efficiency . . . . . . . . . . . . . . . . . . 108
5.4.7 Qualitative Results and Failure Cases . . . . . . . . . . . . . 109
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
iv
6 Conclusion and Future Work 111
6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.1.1 Unsupervised Deep Image Hashing . . . . . . . . . . . . . . 111
6.1.2 Image-Sentence Cross-Modal Hashing . . . . . . . . . . . . . 111
6.1.3 Zero-Shot Cross-Modal Hashing . . . . . . . . . . . . . . . . 112
6.2 Open Questions and Future Work . . . . . . . . . . . . . . . . . . . 112
6.2.1 Binary Representations for Video Sequences . . . . . . . . . 112
6.2.2 Improved Reparametrization Tricks for Discrete Variables . . 113
6.2.3 Binary Representations for Fine-Grained Retrieval . . . . . . 113
A Quantitative Retrieval Evaluation Measures 115
A.1 Precision@K . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
A.2 Mean-Average Precision . . . . . . . . . . . . . . . . . . . . . . . . 115
A.3 HD2 Precision and Recall . . . . . . . . . . . . . . . . . . . . . . . 116
B Qualitative Results of DVB 117
C Qualitative Results of TVDB 120
D Failure Cases of TVDB 129
E Qualitative Comparison between ZSIH and DSH 130
F More Qualitative Results of ZSIH 132
G Failure Cases of ZSIH 135
Bibliography 136
v