1.1 Thesis Statement

236 Pages·2015·2.43 MB·English

http://researchcommons.waikato.ac.nz/

Research Commons at the University of Waikato

Copyright Statement:
The digital copy of this thesis is protected by the Copyright Act 1994 (New Zealand).
The thesis may be consulted by you, provided you comply with the provisions of the
Act and the following conditions of use:
 Any use you make of these documents or images must be for research or private
study purposes only, and you may not make them available to any other person.
 Authors control the copyright of their thesis. You will recognise the author’s right
to be identified as the author of the thesis, and due acknowledgement will be
made to the author where appropriate.
 You will obtain the author’s permission before publishing any material from the
thesis.
A Predictive Model for the Parallel
Processing of Digital Libraries
A thesis
submitted in fulfilment
of the requirements for the degree
of
Doctor of Philosophy
at the
University of Waikato
by
John Thompson
University of Waikato
2015
ii
iii
Abstract
The computing world is facing the problem of a seemingly exponential
increase in the amount of raw digital data, and the speed at which it is being
collected, is eclipsing our ability to manage it manually. Combine this with the
increasing expectations of a growing number of experienced computer users—
including real-time access and a demand for expensive-to-process file types
such as multimedia—and it is not hard to understand why managing data of
thisscaleandprovidingtimelyaccesstousefulinformationrequiresspecialized
algorithms, techniques, and software.
Digital libraries are being used to help address these challenges. Drawing
upon knowledge learned through traditional library science, digital libraries
excel in providing structured user access to a wide variety of documents. They
increasingly include tools for managing, moderating, and marking up these
documents. Furthermore, they often feature phases where documents are
independently processed and so can benefit from the application of parallel
processing techniques—the focus of this thesis. Whether a digital library col-
lection can benefit from parallel processing depends on considerations such
as document type, processing cost per document, number of documents, and
file-system input/output.
To aid in deciding when to apply parallel processing techniques to digital
libraries, this thesis explores the creation a model for predicting key outcomes
of leveraging such techniques. It does so by implementing parallel processing
in three distinct open-source digital library tools, undertaking experiments
designed to measure key processing features (such as processing time versus
number of compute nodes), and applying machine learning techniques to these
features in order to derive a predictive model.
The model created predicts parallel processing performance at 96% ac-
curacy (adjusted r2) for a number of exemplar collection types. The result
is a generally applicable tool for estimating the benefits of applying parallel
processing to a wide range of digital collections.
iv ABSTRACT
v
Acknowledgements
First and foremost I wish to thank my wife, Dr Lin-Yi Chou, for her support,
encouragement, and help. It was her journey to a PhD that inspired me to
startmyown, andherknowledgeonmathematicalmodelsprovedinvaluableas
I tackled developing my own. I would also thank my daughter Victoria, whose
impending birth provided excellent incentive to hurry-up and finish writing...
and whose smiles and giggles since have helped me get through revisions. I
also owe much to my family and friends who never grew tired of my lengthy
explanations/short lectures of what my research entailed (or at least were very
good at pretending not to grow tired).
This thesis would not have happened without the valuable guidance and
feedback from my supervisors, Associate Professor David Bainbridge and Pro-
fessor Ian Witten. My sympathy to the several red pens they have worn out
while error-checking my writing.
I would also like to extend my thanks to the Technical Support Group
at the Department of Computer and Mathematical Sciences—who kindly set-
up the Medusa cluster and provided help and expertise during the (many)
times I encountered/caused technical issues—and acknowledge the support of
the Department and the University of Waikato by way of Scholarships and
resources.
vi ABSTRACT
vii
Contents
1 Introduction 1
1.1 Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Research Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5 Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Very Large-scale Digital Libraries 11
2.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.1 The Digital Library Environment . . . . . . . . . . . . 13
2.2 Real World Case Studies . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 HathiTrust . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.2 CADLIS . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.3 The Internet Archive . . . . . . . . . . . . . . . . . . . 20
2.2.4 US Library of Congress: Web Archive . . . . . . . . . . 22
2.2.5 US Library of Congress: Tweet Archive . . . . . . . . . 23
2.2.6 Elephind . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.7 Informedia . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.8 CiteSeerX . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2.9 Google Search Engine . . . . . . . . . . . . . . . . . . 27
2.3 General Purpose Open-source Digital Libraries . . . . . . . . . 28
2.3.1 DSpace . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.3.2 Greenstone . . . . . . . . . . . . . . . . . . . . . . . . 30
2.3.3 Terrier IR Platform . . . . . . . . . . . . . . . . . . . . 31
2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3 Parallel Processing 37
3.1 Shared and Share Nothing Architectures . . . . . . . . . . . . 40
3.2 Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.3 Federated Search Digital Libraries . . . . . . . . . . . . . . . . 44
3.4 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.1 Measuring Time . . . . . . . . . . . . . . . . . . . . . . 46
viii CONTENTS
3.4.2 Databases . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.4.3 File System . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.4 Video Processing . . . . . . . . . . . . . . . . . . . . . 60
3.4.5 Very Large-scale Corpus . . . . . . . . . . . . . . . . . 61
4 Implementations 65
4.1 Greenstone . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.1.2 Scope of Work . . . . . . . . . . . . . . . . . . . . . . . 70
4.1.3 Open MPI and the SPMD technique . . . . . . . . . . 71
4.1.4 Parallel Greenstone using Open MPI . . . . . . . . . . 73
4.1.5 Greenstone using Hadoop . . . . . . . . . . . . . . . . 77
4.2 DSpace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.1 DSpace using Open MPI . . . . . . . . . . . . . . . . . 85
4.2.2 DSpace using Hadoop . . . . . . . . . . . . . . . . . . 86
4.3 Terrier IR Platform . . . . . . . . . . . . . . . . . . . . . . . . 87
4.3.1 Terrier using OpenMPI . . . . . . . . . . . . . . . . . . 88
4.3.2 Terrier using Hadoop . . . . . . . . . . . . . . . . . . . 90
4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5 Experimental Results 95
5.1 Serial Importing . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.1.1 Serial import of text documents . . . . . . . . . . . . . 96
5.1.2 Serial import of multimedia files . . . . . . . . . . . . . 99
5.2 Parallel Importing . . . . . . . . . . . . . . . . . . . . . . . . 101
5.2.1 CPU Utilisation . . . . . . . . . . . . . . . . . . . . . . 102
5.2.2 Parallel import of text documents . . . . . . . . . . . . 105
5.2.3 Parallel import of image documents . . . . . . . . . . . 108
5.2.4 Parallel import of audio files . . . . . . . . . . . . . . . 110
5.2.5 Parallel import of multimedia files . . . . . . . . . . . . 112
5.2.6 Hadoop Framework . . . . . . . . . . . . . . . . . . . . 114
5.2.7 Compute Nodes . . . . . . . . . . . . . . . . . . . . . . 116
5.2.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.3 Processor versus File I/O Costs . . . . . . . . . . . . . . . . . 119
5.4 Miscellaneous Experiments . . . . . . . . . . . . . . . . . . . . 122
5.4.1 Database performance . . . . . . . . . . . . . . . . . . 122
5.4.2 Batch Size . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.4.3 HDFS Replication Factor . . . . . . . . . . . . . . . . 126
5.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
CONTENTS ix
6 The Predictive Model 131
6.1 Mathematical Modelling . . . . . . . . . . . . . . . . . . . . . 131
6.2 Simple Division Model . . . . . . . . . . . . . . . . . . . . . . 135
6.3 PPoDL Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
6.3.1 Multicolinearity . . . . . . . . . . . . . . . . . . . . . . 147
6.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
7 Application of the Model 151
7.1 Jamendo Audio Collection . . . . . . . . . . . . . . . . . . . . 153
7.2 ReplayMe! Video Collection . . . . . . . . . . . . . . . . . . . 154
7.3 PapersPast Historic Newspaper Collection . . . . . . . . . . . 158
7.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8 Conclusion 163
8.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
8.2 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . 167
8.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
8.4 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . 172
Appendices 195
A Exemplar Digital Collection Processing Measurements 197
B Selected Parallel Processing Visualisations 199
C Software 211