Table Of ContentBIG DATA
MINING AND
COMPLEXITY
THE SAGE QUANTITATIVE RESEARCH KIT
Big Data Mining and Complexity by Brian Castellani and Rajeev Rajaram is the 11th volume
in The SAGE Quantitative Research Kit. This book can be used together with the other titles in
the Kit as a comprehensive guide to the process of doing quantitative research, but is equally
valuable on its own as a practical introduction to Data Mining and ‘Big Data’.
Editors of The SAGE Quantitative Research Kit:
Malcolm Williams – Cardiff University, UK
Richard D. Wiggins – UCL Social Research Institute, UK
D. Betsy McCoach – University of Connecticut, USA
Founding editor:
The late W. Paul Vogt – Illinois State University, USA
BIG DATA
MINING AND
COMPLEXITY
BRIAN C. CASTELLANI
RAJEEV RAJARAM
THE SAGE QUANTITATIVE RESEARCH KIT
SAGE Publications Ltd Brian C. Castellani and Rajeev Rajaram 2021
1 Oliver’s Yard
55 City Road This volume published as part of The SAGE Quantitative
London EC1Y 1SP Research Kit (2021), edited by Malcolm Williams,
Richard D. Wiggins and D. Betsy McCoach.
SAGE Publications Inc.
2455 Teller Road Apart from any fair dealing for the purposes of research,
Thousand Oaks, California 91320 private study, or criticism or review, as permitted under the
Copyright, Designs and Patents Act, 1988, this publication
SAGE Publications India Pvt Ltd may not be reproduced, stored or transmitted in any form,
B 1/I 1 Mohan Cooperative Industrial Area or by any means, without the prior permission in writing of
Mathura Road the publisher, or in the case of reprographic reproduction,
in accordance with the terms of licences issued by the
Copyright Licensing Agency. Enquiries concerning
New Delhi 110 044
reproduction outside those terms should be sent to
SAGE Publications Asia-Pacific Pte Ltd
the publisher.
3 Church Street
#10-04 Samsung Hub
Singapore 049483
Library of Congress Control Number: 2020943076
Editor: Jai Seaman
Assistant editor: Charlotte Bush
British Library Cataloguing in Publication data
Production editor: Manmeet Kaur Tura
Copyeditor: QuADS Prepress Pvt Ltd
A catalogue record for this book is available from the
Proofreader: Elaine Leek
British Library
Indexer: Cathryn Pritchard
Marketing manager: Susheel Gokarakonda
Cover design: Shaun Mercier
Typeset by: C&M Digitals (P) Ltd, Chennai, India
Printed in the UK
ISBN 978-1-5264-2381-8
At SAGE we take sustainability seriously. Most of our products are printed in the UK using responsibly sourced
papers and boards. When we print overseas, we ensure sustainable papers are used as measured by the PREPS
grading system. We undertake an annual audit to monitor our sustainability.
Dedicated to Maggie, Ruby and Swathi
Contents
List of Figures xiii
About the Authors xv
1 Introduction 1
The Joys of Travel 2
Data Mining and Big Data Travel 2
Part I: Thinking Critically and Complex 3
Organisation of Part I 4
Part II: The Tools and Techniques of Data Mining 5
SAGE Quantitative Research Kit 7
COMPLEX-IT and the SACS Toolkit 7
The Airline Industry: A Case Study 8
PART I THINKING CRITICALLY AND COMPLEX 11
2 The Failure of Quantitative Social Science 13
Quantitative Social Science, Then and Now 15
The Three Phases of Science 15
What You Should Have Learned in Statistics Class 16
So, Why Didn’t You Learn These Things? 19
Changing the Social Life of Data Analysis 21
3 What Is Big Data? 23
Big Data as Information Society 24
Big Data as Global Network Society 25
The Socio-Cybernetic Web of Big Data 26
Big Data Databases 28
The Failed Promise of Big Data 31
4 What Is Data Mining? 33
A Bit of Data Mining History 34
The Data Mining Process 35
viii BIG DATA MINING AND COMPLEXITY
The ‘Black Box’ of Data Mining 36
Validity and Reliability 38
The Limits of Normalised Probability Distributions 39
Fitting Models to Data 40
Data Mining’s Various Tasks 40
5 The Complexity Turn 43
Mapping the Complexity Turn 44
Data Mining and Big Data as Complexity Science 46
Top Ten List About Complexity 46
Number 1 47
Number 2 47
Number 3 47
Number 4 47
Number 5 48
Number 6 49
Number 7 49
Number 8 50
Number 9 50
Number 10 50
PART II THE TOOLS AND TECHNIQUES OF DATA MINING 53
6 Case-Based Complexity: A Data Mining Vocabulary 55
Case-Based Complexity 56
COMPLEX-IT and the SACS Toolkit 57
The Ontology of Big Data 59
The Archaeology of Big Data Ontologies 59
The Formalisms of Case-Based Complexity 63
What Is a Case? 63
Two Definitions of a Vector 64
Cataloguing and Grouping Case Profiles 65
Mathematical Distance Between Cases 66
Cataloguing Case Profiles 67
Diversity of Case Profiles 69
The Notion of Time t 69
Profiles That Vary With Time 69
Static Clustering in Time 70
Dynamic Clustering of Trajectories 70
CONTENTS ix
A Vector Field 71
The State Space 71
A Discrete Vector Field of Velocities of Trajectories 72
Why Do We Need a Vector Field? 75
7 Classification and Clustering 79
Top Ten Airlines 80
Classification Versus Clustering 80
Classification Schemes 81
Decision Tree Induction 82
Nearest Neighbour Classifier 83
Artificial Neural Networks 83
Support Vector Machines 84
Clustering 84
Hierarchical Clustering Methods 85
Partitioning Methods 86
Probability Density-Based Methods 87
Soft Computing Methods 87
8 Machine Learning 91
The Smart Airline Industry 92
What Is Machine Intelligence? 93
Machine Intelligence for Big Data 93
Examples of Data Mining Methods Based on Machine Intelligence 94
Artificial Neural Networks 95
Genetic Algorithms and Evolutionary Computation 100
Swarming Travel Routes 100
Overview of Swarm Intelligence 101
9 Predictive Analytics and Data Forecasting 105
Predictive Analytics and Data Forecasting: A Very Brief History 106
Overview of Techniques 107
Bayesian Statistics 107
Bayesian Parameter Estimation 108
Bayesian Hypothesis Testing 110
Decision Trees 112
Neural Networks 112
Regression (Linear, Non-Linear and Logistic) 112
