Understanding complex datasets: data mining with matrix decompositions

257 Pages·2007·5.463 MB·English

by  David Skillicorn

Chapman & Hall/CRC
Data Mining and Knowledge Discovery Series
Understanding
Complex Datasets
Data Mining with
Matrix Decompositions
© 2007 by Taylor and Francis Group, LLC
C8326_FM.indd 1 4/2/07 4:25:36 PM
Chapman & Hall/CRC
Data Mining and Knowledge Discovery Series
SeRieSeDiToR
Vipin Kumar
University of minnesota
department of Computer science and engineering
minneapolis, minnesota, U.s.a
AiMS AND SCoPe
this series aims to capture new developments and applications in data mining and knowledge
discovery, while summarizing the computational tools and techniques useful in data analysis. this
series encourages the integration of mathematical, statistical, and computational methods and
techniques through the publication of a broad range of textbooks, reference works, and hand-
books. the inclusion of concrete examples and applications is highly encouraged. the scope of the
series includes, but is not limited to, titles in the areas of data mining and knowledge discovery
methods and applications, modeling, algorithms, theory and foundations, data and knowledge
visualization, data mining systems and tools, and privacy and security issues.
PubliSHeD TiTleS
Understanding Complex datasets: data mining with matrix decompositions
David Skillicorn
FoRTHCoMiNG TiTleS
CompUtationalmetHodsoF FeatUreseleCtion
Huanliu and Hiroshi Motoda
mUltimediadatamining:asystematicintroduction to Concepts and theory
Zhongfei Zhang and Ruofei Zhang
Constrained ClUstering:advances in algorithms,theory, and applications
Sugato basu,ian Davidson, and Kiri Wagstaff
textmining:theory, applications, and Visualization
Ashok Srivastava and Mehran Sahami
© 2007 by Taylor and Francis Group, LLC
C8326_FM.indd 2 4/2/07 4:25:36 PM
Chapman & Hall/CRC
Data Mining and Knowledge Discovery Series
Understanding
Complex Datasets
Data Mining with
Matrix Decompositions
David Skillicorn
© 2007 by Taylor and Francis Group, LLC
C8326_FM.indd 3 4/2/07 4:25:36 PM
Chapman & Hall/CRC
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487‑2742
© 2007 by Taylor & Francis Group, LLC
Chapman & Hall/CRC is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed in the United States of America on acid‑free paper
10 9 8 7 6 5 4 3 2 1
International Standard Book Number‑10: 1‑58488‑832‑6 (Hardcover)
International Standard Book Number‑13: 978‑1‑58488‑832‑1 (Hardcover)
This book contains information obtained from authentic and highly regarded sources. Reprinted
material is quoted with permission, and sources are indicated. A wide variety of references are
listed. Reasonable efforts have been made to publish reliable data and information, but the author
and the publisher cannot assume responsibility for the validity of all materials or for the conse‑
quences of their use.
No part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any
electronic, mechanical, or other means, now known or hereafter invented, including photocopying,
microfilming, and recording, or in any information storage or retrieval system, without written
permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.
copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC)
222 Rosewood Drive, Danvers, MA 01923, 978‑750‑8400. CCC is a not‑for‑profit organization that
provides licenses and registration for a variety of users. For organizations that have been granted a
photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
Library of Congress Cataloging‑in‑Publication Data
Skillicorn, David B.
Understanding complex datasets : data mining with matrix decompositions /
David Skillicorn.
p. cm. ‑‑ (Data mining and knowledge discovery series)
Includes bibliographical references and index.
ISBN 978‑1‑58488‑832‑1 (alk. paper)
1. Data mining. 2. Data structures (Computer science) 3. Computer
algorithms. I. Title. II. Series.
QA76.9.D343S62 2007
005.74‑‑dc22 2007013096
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
© 2007 by Taylor and Francis Group, LLC
C8326_FM.indd 4 4/2/07 4:25:36 PM
v
For Jonathan M.D. Hill, 1968–2006
© 2007 by Taylor and Francis Group, LLC
Contents
Preface xiii
1 Data Mining 1
1.1 What is data like? . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Data-mining techniques . . . . . . . . . . . . . . . . . . . . 5
1.2.1 Prediction. . . . . . . . . . . . . . . . . . . . . 6
1.2.2 Clustering. . . . . . . . . . . . . . . . . . . . . 11
1.2.3 Finding outliers . . . . . . . . . . . . . . . . . 16
1.2.4 Finding local patterns . . . . . . . . . . . . . . 16
1.3 Why use matrix decompositions? . . . . . . . . . . . . . . 17
1.3.1 Data that comes from multiple processes . . . 18
1.3.2 Data that has multiple causes. . . . . . . . . . 19
1.3.3 What are matrix decompositions used for? . . 20
2 Matrix decompositions 23
2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2 Interpreting decompositions . . . . . . . . . . . . . . . . . 28
2.2.1 Factor interpretation – hidden sources . . . . . 29
2.2.2 Geometric interpretation – hidden clusters . . 29
2.2.3 Component interpretation – underlying pro-
cesses . . . . . . . . . . . . . . . . . . . . . . . 32
2.2.4 Graph interpretation – hidden connections . . 32
vii
© 2007 by Taylor and Francis Group, LLC
viii Contents
2.2.5 Summary . . . . . . . . . . . . . . . . . . . . . 34
2.2.6 Example . . . . . . . . . . . . . . . . . . . . . 34
2.3 Applying decompositions . . . . . . . . . . . . . . . . . . . 36
2.3.1 Selecting factors, dimensions, components, or
waystations . . . . . . . . . . . . . . . . . . . . 36
2.3.2 Similarity and clustering . . . . . . . . . . . . 41
2.3.3 Finding local relationships . . . . . . . . . . . 42
2.3.4 Sparse representations . . . . . . . . . . . . . . 43
2.3.5 Oversampling . . . . . . . . . . . . . . . . . . . 44
2.4 Algorithm issues . . . . . . . . . . . . . . . . . . . . . . . . 45
2.4.1 Algorithms and complexity . . . . . . . . . . . 45
2.4.2 Data preparation issues . . . . . . . . . . . . . 45
2.4.3 Updating a decomposition. . . . . . . . . . . . 46
3 Singular Value Decomposition (SVD) 49
3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.2 Interpreting an SVD . . . . . . . . . . . . . . . . . . . . . 54
3.2.1 Factor interpretation . . . . . . . . . . . . . . . 54
3.2.2 Geometric interpretation . . . . . . . . . . . . 56
3.2.3 Component interpretation . . . . . . . . . . . . 60
3.2.4 Graph interpretation . . . . . . . . . . . . . . . 61
3.3 Applying SVD . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.3.1 Selectingfactors,dimensions,components,and
waystations . . . . . . . . . . . . . . . . . . . . 62
3.3.2 Similarity and clustering . . . . . . . . . . . . 70
3.3.3 Finding local relationships . . . . . . . . . . . 73
3.3.4 Sampling and sparsifying by removing values . 76
3.3.5 Using domain knowledge or priors . . . . . . . 77
3.4 Algorithm issues . . . . . . . . . . . . . . . . . . . . . . . . 77
3.4.1 Algorithms and complexity . . . . . . . . . . . 77
© 2007 by Taylor and Francis Group, LLC
Contents ix
3.4.2 Updating an SVD . . . . . . . . . . . . . . . . 78
3.5 Applications of SVD . . . . . . . . . . . . . . . . . . . . . 78
3.5.1 The workhorse of noise removal . . . . . . . . . 78
3.5.2 Informationretrieval–LatentSemanticIndex-
ing (LSI) . . . . . . . . . . . . . . . . . . . . . 78
3.5.3 Rankingobjectsandattributesbyinteresting-
ness . . . . . . . . . . . . . . . . . . . . . . . . 81
3.5.4 Collaborative filtering . . . . . . . . . . . . . . 81
3.5.5 Winnowing microarray data. . . . . . . . . . . 86
3.6 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.6.1 PDDP . . . . . . . . . . . . . . . . . . . . . . . 87
3.6.2 The CUR decomposition . . . . . . . . . . . . 87
4 Graph Analysis 91
4.1 Graphs versus datasets . . . . . . . . . . . . . . . . . . . . 91
4.2 Adjacency matrix . . . . . . . . . . . . . . . . . . . . . . . 95
4.3 Eigenvalues and eigenvectors . . . . . . . . . . . . . . . . . 96
4.4 Connections to SVD. . . . . . . . . . . . . . . . . . . . . . 97
4.5 Google’s PageRank . . . . . . . . . . . . . . . . . . . . . . 98
4.6 Overview of the embedding process . . . . . . . . . . . . . 101
4.7 Datasets versus graphs . . . . . . . . . . . . . . . . . . . . 102
4.7.1 Mapping Euclidean space to an affinity matrix 103
4.7.2 Mappinganaffinitymatrixtoarepresentation
matrix . . . . . . . . . . . . . . . . . . . . . . . 104
4.8 Eigendecompositions . . . . . . . . . . . . . . . . . . . . . 110
4.9 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.10 Edge prediction . . . . . . . . . . . . . . . . . . . . . . . . 114
4.11 Graph substructures. . . . . . . . . . . . . . . . . . . . . . 115
4.12 The ATHENS system for novel-knowledge discovery . . . . 118
4.13 Bipartite graphs . . . . . . . . . . . . . . . . . . . . . . . . 121
© 2007 by Taylor and Francis Group, LLC
x Contents
5 SemiDiscrete Decomposition (SDD) 123
5.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.2 Interpreting an SDD . . . . . . . . . . . . . . . . . . . . . 132
5.2.1 Factor interpretation . . . . . . . . . . . . . . . 133
5.2.2 Geometric interpretation . . . . . . . . . . . . 133
5.2.3 Component interpretation . . . . . . . . . . . . 134
5.2.4 Graph interpretation . . . . . . . . . . . . . . . 134
5.3 Applying an SDD . . . . . . . . . . . . . . . . . . . . . . . 134
5.3.1 Truncation . . . . . . . . . . . . . . . . . . . . 134
5.3.2 Similarity and clustering . . . . . . . . . . . . 135
5.4 Algorithm issues . . . . . . . . . . . . . . . . . . . . . . . . 138
5.5 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.5.1 Binary nonorthogonal matrix decomposition . 139
6 Using SVD and SDD together 141
6.1 SVD then SDD . . . . . . . . . . . . . . . . . . . . . . . . 142
6.1.1 Applying SDD to Ak . . . . . . . . . . . . . . . 143
6.1.2 Applying SDD to the truncated correlation
matrices . . . . . . . . . . . . . . . . . . . . . . 143
6.2 Applications of SVD and SDD together . . . . . . . . . . . 144
6.2.1 Classifying galaxies . . . . . . . . . . . . . . . 144
6.2.2 Mineral exploration . . . . . . . . . . . . . . . 145
6.2.3 Protein conformation . . . . . . . . . . . . . . 151
7 Independent Component Analysis (ICA) 155
7.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
7.2 Interpreting an ICA . . . . . . . . . . . . . . . . . . . . . . 159
7.2.1 Factor interpretation . . . . . . . . . . . . . . . 159
7.2.2 Geometric interpretation . . . . . . . . . . . . 159
7.2.3 Component interpretation . . . . . . . . . . . . 160
© 2007 by Taylor and Francis Group, LLC
Contents xi
7.2.4 Graph interpretation . . . . . . . . . . . . . . . 160
7.3 Applying an ICA . . . . . . . . . . . . . . . . . . . . . . . 160
7.3.1 Selecting dimensions . . . . . . . . . . . . . . . 160
7.3.2 Similarity and clustering . . . . . . . . . . . . 161
7.4 Algorithm issues . . . . . . . . . . . . . . . . . . . . . . . . 161
7.5 Applications of ICA . . . . . . . . . . . . . . . . . . . . . . 163
7.5.1 Determining suspicious messages . . . . . . . . 163
7.5.2 Removing spatial artifacts from microarrays. . 166
7.5.3 Finding al Qaeda groups . . . . . . . . . . . . 169
8 Non-Negative Matrix Factorization (NNMF) 173
8.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
8.2 Interpreting an NNMF . . . . . . . . . . . . . . . . . . . . 177
8.2.1 Factor interpretation . . . . . . . . . . . . . . . 177
8.2.2 Geometric interpretation . . . . . . . . . . . . 177
8.2.3 Component interpretation . . . . . . . . . . . . 178
8.2.4 Graph interpretation . . . . . . . . . . . . . . . 178
8.3 Applying an NNMF . . . . . . . . . . . . . . . . . . . . . . 178
8.3.1 Selecting factors . . . . . . . . . . . . . . . . . 178
8.3.2 Denoising . . . . . . . . . . . . . . . . . . . . . 179
8.3.3 Similarity and clustering . . . . . . . . . . . . 180
8.4 Algorithm issues . . . . . . . . . . . . . . . . . . . . . . . . 180
8.4.1 Algorithms and complexity . . . . . . . . . . . 180
8.4.2 Updating . . . . . . . . . . . . . . . . . . . . . 180
8.5 Applications of NNMF . . . . . . . . . . . . . . . . . . . . 181
8.5.1 Topic detection . . . . . . . . . . . . . . . . . . 181
8.5.2 Microarray analysis . . . . . . . . . . . . . . . 181
8.5.3 Mineral exploration revisited . . . . . . . . . . 182
© 2007 by Taylor and Francis Group, LLC