Spark GraphX in Action

282 Pages·2017·8.49 MB·English

Michael S. Malak
Robin East
M A N N I N G
Spark GraphX
in Action
MICHAEL S. MALAK
ROBIN EAST
MANNING
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: [e-mail]
©2016 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial caps
or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.
Manning Publications Co. Development editor: Marina Michaels
20 Baldwin Road Technical development editors: Michael Roberts
PO Box 761 and John Guthrie
Shelter Island, NY 11964 Copyeditor: Corbin Collins
Proofreader: Melody Dolab
Technical proofreader: Antonio Magnaghi
Typesetter: Dottie Marsico
Cover designer: Marija Tudor
ISBN 9781617292521
Printed in the United States of America
1 2 3 4 5 6 7 8 9 10 – EBM – 21 20 19 18 17 16
brief contents
PART 1 SPARK AND GRAPHS ......................................................... 1
1 ■ Two important technologies: Spark and graphs 3
2 ■ GraphX quick start 24
3 ■ Some fundamentals 32
PART 2 CONNECTING VERTICES..................................................59
4 ■ GraphX Basics 61
5 ■ Built-in algorithms 90
6 ■ Other useful graph algorithms 110
7 ■ Machine learning 125
PART 3 OVER THE ARC .............................................................165
8 ■ The missing algorithms 167
9 ■ Performance and monitoring 187
10 ■ Other languages and tools 216
iii

contents
preface xi
acknowledgments xiii
about this book xiv
about the cover illustration xviii
PART 1 SPARK AND GRAPHS..............................................1
1 Two important technologies: Spark and graphs 3
1.1 Spark: the step beyond Hadoop MapReduce 4
The elusive definition of Big Data 6 ■ Hadoop: the world before
Spark 6 ■ Spark: in-memory MapReduce processing 7
1.2 Graphs: finding meaning from relationships 9
Uses of graphs 10 ■ Types of graph data 12
Plain RDBMS inadequate for graphs 14
1.3 Putting them together for lightning fast graph processing:
Spark GraphX 14
Property graph: adding richness 15 ■ Graph partitioning: graphs
meet Big Data 17 ■ GraphX lets you choose: graph parallel or data
parallel 19 ■ Various ways GraphX fits into a processing flow 19
GraphX vs. other systems 21 ■ Storing the graphs: distributed file
storage vs. graph database 23
1.4 Summary 23
v
vi CONTENTS
2 GraphX quick start 24
2.1 Getting set up and getting data 25
2.2 Interactive GraphX querying using the Spark Shell 26
2.3 PageRank example 29
2.4 Summary 31
3 Some fundamentals 32
3.1 Scala, the native language of Spark 33
Scala’s philosophy: conciseness and expressiveness 33
Functional programming 34 ■ Inferred typing 38
Class declaration 39 ■ Map and reduce 40
Everything is a function 41 ■ Java interoperability 42
3.2 Spark 43
Distributed in-memory data: RDDs 43 ■ Laziness 44
Cluster requirements and terminology 47 ■ Serialization 48
Common RDD operations 48 ■ Hello World with Spark
and sbt 51
3.3 Graph terminology 52
Basics 52 ■ RDF graphs vs. property graphs 55
Adjacency matrix 56 ■ Graph querying systems 56
3.4 Summary 57
PART 2 CONNECTING VERTICES......................................59
4 GraphX Basics 61
4.1 Vertex and edge classes 61
4.2 Mapping operations 67
Simple graph transformation 67 ■ Map/Reduce 68
Iterated Map/Reduce 72
4.3 Serialization/deserialization 74
Reading/writing binary format 74 ■ JSON format 76
GEXF format for Gephi visualization software 78
4.4 Graph generation 80
Deterministic graphs 80 ■ Random graphs 81
4.5 Pregel API 83
4.6 Summary 89
CONTENTS vii
5 Built-in algorithms 90
5.1 Seek out authoritative nodes: PageRank 91
PageRank algorithm explained 91 ■ Invoking PageRank in
GraphX 92 ■ Personalized PageRank 94
5.2 Measuring connectedness: Triangle Count 95
Uses of Triangle Count 96 ■ Slashdot friends and foes
example 96
5.3 Find the fewest hops: ShortestPaths 99
5.4 Finding isolated populations: Connected
Components 100
Predicting social circles 101 ■ Reciprocated love only, please:
Strongly Connected Components 106
5.5 Community detection: LabelPropagation 107
5.6 Summary 108
6 Other useful graph algorithms 110
6.1 Your own GPS: Shortest Paths with Weights 111
6.2 Travelling Salesman: greedy algorithm 115
6.3 Route utilities: Minimum Spanning Trees 117
Deriving taxonomies with Word2Vec and Minimum
Spanning Trees 121
6.4 Summary 124
7 Machine learning 125
7.1 Supervised, unsupervised, and semi-supervised
learning 126
7.2 Recommend a movie: SVDPlusPlus 128
Explanation of the Koren formula 134
7.3 Using GraphX With MLlib 135
Determine topics: Latent Dirichlet Allocation 135 ■ Detect spam:
LogisticRegressionWithSGD 143 ■ Image segmentation (for
computer vision) using Power Iteration Clustering 147
7.4 Poor man’s training data: graph-based semi-supervised
learning 151
K-Nearest Neighbors graph construction 154 ■ Semi-supervised
learning label propagation 160
7.5 Summary 164
viii CONTENTS
PART 3 OVER THE ARC.................................................165
8 The missing algorithms 167
8.1 Missing basic graph operations 168
Common sense subgraphs 168 ■ Merge two graphs 169
8.2 Reading RDF graph files 172
Matching vertices and constructing the graph 173
Improving performance with IndexedRDD, the RDD
HashMap 174
8.3 Poor man’s graph isomorphism: finding missing
Wikipedia infobox items 179
8.4 Global clustering coefficient: compare
connectedness 184
8.5 Summary 186
9 Performance and monitoring 187
9.1 Monitoring your Spark application 188
How Spark runs your application 188 ■ Understanding
your application runtime with Spark monitoring 191
History server 201
9.2 Configuring Spark 203
Utilizing all CPU cores 206
9.3 Spark performance tuning 207
Speeding up Spark with caching and persistence 207
Checkpointing 210 ■ Reducing memory pressure with
serialization 211
9.4 Graph partitioning 213
9.5 Summary 215
10 Other languages and tools 216
10.1 Using languages other than Scala with GraphX 217
Using GraphX with Java 7 217 ■ Using GraphX with
Java 8 222 ■ Whether GraphX may gain Python or R bindings in
the future 222
10.2 Another visualization tool: Apache Zeppelin
plus d3.js 222
CONTENTS ix
10.3 Almost a database: Spark Job Server 225
Example: Query Slashdot friends degree of separation 226
More on using Spark Job Server 228
10.4 Using SQL with Spark graphs with GraphFrames 229
Getting GraphFrames, plus GraphX interoperability 230
Using SQL for convenience and performance 231
Searching for vertices with the Cypher subset 232
Slightly more complex isomorphic searching on YAGO 234
10.5 Summary 237
appendix A Installing Spark 238
appendix B Gephi visualization software 242
appendix C Resources: where to go for more 246
appendix D List of Scala tips in this book 249
index 251