An Empirical Comparison of Big Graph Frameworks in the Context of Network Analysis

1.5 MB·

by  Jannis Koch

An Empirical Comparison of Big Graph Frameworks in the Context of
Network Analysis
Jannis Koch Christian L. Staudt Maximilian Vogel Henning Meyerhenke
Abstract
Many applications – think for example of the networks
collectedbyonlinesocialnetworkingsites–produceenor-
Complex networks are relational data sets commonly rep- mousamountsofnetworkdatathatexceedthememoryof
resented as graphs. The analysis of their intricate struc- any single computer, calling for distributed solutions. A
6
ture is relevant to many areas of science and commerce, distributedsystemconsistsofasetofautonomousproces-
1
anddatasetsmayreachsizesthatrequiredistributedstor- sors, each with their own memory, running software that
0
age and processing. performs computations and exchanges data as messages
2
We describe and compare programming models for dis- via a network. Distributed systems are highly scalable
n
tributed computing with a focus on graph algorithms for in the sense that additional computers can be added eas-
a
J large-scale complex network analysis. Four frameworks – ily. Of course, additional computers may mean additional

3 GraphLab, Apache Giraph, Giraph++ and Apache Flink overhead as well.
Overview ProgrammingModels&Frameworks
– are used to implement algorithms for the representative |

]
problemsConnectedComponents,CommunityDetection,
C DistributedComputing
PageRank and Clustering Coefficients. The implementa-
D
General-Purpose Graph-Specific
tions are executed on a computer cluster to evaluate the
.
s frameworks’ suitability in practice and to compare their
c
performancetothatofthesingle-machine,shared-memory MapReduce PACT Vertex- Graph-
[
parallel network analysis package NetworKit. Out of the Centric Centric

1 distributed frameworks, GraphLab and Apache Giraph
v generally show the best performance. Pregel GAS
9
8 In our experiments a cluster of eight computers run-
2 ning Apache Giraph enables the analysis of a network
Hadoop ApacheFlink Apache GraphLab Giraph++
0 with about 2 billion edges, which is too large for a sin-
Giraph
0 gle machine of the same type. However, for networks that
.
1 fit into memory of one machine, the performance of the
0 shared-memory parallel implementation is far better than
6
thedistributedones. Thestudyprovidesexperimentalev-
1 FStaiugdtu–Cromeplex1N:etwoOrkAnvalyesirsovnDiiestrwibutedaSynstedms classification of distributed pro- 9
idence for selecting the appropriate framework depending
: grammingmodels(black)andimplementations(blue)con-
v on the task and data volume.
sidered in this paper.
i
X
r Keywords: big graph frameworks, distributed com-
a
puting, graph algorithms, complex networks, network Multiple programming models for distributed par-
analysis allel computing have been introduced, some general-
purpose (like MapReduce), others focus on the pro-
cessing of large graphs (like Pregel) – sometimes
1 Introduction
called Big Graph frameworks to express their re-
lation to Big Data. Among the software frame-
Complexnetworksaredatasetswhich mapitemsandthe works implementing these programming models, we
links among them, forming complex relational patterns. consider Apache Giraph [Apache, 2015b], Apache Gi-
They are typically represented as graphs, i.e. a set of n raph++ [Tian et al., 2013], GraphLab [Dato, 2015] and
vertices V and m edges E. This abstraction is as gen- Apache Flink [Apache, 2015a] in our study. An overview
eral as it is powerful, and has been employed in numerous of programming models and their implementations is
problemdomains[Costa et al., 2011]. Thetaskofnetwork shown in Figure 1.
analysis is to characterize the structure of the network as Consideringperformance,thescalabilityandexpressive-
a whole or identify important elements in it, typically by nessenabledbysuchframeworkscomesatthecostofover-
applying a collection of graph algorithms tailored to this head: A recent experimental study [Satish et al., 2014]
purpose (see Section 4). compares implementations of graph algorithms based on
1
2 Programming Models for Dis-
frameworks (including GraphLab and Giraph) to ad hoc,
hand-optimized code and observes a substantial perfor- tributed Graph Processing
mance gap. Similar algorithms chosen for our study were
also used recently by [McColl et al., 2014]. They experi-
2.1 General-purpose Models
ence the aforementioned performance gap, too. However,
they test only on graphs generated with the synthetic
MapReduce and its Limitations. MapRe-
R-MAT model, which lacks realism in several respects.
duce is a programming model for general distributed
Moreover, our focus is on distributed frameworks and we
computation [Dean and Ghemawat, 2008], and is of-
include some not used in their study.
ten considered the de facto standard for this pur-
pose [Karloff et al., 2010]. A basic MapReduce program
Complexnetworkspresentcertaininherentperformance
is defined by two user functions - a map and a re-
challenges for distributed systems: Computation is data-
duce function. Both functions receive and emit data
driven in this case, i.e. it is largely determined by the
as key-value pairs. A program is executed in three
graph structure. This makes it difficult to exploit oppor-
phases - a map, a shuffle and a reduce phase. Although
tunities for parallelism, because they depend heavily on
MapReduce is able to express many common graph al-
the input. Real-world networks often have a skewed de-
gorithms [Lin and Dyer, 2010], it has limited practicality
gree distribution, which adversely affects load balance. In
in this area: Many graph analysis algorithms are itera-
synchronous execution models, the few high-degree ver-
tive. Since the MapReduce model does not provide seam-
tices will delay the whole system. Performance depends
less support for iteration, this can only be reproduced by
also on how vertices are distributed among the processor
schedulingseveralconsecutivejobs,resultinginsignificant
nodes, i.e. the graph must be partitioned so as to re-
overhead. Because each iteration needs the initial input
ducecommunicationvolumeandtobalancetheworkload
graph data, the map function does not only emit an in-
– a problem whose optimal solution is NP-hard to com-
termediate result for the input vertex, but also the vertex
pute. Given these challenges, performance experiments
data itself, i.e. the vertex adjacency list and its current
on real-world networks are crucial for choosing the right
state. Consequently, graph data is sent over the network
framework for the problem at hand.
unnecessarily. Workaroundslikeemittingthegraphstruc-
turemakeMapReduceimplementationshardertodevelop
and understand. We do not include MapReduce in the
experimental study, since the models we discuss in the
following offer more intuitive ways of expressing graph al-
gorithms.
Contribution. We address the shortage of compar-
ative studies and report performance results for several PACT Model. The PACT ("Parallelization Con-
distributed frameworks. Apache Giraph, GraphLab, Gi- tract")model[Battré et al., 2010]wasproposedtoextend
raph++andApacheFlinkaredescribedandcomparedre- MapReduceinbothfunctionalityandoptimizationoppor-
garding both theoretical and practical aspects. The theo- tunities. It can be seen as a generalization of MapReduce
reticalareasofcomparisonincludetheprogrammingmod- that offers additional operators. A PACT operator re-
els and their suitability for expressing graph algorithms. ceives a user-defined function and an input data set con-
Foranexperimentalcomparison,severalwell-knowngraph taining tuples of predefined types. It applies the user-
algorithmswereimplementedusingtheframeworks. With functiontotheinputdatainparallelandreturnstheout-
real-world complex networks (mainly from web applica- put. The operator is described by a) the Input Contract,
tions) as input, the implementations were executed on a which defines how the user function can be applied to the
computer cluster to reveal how the frameworks compare input data set in parallel and b) the Output Contract,
inpracticewithregardstoperformanceandmemoryfoot- which gives additional information on the data output. It
print. Additionally, we compare performance with imple- can be optionally attached by the programmer and allows
mentations in a shared-memory parallel framework, Net- the PACT compiler to apply optimization techniques to
worKit [Staudt et al., 2014]. While the small-scale clus- the execution. When writing a PACT program, the pro-
ter we had available for this study is not enough to test grammer uses an operator by defining its user function
out the true scalability of distributed frameworks, our re- andtheinputdataitoperateson. Theoutputofanoper-
sults enable an informed choice on which framework to ator can be used as the input of another, which creates a
use for a given network analysis problem. It becomes directed acyclic graph of operators (see Figure 3). Unlike
clearthat,comparedtoshared-memoryparallelsolutions, MapReduce,thePACTmodeldoesnotforceaspecificor-
distributed frameworks come with a significant overhead. der of the operators. The PACT compiler examines the
Consequently, opting for a distributed solution must be structure of the resulting operator graph and infers an
justifiedbythefactthatthegraphstobeprocessedcannot optimization strategy for the execution, e.g. rearranging
fit into main memory, which currently applies to graphs operators in the graph. A PACT program therefore has a
above the size range of several billion edges. declarative style, reminiscient of the SQL language. The
2
PACT model is implemented by the Apache Flink frame-
P P
work [Apache, 2015a]. 1 2
4 6
1 2
2.2 Vertex-centric Models
5
Vertex-centricmodelsexpressprogramsfromavertexper-
G 3 7
spective. Theseprogramsareexecutediterativelyforeach 1
vertex in the graph. The models are tailored to graph al-
gorithms and, by incorporating vertices and edges, they
include the data dependencies of graphs in the program- Figure 2: A graph with two partitions P and P . The
1 2
ming model. dashed vertices in P are boundary vertices of P and
2 1
therefore part of the subgraph G .
1
Pregel. The Pregel model was introduced by Google
in 2010 [Malewicz et al., 2010]. It is based on the Bulk
Synchronous Parallel model (BSP), which is a general,
iterative model for parallel computation [Valiant, 1990].
abstraction which provides information on the partition
An iteration consists of an algorithm that is executed on
structure to the programmer: A program is no longer ex-
each processor, followed by a communication phase for
pressed for one vertex, but for the whole block (or several
data exchange between processors. Such an iteration is
blocks) of a partition. Each vertex of the graph is part of
called a superstep. The execution is synchronous, thus
exactly one block. The graph is divided into k subgraphs
a processor will only start executing a superstep if all
G (the blocks), each of which contains the vertices of a
other processors have finished with the previous one. The i
block P as well as all vertices adjacent to vertices of P .
PregelmodelisimplementedbytheApacheGiraphframe- i i
The vertices in P are called internal vertices of G , all
work [Apache, 2015b]. i i
others are called boundary vertices (see Figure 2 for an
example).
Gather-Apply-Scatter (GAS). The GAS model
was proposed as part of the PowerGraph abstrac- A graph program is executed iteratively and syn-
tion [Gonzalez et al., 2012], which is implemented in the chronously, just like a Pregel program. It is defined by
GraphLab framework. GAS programs are iterative and a compute function, which can access the data of all ver-
vertex-centric like Pregel programs, but decompose an it- tices, both internal and boundary. Yet, only internal ver-
erationintothegather,applyandscatterphases. Accord- tices can be changed. Updates to boundary vertices are
ingly, a GAS program is defined by a gather, apply and only possible by sending a message to the corresponding
scatter and an optional gather_sum function. A further internalvertex. Theownerofthevertexreceivesincoming
decomposition of the vertex program enables the frame- messages and can update it accordingly.
worktoexecutegatherandscatterconcurrentlyonseveral
machines to balance the workload of high-degree vertices. Thelower-levelgraph-centricabstractionisusefulinthe
Inthegatherphase, avertexcangatherinformationfrom following areas: In vertex programs, data can be passed
neighbors. The gather function is called for each edge of toneighborsonly,soitcantakemanysuperstepstoprop-
the vertex and returns the desired data. The values re- agate information across a graph. This is slow and could
turned are aggregated by the gather_sum function. The be accelerated for the vertices that are in the same parti-
apply function is called once per vertex on the result of tion. Inthegraph-centricmodel,messagesneedtobesent
thegatheraggregation. Applyistheonlyphasewherethe to boundary vertices only, as we can directly update ver-
vertex data can be changed. Finally, the scatter func- texstatesofinternalvertices. Thisreducestheamountof
tion is called for each edge again. Scatter can be used to messages that are sent and processed. It may also reduce
update edge data. The gather and scatter functions only the number of supersteps: Information of one vertex is
have read access to vertex data. This allows them to be availabletoallverticesinthepartitioninthesamesuper-
executed concurrently without the need for synchroniza- step, which leads to faster convergence and therefore exe-
tion. The GAS model is implemented by the GraphLab cution time for some problems. The graph-centric model
framework [Dato, 2015]. alsoallowstheuseofsequentialalgorithmsastheycanbe
easily applied to the subgraph represented by the parti-
tion. Graph-centricprogramsareexecutedsynchronously,
2.3 Graph-centric Model
as a synchronization barrier waits for the last computa-
One of the reasons why vertex programs are so intuitive tion to finish after each superstep. However, local asyn-
is that the abstraction hides details of the distributed ex- chrony is possible, as updates made to vertex data are
ecution, like graph partitioning. However, this informa- available within the computation of one partition imme-
tion could be used for algorithm-specific optimizations. diately. The graph-centric model is implemented by the
Thegraph-centricmodel[Tian et al., 2013]isalower-level Giraph++ [Tian et al., 2013] framework.
3
3 Frameworks 3.4 Giraph++
The graph-centric model originates from a paper by Tian
This section introduces the software frameworks that im-
et. al. [Tian et al., 2013], who also put forward Gi-
plement the aforementioned programming models. All
raph++, an implementation of this model developed on
of them share some common implementation principles,
top of Apache Giraph. Currently Giraph++ is not avail-
whichwediscussbrieflybeforethespecificsofeachframe-
able as a separate framework, but as a patch for Apache
work: One is the use of a distributed file system, which
Giraphwhichaddssupportforthegraph-centricmodelto
provides the means to store data sets that are too large
Giraph [Apache, 2014]. It is proposed to become part of
for a single machine and enable data locality and fault-
the Apache Giraph project.
tolerance. Files are split up into blocks which can dis-
tributedacrossacluster. Thefilesystemcoordinateshow
clients access and write to files. All frameworks presented
4 Network Analysis Algorithms
support the Hadoop Distributed File System (HDFS).
Theframeworksofmessage-basedprogrammingmodels
For the experimental study, we select four common and
also implement message combiners as follows: In many
typical network analysis problems, Connected Compo-
cases, the message receiver is not interested in each indi-
nents, Community Detection, PageRank, and Clustering
vidualmessage value, butsome kindofaggregationof the
Coefficients. The former two call for a decomposition of
messages. Therefore, messages can be combined on the
the entire network into cohesive parts, and we opt for al-
sender node to reduce network traffic. The principle of
gorithms based on label propagation. Thus, both lend
combiners can be used in vertex- and graph-centric mod-
themselves to vertex-centric implementations. The latter
elsbyimplementingacombinefunctionwhichiscalledby
two yield a ranking of vertices by structural relevance,
the framework on messages with the same receiver.
PageRank through a simple iterative rule, Clustering Co-
efficients based on the somewhat more intricate counting
of triangles. We consider the program flow of these algo-
3.1 Apache Flink
rithms representative for many algorithmic kernels used
for network analysis. Moreover, their algorithmic struc-
Apache Flink [Apache, 2015a] (previously called Strato-
ture is able to highlight challenges and benefits of dis-
sphere[Apache, 2015c])isanopen-sourceframeworkwrit-
tributed processing.
teninJava. ItimplementsthePACTprogrammingmodel.
PACTwasputforwardbythisprojectwiththegoalofex-
tending the MapReduce model with new operators and a 4.1 Connected Components via Label
flexiblecompilerabletoinferoptimizationstrategiesfrom
Propagation
descriptive programs. Apache Flink offers a Java and a
Scala API to write PACT programs. Many real-world networks have several connected com-
ponents, but one of them usually dominates by far in
size [Newman, 2010]. For network analytic purposes it is
3.2 Apache Giraph interesting to determine whether a vertex belongs to this
giant component or not. Thus, subdividing the network
Apache Giraph [Apache, 2015b] is an open-source imple- intoconnectedcomponentsisacommonanalysistaskand
mentation of the Pregel programming model. It is writ- can be implemented as attaching a component label to
ten and implemented in Java and uses Apache Hadoop, each vertex. The label propagation algorithm used in this
a MapReduce implementation, for the execution of vertex study starts with each vertex in its own component and
programs. Facebook uses Apache Giraph in practice for communicates its component ID to all neighbors. A ver-
theanalysisofitssocialgraphandisanactivecontributor tex then applies the lowest component ID to itself. If the
to the project [Avery Ching, 2013]. labelhaschanged,thevertexwillnotifyallneighborswith
the new component ID.
Figure 3 illustrates the principle of the algorithm in
3.3 GraphLab PACT: The algorithm works on tuples of vertex IDs and
componentlabels,whereastheedgeinformationisusedto
TheGraphLab[Dato, 2015]packagewasinitiallyintended create these tuples.
to be used on shared memory systems, but was ex- The implementations in Pregel and GAS can be seen
tendedtosupportdistributedexecution[Low et al., 2012]. in Algorithms 1 and 2: After the initialization with the
Programs utilizing GraphLab are written in C++; vertexIDascomponentlabel,theverticesdistributetheir
Python bindings are also available. With GraphLab component label along their edges to neighbors and in
2, support for the GAS model was introduced, which turn apply the lowest component label from all incoming
was proposed as part of the PowerGraph abstraction labels. Figure4showsanexamplerunofthealgorithmin
[Gonzalez et al., 2012]. the Pregel model.
4
Vertices Iteration Output Algorithm 2: Connected Components in GAS model
1 init(vertex)begin
Edges Join Group Reduce 2 vertex.component←vertex.id
3 gather(vertex, edge)begin
4 returnvertex.component
Exampledataflow Join: Group: Reduce:
foofrththeeinfiprusttigtrearapthi:on ((21,,cc21)) ((21,,cc21)) ((21,, [[cc11,,cc22,]c)3]) ((12,, cc11)) 5 gather_sum(v1, v2)begin
(2,c3) (3,c2) (3, [c2,c3]) (3, c2) 6 returnminimum(v1, v2)
1 2 3 (3,c3)
7 apply(vertex, minimum)begin
8 changed false
Figure 3: PACT operator graph of the Connected Com- 9 ifvertex←.label > minimum then
ponents example 10 changed true
←
11 vertex.component minimum
←
Algorithm 1: Connected Components in Pregel 12 scatter(vertex, edge)begin
13 ifchanged then
model
// signal to inform neighbour of change
1 compute(vertex, messages)begin 14 signal(edge.target())
2 ifgetSuperstep() = 0 then
3 vertex.component vertex.id
←
4 changed true
←
5 else Active
6 minMsg getMinimum(messages)
← 1) 1 2 3 4 5 [1, 2, 3, 4, 5]
7 changed false
←
8 ifminMsg < vertex.component then
// found smaller component
9 changed true 2) 1 2 1 4 4 [1, 2, 3, 4, 5]
←
10 vertex.component minMsg
←
11 ifchanged then 3) 1 1 1 4 4 [2]
// component has changed - send new id to neighbors!
12 sendToAll(vertex.component)
4) 1 1 1 4 4 [ ]
In the graph-centric model, the algorithm does the fol- Figure4: Examplecomputationofconnectedcomponents
lowing: In the first superstep, a sequential algorithm is in the vertex-centric Pregel model. The two connected
used to find all connected components in one partition. components are separated by a larger gap in the middle.
Foreachboundaryvertex,amessagewiththenewlyfound Gray arrows represent graph edges, black arrows repre-
component label is sent to the partition with the corre- sent message passing, inactive vertices are indicated by a
sponding internal vertex. If any of the incoming messages dotted line.
have a smaller label for a vertex, then this vertex and all
other vertices within the same component and partition
will be updated at once. Figure 5 shows an example run idea of label propagation is also useful for partitioning
of the computation. a network into equally sized blocks with few edges run-
ning between different blocks [Meyerhenke et al., 2014,
Slota et al., 2014]. Such results are potentially useful for
4.2 Community Detection via Label
computing partitions in the graph-centric model.
Propagation
InPACT,thealgorithmisexpressedasfollows: Thela-
Many real-word networks exhibit a community structure, belofavertexisinitializedwiththevertexIDofarandom
i.e. theycanbedecomposedintointernallydenseandex- neighbor. A bulk iteration is performed on the resulting
ternally sparse subgraphs named communities. Among data set. In each iteration, a join operator is used to
heuristics for detecting communities, a label propaga- get tuples (vertexId, neighborLabel) for every vertex.
tionalgorithm[Raghavan et al., 2007]isamongthefastest A group operator on both vertex and label followed by a
and easiest to parallelize: Each vertex is initialized with sumaggregatorcountsthelabelsforeachvertex. Thisdata
a unique community label, and vertices propagate their set is then grouped by the vertex ID, a reduce operator
label to all neighbors in each iteration and update it finds the most frequent labels and emits the result.
with the most frequent of the received labels. Ties The label propagation algorithm can be naturally ex-
are broken systematically, e.g. uniformly at random pressed in the vertex-centric model of Pregel. In the algo-
or by preferring the minimum label. The algorithm rithm’sinitializationsuperstep,anoptimizationispossible
can be executed synchronously or asynchronously. As since in Pregel each vertex can access its neighbors’ IDs.
the synchronous execution can lead to an oscillation of Since these IDs are equal to the initial label value, there
labels [Raghavan et al., 2007], the asynchronous execu- cannot be a most frequent one. Thus, the label of each
tion is preferred. When adapted suitably, the general vertex is initialized with the label of a randomly selected
5
P P incoming edges and returns the neighbor vertex score di-
1 2
vided by its outdegree. apply receives the sum of these
1 2 3 4 5 6
values, which is used to calculate the new score and
the delta value. If delta is below the tolerance, the
1) 1 1 1 3 3 3
converged flag is set to true. scatter, which is called
on outgoing edges, signals neighbor vertices, but only if
2) 1 1 1 1 1 1 the vertex score has not converged. In Pregel, the termi-
nation is handled differently as all vertices remain active
3) 1 1 1 1 1 1 as long as the global termination criterion is not met (Al-
gorithm 4).
Figure5: Examplecomputationofconnectedcomponents
inthegraph-centricmodelofGiraph++: Foreachbound- Algorithm 3: PageRank in GAS model
ary vertex, a message with the component label is sent to // gather on incoming edges
the partition with the corresponding internal vertex. 1 gather(vertex, edge)begin
2 returnedge.source.score / edge.source.numOutEdges
3 apply(vertex, sum)begin
4 newScore=α+(1-α)*sum
neighbor. This label will be sent to all neighbors, which
5 delta newScore-vertex.score
selectthemostfrequentoftheincominglabels. Ifthenew ←
6 vertex.score newScore;
labeldiffersfromthepreviousone, alocal convergedflag 7 if|delta| >←TOLERANCE then
is set to false. Global aggregation of this flag is used to 8 converged FALSE
←
9 else
check for termination.
10 converged TRUE
In GAS the algorithm works similarly, as gather and ←
gather_sum determine the label frequency, apply applies // scatter on outgoing edges
11 scatter(vertex, edge)begin
thenewlabelandscatterdistributesitamongneighbors.
12 ifnot converged then
Thegraph-centricimplementationisagainsimilar,with 13 signal(edge.target)
the major difference that within a partition the labels
// alternative scatter using GraphLab’s delta caching
can be updated directly, because messages have to be ex-
14 scatter(vertex, edge)begin
changed for boundary vertices only. Since the label up- 15 ifnot converged then
datesarenotappliedgloballyinasynchronizedway,label 16 postDelta(edge.target,delta); /* post delta only if
vertex has not converged */
oscillation is less likely to occur.
4.3 PageRank
Algorithm 4: PageRank in Pregel
PageRank is a well-known centrality algorithm, enabling
a ranking of vertices by their structural importance – 1 compute(vertex, messages) begin
and taking the importance of neighbors into account. 2 if getSuperstep() = 0 then
It was originally developed to rank websites in the web 3 vertex.rank ← 1
4 else
graph [Brin and Page, 1998], but due to Google’s suc-
5 newScore ← α + (1 - α) * sum(messages)
cess it has experienced much wider attention and ap-
6 delta ← newScore - vertex.rank
plications [Newman, 2010]. The PageRank corresponds
7 vertex.rank ← newScore
to the dominant eigenvector of a transition matrix mod-
8 if |delta| > TOLERANCE then
eling a random surfer. In practice one often approxi- 9 converged ← FALSE
mates a vertex score P(v) by using the power iteration 10 else
algorithm, i.e. by computing the iteration Pi+1(v) = 11 converged ← TRUE
αr(v)+(1−α)·(cid:80) Pi(u) , where α is a dampening 12 setAggregatorValue(CONVERGENCE, converged)
{u,v}∈E deg(u)
factor that models a random jump to an arbitrary web-
13 if getAggregatorValue(CONVERGENCE) then
page (as opposed to following a link). In the following we // algorithm converged
make the usual assumption of r(v) = 1 for all v ∈ V and 14 voteToHalt()
call P(v) the score of vertex v. 15 else
The PACT implementation works on different sets of 16 sendMessageToAllEdges(vertex.rank /
tuples: 1) vertex ID and score and 2) vertex ID and a list vertex.numEdges)
of neighboring vertices. A bulk iteration for the vertices
17 combine(msg1, msg2) begin
is then defined by a join over the neighbor tuple and the
18 return msg1 + msg2
corresponding scores. Afterwards, a group by vertex ID
and the aggregation by computing the sum follows.
In GAS, the PageRank algorithm is easily expressed as The graph-centric implementation is again similar to
follows (also see Algorithm 3): gather is called for all the vertex-centric models, in this case Pregel. However,
6
in Giraph++ the support for local asynchrony is used: do the final computation. Aggregation of the vertices’
Values from vertices within the partition can be accessed values is then done using mapReduce functions. The
directly while for boundary vertices messages need to be graph-centric model does not yield any advantages over
exchanged. Also, here we can make use of a minor opti- the vertex-centric models in this application, so the im-
mization [Zhang et al., 2012] that was shown to converge plementationbasicallyfollowsthePregelimplementation.
faster than the traditional iterative approach. Again, the main difference is that the computation for
edges in a partition can be done directly, whereas explicit
message exchange is necessary for boundary vertices.
4.4 Clustering Coefficients
The implementation of the approximation algorithm
Transitivity is a network analytic concept to express if principally follows the implementation of the exact algo-
two neighbors of a vertex are likely to be neighbors as rithm with a few exceptions. Depending on whether the
well or not. In social networks, for example, two friends global or the average local clustering coefficient is com-
of the same person are more likely to be friends as well puted, the probability for each vertex needs to be set.
than a random pair of vertices [Newman, 2010]. Thus, as Then,insteadofexchangingthewholeneighborhood,each
one formalization of the transitivity, we can use the av- vertex will only draw a fixed number of vertex pairs and
erage local clustering coefficient C(G) for an undirected sendittothesevertices, e.g.duringasuperstepinPregel.
graphGtoexamineitslocalconnectivity. Thisparticular Another superstep is then used to intersect the neighbor-
measure is defined as the mean over the local clustering hoods and the incoming messages and for each match the
coefficient over all vertices, i.e. C(G) = 1 (cid:80) δ(v), link counter will be incremented. Afterwards, the usual
|V(cid:48)| v∈V(cid:48) τ(v)
where V(cid:48) := {v ∈ V | deg(v) ≥ 2), δ(v) is the num- global aggregation will compute the final values.
ber of triangles v belongs to, and τ(v) := (cid:0)deg(v)(cid:1) (see
2
e.g. [Schank and Wagner, 2005]). This is different from
the global clustering coefficient, which contrasts the num- Algorithm 5:ClusteringCoefficientsinPregelmodel
bersofclosedtrianglesandofconnectedtripletsinagraph 1 compute(vertex, messages)begin
(times a constant factor). 2 ifgetSuperstep() = 0 then
3 foredge vertex.edges do
All of the described measures can be computed with ∈
4 sendMessageToAllNghbrs(edge.otherVertex.id)
thesamealgorithmicidea,asthebasicoperationiscount-
ing triangles. Since the exact algorithm is computa- 5 else ifgetSuperstep() = 1 then
6 neighborEdges 0
tionally expensive for large complex networks, we are ←
7 formsg messagesdo
also interested in an efficient approximation algorithm. 8 ifms∈g vertex.edges then
∈
[Schank and Wagner, 2005]proposedasequentialrandom 9 neighborEdges neighborEdges+1
←
sampling algorithm to approximate the clustering coeffi-
10 neighborEdges neighborEdges/2 // each link has a
cient with a (probabilistic) additive error bound. For the message from b←oth vertices
average local clustering coefficient, it randomly selects a 11 possibleEdges vertex.numEdges*(vertex.numEdges-
←
vertex v, and two different neighbors of v, u and w. If 1)/2
12 vertex.localClustering neighborEdges/possibleEdges
u and w are connected, a counter is incremented. This ←
// Use aggregators for global values
process is repeated for a fixed number of samples. The 13 setAggregatorValue(POSSIBLE,possibleEdges)
approximation value of the average local clustering coef- 14 setAggregatorValue(ACTUAL,neighborEdges)
15 setAggreagtorValue(LOCALCC,vertex.localClustering)
ficient is the number of links found divided by the num-
ber of samples. For the global clustering coefficient the 16 masterCompute()begin
samples will be chosen with a probability proportional to 17 avgLocalCC getAggregatorValue(LOCALCC)/
←
numVertices
deg(v)·(deg(v)−1) for a vertex v, where deg(v) denotes
18 globalCC getAggregatorValue(ACTUAL)/
the degree of v. ←
getAggregatorValue(POSSIBLE)
In PACT, to count the relevant edges, two join opera-
tions are necessary. The first is carried out on the edges
with themselves resulting in a set of neighbors of neigh-
borsofavertex. Thesecondjointhenjoinstheresulting
setwiththeedgesagainwithrespecttoavertex. Therest
of the algorithm is straightforward.
4.5 Discussion
The Pregel implementation (Algorithm 5) consists of
two supersteps where the first one sends the edge infor-
mationtoincidentverticeswhilethesecondonecomputes In summary, we confirm that graph-specific frameworks
thelocalclusteringcoefficientandsetstheaggregatorsfor (Giraph, Giraph++, GraphLab) offer a more concise and
the global values. intuitive way to express graph algorithms. Not surpris-
The GAS implementation (Algorithm 6) is similar as it ingly, vertex-centric programming models are well suited
uses one step to calculate the common neighbors of two for vertex-centric algorithmic techniques as in label prop-
vertices and uses this information in the second step to agation or PageRank’s iterative method.
7
Algorithm 6: Clustering Coefficients in GAS model a heartbeat message indicates a failure. In this case, the
master starts up new workers replacing the failed ones.
// gather on all edges
Then, all workers load the last checkpoint from the file
1 gather1(vertex, edge)begin
2 return[edge.otherVertex] system and resume the computation. The checkpointing
3 gather_sum1(list1, list2)begin mechanism was tested in experiments for this particular
4 append(list1,list2) purpose by killing off one worker process. The master de-
5 apply1(vertex, neighbors)begin tects the missing worker immediately, and computation
6 vertex.neighbors=neighbors resumes without errors. However, the overhead for saving
// scatter on all edges checkpoints regularly slows down the performance. Al-
7 scatter1(vertex, edge)begin though GraphLab has a similar checkpoint mechanism,
8 otherVertex=edge.otherVertex
it does not automatically recover in the case of a fail-
9 commonNeighborsList=intersection(vertex.neighbors,
otherVertex.neighbors) ure. GraphLab periodically saves a binary serialization
10 edge.commonNeighbors=commonNeighborsList.size snapshot of the graph. The algorithm has to be restarted
11 manually using the snapshot as the input graph.
12 gather2(vertex, edge)begin
13 returnedge.commonNeighbors
A Shared-Memory Framework for Comparison.
14 apply2(vertex, neighborEdges)begin
NetworKit [Staudt et al., 2014] is a tool suite of algo-
15 vertex.neighborEdges neighborEdges
16 vertex.possibleEdges ←vertex.numEdges*(vertex.numEdges rithms and data structures for the analysis of large com-
←
-1)/2 plex networks, implemented in C++, OpenMP in the
17 vertex.localClustering neighborEdges/
← backend and in Python for a user-friendly frontend. Net-
vertex.possibleEdges
worKitrunsonasinglecomputer,itsOpenMPimplemen-
18
tations exploit multicore capabilities. It is based at Karl-
19 main(graph)begin
20 executeVertexPrograms(graph) sruhe Institute of Technology and developed with inter-
21 avgLocalCC mapReduce(vertex.localClustering)/ national contributors under an open-source model.1 The
←
numVertices aimofthepackageistoprovideefficient,oftenparallelized
22 globalCC mapReduce(vertex.neighborEdges)/
← graph algorithms and lean data structures for network
mapReduce(vertex.neighborEdges)
analysis.
For this study, we let efficient shared-memory paral-
lel implementations of network analysis algorithms from
5 Experimental Comparison
NetworKit compete with implementations in distributed
frameworks. The aim of the comparison is to see when it
In the following we present a comparative experimental
is worth to move to distributed systems.
study on the performance of four distributed and one
single-machine framework.
Test data sets. Real world graph data sets used
for the experiments are listed in Table 1. The
5.1 Experimental Setup
uk-2002 data set [Boldi and Vigna, 2004] originates from
a web crawl of the .uk domain. The remaining data
Fortheexperimentsarathermodestcomputerclusterof8
sets were taken from the KONECT graph collection
nodeswasused. Eachsystemhas24GBRAMandrunsan
Intel(cid:13)R Xeon(cid:13)R X5355 CPU with 2.66 GHz and 8 cores on [Kunegis, 2013]. The Wikipedia data set consists of
linked encyclopedia pages of the English Wikipedia.
two processors. The framework versions are GraphLab
The orkut-links, livejournal-links, flickr-links
2.2, Apache Giraph 1.1.0, Apache Flink 0.6, and Net-
graphs represent the users and friendships of the respec-
worKit 3.3. (The NetworKit version has been patched
tive social networks. twitter [Kwak et al., 2010] and
with a fix for an earlier performance bug in the PageR-
twitter-l[Cha et al., 2010]aresnapshotsofthefollower
ank implementation.) Apache Giraph was deployed in a
graph of the social network Twitter. twitter-l is the
Apache Hadoop MapReduce environment, using Hadoop
largest data set used in the experiments with almost two
0.20.203.0 and Apache ZooKeeper 3.4.6. For scaling ex-
billion edges, taking up over 36 GB of disk space when
periments,thealgorithmswererunon1,2,4and8cluster
represented in an edge list file. The relatively small
nodes. NetworKit was run on a single node of the same
flickr-edgesgraphisbasedonimagesoftheFlickrweb-
type.
site,adjacentimagessharecommonmetadata. Addition-
ally, synthetic graphs (scale-free graphs generated using
Fault tolerance. Apache Giraph is the only one of the
NetworKit’s Dorogovtsev-Mendes generator) are used for
examinedframeworkswhichsupportstherecoveryincase
weak scaling experiments. These synthetic data sets con-
of a failure during the algorithm’s execution. It uses a
tain 16 million edges per compute node, i.e. 16, 32, 64
checkpoint mechanism which saves the current computa-
and 128 million edges for the experiments on a single, 2,
tionstateofthealgorithmexecutiontothedistributedfile
4 and 8 compute node(s).
system. Workersregularlysendheartbeatmessagestothe
Mastertoindicatetheyareexecuting, thustheabsenceof 1http://networkit.iti.kit.edu
8
flickr-links
11000 5.46.47.610.3 24.427.932.441.5 34.226.223.724.1 60.135.222.923
1 0.8
5.2 Results least 4 nodes by any of the distributed frameworks. The
0.1
NetworKit GraphLab Giraph Giraph++ Flink billion-edge twitter network can only be processed with
orkut-links GraphLab and Giraph (besides NetworKit), and the two
101101000000 flickr-links 555..46367..45275..652120..83 2412.472878.96302.45341.5 3146.2412165.29283.76824.1 6076.1933758.222128.194223 dswhiesoatwrkisbsuctahtelaidntgfornbalemyheatwvhioeorrGk,siarrnaedpqhuthiirametpaGltermlaepaeshnttLa8atbinoohndahsesat.shFeailgasutragrebesl7et
10
1 02..38 relative increase, with the time quadrupling from the sin-
1 gle node to the eight node execution.
NetworKit GraphLab Giraph Giraph++ Flink
0.1
NetworKit GraphLab Giraph Giraph++ Flink
uk-2002
11000000 orkut-links 141123 329282 619469 769377808570 400
1101010000 21 55.637.525.522.8 12788 60 53 16411598 68 218142 Timeins1100020f0lickr-links 5.46.47.610.3 24.427.932.441.5 34.226.223.724.1 60.135.222.923
1 2.3
1 NetworKit GraphLab Giraph Giraph++ Flink 1 0.8
NetworKit GraphLab Giraph Giraph++ Flink 0
wikipedia-links-en twitter 0.1 1 2 4 8
11000000 u13k0-1200602 249236701562233220...7246440.22654438.2412.....30..92.421...43271549 598452 362797258520 200619469134.48 70818557.03 oNrkeutwt-olriKnkits GrGaprahpLhaLbab GiGrairpahph GiGrairpahp+h+++ FliFnlkink
1101010000 21 141123 100 Fe1ri01ag00tu00erde7d:atWaesaektssco5af5.6l31i7n.652g5.m52o2f.i8lclioonn12n7ee88dc6tg0ee5ds3 cpoem1r64pn11oo5n9d8ee6n8tsw76i9t3h782g18e14n2-
1 0 10
1 GraphLab Giraph Giraph++ Flink GraphLab Giraph 2.3
NetworKit GraphLab Giraph Giraph++ Flink
1
1 2 4 8 nodes NetworKit GraphLab Giraph Giraph++ Flink
wikipedia-links-en twitter flickr-edges
Fpo1ig01n00ue00rnets6.1:30R106unni2n492360156722g33220...7244406.22446583.2412.....30..92.421...34271549time5s98[4i5n2 sec6o77n5d50s] of1200C00onn1e34c.4t8ed C1o85m.3- 341010000 uk-2002 1.9721.041121.9293 1010000329152981248.680.75 69 619469347 122878068555710
10 2 21 1.42
10 10
Connected Components via Label Propagation. 1
1 0 0.54
AsFiguGrraeph6LasbhowGisra,pthheGsihraaphr+e+d-mFelminkoryparalGlerlapihmLapblemGireapnh- 0 1 1
NetNweotrKwiotrKit GGrarpahpLhaLbab GiraphGiraphGiraGpihra+ph+++ FFlliinnkk
tation in NetworKit is much faster than the distributed
1 2 4 8
frameworks. Out of the latter, GraphLab performs best, livejouwrnikailp-leindkias--ulinks-en twitter
followed by Apache Giraph. Giraph++ cannot benefit 1120000 59845210000677550 200 185.3
from graph-centric optimizations and although it reduces 90 130106 249236701562233220...7246440.22654438.2412.....30..92.421...43271549 87 1000 698 633 131441.418
100 322233
the number of supersteps, it performs worse. Apache 60 57 100 100
Flink is much slower than the other frameworks on a sin- 10 48 41
30 10
16.7
gle node (over 13x slower than GraphLab and 6x slower 1 0
0 GraphLab Giraph Giraph++ 1Flink GraphLab Giraph
than Giraph for the orkut-links graph). It catches up NetworKit GraphLab Giraph Giraph++ Flink
in the cluster setup due to higher speedup rates, but re- orkut-links 1 2 4 8 nodes
mains slower than GraphLab and Giraph. For smaller 120 10000
graphs such as the flickr-links graph, GraphLab and Figure 8: Runni1n00g.5 times [in sec1o46n2d13s18] of Comm2511unity De-
tec90tion on 1, 2, 4 an85.d2 8 nodes1.00T0 he fla64s8hsymbol indicates
Apache Giraph running times increase with adding addi-
179
tional compute nodes due to overhead, while Giraph++ th6a0t a graph is too la52r.7g50e.3to be10p0rocessed with the respec-
tive tool using the given number of nodes.
and Apache Flink do achieve a speedup. The connected 30 24.9 10
components on uk-2002 can only be computed with at 0 1
NetworKit GraphLab Giraph Giraph++ Flink
uk-2002 wikipedia-links
name(directed) n m size
ttwwiitttteerr-(l1)(1) 5421MM 11,,946638MM 3262..1365 112500 14012369000001210148000.559500020...002.753 1856 300 258 110000000 1856
wikipedia-links-en(1) 27M 601M 9.48 200
uk-2002(1) 18M 261M 8.26 90 100 100
orkut-links(1) 3M 117M 1.65 60 100
livejournal-links-d(0) 4.8M 68M 0.99 30 27.2 10 10
livejournal-links-u(1) 5M 49M 0.67
flickr-links(0) 1.7M 15M 0.18 0 1 0 1
flickr-edges(0) 105k 2.3M 0.02 NetworKitGraphLab Giraph GraphLab Giraph
1 2 4 8
Table 1: Properties of the networks (with n vertices and
m edges) used in the experiments. Size denotes the size
of the graph stored in edge list format in GB.
9
flickr-links
11000 5.46.47.610.3 24.427.932.441.5 34.226.223.724.1 60.135.222.923
1 0.8
0.1
NetworKit GraphLab Giraph Giraph++ Flink
orkut-links
1000 769378
100 55.637.525.522.8 12788 60 53 16411598 68 218142
10
2.3
1
NetworKit GraphLab Giraph Giraph++ Flink
Community Detection via Label Propagation.
uk-2002
Out of the distributed frameworks, GraphLab has a
clear advantage over the other frameworks with its 1000 141123 329282 619469 708570
100
asynchronous model, which is evident in the running
21
times (Figure 8). NetworKit is again much faster. 10
Forflickr-edgesandlivejournal-links-u,GraphLab
1
gets slower when more computing nodes are added. NetworKit GraphLab Giraph Giraph++ Flink
Flink can only handle the flickr-edges data set. Gi-
wikipedia-links-en twitter
raph++computesdatasetsuptotheorkut-linksgraph, 1000 598452 677550 200 185.3
but is very slow (6x the execution time of Giraph for 130106 249236701562233220...7246440.22654438.2412.....30..92.421...43271549 134.48
100
livejournal-links-u). BothGraphLabandGiraphpro-
100
cess graphs up to the size of wikipedia-links-en on a 10
cluster of eight nodes. The synchronous execution of this
1 0
algorithmcancauseconvergenceproblemsthatprevented GraphLab Giraph Giraph++ Flink GraphLab Giraph
some implementations from terminating. Label oscilla-
1 2 4 8 nodes
tions can occur as two neighboring vertices join the com-
munityoftheothervertexandbecausethecommunication Figure 9: Running times [in seconds] of PageRank.
happens synchronized after local computations are done, flickr-edges
they effectively exchange their labels. This is a disadvan- 1000 789 760
410
tageofframeworksonlysupportingsynchronousexecution 183 154 219
(Giraph,Giraph++andFlink)forthistypeofalgorithms. 100 55 61
34
10
PageRank. With the exception of NetworKit’s imple-
mentation, the PageRank implementations tested run for
1
a fixed number of iterations. PageRank was executed
NetworKit GraphLab Giraph
on the directed test graphs. The livejournal-links-d
1 2 4 8 nodes
graph could be processed by all frameworks in every set-
ting. uk-2002 and wikipedia-links-en could be pro- Figure 10: Running times [in seconds] of the exact clus-
cessed by the distributed frameworks with 4 and 8 nodes tering coefficient for the flickr-edges graph.
only, twitter only in the 8 node setting. The largest
graph,twitter-l,couldonlybeprocessedbyApacheGi-
8 nodes to be faster than NetworKit, and Apache Giraph
raph. GraphLab stopped with memory allocation errors
still takes four times as long on 8 nodes.
and Flink aborted the computation.
GraphLab and NetworKit achieve the best perfor- As expected, the approximation of the clustering co-
mance for PageRank (see Figure 9): Apache Giraph efficients allows much larger graphs to be analyzed (see
is slower than GraphLab by a factor of 1.6 to 2.5 for Figure 11). NetworKit’s implementation is by far the
the smaller livejournal-links-d, and 2.8 to 3 for the fastest as it can access the graph structure much faster,
larger wikipedia-links-en and twitter. The differ- while in distributed frameworks this requires message ex-
ences to Apache Flink are even larger, the latter’s run- change, possibly over the network. In most cases, adding
ning time is up to 12 times (uk-2002) higher than that additionalclusternodesdoesnotbenefittherunningtime
of GraphLab. The Giraph++ algorithm exhibits poor (with the exception of the Giraph implementation).
performance. GraphLab is sometimes faster than Net-
worKitduetothefactthatNetworKititerateslongeruntil
Memory Usage. The memory footprint of the frame-
a higher accuracy is achieved.
worksiscrucialtotheiruseinpractice–allthemoreaswe
often experience a rather poor scaling behavior in terms
Clustering Coefficients. The exact local clustering of running time for the graph computations we investi-
coefficientsalgorithmisthemostchallengingalgorithmfor gate here. Factors important for memory usage are the
the frameworks. It needs a lot of memory, as every vertex underlying programming language, data structures, and
exchanges its total neighborhood with every neighbor, so the framework structure and functionality. In Apache Gi-
the number of messages sent is in O(n · d2 ). Of the raph for example, messages sent in a vertex program are
max
test data sets, only the small flickr-edges graph can be cached and only sent at the end of a superstep, which
handled by NetworKit, GraphLab, and Apache Giraph, requires additional memory.
even when the distributed frameworks use all 8 servers GraphLabcanonlyprocessgraphswhichcanbeloaded
(see Figure 10). Apache Flink runs the algorithm for two into memory entirely, i.e. up to twitter. Giraph, on
hours, after which the execution is aborted. NetworKit the other hand, offers an optional graph swapping func-
has again the fastest execution time on a single node, fol- tionality. With this feature enabled, Giraph is able to
lowedbyGraphLabandApacheGiraph. GraphLabneeds load the twitter-l graph, but only the memory efficient
10