Table Of ContentAn edge density definition of overlapping and weighted graph communities
Richard K. Darst,1 David R. Reichman,1 Peter Ronhovde,2 and Zohar Nussinov2
1Department of Chemistry, Columbia University, 3000 Broadway, New York, NY 10027, USA.
2Department of Physics, Washington University in St. Louis,
Campus Box 1105, 1 Brookings Drive, St. Louis, MO 63130, USA
Community detection in networks refers to the process of seeking strongly internally connected
groups of nodes which are weakly externally connected. In this work, we introduce and study a
community definition based on internal edge density. Beginning with the simple concept that edge
density equals number of edges divided by maximal number of edges, we apply this definition to a
varietyofnodeandcommunityarrangementstoshowthatourdefinitionyieldssensibleresults. Our
communitydefinitionisequivalenttothatoftheAbsolutePottsModelcommunitydetectionmethod
3 (Phys. Rev. E 81, 046114 (2010)), and the performance of that method validates the usefulness of
1 ourdefinitionacrossawidevarietyofnetworktypes. Wediscusshowthisdefinitioncanbeextended
0 toweighted,andmultigraphs,andhowthedefinitioniscapableofhandlingoverlappingcommunities
2 and local algorithms. We further validate our definition against the recently proposed Affiliation
GraphModel(arXiv:1205.6228[cs.SI])andshowthatwecanpreciselysolvethesebenchmarks. More
n
than proposing an end-all community definition, we explain how studying the detailed properties
a
of community definitions is important in order to validate that definitions do not have negative
J
analytic properties. We urge that community definitions be separated from community detection
4
algorithms and and propose that community definitions be further evaluated by criteria such as
1
these.
]
h
p I. INTRODUCTION thingtodo,butmuchmorecouldbelearnedifthescope
- and definitions of communities were broader.
c
o It is standard to use the concepts of graph theory The Girvan-Newman modularity is one of the few
s in order to represent the interactions of complex sys- heavily-studied community definitions [12–16]. Modu-
.
s tems. Here, the nomenclature of “nodes” and “edges” larity weights internal edges against external edges in
c
is used to represent generic items and the interactions an attempt to indicate, without any user-input parame-
i
s between them[1]. One important form of structure in ters,a“best”communitystructureforanytypeofgraph.
y
graph theory is that of communities, or strongly con- Modularity is one of the oldest and most-emphasized of
h
p nected subgroups[2]. There is no single agreed upon def- the existing community definitions, but despite its util-
[ inition for communities, but it is generally accepted that ity, there is a useful caveat to be made in the study of
communities are groupings of nodes that are strongly (ground-state) definitions of modularity. In 2006 For-
1
connected to each other and weakly connected to nodes tunato and Barth´elemy showed that modularity has a
v
in other communities. It is important to note the two very interesting property: namely an implicit depen-
0
2 uses of the term “community” here. The first is a real- dence on the total size of the network to which it is
1 worldgroupingofobjects,sometimesknownas“ground- applied[17, 18]. This prevents modularity-based commu-
3 truth” in other works[3]. This grouping is not precisely nitydetectionmethodsfromresolvingsmallcommunities
1. mathematically defined, but is empirically defined based inalargegraph. Inshort,theoptimalcommunityinone
0 on the, e.g., social, anthropological, biological, etc. data partitionofagraphdependsuponpropertiesofthegraph
3 upon which the graph is created[4–10]. The second form far away. This behavior is non-intuitive and referred to
1 of “community” is a mathematical construction of nodes asa“resolutionlimit.” Severalattemptshavebeenmade
v: andedges. Thecommunitydetectionfieldhastwogoals: to produce multi-resolution modularity measures which
i the definition of mathematical communities that most have a tunable parameter that “zooms” in or out and
X
correspond to real-world communities, and the develop- controls the size of detected communities [19], but these
r ment of algorithms that can locate these mathematical too have been shown to suffer resolution limits [20, 21].
a
communities within graphs. Modularity has taught us several things. First and fore-
most,communitydefinitionsneedtheoreticalstudy. Sec-
Manycommunitydefinitionshavebeenproposedinthe
ond, any global community definition may have a res-
literature. Several reviews are dedicated to an overview
olution limit, necessitating local community definitions
and comparison of community detection (CD) methods,
[18, 20, 22, 23].
and more specifically, of community definitions them-
selves [2, 3, 11]. Often, primary emphasis is placed on In order to understand the relevance of the work pre-
thedescriptionandworkingsofthecommunitydetection sentedinthiswork,itisusefultolookatpastcommunity
method itself, and the method’s particular community detection approaches from the standpoint of optimizing
definition is only implicitly defined as the practical re- oversomeconfigurationalspace. Insteadoftryingtoim-
sultofapplyingtheCDalgorithm, andnotasaseparate prove sample techniques, we are looking at a different
formal definition. This has been accepted as a practical cost function which gives a different ground state, which
2
may be “more correct” than another ground state. As and Leskovec (YL), who have proposed a new model of
an example, after the introduction of modularity as a socialnetworks, theAffiliation Graph Model (AGM)[40].
measure of community structure, many attempts were This model takes into account features observed in real-
put forth to improve community detection via the use of world social networks found via comparison with online
enhanced sampling of the configuration space of commu- social networks. YL claim that no current community
nity assignments in order to better optimize modularity detection algorithms can describe and detect communi-
[14, 15, 24–32]. As useful as these methods are, they all tiesinthesegraphs. Weconceptuallyshowthattheedge
share one fundamental limit: they rely on the assump- densitycommunitydefinitionmodelsthisgraphproperly,
tion that modularity is the correct cost function to op- and then perform actual community detections to prove
timize and that community detection is limited by the that this is the case.
ability to properly sample configuration space [21]: ba- We do not claim here that edge density is a universal
sically, overcoming kinetic barriers in minimization dy- definition of community that applies to all possible com-
namics. On the other hand, it is natural to place an munities in all possible graphs. Instead, we provide the
emphasis on understanding other ground state commu- tools to determine if any one particular class of graphs
nity definitions and properties, i.e. the mapping from iswelldescribedbyan edgedensity picture, eitherbyan
real-world to mathematical communities, before focus- analysisofthe graphgenerationprocess, orpropertiesof
ing on the search for optimal partitions. The viewpoint the edge structure. We hope to inspire similarly detailed
weespouseisthatonlyafterunderstandinggroundstate analysis, and more rigorous comparison, of existing and
characteristicsshouldonefocusonthesamplingrequired future community definitions.
tofindingthatgroundstate. Therearefewsuchanalyzes This remainder of this work is organized as follows:
in existing literature, but the number is growing. In section II, we define the basic tools needed to quan-
Inthiswork,weproposeacommunitydefinitionbased tify edge density, then state our edge density definition.
on internal edge density (abbreviated as simply “edge In section III, we describe historic and new edge den-
density”). Edgedensity-baseddefinitionshavebeencon- sity models, and the Potts model framework we use to
sideredbeforeinawidevarietyofcontexts[3,11,33–37]. perform actual community detection. In section IV, we
The particular edge density definition we focus on has describesomeuniversalpropertiesofedgedensity,which
been used in an implicit fashion previously [38, 39], but areindependentoftheexactmodelusedtoperformcom-
has never been explicitly defined and extensively stud- munity detection. In section V, we describe how certain
ied, as we do here. Our definition affords us the ability models handle the boundaries between communities dif-
to compare the properties of graph-based communities ferently, and explain our model of choice. In section VI
to the properties of real-world communities as a means and Appendix A, we discuss practical considerations for
of quantifying the degree to which they match. Cur- constructing an actual algorithm employing edge den-
rentlythemostcommonmethodofassessingcommunity sity. In section VII, we validate the use of edge den-
detection algorithms is to choose a test graph and run sity as a scale parameter. In section VIII, we discuss
community detection. This gives a biased perspective general considerations for overlapping communities, and
of community detection, since it takes some initial belief contrasting with some recent work, we show that edge
about the structure of communities and optimizes meth- densities detect the correct conceptual behavior. In sec-
ods towards that definition. Recently, there has been tion IX, we rigorously correlate our edge density to af-
much needed emphasis on the comparison of commu- filiation graph models, and see that we can detect these
nity definitions to real-world communities. In order to communities easily. In section X, we describe the ex-
make progress, we show how edge density relates to ac- tension to weighted graphs, and how our new variable
tual properties of model graphs and communities. topologyPottsmodelprovidesasignificantimprovement
over older models. In section XI, we discuss various pos-
In addition to the above benefit of edge density-based
sible extensions. In section XII, we discuss some gen-
communities, edge density can be quite naturally ex-
eral background and address some possible limitations
tended to weighted and multi-graphs, and handle over-
of edge density. In section XIII, we discuss the danger
lapping communities, all important areas of modern re-
of over-optimizing to particular benchmark models and
search in community detection. In particular, few algo-
why studies of edge density are useful nonetheless. In
rithms have been proposed that are capable of detecting
section XIV, we provide concluding remarks and discuss
overlapping communities. Thus, our work not only pro-
directions for future research.
vides a conceptual breakthrough in terms of a rigorous
analysis of new community definitions, but also provides
apracticalbenefitincommunitydetectionability. Asan
outgrowthoftheseextensions,weproposeanewvariable II. EDGE DENSITY
topology Potts model,whichallowsamorenaturalmeans
of community detection in heavily weighted graphs. Wemustfirstdefinethenomenclatureandmeasureswe
To provide concrete illustration of these claims, we will use. We will specify communities or groups of nodes
turn to a recently proposed proposed social network by capital latin letters such as A and B, and individual
model. In particular, we focus on recent work of Yang nodes by lowercase letters such as a, b, x, and y. In
3
particular, the node a shall represent an arbitrary node (a) (b)
of community A, and specific nodes of community A are A B
ρ=4/8
indexed as ai. The nodes x and y generally represent x
nodes which are not currently in any community. The (e)
community A will have n nodes. ρ=6/10 l=8, e=4 E
A
We use a set terminology to discuss communities and l=10, e=6
groups of nodes. Set union is denoted with “ ”, indi- (c) (d)
∪ C C’ D’ e’
cating every node on the left or right or both sides of D y ρ=4/6
the operator. Set intersection is denoted “ ”, indicat- l=6, e=4
∩
ingeverynodethatisonbothsidesofthisoperator. We
usethisnomenclatureloosely,allowingconstructssuchas
ρ=5/20
ρ=2/3 ρ=2/4
“A x”eventhoughweareoperatingonacommunityon l=10, e=5 l=3, e=2 l=4, e=2
∪
theleftandanindividualnodeontheright. Thisshould
betakentomeanunionofthesetofnodesincommunity FIG. 1: Edge density calculations in a variety of situations.
A and the set of nodes containing x. Foreachsituation,listhemaximalpossiblenumberofedges,
Foranygroupofnodes(explainedbelow), l represents eistheactualnumberofedges,andtheedgedensityρ=e/l.
In (a), we see the edge density of one community A, a group
the maximal number of possible edges (“links” in our
of n = 5 nodes, with l = 1n(n − 1) = 10 total possible
nomenclature). Thevariableerepresentstheactualnum- 2
links(dottedandsolidlines),e=6actualedges(solidlines),
ber of edges. The edge density ρ is the fraction of these
thus leading to a ρ = 6 . In (b), we see the edge density
links which have edges actually present, A 10
definition applied between one node x and a community B.
e The community B contains eight nodes, for a total number
ρ= . (1) of possible links to node x of l = 8. We see four actual
l Bx
links in place, leading to a ρ = 4. Note that the internal
Bx 8
connectivityofthecommunityB isirrelevantandnotshown.
We can calculate the edge density within and between
In(c),weseetheedgedensityasappliedtotwocommunities
a variety of different types of groupings of nodes, all of
C (5 nodes) and C(cid:48) (4 nodes). Internal edge structure is
whichcanbeusefulunderdifferentcircumstances. Fig.1
illustrates the most common situations. (a) We can cal- again irrelevant and not shown. There are a total of lCC(cid:48) =
nCnC(cid:48) = 20 possible links between these two communities,
icnulwatheicthhecaesdegle=den21sniAty(nwAit−hi1n).onClyomonmeuncoitmymedugneitydeAn-, (edC)Cs(cid:48)h=ow5stehdegeedsgiendpelnasciet,yfoofroannenedodgeeddebnesiintgypoufllpeCdCb(cid:48)e=twe25e0n.
sity is used to quantify the absolute community size and two communities D and D(cid:48). Edge density for each half is in
scale. (b) We can calculate the edge density between a analogyto(b). In(e),weseetheedgedensitybetweenanode
community B and a node x not currently in community e and a community which contains it, E. This is similar to
B, in which case l = nB, since that node can connect (b) except that, by convention, we use nE −1 links as our
once to every node in B. This is useful when deciding if basis.
a node should enter or leave a community. (c) We can
calculate the edge density between two communities C
and C(cid:48), in which case l = nCnC(cid:48), since every node in C aspossible(absorbingasmanynodesaspossible)aslong
canconnectoncetoeverynodeinC(cid:48). Thisisusefulwhen asρ>ρ∗. (c)AcommunityAofscaleρ∗ musthaveeach
testingiftwocommunitiesshouldmerge. (d)Wecancal- individual node a’s edge density ρAa >ρ∗.
culate the edge density for the case of one node y which Part (a) is the essence of the definition, establishing a
has edges between two communities D and D , in which scale parameter of the communities. Part (b) is neces-
(cid:48)
case lBy = nB and lD(cid:48)y = nD(cid:48). This is useful for decid- sary because without it, we could always prefer smaller,
ing which of two communities the node y would prefer more dense communities. For example, a graph could be
to join. (e) By convention, when we calculate the edge partitioned into two-node cliques, all of which have an
density between a community E and node e which is edge between the nodes, to get communities which all
(cid:48)
currently within that community, we only consider links have ρ = 1 but do not provide useful information con-
between e and the n 1 other nodes of the communi- cerning the overall structure of the graph. Part (c) is
(cid:48) E
−
ties. Thus, in this case, lEe(cid:48) = nE 1 instead of nE, in necessary to ensure that every node is well-connected to
−
contrast to case (b). the community, preventing nodes only loosely connected
from being added to any community. In future sections,
we will rigorously show that the absolute Potts model
A. The edge density community definition and the variable topology Potts model return communi-
ties satisfying these criteria. Part (a) will be shown via
The most basic form of an edge density community Eqs. (11, 18) in Sec. III. Parts (b) and (c) are discussed
definition (EDCD) for overlapping or isolated communi- in Sec. IIID.
ties consists of three parts. (a) A community A of scale Thisdefinitionissufficientwhencommunitiesarecon-
ρ is a subset of nodes with an internal edge density ρ sideredinisolation,i.e. whenweneverconsideronenode
∗ A
greaterthanρ . (b)Communitiesaretakentobeaslarge and must decide which of several communities it might
∗
4
join,ifitwouldincreaseρforanyofthem(Fig.1(d)). In we approach mean-field ρ = p with decreasing standard
these cases, one may choose to add the node to a larger deviationaroundthismean. Thisrelationbetweenρand
community resulting in a still larger resultant commu- anunderlyingpisvalidforanycasewherealllinksshare
nity, or to the community to which it shares the higher the same edge probability.
edge density. Both of these are reasonable options. One
simplecriteriawouldbetousethetotalnumberofedges
as a criteria: a node x joins community A instead of III. EDGE DENSITY MODELS
community B if e > e . Existing edge density com-
Ax Bx
munity definition methods make different choices here, In the previous section, we stated a basic edge density
which will be discussed in Section V. definition. This definition has been implicitly used in
severalpreexistingcommunitydetectionmethods. Thus,
the edge density community definition is not new, but it
B. Connection to graph generation processes
has never been as explicitly stated and analyzed. Before
weproceedtomorespecificusesoftheedgedensitycom-
Theedgedensityvariableρreferstoactualedgedensi- munity definition, we will review existing Potts-model
tiesinaspecificgraphinstance. Manybenchmarkgraphs based edge density community definitions.
are designed stochastically, with edges placed within or Recent edge density-based community definitions are
betweencommunitieswithasomespecifiedprobabilityof formulated in terms of Potts models [42–44]. The Potts
p. Sinceeachlink(possibleedge)hasanedgeplacedwith model is a spin system, with each site having one of up
aprobabilityofpindependentofallotheredges,thevari- to q associated spin flavors σ = 0,...,q 1. It is sim-
ablee(numberofedgesplaced)isbinomiallydistributed ilar to the Ising model except that the I−sing model al-
with mean lp (and probability of success p). Each of lows spins of only 0 or 1 (i.e., the Ising model is a Potts
the situations depicted in Fig. 1 can have a defined p. model with q = 2). For community detection, we take
For example, planted l-partition model (also known as the spin flavor of each site (node) as corresponding to a
Stochastic Block Models (SBMs)) graphs are defined by community assignment. That is, if σ = p then node
a
specifying intra-community edge densities pA, pB, etc. a lies in community number p. (The index p clearly
(Fig. 1 (a)), and inter-community edge densities pAB, lies in the range 0 p (q 1)). Inter-site interac-
pAC, pBC, etc. (Fig. 1 (c)) [41]. Then, each respective tions which define a≤Ham≤iltoni−an are minimized to find
e will have a binomial distribution with (its respective) l an optimal community assignment. This may be done
trials and (its respective) p chance (of an edge) per trial, by penalizing inter-community edges and favoring intra-
community edges. Reichardt and Bornholdt explain the
P[e=x]= (x;l,p) (x;lp,lp(1 p)). (2)
balances and equivalences of sums of internal and exter-
B ≈N −
nal edges in [19]. The Reichardt-Bornholdt (RB) Potts
(l,p)representsabinomialdistributionofl trialsandp
B model was the first explicit Potts model used for com-
probability per trial, and ( x ,σ2) represents a normal
N (cid:104) (cid:105) munity detection, but it does not use our edge density
distribution with a mean of x and a variance of σ2
(cid:104) (cid:105) community definition [19, 22]. Instead, the RB Potts
after we make a normal approximation to the binomial
model is equivalent to the Girvan-Newman modularity.
distribution. With ρ = e/l, we have a distribution of ρ
Because the RB Potts model does not correspond to the
of
edgedensitydefinitionsinthispaper,itisnotconsidered
(cid:104) x(cid:105) in the following analysis.
P ρ= = (x;l,p), (3)
l B Potts models are ideal for edge density definitions,
(cid:18) p(1 p)(cid:19) because they consist primarily of a sum over all edges
P [ρ=x] x;p, − . (4) [45]. The models allow us to select only internal edges,
≈ N l
although it is equally possible to select only external
In the above, we employed the following shorthand for edges. One possible limitation of Potts models is that
(a normalized) binomial distribution, each node can only be associated with one community.
Thiswouldseemtoexcludethepossibilityofoverlapping
(cid:18) (cid:19)
n communities. In order to expand the Potts models to al-
(m;n,p)= pm(1 p)n m. (5)
B m − − lowoverlappingcommunities, wereformulatethemodels
away from an edge-centric definition towards an equiva-
The normal distribution is, explicitly, given by lent community-centric definition.
1 (cid:20) [x x ]2(cid:21)
(x; x ,σ2)= exp −(cid:104) (cid:105) . (6)
N (cid:104) (cid:105) √2πσ2 − (2σ2) A. Absolute Potts model
Clearly, ρ is a random variable with a mean of p and
(cid:113) The absolute Potts model (APM) is a spin-based fer-
has a standard deviation of p(1−p). We see the intu- romagnetic community detection method[38, 39]. It has
l
itive result that as we approach larger community sizes, been shown to be very accurate at community detection
5
underawidevarietyofcircumstances[38,39,46,47]. The The energy is now written as sum over edge densities.
APM implicitly uses the edge density community defini- We see that for the energy of any one community to be
tion. In the Potts representation of the APM, the CD negative (attractive and having a binding energy), the
problem is mapped onto an energy minimization of a term ρ γ(1 ρ ) must be positive. Rearranging, this
A A
− −
Hamiltonian defined over pairs of spins gives us a correspondence between the APM variable γ
and the critical (minimum) edge density ρ
∗
(cid:88) 1
E =− 2(Aaa(cid:48) −γBaa(cid:48))δ(σa,σa(cid:48)). (7) γ
a,a(cid:48)=a ρA > =ρ∗. (11)
(cid:54) 1+γ
The sum above is over distinct nodes a, a with, as ex-
(cid:48) Eq. (11) is the fundamental relationship between the
plainedabove,σ denotingthecommunityassignmentof
a APMvariableγ andtheedgedensitycriticalvalueρ . It
nodea,andAaa(cid:48) beingtheweighted“adjacencymatrix”. shows that any community returned by the APM m∗ust
In unweighted graphs, Aaa(cid:48) = 1 if an edge is present satisfy the edge density community definition part (a).
between nodes a and a(cid:48), and Aaa(cid:48) = 0 otherwise. For We can now state two identical relationships which give
weighted graphs, described further in Sec. X, Aaa(cid:48) = w, equivalences of the APM and the edge density view-
where w is the respective edge weight. In unweighted
points:
graphs, we will define the “inverse adjacency matrix” to
have elements of Baa(cid:48) =1 if there is no edge preset, zero γ
ρ = , (12)
otherwise. For more general unweighted graphs, we will ∗ 1+γ
set Baa(cid:48) =1 Aaa(cid:48). In these models, lower energies are ρ
− ∗
attractive, and any shifts in community assignment that γ = . (13)
1 ρ
lower energy are favorable. − ∗
This Hamiltonian in Eq. (7) involves a sum over all We can rewrite the APM Hamiltonian in terms of ρ
∗
edges. However, due the presence of the Kronecker delta instead of γ using Eqs. (10, 13),
(i.e., δ(σaσa(cid:48))=1 if a=a(cid:48) and zero otherwise), the sum
is inherently local. That is, the sum contains only edges (cid:88) (cid:18)ρA ρ∗(cid:19)
E = l − . (14)
between nodes which are in the same community. An A
− 1 ρ
inter-community edge has a favorable (negative) energy A − ∗
of 1, and each missing edge has an energy penalty (pos-
We can now relate the previously existing APM to
itive energy) of γ. While typically represented within a
our new edge density community definition. First, if
spinformulation,thiscanberecastwiththeprimarysum
ρ > ρ , the energy of community A is negative, and
A ∗
over communities instead of over edges:
thus has a binding energy. Therefore, all communities A
(cid:88) (cid:88) 1 must have ρA > ρ∗, corresponding to the edge density
E =− A (a,a(cid:48)(cid:54)=a)∈A2(Aaa(cid:48) −γBaa(cid:48)), (8) clAom=mu21nniAty(ndAefi−ni1t)ioanrepasrctal(ea)f.acStoercso.ndI,n1o/r(d1e−r tρo∗)mainnid-
mize energy (and create the best partition according to
with (a,a) being all ordered pairs of nodes in commu-
(cid:48) the APM), we want l to be as large as possible (larger
A
nity A. By the use of the term “ordered”, we make
communities),andalsoρ tobeaslargeaspossible. Fur-
A
evident that we include both the pair (a,a) and the
(cid:48) thermore, because of the factor (ρ ρ ), increasing the
A ∗
pair (a,a) (for a = a) in the sum. The spin formu- −
(cid:48) (cid:48) edge density of the community also results in lower en-
(cid:54)
lation appears simpler conceptually, and may be imple-
ergy. As we will see in Sec. VII, larger community sizes
mented much more efficiently, although the community-
generally tend to imply smaller ρ, thus larger commu-
centricdefinitionallowsustomakeadditionaltheoretical
nity size and larger ρ are competing factors which must
progress.
be balanced. Eq. (14) quantifies that balance. Commu-
Since the inner sum of 12Aaa(cid:48) is the number of edges nities with single nodes (“size one” communities) have
in community A, and the sum of 12Baa(cid:48) is the number of E =0 according to Eq. (7), and this can be represented
missing edges in community A, we can rewrite this as in Eq. (14) if we define a community consisting of only
(cid:88) one node to have ρ=1.
E = (e γ(l e )), (9)
A A A
− − −
A
B. Variable topology Potts model
where the inner sum has been rewritten in terms of the
number of existing edges e and the number of missing
A
edges l e . With minor rearrangement and invoking The absolute Potts model [38, 39] has, by now, been
A A
our edge−density definition ρ = e/l, we may rewrite the introduced and studied in earlier works. The variable
energy in terms of edge density with a prefactor of l , topologyPottsmodel(VTPM),whichwenowintroduce,
A
has been hinted at in previous literature, but never de-
(cid:88)
E = l (ρ γ(1 ρ )). (10) fined nor extensively analyzed[22, 48]. A somewhat sim-
A A A
− − − ilar technique was used in Ref. [47], where weights were
A
6
shiftedbyavalueV. TheapproachofRef.[47]technique then be subdivided at their weakest point. The VTPM
requires an adjustment of two variables, V and γ, in or- is so named because it is able to “change the topology”
dertofindoptimalpartitions,whilethetechniqueofthis of these cliques.
section only requires adjustment of ρ . The core advantages of the VTPM over the APM
∗
The defining property of the variable topology Potts are the ability to break up cliques and handle weighted
modelisthatitassignsaconstantpenalty(ofsizeρ )to graphs more naturally. Otherwise, it contains the same
∗
all links, as opposed to only missing edges. The VTPM information as the APM. Instead of the control parame-
Hamiltonian is ter γ = ρ∗ , it uses ρ directly, leading to a much more
1 ρ∗ ∗
(cid:88) 1 natural in−terpretation of the community scale. Because
E = (Aaa(cid:48) ρ∗)δ(σa,σa(cid:48)). (15) of this, the VTPM provides a compelling community de-
− 2 −
a,a(cid:48)=a tectionalgorithmforfutureuse,whichwillbeelaborated
(cid:54)
on in future sections.
AsintheAPM,thiscanberecastintoaformwhichsums
over communities first, instead of over all pairs of nodes
(cid:88) (cid:88) 1 C. Discussion of Potts models
E = (Aaa(cid:48) ρ∗), (16)
− 2 −
A (a,a(cid:48)=a) A
(cid:54) ∈ The APM has been extensively studied and shown
to have acceptable performance in community detection
with all variables analogous to the APM nomenclature
across a wide variety of conditions and problems, in-
in Eq. (8). We proceed as in the APM to turn this into
cluding common equal-sized and power law distributed
an edge-density based definition using the same relation
(cid:80) graphs [39, 46]. Since we have shown that the APM
eA = Aaa(cid:48)
directly uses the edge density community definition (al-
(cid:88) thoughunstatedexplicitlyuntilnow),evenwithoutaddi-
E = l (ρ ρ ). (17)
A A ∗
− − tional tests, we have strong support for the edge density
A
communitydefinition. WhiletheVTPMhasnotbeenas
We,again,haveacriteriawhichmustbesatisfiedforany extensivelystudied,bycomparingEq.(14)andEq.(17),
VTPM community to have a binding energy, weseethattheHamiltoniansarethesameforunweighted
graphs, save a scale factor of 1 . This means that the
1 ρ∗
ρA >ρ∗ (18) VTPM also will be equally po−werful in all of the above
cases.
which is identical to the criteria of the APM and of edge
density community definition (a). In this form, we see
thattheVTPMinsiststhatallcommunitieshaveanedge
D. Energy changes upon community assignment
densitygreaterthanρ ,whichverynaturallycorresponds
∗ perturbation
to the edge density community definition. Communities
are weighted by the number of links l = 1n (n 1).
It is illuminating to contrast the VATPM2HAamAilt−onian TheAPMandVTPMaresumsoveralledgesprovided
ofEq.(17)withtheAPMHamiltonianofEq.(14). They they connect nodes in the same community. Thus, when
appear identical, with the exception of the APM’s inclu- we make some change to the community assignments,
sion of a scale factor of 1 . This means that all past for example, by combining two communities, the energy
work on the APM applies1−eρq∗ually to the VTPM. In par- change is completely represented by the sum of energies
fromedgeswhichwerejustmovedintothesamecommu-
ticular, we do not need extensive additional testing on
nity, minus sum of energies from edges which have been
the VTPM in order to show that it achieves the same
performance. The APM scale factor of 1 becomes removed from the same community. As an example of
1 ρ∗ this,consideraddingnodextocommunityB,Fig.1(b).
negative when ρ > 1. In all of our analy−sis thus far,
∗ The entire APM energy remains unchanged except for
thiswouldnotseemtobealimitation,butwhenwecon-
the n links between x and the nodes of B. Thus, the
sider weighted graphs in Sec. X, it will. This is the first B
energy change can be represented as
advantage of the VTPM over the APM.
A “clique” is a group of nodes all mutually connected (cid:88)
∆E = (A γB )
(thus, ρ = 1). Normally, community detection methods bx bx
−
donotbreakupcliques,becausetheyareasstronglycon- b B
∈
nected as possible. However, if the edges are weighted, = eBx γ(lBx eBx)
− −
theremaybecertain“weak”edgeswhereitisreasonable ρ ρ
Bx ∗
= l − . (19)
to separate communities. In the APM, weighted cliques Bx 1 ρ
can not be subdivided, since it only places energy penal- − ∗
ties at missing edges. The VTPM overcomes this lim- In the above, A and B correspond to the adjacency
bx bx
itation by allowing the least weighted edges to become and inverse adjacency matrices as defined previously,
repulsive repulsive first as ρ is increased. Cliques can while other uses of B within subscripts correspond to
∗
7
the community B in Fig. 1. For the VTPM, the energy
A
change upon perturbation is also analogous,
∆E =l (ρ ρ ). (20)
Bx Bx ∗
−
Theenergychangesviaperturbationhavethesameform
as for the global energy, with the sum only over the sub-
set of edges created, and the subtraction of only a sum
overthesubsetofedgesremoved. Thelocalnatureofen-
ergy makes it fast to compute energy changes under sys-
tem perturbations. Community detection then becomes
a fairly well understood problem of sampling an energy
FIG. 2: Illustration of average mean community edge den-
landscape with local interactions, using dynamics of the sity. Community edge density ρ 9 is equal to the mean of
user’s choice. all node edge densities 1(cid:0)5 + 2A+164 + 3 + 2 + 2(cid:1) = (cid:104)ρ (cid:105)
Using Eq. (19), we can see that a node x will join a 6 5 5 5 5 5 5 Aai
This, and other similar invariants, are among the properties
communityB (Fig.1(b))ifthenode-to-communityedge of edge density.
density is ρ > ρ . A node e will leave a community
Bx ∗ (cid:48) .
E (Fig. 1 (e)) if the node-to-community edge density is
ρEe(cid:48) <ρ∗. WecanalsoshowthattwocommunitiesC,C(cid:48)
and evaluate the average in Eq. (21). This leads to
will merge (Fig. 1 (c)) if the inter-community edge den-
sity is ρCC(cid:48) >ρ∗. Thus, edge density naturally describes 1 (cid:88)eAa
allpossiblecommunitymerges,withρ∗ beingthecritical (cid:104)ρAai(cid:105) = n l ,
edge density for all possible merges or splits. A a Aa
e
The above shows that communities returned by the = A . (22)
Pottsmodelssatisfy(b)and(c)oftheedgedensitycom- 12nA(nA−1)
munitydefinition, Sec.IIA.Criteria(c)statesthatcom-
In the above, the number of links for each node is con-
munity B will have every node ρ > ρ . If this was
Bx ∗ stant at l =n 1, and 1n (n 1) is the number of
not true in a community, energy would be lowered by a A− 2 A A−
links in a community of n nodes.
removing node x from the community via the inverse of A
Next, when a node x joins a community A to form
Eqs. (19,20). Criteria (b) states that a community B
community A x, there is a relationship between the
willgrowaslongasρB staysgreaterthanρ∗. Thisisev- ∪
edge density of the combined unit ρ and that of the
idenced by the fact that any node x with ρ > ρ will A x
Bx ∗ component edge densities ρ and ρ ∪,
have a favorable energy change upon joining B, growing A Ax
B is large as possible as long as all nodes have sufficient e e +e
A x A Ax
node-to-communityedgedensity. Eq.22ensuresthatthe ρA∪x = lA∪x = lA+lAx ,
resulting community has ρ >ρ . ∪
B ∗ l l
A Ax
= ρ + ρ , (23)
A Ax
l +l l +l
A Ax A Ax
IV. MODEL-INDEPENDENT COMMUNITY which is simply the mean of ρ and ρ weighted by the
A Ax
PROPERTIES respective numbers of links l and l .
A Ax
A similar relationship can be shown for two non-
The edge density ρ is not just an arbitrary variable overlappingcommunitiesAandBwithinitialedgedensi-
selected because it is simple and leads to a consistent tiesρ andρ andainter-communityedgedensityρ .
A B AB
definition of community. It has many theoretical prop- When two communities merge, the new edge density is
erties which can be compared to real communities, and the average of the edge density of A, the edge density of
helps in the formulation of a consistent community de- B, and the inter-community edge density ρ , weighted
AB
tectionframework. Inthissection, wewillderivevarious by the corresponding number of links l , l , and l
A B AB
properties of ρ which will prove useful in the exploration
l ρ +l ρ +l ρ
of the edge density community definition. ρ = A A B B AB AB. (24)
A B
Consider a community A with nA members ai (Fig. ∪ lA+lB +lAB
2). The edge density within A is ρ . Each node a has
A i These properties are useful for considering dynamics
an edge density to the rest of the community ρ . The
ai of community detection algorithms. For example, for a
firstpropertywhichwewillshowisthattheedgedensity
communityAandanodex,ifweknowthatxshouldjoin
of the community is equal to the average edge density of
A(ρ >ρ ),thenthefinaledgedensityofA(ρ )can
Ax ∗ A x
the component nodes to the community, notdecreasebelowρ . Furthermore,ifwehavetw∪ocom-
∗
munities A and B (with ρ > ρ and ρ > ρ ), if the
ρ = ρ , (21) A ∗ B ∗
A (cid:104) Aai(cid:105) inter-community edge density ρ >ρ (the criteria for
AB ∗
with the average taken over different community mem- community merging), then, after merging, we are guar-
bers a . To show this, we apply the edge density Eq. 1 anteed that the new community A B must have ρ
i A B
∪ ∪
8
B A or B, as in Fig. 3. As a precondition , we must have
A ρ > ρ and ρ > ρ , otherwise x can not join both
Ax ∗ Bx ∗
communities. According to the edge density community
x
definition, with ρ > ρ and ρ > ρ , in isolation,
Ax ∗ Bx ∗
or if overlapping community assignments were allowed,
it would be allowable for x to join either community,
andthiswouldbethepreferredenergy-minimizingmove.
n =4 n =6 When overlaps are not allowed, x must choose one of A
A B
ρ =3/4 ρ =4/6
A B or B to join.
A similar situation occurs when node b, part of com-
FIG. 3: Balance between placing a node x into community munity B, has a sufficient edge density to also join A.
A or B. In this plot, x has a sufficient edge density to join
The energy of addition to A must at least compensate
either community, but when overlapping community assign-
for the energy of removal of b from B. We imagine this
ments are not allowed, we can have x join both. The text
in two parts: first, the removal from B, and second, the
discusses the criteria that edge density models use to assign
choice between addition to A or B, allowing us to con-
x to either A or B.
sideronlytheA,B,xsituationofthepreviousparagraph
without loss of generalization.
satisfying the edge density condition ρ > ρ . These When using the VTPM, the node x will choose to join
A B ∗
properties serve as a basic check on the s∪ensibility of our the community which most minimizes the energy. It will
community definition. join A when EAx <EBx, which can be rearranged to
When ρ =1, then all communities must be fully con-
∗
nected (only graph cliques are allowed as communities). nA[ρAx ρ∗] > nB[ρBx ρ∗]. (25)
− −
When ρ >1, communities can not exist as edge density
∗
cannotbegreaterthanone. Inthiscase,mostCDmeth- WeseethatxwilljointhecommunityAorB towhichit
ods will return “communities” which actually consist of hasthelargestexcessedgedensityρCx ρ∗,weightedby
−
single nodes, since there is never a case that multiple thecommunitysizenA ornB. Wenotethefollowing: (a)
nodes can join together. When ρ∗ =0, there is no lower we assume ρ∗ < 1. The cases where ρ ≥ 1 are discussed
boundforcommunitysize,andallnodescancollapseinto above. (b)weassumeρAx >ρ∗andρBx >ρ∗. Ifbothare
one large community spanning the system, however, this less than ρ∗, x can join neither community, if one is less
may not happen if the graph consists of disjoint subsets than ρ∗, we see the node will join the other community
of nodes and the exact dynamics of the CD process does (c) If ρ∗ = 0, then x will join the community of greater
not attempt to join disjoint sets of nodes together. size. (d) If ρAx = ρBx, x will join the community of
greatersize. (e)ifn =n ,accordingtoEq.(25),xwill
A B
join the community to which it has greater ρ.
V. MODEL-DEPENDENT COMMUNITY Weseethattheseresultssupporttheideathatanode,
PROPERTIES when faced with other communities of equal sizes, will
join the community to which it shares the greatest edge
density, however, there is also a competing preference
Thediscussionofthepropertiesintheprevioussection
towards smaller, more dense, communities.
makes one critical assumption: when nodes are added
It deserves emphasis that the results from this section
to communities, they are previously “unassigned,” or
arederivedforoneparticularinstanceoftheedgedensity
in single-node zero-energy communities. In these cases,
community definition, the one derived from the VTPM.
there is no energy barrier for removing nodes from the
For unweighted graphs, this model will give equivalent
previous community to which they are bound. This is
results to the APM. The APM has been shown to be an
the case when constructing local communities, or when
effective community detection method in a wide variety
overlapping communities are allowed. When this is not
of situations [38, 39, 46].
the case, in order to add a node b to community A, it
must first be removed from some other community, say
B. Since the node b is in community B, b must have a
binding energy to B that must first be overcome. Any VI. SIMPLE EDGE DENSITY COMMUNITY
energyreleasedbymovingbtoAmustfirstoffsettheen- DETECTION ALGORITHMS
ergy needed to remove b from B. In order to determine
trade-offs between larger communities and greater edge Thepurposeofthispaperisnottodiscussspecificedge
density,wemustusetheHamiltonianfromoneofourset density community detection algorithms, instead, we fo-
of models. Towards this end, without loss of generality, cusontheground-statepropertiesofthiscommunitydefi-
we may use the APM. nition. Therearevariouspublishedalgorithmswhichcan
In order to demonstrate the choices which the edge beuseddirectlywithedgedensities. Previousworkfrom
density models make, we will use a simple thought ex- our group used a global algorithm, beginning with every
periment of one node x which can either join community node in a different community and merging communities
9
untilsomeenergyminimumisreached[38,39,49,50]. Al- density on average decreases with each additional node,
ternatively, there are various local methods, which build lowering the edge density. As we keep adding nodes to
up a single community around single nodes[51–53]. the community, eventually, the community will grow so
Unlike other methods, there is no heuristic for deter- large that adding extra nodes will decrease ρ below ρ .
∗
mining the proper ρ . Instead, we use a multi-replica in-
∗
ference method where independent “replicas” are solved
atthesamevalueofρ∗,andtheirsimilarityiscompared. VIII. OVERLAPPING COMMUNITIES
If, for a given value of ρ , the independent replicas min-
∗
imize to give similar community structures, this is con-
Many networks have community structure which can
sideredtobea“good”valueofρ . Thishasproventobe
∗ mostnaturallybedescribedasoverlapping. Inthisview-
a reliable and robust method for community detection.
point,therearecertainnodeswhichcanbereasonablyin-
For addition information, see Sec. A4.
cludedinmultiplegroups. Perhapsthemoststandardex-
Further information and considerations about these
ampleofthissituationissocialnetworks: anyoneperson
methods is found in Appendix A.
willbeinvolvedingroupscorrespondingtowork, family,
hobbies, etc. The overlaps can consist of single nodes
or larger subsets. Not all community detection meth-
VII. EDGE DENSITIES IN REAL NETWORKS
ods can be extended to handle overlapping nodes. For
example, the Newman betweenness algorithm progres-
Our edge density definition assumes that ρ is a use-
∗ sively cuts edges until modularity maximization states
ful resolution parameter, which can select for larger or
that final communities are found [13]. Because each cut
smaller communities. We have operated on an intuition
is final, there is no ability to create overlapping com-
that a larger ρ selects for a smaller, more densely con-
∗ munities. Different methods, such as clique percolation
nected communities, while a smaller ρ selects larger,
∗ [7], attempt to detect overlapping community structure.
less densely connected communities. In this section, we
There are a variety of local community definitions, in-
will directly consider this assumption. Recently, Yang
cluding the community-centric Potts model formulations
and Leskovec (YL) have taken a variety of real net-
of Eq. (8), can detect overlapping communities by virtue
works, mainly social networks, and rigorously studied of independently detecting each community[51].
theirpropertieswithrespecttosize,overlapregions,and
The work of YL also looked at the characteristic of
otherparameters[40]. Theyfindthatthenumberofedges
overlapping regions of communities[40]. According to
within a community tends to grow with a power in the
the behavior seen by YL, regions of overlap between
range (1,2), with observed values of 1.1 and 1.5. For
communities have edge density contributions from both
some exponent value ν, it is thus found
communities[40,54,55]. Thus,theseoverlapregionshave
a greater density than the individual communities. It
e nν. (26)
∝ had previously been assumed that these regions of over-
lapping communities had a smaller edge density than ei-
However, maximal number of edges (l in our nomencla-
ther of the non-overlapping regions, and past methods
ture) grows as
of community detection make the opposite assumption,
1 and thus not suitable for community detection with the
l= n(n 1) n2, (27)
observed behavior [7, 56, 57]. We can show directly that
2 − ∝
the edge density community definition properly handles
thus we find a scaling relation of edge density of this case.
Let us look at a diagram of two overlapping commu-
e nν
ρ= nν 2. (28) nities A and B, embedded in a universe of nodes (Fig 4,
−
l ∝ n2 ∝
upper). WedenotetheuniverseofnodesU,thetwocom-
munities A and B, the region of community A excluding
As long as ν (1,2), we find ρ decreases as n increases.
∈ B as A B and vice versa, the overlap region A B,
For example, YL observed social networks to have an
− ∩
and the universe of nodes excluding either community,
exponent value of ν 1.5, giving us
≈ U A B. Fig.4isanschematicofthissituation. Com-
− −
ρ n .5. (29) munitiesAandB bothhaveinternaledgedensitiesof.5.
−
∝ Theoverlappingregion A B hasanedgedensityof.75.
∩
As we see, this validates one of the central tenets of the The edge density between A B and A B is .5, and
− ∩
edgedensitycommunitydefinition: ascommunitiesgrow vice versa for B and A.
larger, the edge density tends to decrease. By specifying With ρ =.75, only the overlapping region A B will
∗
∩
aρ ,weimplicitlyspecifyasizescaleofcommunitywhich be detected, with ρ = .5, communities will expand to
∗ ∗
we will then detect. Another way to view this is from cover the region A or the region B, but not A B. The
∪
thestandpointofanagglomorativecommunitydetection reason for this is that, if we have currently detected the
alogrithm. Startingfromadensecore,aseachadditional community A, any one one node x in B A will have a
−
nodeisaddedtoagivencommunity,thecommunityedge p=.5edgedensitytoA B,butonlyp=.5edgedensity
∩
10
U (a) SBM (b) AGM
A B
A−B B−A A∩B U −A−B
FIG. 5: Schematic of stochastic block modes (SBM) vs af-
A−B .5 0 .5 0
filiation grapm model (AGM) graphs. (a) The SBM graphs
B−A 0 .5 .5 0
are non-overlapping, while (b) the AGM graphs allow over-
A∩B .5 .5 .75 0 laps. NotethatinourspecificinstancesofAGMgraphs,every
U −A−B 0 0 0 0 node is it at least one community.
FIG. 4: Schematic of two communities A and B with p =
A
pB = .5, and their overlap edge densities. This illustrates verse (yielding a planted l-partition, or Stochastic Block
theaffiliationgraphmodeledgedensitiesbetweeneachofthe Model), or any assortment of hierarchical, overlapping,
distinct regions of the figures, with a “dense overlap” in the subset-containing, or other arrangements. Then, we add
A∩B region, as determined by YL [40]. According to YL, an edge with probability p for each shared affiliation.
currentlyexistingcommunitydefinitionmodelscannotdetect
Sincetherecanbemultiplesharedcommunitiespernode,
thesecommunities. Thetableindicatestheprobabilityforan
we allow each shared affiliation a chance to produce an
edge being present between any two regions of the plot: the
edge. Thisgivesusanultimateedgeprobabilitybetween
“−” operator indicates set difference, A−B indicates the
nodes x and y of
region of A excluding B.
(cid:89)
p =1 (1 p ) (30)
xy C
− −
to A B, thus the average connection probability to the C
entire−setAisp .05nA−B+.5nA∩B <.5,thus,p <ρ =
Ax nA Ax ∗ where the product is over all communities C which con-
.5. This is the desired behavior. The same will be true
tains both nodes x and y. This is the complement of the
of a node y in A B attempting to join community B.
probability thatnone of the sharedaffiliations producea
−
Thus,byapplyingtheedgedensitycommunitydefinition,
community.
we can get exactly the desired behavior: for ρ = .5, we
∗ The instance of AGM networks we study here is con-
detect A and B, while for ρ =.75, we detect A B.
∗ structed as such: A universe of N nodes is taken. We
∩
take q communities of n nodes each, with nq N, in
≥
two steps: first, we initialize the communities with non-
IX. THE AFFILIATION GRAPH MODEL
overlapping communities of N/q n nodes. Then, for
≤
each community, we add additional nodes necessary to
Inordertomodelthepropertiestheyobserved,YLde- makennodespercommunitybyrandomlychoosingfrom
scribed the “Affiliation Graph Model” (AGM)[40]. This all other nodes. There are no restrictions to the maxi-
model is similar to stochastic block models in that edges mum number of communities to which a node can be-
are drawn in only based on the community memberships long. The edge probability between any two nodes is
of the two nodes. However, nodes are allowed to be- given by Eq. (30). Our particular AGM graphs can be
long to multiple communities, and the communities are identified by the parameters (q,n,p). The AGM bench-
not restricted to being disjoint. This can be interpreted mark graphs differ from stochastic block model graphs
as a bipartite graph from “people” to shared “affilia- by allowing overlaps, Fig. 5.
tions”. IntheAGMitself,theaffiliationsarenotpresent, Further, we may produce multi-layer benchmark
andinsteadeachsharedaffiliationincorporateschanceof graphs. It is traditional to produce hierarchical graphs
shared edge between “people” (nodes in the graph). A forconsideration,wheresmallcommunitiesarecontained
similar concept of affiliations grouping has been consid- within single large communities. To create our multi-
ered before in a sociological and network context[55, 58], layer graphs, we create AGM graphs independently over
and such work has hinted that under certain generation the same universe of nodes, and then merge the set of
processes,suchmodelscouldproducepower-lawdistribu- edges. There is no requirement that each small commu-
tion of node degrees[40]. Because this benchmark model nitybecontainedwithinasinglelargecommunity,andin
incorporates dense overlap, YL claim that current com- fact, for each community, each small community is likely
munity detection methods are not able to successfully to overlap with every large community. The amount of
detect communities in this type of graph. overlap present here is unparalleled in the existing lit-
The basic idea behind AGMs is that communities erature. Fig. 6 illustrates the difference between hi-
(“affiliations”) are chosen from a universe of nodes. erarchical and multi-layer community assignments. To
These could be non-overlapping and spanning the uni- produce multi-layer AGMs, we take two AGMs inde-
