Computational Feasibility of Clustering under Clusterability Assumptions

0.2 MB·English

by  Shai Ben-David

Computational Feasibility of Clustering under
Clusterability Assumptions
5
1
0
“Vanityofvanities, saithKoheleth;vanityofvanities, allisvanity.”(Ecclesiastes, Ch1;2)
2

n
a
Shai Ben-David
J

2 School of Computer Science, University of Waterloo,

] Waterloo,Ontario, N2L 3G1, Canada
C
[email protected]
C
.
s
c January 5, 2015
[

1
v
Abstract
7
3 Itis wellknown that mostof the common clustering objectives areNP-
4
hard to optimize. In practice, however, clustering is being routinely carried
0
0 out.
1. One approach for providing theoretical understanding of this seeming
0 discrepancy is to come up with notions of clusterability that distinguish re-
5
alistically interesting input data from worst-case data sets. Thehope isthat
1
: therewillbeclustering algorithms thatareprovably efficientonsuch“clus-
v
terable” instances. In other words, hope that “Clustering is difficult only
i
X when it does not matter”1 (the CDNM thesis for short). We believe that to
r
someextentthismayindeedbethecase. Thispaperprovidesasurveyofre-
a
centpapers along this lineofresearch andacritical evaluation their results.
Our bottom line conclusion is that that CDNMthesis is still far from being
formallysubstantiated.
We start by discussing which requirements should be met in order to
provide formal support the validity of the CDNM thesis. In particular, we
listsomeimpliedrequirements fornotions ofclusterability. Wethenexam-
ineexisting resultsinviewoftheserequirements andoutline someresearch
challenges andopenquestions.
1Thisphraseisinfactatitleofarecentpaper–[18].
1
1 Introduction
Thegoalofthisnoteistwo-fold. First,Iwouldliketoprovideapersonallybiased
overview of the research concerning the computational complexity of clustering
underdataniceness assumptions. Having worked in thisarea forquitesometime
now, I feel that while the TCS community appreciates work that may have prac-
tical relevance (and clustering is clearly a task that arises in many applications),
sometimes in this area there is a significant gap between research motivationand
the actual technical results it yields. A secondary aim of this paper is to call the
attentionofthetheoretical research communitytosomesuch gapsandencourage
furtherworkalong directionsthatmighthaveotherwiseseemed resolved.
1.1 Alternativestoworst-caseformeasuringcomputationalcom-
plexity
Computationalcomplexitytheory aimsto providetoolsfor thequantificationand
analysisofthecomputationalresourcesneededforalgorithmstoperformcompu-
tational tasks. Worst-case complexity is by far the best known, most researched
and best understood approach to computational complexity theory. In particular,
NP-hardness is a worst-case-instance notion. By saying that a task is NP-hard
(andassumingP = NP),weimplythatforeveryalgorithm,thereexistinfinitely
6
many instances on which it will haveto work hard. However, for many problems
this measure is unrealistically pessimistic compared to the experience of solving
themforpracticalinstances. AproblemmaybeNP–hardandstillhavealgorithms
that solve it efficiently for any instance that is likely to occur in practice or any
instanceforwhich onecares tofind an optimalsolutionfor.
Severalapproacheshavebeenproposedtobringingcomputationalcomplexity
theoryclosertotheactualhardnessfacedwhensolvingoptimizationproblemson
real data. Average Case Complexity ([20], [11]), analyzes run time w.r.t. some
given probability distribution over the input instances. Smoothed Analysis ([25])
examines the running time of a given algorithm by taking the worst case over all
inputs of the average runtime of the algorithm over some vicinity of the input. A
differentapproach istohaveanotionof“well-behaved-instances”,so thatonone
handitisreasonabletoexpectthatinstancesonecomesacrossinapplicationsare
so well behaved, and on the other hand there exist algorithms that can solve any
wellbehavedinputinpolynomialtime. Variousearlierapproacheshaveaddressed
computational hardness by defining subset of relatively-easy instances (most no-
2
tably, the area of parameterizedcomplexity ([19])). [9], and [16] proposegeneral
notions of tamed instances that apply across different problems. Both of these
papersapplysometypeofrobustnesstoperturbationsasthekeypropertyofsuch
well behaved instances. Algorithms that efficiently solve NP-hard problems on
such perturbation robust instances have been shownto exist for agnosticlearning
ofhalf-spaces ([13])and forgraphpartitioningproblems([17]). [6]formalizeda
uniquenessoftheoptimalsolutioncriterionasanotionofwellbehavedclustering
instances, which can also be applied to other types of problems. In this note we
will focus on the application of such approaches to clustering. We will discuss
those, as well as other notions of niceness-of-instances that are specific to clus-
tering problems, as a basis for alternatives-to-worst-case-complexity analysis of
clusteringtasks.
1.2 A focus on clustering tasks
Clustering is a very useful paradigm that is being applied in a wide range of data
exploration tasks. The term “clustering” should be thought of as an umbrella no-
tion for a big and varied collection of tasks and algorithmicparadigms. Here, we
focusonclusteringtasksthataredefined asdiscreteoptimizationproblems. Most
of those optimization problems are NP-hard. We wish to examine whether this
hardness remains an issue when we restrict our attention to “clusterable data” -
dataforwhichameaningfulclusteringexists(onecanarguethatwhenthereisno
clusterstructureinagivendataset,thereisnopointinapplyingaclusteringalgo-
rithmtoit). Inotherwords,wewishto evaluatetowhat extentcurrent theoretical
work supportsthe “Clustering is difficult only when it does not matter”(CDNM)
thesis.
For the sake of concreteness, we will focus on two popular clustering objec-
tives,k-means andk-median.
1.3 Outline of the paper
We start this note by listing, in Section 2, what we think are requirements from
notions of clusterability aiming to substantiate the CDNM thesis. In Section 4,
we list various notions of clusterability that have been proposed in the context
of this line of research. These include: Additive perturbation robustness (APR),
[1]; Multiplicative perturbation robustness (MPR), [16]; (α,ǫ) Perturbation Re-
silience, [8]; ǫ -Separatedness, [22]; Uniqueness of optimum, [6] (they call it
3
(c,ǫ)-approximation-stablility); α-center stability, [5]; and (1 + α) Weak Dele-
tionStability,[4].
The main body of this paper is an examination, in Section 5, of how well do
the current notions and results meet the requirements (of Section 2). To get a
senseofhowstrictaclusterabilityconditionis,weconsideran optimalclustering
of data sets that satisfy that condition and examine the implied bounds on the
ratio between the average distance of a data point to its own cluster center and
the distance between centers of different clusters (or the distance of a point from
centers of clusters it does not belong to). By analyzing the results pertaining to
theproposed notionsofclusterabilitylistedabove,we show,forexample,that,
The values of ǫ for which ǫ -Separatedness is shown (in [22]) to allow
•
poly(k) clustering algorithms imply that, in the optimal clustering, the av-
erage distance of a point from its cluster center should be smaller than the
minimaldistancebetweendistinctclustercentersbyafactorofatleast200.
The values of parameters for which (c,ǫ) approximation stability is shown
•
(in [6]) to allow poly(k) clustering algorithms imply that, in the optimal
clustering, for all but an ǫ-fraction of the input points, the distance of a
pointto itsownclustercenterissmallerthan itsdistancetothenextclosest
centerbyat least 20timestheaveragepoint-to-its-cluster-center-distance.
Thevalues of α for which (1+α) weak deletion stabilityis shown (in [4])
•
toallowpoly(k)clusteringalgorithmsimplythat,intheoptimalclustering,
thevastmajorityoftheclustersaresodistantfromtherestofthedatapoints
thatanypointoutsidesuchaclusterisfurtherfromthecenterofthatcluster
byat least log(k)timesthe”averageradius”ofitsowncluster.
Our conclusion is that the currently available theory is still far from substan-
tiating the CDNM thesis. In particular, while additive perturbation robustness,
with any non-zero robustnessparameter, givesrise to algorithmsthat find the op-
timal clusterings in time polynomial in the input size and its dimension, as far as
currentlypublishedresultsgo,nonoftherequirementslistedaboveallowsfinding
optimalclusteringsolutionsintimepolynomialinthenumberoftargetclusters,k,
unlessthecorrespondingparametersaresettovaluesthatholdonlyforextremely
wellclusterabledatasets2.
2Theaboveconsequencesoftherequiredclusterabilityconditionsareobtainedbyexamining
theparametervaluesandconstantsthatareimplicitintheasymptoticformulationoftheefficiency
resultsintheabovecitedpapers. Oneshouldnotethatthesenegativestatementsreflectonlythe
4
In Section 6 wediscussthesediscouragingresultsfurther, highlightsomeim-
pliedopen problemsand proposedirectionsinwhich thislineofresearch should,
inouropinion,proceed.
2 Requirements from notions of clusterability
We begin this discussion by stating requirements that (we believe) a notion of
clusterability should satisfy to be applied for supporting the “Clustering is Diffi-
cult only when it does Not Matter” (CDNM, in short) thesis. At this point those
requirementswillbestatedasqualitative,highlevel,statements. Wediscussmore
concretequantitativeformulationsinSection 5.
1. It should be reasonable to assume that most (or at least a significant pro-
portionof)theinputsonemaycaretoclusterinpracticesatisfythecluster-
abilitynotion.
Some disclaimer is in place here; Of course, we do not have any way to
guaranteethatunseenpracticalinstanceswillsatisfyanynon-trivialrequire-
ment. However, this type of consideration can serve as a way to filter out
clusterability conditions that are too restrictive. Furthermore, when a good
datagenerativemodelis available, onecan formalizerequirements pertain-
ing to a high probably of having the generated instances satisfy the given
clusterabilitynotion.
2. In order to support the CDNM thesis, a notion of clusterability should be
such that there exist efficient algorithmsthat are guaranteed to find a good
clustering(minimizingtheobjectivefunction,orgettingveryclosetoit)for
anyinputthatsatisfiesthatclusterabilityrequirement.
The next two requirements may be more debatable. Their significance is moti-
vated by considering practical aspects of clustering applications. Assume we do
havesomeclusterabilityconditionandaguaranteethatthealgorithmweareabout
to run is efficient on instances satisfying it. Still, when we get some real input,
thereisnoguaranteethatitsatisfiesthatclusterabilitycondition. Ifitdoesnot,and
currentstate of knowledge,andare notprovenlower bounds. For some of the abovenotionsof
clusterability, we also discuss lower bounds on the parameter values required to overcome the
NP-hardnessoftheclusteringtasks.
5
we run our algorithm, it may either run for too long or terminate with some sub-
optimal solution. However, for most of the NP-hard clustering problems, there is
no efficient way of measuring how far from optimal a given clustering solution
is. We are therefore in the risk of not being able to protect against bad solutions.
Thisconsiderationimpliesathirddesirablerequirement–theabilitytodistinguish
between clusterableand non-clusterableinputdatasets. Namely,
3. There exists an efficient algorithmfortestingclusterability. Namely, given
aninstance(X,d),thealgorithmdetermineswhetheritsatisfiestheclusterability
requirementor not.
Anotheradvantageofhavinganotionofclusterabilitysatisfythisrequirement
is that it will allow a direct evaluation of the extent to which the notion satisfies
Requirement 1 above. Namely, having an efficient clusterability -checking algo-
rithm,onecouldapplyittocollectionsofrepresentativepracticalclusteringinputs
fromvariousdomainandevaluatetowhatextenttheclusterabilityrequirementac-
tuallyholdsfor suchclusteringtasks.
A forth, somewhat orthogonal, desiderata relates to existingcommon cluster-
ingalgorithms. Namely,
4. Some commonly used clustering algorithm can be guaranteed to perform
well(i.e.,runinpolytimeandfindclose-to-optimalsolutions)onallinstancessat-
isfyingtheclusterabilityassumption.
Requirement 4 is important if our goal is to understand what is happening
nowadays in clustering work by providing a theoretical explanation for the suc-
cessofcommonclusteringalgorithmsonrealdata. However,evenwhenfailingit,
requirement 2 may lead to the development of new clustering algorithms, which
mayhaveindependentmerits.
The main Open Question: Find a notion of clusterability that satisfies the re-
quirementsabove(oreven justthefirsttwo).
3 Definitions and basic notions
Weconsiderclusteringtasksthat can bedescribed asfollows:
6
TheinputisafinitesubsetX ofametricspace(Y,d)3,andsomenumberk.
•
When X = Y or Y is someEuclidean space (with theEuclidean distance),
weomitmentioningitexplicitly.
The solution space is a collection of partitionings of the input set X into
• S
k subsets(a.k.a. k-clusterings).
The problem is determined by an objective function, , that maps pairs of
• O
(Instance, Clustering) to the real numbers. The goal of the algorithm is
to find a clustering in the solution space that minimizes this objective for
the given input instance. We let C (X,d) be ArgMin (C,(X,d))
O C∈SO
(namely the set of all clusterings in the solution space that minimize the
objective cost for the input). Finally, given some objective function, let
OPT(X,d) denote the cost of an optimal clustering or (X,d) (this value,
depends,of course, on theobjectivefunctionin question. However,to sim-
plify the notation, we suppress this dependence on the objective). We also
suppressthedistancefunctiond whenitisclear fromthecontextand when
itistheEuclideandistance.
We zoom in even further and consider only “center based” clustering objectives.
For such problems, a clustering is defined by a set of k points (centers), the par-
titionassociated with such a set of centers is theVoronoi partition it induces over
the input set, and the objective function has the form ((X,d),(c ,...c )) =
1 k
O
F(min d(x,c )), forsomenon-decreasing functionF : R R.
Px∈X i≤k i →
Thisfamilyofclusteringobjectivesincludescommontasks suchas,
k-means,where ((X,d),(c ,...c )) = (min d(x,c )2,
• O 1 k Px∈X i≤k i
k-median,where ((X,d),(c ,...c )) = min d(x,c ) and
• O 1 k Px∈X i≤k i
k-medoids, where the objective as the same as in k-median, but the cluster
•
centers, c ,...,c , are requiredto bemembersofX.
1 k
Givensuchaclustering,wecallthesubsetofX intheVoronoicellofeachcenter
c the i’th cluster and denote in by C . By extending the format of the objective
i i
functionto G( C ) F(d(x,c )),forsomenon-decreasingG,onecan
Pi≤k | i| Px∈Ci i
capture some additional common clustering objectives like the sum-of-incluster-
distances (MinSum). In this note we focus on the k-means, k-median and the
k-medoidsobjectives.
3In some cases, d is notrequiredto satisfy the triangle inequality, in which case we call it a
dissimilarityfunctionratherthanametric.
7
Allofthesethreeclustering-motivateddiscreteoptimizationproblemsareknown
to beNP-hard, and evenNP-hard to approximate(somesuch hardness results are
statedquantitativelylater).
Throughout this paper, we will use m to denote the input size (that is m =
X ) and use D(X) to denote the diameter of the input set, namely, D(X) =
| |
max d(x,y) : x,y X . When the data is a subset of some Euclidean space,
{ ∈ }
Rn, we use n to denote that dimension. Furthermore, unless otherwise stated,
wheneverwediscussthek-means objective,weassumethat X Rn.
⊆
3.1 Measures of clustering approximations
Whenitcomestoapproximationalgorithmsforclusteringthereisanothertechni-
calpointtobeawareof,namely,thewayinwhichonemeasuresthedifferencebe-
tweenanoptimalsolutionandanapproximateone. Thereareatleasttwodifferent
approachesofquantifyingthatgap. Thefirst,andprobablyalsothemostcommon
one, is to consider only the cost of the solutions. In other words, given a cluster-
ing objective function an input set X and a clustering, C of it, say that C is
O
an ǫ cost-approximategood solutionif (C,X) Opt(X)(1+ǫ) (alternatively,
O ≤
one could consider additive approximations to the cost, namely, requiring that
(C,X) Opt(X)+ǫ. Additiveapproximationsarisenaturallyinthecontextof
O ≤
statisticalmachinelearning, where approximatesolutionsare computed based on
smallsamplesoftheinputdata). Adifferenttypeofapproximations,morespecific
to clustering problems, is to define some measure of distance between solutions,
suchassomedistancebetween thecentervectorsoftwocenterbasedclusterings,
say (C,C′) d=ef inf max d(c ,c′ ),whereΠisthesetpermutations
Dcenters π∈Π i≤k i π(i)
of the cluster indices 1,...,k , and c ,...c , c′,...c′ are the clustercenters of
{ } 1 k 1 k
C andC′ (respectively),or (C,C′) d=ef (1/ X )inf C ∆C′ .
Derr | | π∈ΠPi∈{1,...k}| i π(i)|
Having such a measure of distance between clustering solutions, an approx-
imation algorithm is required to come up with a clustering that is close to an
optimalclusteringw.r.t. that measure.
There are some implications between these notions of clustering approxi-
mations. In particular, note that for, say, the k-means objective (C,X)
|O −
(C′,X) (C,C′) (for every input set X and clusterings C, C′).
centers
O | ≤ D
Roughly speaking, approximation is hardest with respect to the distance.
err
D
Someof theclusterabilityconditionsdiscussedbelow implythat such an approx-
imation follows from approximations w.r.t. the other measures. For example, the
8
Uniqueness of optimum condition (see below) explicitly requires that clustering
solutions that have objective cost close the optimal one are also close w.r.t. the
measure. Also, the Additive perturbation robustness clusterability implies
err
D
that any good enough approximation(of an optimal clustering) w.r.t. the
centers
D
isin itselfan optimalclustering.
4 Notions of clusterability
In the past few years there have been several interesting publications along the
lines described above, showing that for various notions of clusterability there are
indeed algorithms that find optimal clusterings in polytime for all appropriately
clusterableinstances. Belowisa(possiblynotexhaustive)listofmajornotionsof
clusterability that have been discussed in that context4. Most of these definitions
can beappliedto anyoftheabovementionedclusteringobjectives.
1. PerturbationRobustness: Aninputdatasetisperturbationrobustifsmall
perturbationsofitdonotresultinachangeoftheoptimalclusteringforthat
set.
(a) Additive perturbation robustness (APR) [1]5: An input set (X,d) is
ǫ-APR if for some optimal k-clustering C, for every d′, if d(x,y)
| −
d′(x,y) ǫ for every x,y X, then C C (X,d′). Namely, an
O
| ≤ ∈ ∈
optimal clustering of the input (X,d) remains optimal for any small
(additive) perturbation of this input6. Since this additive condition is
notscaleinvariant,weimplicitlyaddtheassumptionthatthediameter
of theinputset, max d(x,y),isat most1 (otherwisethestability
x,y∈X
parametershouldbemultipliedbythatdiameter).
(b) Multiplicativeperturbationrobustness(MPR)[16]: Aninputset(X,d)
is α-MPR if for some optimal k-clustering C such that for every d′,
4Thereadershouldbeawarethatdifferentpapersusedifferentterminologyforsimilarnotions
(andsimilarterminologyfordifferentnotions),somychoiceofterminologybelowisnotalways
consistentwithotherpublications.
5Thedefinitionofrobustness,as wellastheimpliedefficiencyofclusteringresult, in[1]are
particular cases of a more general definition and more general results of [9]. For the sake of
concisenessand dueto its similarity to othernotionsdiscussed below, we presenthere onlythis
case.
6 Thedefinitionin[1]isformulatedasrobustnessw.r.t. perturbationsoftheclustercentersof
theoptimalsolution. However,itcanbereadilyseenthatthetwodefinitionsareequivalent.
9
if d(x,y) d′(x,y) αd(x,y) for every x,y X, then C
≤ ≤ ∈ ∈
C (X,d′). Namely, an optimal clustering of the input (X,d) that re-
O
mainsoptimalforanysmall(multiplicative)perturbationofthisinput.
(c) [8] propose the following relaxation of the MPR requirement: A data
set (X,d) is (α,ǫ)-perturbationresilient if there exists some optimal
k-clusteringC suchthatforeveryd′,if d(x,y) d′(x,y) αd(x,y)
≤ ≤
forevery x,y X,then forsomeC′ C (X,d′), (C,C′) ǫ.
O err
∈ ∈ D ≤
2. ǫ -Separatedness: [22]7 discuss clustering w.r.t. the k-means objective.
They define an input data set (X,d) to be ǫ-separated for k if the k-means
costoftheoptimalk-clusteringof(X,d)islessthenǫ2 timesthecostofthe
optimal(k 1)-clusteringof(X,d).
−
3. Uniqueness of optimum: [7]8 define a data set to be (c,ǫ)-approximation-
stablewith respect to sometarget clusteringC ifevery clustering C of X
T
whose objective cost over (X,d) is within a factor c of the objective cost
of C (on (X,d)) is ǫ-close to C w.r.t. some natural notion of between -
T T
clusteringdistance. Itiseasilyseenthatsuchaconditionholdswithrespect
to any C if and only if it holds (up to constant factors) w.r.t. the opti-
T
mal clustering for (X,d) (see Fact 2.2. of [7]). We relate to this property
as Uniqueness of Optimum since it rules out the possibility of having two
significantlydifferent close-to-optimal-costsolutions.
4. α-centerstability: [5]defineaninstance(X,d)tobeα-centerstable (with
respecttosomecenterbasedclusteringobjective )ifforanyoptimalclus-
O
teringC C (X,d) defined bycenters c ,...c (oftheclustersC ,...C
O 1 k 1 k
∈
respectively),foreveryi kandeveryx C ,andeveryj = i,αd(x,c ) <
i i
≤ ∈ 6
d(x,c ). Namely, pointsare closer by a factor α to their own cluster center
j
thanto anyotherclustercenter.
5. (1+α)WeakDeletionStability: [4]defineaninstancefork-clusteringto
satisfythe(1+α) WeakDeletionStabilityconditionif, foralli = j,
6
OPT(i→j) > (1+α)OPT,
7This is a journal version of [21], where the definition and the main results were initially
introduced.
8Thisisajournalversionof[6],wherethedefinitionandthemainresultswereinitiallyintro-
duced.
10