Variable selection for clustering with Gaussian mixture models: state of the art

0.26 MB·

by  Abdelghafour Talibi

Variable selection for clustering with Gaussian mixture models: state of
the art
AbdelghafourTalibi1,BoujemaˆaAchchab1,RafikLasri2
1LAMSADLaboratory,UniversityofHasan1st,ESTB,BerrechidMorocco
2SAATLaboratory,UniversityofAbdelmalekEssadi,FPL,LaracheMorocco
Correspondingauthor: AbdelghafourTalibi,[email protected]
7
1 Abstract
0
Themixturemodelshavebecomewidelyusedinclustering, givenitsprobabilisticframeworkinwhich
2
itsbased,however,formoderndatabasesthatarecharacterizedbytheirlargesize,thesemodelsbehave
n
a disappointinglyinsettingoutthemodel,makingessentialtheselectionofrelevantvariablesforthistype
J ofclustering. Afterrecallingthebasicsofclusteringbasedonamodel,thisarticlewillexaminethevari-

1
ableselectionmethodsformodel-basedclustering,aswellaspresentingopportunitiesforimprovement
3
ofthesemethods.

]
L
Keywords
M
Model-basedclustering;MixtureModel;high-dimensionaldata;EMalgorithm;variableselection;model
.
t
a selection;penalization
t
s
[
I INTRODUCTION

1
Clustering aims to classify objects of a population in groups, where the objects in the same
v
6 group are similar to each other, and the objects in different groups are dissimilar. Unlike the
4 supervised classification where the number of groups is known in advance, at least for a sam-
9
ple, in the case of clustering, it is unknown how many groups and it remains to be estimated.
8
0 In fact, many fields of research used clustering methods on the data, in order to obtain groups
.
1 thatallowunderstandingandinterpretingthephenomenonstudied. Thereisaverylargefamily
0
of clustering methods. One of these is called partitioning methods that are based on heuristics
7
1 or geometric procedures defined by measure of proximity between observations in the same
:
v group, or between the observations in different groups, such as hierarchical clustering (Ward
Xi [1963]) and K-means (Macqueen [1967]) based on the minimization of the within-cluster sum
r of squares (WCSS) which is the distance between the observations belonging to the same clus-
a
ter. Another family is based on a probabilistic framework called mixture model, in this family;
theclassificationproblemisapproachedbyaprobabilisticapproach. Thisapproach,asitsname
suggests, uses probabilistic modeling. The goal is always the same: to establish an automatic
classificationofindividualsinhomogeneousgroups. Herethemeaninggiventothehomogene-
ity of the groups is different: it is no longer based on geometric considerations but relies on
the analysis of the probability distribution of the population. The Gaussian mixture models is
the most commonly used model. The notion of homogeneity is reflected by the fact that the
observationswhichareinthesamegrouparecomingfromthesameGaussiandistribution.
1
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal
These mixing models presented by Wolfe [1963], Scott and Symons [1971] and Duda et al.
[2000], and then studied by Mclachlan and Basford [1988], Mclachlan and Basford [1988],
Banfiled and Raftery [1993] or Fraley [1998] and Fraley and Raftery [2002], have many
advantages, such as their flexibility allowing to model a wide variety of random phenomena,
due to their ability to adapt to many situations, and the possibility of being statistically inter-
preted,but,giventhelargenumberofvariablespresentinmoderndatabases,thesemodel-based
methods are over-parameterized in such a situation, resulting a poor performance. Moreover,
this abundance of descriptive variables may seem an asset to determine a proper clustering of
data. However,onlyasubsetofthesedescriptivevariablesmayberelevantforclustering,other
variables may be redundant or even non-significant for the classification. In order to consider
onlytheinformationrequiredfortheclustering,theselectionofrelevantvariablesmustbecon-
sidered, which will both improve the clustering process and facilitate the interpretation of the
clusteringresultsobtained.
Somestudieshavebeenfocusedontheselectionvariablesinclustering;themaindifficultylies
in the construction of a criterion to guide the selection of variables but cannot be based on the
labels. The proposed methods are generally classified into filter or wrapper by the terminology
borrowedfromthesupervisedcase,introducedbyKohaviandJohn [1997]. Methodsknownfil-
ter treat the problem of variable selection regardless of the classification process. Among these
methods, one can cite the work of Dash et al. [2002] and Jouve and Nicoloyannis [2005]. In
contrast,thewrappermethodsofvariableselectionproceduresareincludedintheclassification
process.
Thisarticleisorganizedasfollows. Section2brieflyreviewsthebasicsofclusteringbymixture
model and its estimation with the EM algorithm. Next, the section 3 respectively present the
wrapper methods for selecting variables in the Bayesian approach (model selection) and under
the frequentist approach (penalization of the likelihood). Finally, some methods are tested on
realdataset.
II THEMIXTUREMODELANDTHEEMALGORITHM
2.1 Themixturemodel
Mixturemodelshaverecentlyreceivedmuchattention,giventheintuitiveideathatapopulation
iscomposedofseveralgroupsaswellastheirflexibilitytomodelawidevarietyofphenomena.
The Gaussian mixture models are based on the idea that each group is represented by a mul-
tivariate Gaussian distribution, since each observation x i = 1,...,n is a vector (x ,...,x )
i i1 iJ
withx thevalueofthevariablej fortheobservationx ,withdifferentparametersofthedistri-
ij i
butions of other groups, and the total population is represented by a mixture of these Gaussian
distributions. The general form of the likelihood of a mixture model with K component for a
singleobservationx :
i
K
(cid:88)
L(x ;θ) = π f (x ;θ ) (1)
i k k i k
k=1
With,π ,...,π : themixtureproportions,
1 K
f : theGaussiandistributionofthecomponentk,
k
1 1
f (x ;θ ) = exp(− (x−m )tV−1(x−m )) (2)
k i k (2π)J/2|V |1/2 2 k k k
k
2
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal
θ = {m ,V }: theparametersoff theGaussiandistributionofthecomponentk,
k k k k
m : themeansvectorofthecomponentk,
k
V : thecovariancematrixofthecomponentk.
k
And the general form of the likelihood of a mixture model with K components for n observa-
tionsx :
i
n K
(cid:89) (cid:88)
L(x;θ) = [ π f (x ;θ )] (3)
k k i k
i=1 k=1
Theparametervectoristhenθ = (π ,...,π ,m ,...,m ,V ,...,V ).
1 K 1 K 1 K
2.2 Themixturemodelandclustering
Clustering aims to classify the observations x , i = 1,...,n, representing a population com-
i
posedofnobservations,wheretheobservationsx aremeasuredonJ variables(x ,...,x ),to
i i1 iJ
K groups. lets (G ,...,G ) be a partition of the data. This data partition can be formalized by
1 K
z = (z ,...,z ), where the n vectors z = (z ,...,z ) are such that; z = 1 if the observation
1 n i i1 iK ik
x belongtotheclassk andz = 0otherwise.
i ik
The data clustering can be obtained by ”mixture approach”, which aims to estimate the param-
eters of the mixture and then classify the data by assigning each observation to the class with
the highest membership probability. Or, by ”classification approach”, which consider z also as
aparametertobeestimated.
2.2.1 TheEMalgorithm
Eveninthemixtureapproach,themaximumlikelihoodisdifficulttosolvebecauseitrepresents
aproducts,itiscommontomaximizethelog-likelihood:
n K
(cid:88) (cid:88)
logL(x;θ) = log[ π f (x ;θ )] (4)
k k i k
i=1 k=1
Maximizing the log-likelihood is even difficult, using the EM algorithm (Expectation Maxi-
mizationalgorithm)proposedbyDempsteretal. [1977],andstudiedbyMclachlanandKrish-
nan [1997],isthemostcommonlyusedwhichisbasedonthecompletedlog-likelihood:
n K
(cid:88)(cid:88)
logL (x;θ) = z [logπ +logf (x ;θ )] (5)
c ik k k i k
i=1 k=1
With: z = 1iftheobservationx belongtothecomponentk,otherwisez = 0.
ik i ik
3
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal
The EM algorithm is based on maximizing successive iterations of the expectation of the com-
pletepenalizedlog-likelihoodconditionallytotheobservationsxandacurrentvalueθ(r) ofthe
parametervector,
Q(θ|θ(r)) = E[logL (x,z;θ)|x,θ(r)] (6)
c
After the initialization of the vector of parameters θ(1), this algorithm alternates between the
followingtwosteps. Attherth iteration,
• E-step: This step consists to calculate the expectation Q(θ|θ(r)), returning to express the
conditionalprobabilitiesdenotedzˆ(r) thatx belongstothecomponentk:
ik i
πˆ(r)f (x ;mˆ(r),Vˆ(r))
zˆ(r) = P(z = 1|x ,θˆ(r)) = k k i k k (7)
ik ik i (cid:80)K πˆ(r)f (x ;mˆ(r),Vˆ(r))
k=1 k k i k k
• M-step: This maximization step consist to determine the vector of the parameters θˆ(r+1)
maximizingQ(θ|θ(r)). Thisisequivalenttodetermine:
TheproportionsvectormaximizingQ(θ|θ(r)):
∂Q (cid:80)n zˆ(r)
= 0 ⇒ πˆr+1 = i=1 ik (8)
∂π k n
k
ThemeansvectorsmaximizingQ(θ|θ(r)):
n
∂Q 1 (cid:88)
= 0 ⇒ mˆr+1 = zˆ(r)x (9)
∂mk k (cid:80)n zˆ(r) ik i
i=1 ik i=1
ThevariancematricesmaximizingQ(θ|θ(r)):
n
∂Q 1 (cid:88)
= 0 ⇒ Vˆr+1 = zˆ(r)(x −mˆr+1)(x −mˆr+1)t (10)
∂Vk k (cid:80)n zˆ(r) ik i k i k
i=1 ik i=1
2.2.2 Classificationrule
Oncetheestimationofthevectorofparametersθ isdone,wedeterminethebestpartitionofthe
observationsbyassigningeachindividualtotheclassforwhichithasthehighestprobabilityof
belonging.
ˆ
πˆ f (x ;mˆ ,V )
k k i k k
zˆ = (11)
ik (cid:80)K πˆ f (x ;mˆ ,Vˆ )
k=1 k k i k k
Each observation is finally assigned to the class for which the conditional probability is the
largest.
4
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal
III VARIABLESELECTIONFORGAUSSIANMIXTUREMODELS
Several recent studies were interested on variables selection for the clustering. The underlying
ideaoftheseworksisthatonlyasubsetofalltheexistingvariablesisrelevantfortheclustering,
theothervariablesareonlyharmfultothetheclustering. Theclusteringtaskshouldbetherefore
made on the basis of the relevant variables, eliminating insignificant variables improved at one
hand, the the clustering results, on the other hand, the interpretation of the resulting groups
should be mitigated by the meaning of the selected variables. In clustering based on Gaussian
mixturemodels,theselectionofrelevantvariableswastreatedintwoways.
In the first way, authors such as Law et al. [2004], Raftery and Dean [2006], Maugis et al.
[2009], Maugis et al. [2009] and Maugis and Michel [2009] treated the problem within a
Bayesian framework, in fact, the selection of the relevant variables is restated here as a model
selection problem with a determination of the role of each variable. In the second way, authors
suchasPanandShen [2007],WangandZhou [2008],Xieetal. [2008]andXieetal. [2007]
dealt with the problem within a Frequentist framework, by introducing a penalty term in the
likelihoodfunctioninordertoselecttherelevantvariables.
3.1 Variableselectionasamodelselectionproblem
Law et al. [2004], Raftery and Dean [2006] and Maugis et al. [2009], treat the variable
selection problem for model-based clustering by determining the role of each variable. This
determination of variables roles is recast in the work of Law et al. [2004] as an estimation
problem,whileintheworksofRafteryandDean [2006]andMaugisetal. [2009],itistreated
as a model selection problem in the context of Gaussian mixture models, where they consider
a parsimonious models based on a decomposition of the covariance matrix proposed by Fraley
andRaftery [1999]andCeleuxandGovaret [1995]:
V = λ D A Dt (12)
k k k k k
Whereλ isthelargesteigenvalueofV whichcontrolsthevolumeofthekth cluster,D isthe
k k k
eigenvectorsmatrixofV ,whichcontroltheorientationofthatclusterandA isadiagonalma-
k k
trixwiththescaledeigenvaluesasentries,whichcontroltheshapeofthatcluster. Byimposing
constraints on the various elements of this decomposition, a large range of models is available,
rangingfromthesimplesphericalmodelsthathavefixedshapetotheleastparsimoniousmodel
whereallelementsofthedecompositionareallowedtovaryacrosstheclusters.
Lawetal. [2004]proposeasolutiontothevariableselectionprobleminmodel-basedclustering
under the assumption that the irrelevant variables are independent of the relevant variables,
by treating it as an estimation problem, which prevents any combinatorial search. Instead of
selectingasubsetofvariables,theyestimateasetofactualvaluesϕ ’s(ϕ ∈ [0,1]),withϕ = 1
j j j
if the variable j is relevant for clustering (ϕ = 0 otherwise) they define the quantities ρ =
j j
P(ϕ = 1),theprobabilitythatthevariablej isrelevant,thesequantities(oneforeachvariable)
j
thattheycallfeaturesaliencies.
K J
(cid:88) (cid:89)
p(x|θ) = π (ρ f(x |θ )+(1−ρ )f(x |ζ )) (13)
k j j kj j j j
k=1 j=1
Where f(x |θ ) is the density function of the jth variable in the component k, and f(x |ζ )
j kj j j
a common density independent of the components. Since they are in the presence of a model
5
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal
selection problem, it is necessary to avoid the situation where all saliencies take the maximum
possible value. This is achieved by adopting the Minimum Message Length penalty(MML).
TheMMLcriterionencouragesthesalienciesofirrelevantvariablestobeequaltozero.
For Raftery and Dean [2006], the basic idea is to recast the variable selection problem as a
comparison problem between competing models for all variables considered initially. Compar-
ingtwonestedsubsetsisequivalenttocomparingtwomodels,inoneallvariablesthatareinthe
largest subset are informative for the clustering, while in the other, the variables considered for
exclusion are conditionally independent of the clustering given the variables included in both
models. ThiscomparisonisperformedusinganapproximationofBayesFactors.
ContrarytoLawetal. [2004],RafteryandDean [2006]donotconsiderthattheirrelevantvari-
ablesareindependentwiththerelevantvariables,buttheydefinethatalltheirrelevantvariables
subsetScisdependenttoallclusteringvariablessubsetS. thecompetingmodelsarecompared
through Bayes factor of the log-likelihood by the BIC approximation. And the selected model
maximizesthefollowingquantity:
(Kˆ,mˆ,Sˆ) = arg max {BIC (xS|K,m)+BIC (xSc|xS)} (14)
clustering regression
K,m,r,l,V
Where K is the number of components and m ∈ M is a model that belongs to the family of
parsimoniousmodelsavailableintheSoftwaremclust(FraleyandRaftery [1999]).
The first term of (14) corresponds to the BIC approximation of Gaussian mixture model with
K components, the second, to the BIC approximation of a linear regression of the irrelevant
variables in relation to the irrelevant variables. They propose an algorithm, which every step,
seeks to add the variable that improves the clustering as measured by BIC and evaluates if any
of the current grouping variables can be eliminated. At each step, the best combination of
number of components and clustering model is chosen. The algorithm stops when there is no
improvement.
Indeed, as in the work of Law et al. [2004], the dependence of all the irrelevant variables
to relevant ones seems questionable. To overcome the limits of the method of Raftery and
Dean [2006], Maugis et al. [2009] consider firstly the subset S which represents the relevant
variables,andwhichincludesasubsetRoftherelevantvariablesrelatedtoasubsetofirrelevant
variables, and secondly, Sc the complement of the subset S which is divided into two subset: a
subsetU ofirrelevantvariableswhichcanbeexplainedbylinearregressiontothesubsetRand
subset W of irrelevant variables that is completely independent of all relevant variables, and
try to find the subsets F = (S,R,U,W), so their model is called SRUW. The selected model
maximizesthefollowingquantity:
(Kˆ,mˆ,rˆ,hˆ,Fˆ) = arg max {BIC (xS|K,m)+
clustering
K,m,r,h,V (15)
BIC (xU|r,xR)}+BIC (xW|l)}
regression ind
The quantity (15) includes three terms, the first is the model-based clustering by a Gaussian
mixture model with K components on the subset S and m its shape chosen from a collection
of 28 parsimonious models available in Mixmod software (Biernacki et al. [2006]), the sec-
ond term represents a BIC approximation of the linear regression of the subset U of irrelevant
6
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal
variables to the subset R of relevant variables, r is the form of the covariance matrix of the
regressionassumedtobespherical,diagonalorunconstrained. Thelasttermcorrespondstothe
BIC of a Gaussian distribution of the subset of irrelevant variables that are assumed to be inde-
pendent of all relevant variables with l the shape of its variance matrix assumed to be diagonal
orspherical.
Also, Maugis and Michel [2009] present a new variable selection method for clustering. They
re-formthevariableselectionproblemofclusteringasamodelselectionprobleminthecontext
of density estimation. They assume that the observed sample come from an unknown proba-
bility distribution with density s. A specific model collection is defined: Each model S(K,v)
corresponds to a particular clustering situation where K is the number of cluster and v is the
subset of relevant variables. A density t belonging to S(K,v) is decomposed in a density of
a Gaussian mixture model with K components on the subset v of the relevant variables and a
multidimensional Gaussian density on the other variables. The problem is reformulated as the
choice of a model from a collection, as this choice automatically leads to a clustering of the
data and a selection of variables. Thus, a data-driven criterion is necessary to select the ”best”
model from a model collection. This criterion depends on unknown multiplicative constant to
beevaluatedinpractice. Aheuristicmethodcalled”slope”isproposedandtestedforthisprob-
lem. Their idea is that on the irrelevant variables, since the data are centered, individuals have
a mean equal to zero, and these variables do not allow distinguishing different groups. So on
these variables, the data is assumed to follow a common spherical Gaussian distribution with
mean vector equal to zero. While, on the relevant variables, the means vector of the different
components are free and the data are assumed to have a completely free and positive-definite
covariancematrix. Onthesevariablesthemixturemodelisselectedfromthefamily:
K
(cid:88)
L = x ∈ Rα → π f(x;m ,V ) (16)
(K,α) k k k
k=1
K
(cid:88)
With∀k ∈ {1,...,K},π ∈]0,1[, π = 1,m ∈ [−a,a],(V ,...,V ) ∈ D+ . (17)
k k k 1 K K,α
k=1
Where K is the number of components, v is the index of the subset of relevant variables which
theirCardinalisdenotedα,a > 0andD+ denotesasymmetricpositive-definitematrixrelated
K,α
to the specified form of Gaussian mixture. On the irrelevant variables, a spherical Gaussian
densitybelongingtothefollowingfamilyisconsidered:
G = x ∈ RJ−α → f(x;0,σ2I ),σ2 ∈ [λ ,λ ] (18)
(α) J−α m M
Thus,theGaussianmixturefamilyassociatedtothepairs(K,v)isdefinedby:
S = {x ∈ RJ → f(x )g(x );f ∈ L g ∈ G } (19)
(K,v) [v] [vc] (K,α) (α)
7
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal
3.2 Variableselectionbylikelihoodpenalization
On the other hand, Pan and Shen [2007], Wang and Zhou [2008], Xie et al. [2008]and Xie et
al. [2007] select the relevant variables and perform clustering by penalizing the log-likelihood
functiontomaximize. Thepenalizedlog-likelihoodfunctionhasthefollowingform:
logL (x;θ) = logL(x;θ)−p (θ) (20)
p λ
WherelogL(x ;θ)isthelog-likelihoodfunctionandp (θ)isthepenaltyfunction
i λ
The variable selection method of Pan and Shen [2007] is in the case of clustering for a small
samplesizeandhighdimension,whenthedatasizeexceedsthesamplesize.
Inspiredbythepenalizedregressionforselectingvariables(Tibshirani,1996;FanandLi,2001),
theyassumethatthepenalizationcanbeasviableforvariableselectioninthecontextofmodel-
based clustering and consequently they propose a clustering approach based on a penalized
model. Specifically,themeansm specifictoeachclusterareadaptedtoaglobalmeanm;with
k
apenaltyfunctionappropriatelyselected,somevariablesmeansonthevariouscomponentsare
estimated to be exactly m, allowing a selection of variables. To facilitate the variable selection
problemfor”J (Numbervariables)large,n(samplesize)small,”PanandShen [2007]consider
adiagonalcovariancematricescommonbetweenalltheclustersandtheyreduceandnormalize
thedatasothateachvariablehasameanequalto0andavarianceequalto1. Theformoftheir
l1-normpenaltyfunctionis:
K J
(cid:88)(cid:88)
p (θ) = λ |m | (21)
λ kj
k=1 j=1
Where λ an hyper parameter which controls the level of desired sparsity and m the mean
kj
of the jth variable in the component k. Thus, given that the observations are normalized, if
the means of a variable j in each component are equal m =,...,= m , this variable is
1j Kj=0
considered to be irrelevant. To select the value of the K the number of components, and the
valueofthehyperparameterλ,theyproposeamodifiedBICcriterion.
ˆ
BIC = −2logL (θ)+log(n)d (22)
p
ˆ
Where θ is the Maximum Likelihood Estimator (MLE) and d = dim(θ) is the total number of
unknownparameters(FraleyandRaftery,1998).
Illuminated by the method of Pan and Shen [2007], Wang and Zhou [2008] also provides
a method of variable selection for model- based clustering for a low sample size and large
dimension,theyconsideracommondiagonalcovariancematricesbetweentheclustersandthey
reduce and normalize the data so that each variable has a mean equal to 0 and a variance equal
to1. Consideringthatthemeanparametersinclustersassociatedwiththesamevariablecanbe
naturally grouped together, and intuitively should be treated as a group, they propose two new
penalty functions, different from the penalty function of Pan and Shen [2007], which does not
8
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal
take into consideration the ”grouping” information in the data. To eliminate non-informative
variables,allm ,k = 1,...,K,shouldbeequaltozero. However,thel1-normpenaltyfunction
kj
proposedbyPanandShentreatedm individually,and,itdoesnotusetheinformationthatm
kj kj
andm areassociatedwiththesamevariablex ,andintuitively,theybelongtoa”group”and
k(cid:48)j j
should be treated differently from m , which are associated with another variable x . When
kj(cid:48) j(cid:48)
the jth variable is uninformative, the l1-norm penalty function tends to shrink only a portion
of m , but not all to zero, where it fails to consider the jth variable as being irrelevant. This
kj
brought Wang and Zhou [2008] to propose the l∞-norm penalty function which shrinks, for
eachvariable,themeansinallclustersinordertoidentifytheirrelevantvariables:
J
(cid:88)
p (θ) = λ max |m | (23)
λ ∞ kj
k∈1,...,K
j=1
The l∞-norm penalty function penalizes the maximum absolute value of m , k = 1,...,K, if
kj
the maximum absolute for the jth variable is equal to zero, all other means for this variable in
thedifferentclustersareautomaticallyreducedtobeequaltozero.
ForthesamepurposeasWangandZhou [2008],Xieetal. [2007]triedtoovercomethelimits
of the l1-norm penalty function but in another way, they also found that the l1-norm penalty
function treats the m individually, on the other side, a variable is irrelevant if m =,...,=
kj 1j
m = 0, in fact, to make a selection of relevant variables, it is natural to treat the means
Kj
m ,...,m asagroupofparametersandtoconstructapenaltyencouragingallthesemeansof
1j Kj
a variable in the different clusters be equal to zero. They observe that if considering the means
in the clusters as a row vector, the direction of the regrouping m ,...,m is vertical and they
1j Kj
callittheverticalmeansgrouping(VMG),forwhich,theyproposethefollowingpenalty:
√ J
(cid:88)
p (θ) = λ K (cid:107)m (cid:107) (24)
λ .j
j=1
(cid:113)
Wherem = (m ,m ,...,m )tand(cid:107)m (cid:107) = (cid:80)K (m )2isthel2-normpenaltyfunction
.j 1j 2j Kj .j k=1 kj
onthemeansm ’sfork = 1,2,...,K.
kj
On the other hand, they consider also, that in some cases, through prior information, a group
ofvariables issusceptibleto berelevantor not,thus,they proposeanothergroup thatconsiders
this prior information, named the horizontal means grouping (HMG), in the case of common
diagonal covariance matrices between the clusters. The grouping penalty proposed has the
following:
K G
(cid:88)(cid:88)√
p (θ) = λ q (cid:107)mg(cid:107) (25)
λ g k
k=1 g=1
Where mm corresponds to the mean of a group of variables, dim(mg) = q and (cid:80)G q = J,
k k g g=1 g
with J the number of variables. A modified BIC is proposed as a model selection criterion to
selectthenumberofcomponentsK.
9
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal
Alltheseworksalreadymentioned,assumethattheclustershaveacommondiagonalcovariance
matrices,thecommonmatriximpliesthattheclustershavethesamesize,whichmaybewrong
inpractice. Indeed,Xieetal. [2008]extendthemethodofPanandShen [2007],byconsidering
a cluster-specific diagonal covariance matrices, for which they have presented the following
penaltiesfunctions:
K J K J
(cid:88)(cid:88) (cid:88)(cid:88)
p (θ) = λ |m |+λ |σ2 −1| (26)
λ 1 kj 2 kj
k=1 j=1 k=1 j=1
K J K J
(cid:88)(cid:88) (cid:88)(cid:88)
p (θ) = λ |m |+λ |logσ2 | (27)
λ 1 kj 2 kj
k=1 j=1 k=1 j=1
The l1-norm penalty is used to force the irrelevant variables to have means m equal to 0, and
kj
avariancesσ2 thatarecloseto1tobeexactly1.
kj
Toinsertthepenalty,theyproposeamodifiedBICasamodelselectioncriterion.
Also, the diagonal covariance matrices assumption implies that the clusters have the same ori-
entation,whichmaybealsowronginpractice,inciteZhuandPan [2009]toproposeapenalized
likelihood approach for models with unconstrained covariance matrices. The first penalty pro-
posed,allowthecommoncovariancematricestobeunconstrainedandhavethefollowingform:
K J J J
(cid:88)(cid:88) (cid:88)(cid:88)
p (θ) = λ |m |+λ |W | (28)
λ 1 kj 2 jl
k=1 j=1 j=1 l=1
WhereW aretheelementsofW = V−1 theinverseofthecovariancematrix.
kj
Thesecondpenaltyfunction,allowalsotothecovariancetobedifferentacrosstheclusters,and
havethefollowingform:
K J K J J
(cid:88)(cid:88) (cid:88)(cid:88)(cid:88)
p (θ) = λ |m |+λ |W | (29)
λ 1 kj 2 k,jl
k=1 j=1 k=1 j=1 l=1
3.3 Variableselectionbycombininglikelihoodpenalizationandmodelselection
Meynet and Maugis [2012] suggest the Lasso-MLE procedure, which combines both the
method of Pan and Shen [2007] and the method of Maugis and Michel [2009] The first step
of their approach is to create a model sub-collection. As Pan and Shen [2007], a l1-norm
penalty is considered to get a sub-collection of models {S(K,J ),(K,J ) ∈ Ml}, where K is
r r
the number of components, and J the subset of variables selected as relevant by the penalized
r
maximum likelihood and Ml is the penalized maximum likelihood. By changing each time K
thenumberofcomponentsofthemixtureandtheregularizationparameterλ,anEMalgorithm
isusedtomaximizethepenalizedlog-likelihood.
10
JournalofDataMiningandDigitalHumanities http://jdmdh.episciences.org
ISSN2416-5999,anopen-accessjournal