Model-based Classification and Novelty Detection For Point Pattern Data

1.1 MB·

by  Ba-Ngu Vo

Model-based Classification and Novelty Detection
For Point Pattern Data
Ba-Ngu Vo∗, Quang N. Tran∗, Dinh Phung† and Ba-Tuong Vo∗
∗Curtin University, Australia
†Deakin University, Australia
Abstract—Point patterns are sets or multi-sets of unordered other hand, the BS paradigm addresses the problem at the
elements that can be found in numerous data sources. However, most fundamental level by operating directly on the point
indataanalysistaskssuchasclassificationandnoveltydetection,
patterns. The philosophy of the BS paradigm is to preserve
appropriate statistical models for point pattern data have not
the information content of the data, which could otherwise be
received much attention. This paper proposes the modelling of
point pattern data via random finite sets (RFS). In particular, corrupted through the data transformation process. However,
we propose appropriate likelihood functions, and a maximum existingmethodsintheBSparadigmareconfinedtodistance-
likelihood estimator for learning a tractable family of RFS based approaches [9], while statistical modelling tools and
7 models.Innoveltydetection,weproposenovelrankingfunctions
model-based approaches have been overlooked.
1 basedonRFSmodels,whichsubstantiallyimproveperformance.
0 In this paper, we introduce statistical models for point
2 Index Terms—Classification, novelty detection, naive Bayes pattern data using Random Finite Set (RFS) theory [11],
b model, point pattern data, multiple instance data, point process, [12], [13]. In particular, we propose appropriate likelihood
e random finite set. functions, and a maximum likelihood estimator for learning
F (from training data) a tractable family of models, called iid-

8 clusterRFSs.Further,innoveltydetectionwhereobservations
I. INTRODUCTION are ranked according to their likelihoods, we show that the

] standard RFS densities are not suitable for point patterns and
G Point patterns are sets or multi-sets of unordered points (or
proposed novel ranking functions that substantially improve
L elements) that can be found in numerous data sources. In nat-
performance.
ural language processing and information retrieval, the ‘bag-
.
s
of-words’ representation treats each document as a collection
c
[ or set of words [1], [2]. In image and scene categorization, II. AMOTIVATINGEXAMPLE

the ‘bag-of-visual-words’ representation–the analogue of the
2 The objective of a classifier is to assign a class label
‘bag-of-words’–treats each image as a set of its key patches
v yˆ ∈ {1,...,C} to an unseen data point X that consists of
3 [3]. In data analysis for the retail industry as well as web m feature vectors x ,...,x . In the Bayesian framework, the
1 m
7 management systems, transaction records such as market-
optimal class label is given by yˆ= argmax p(y |X), where
4 basketdata[4],[5]andweblogdata[6]aresetsoftransaction y
p(y |X) ∝ p(y)p(X | y) is the class posterior probability,
8
items. Other examples of point pattern data could be found in
0 p(y) is the prior probability of class y, and
drug discovery [7], protein binding site prediction [8].
.
1 One simple approach to the classification problem for point p(X |y)=p(x1,...,xm |y) (1)
0
patternsisviathena¨ıveBayes(NB)classifier,seeforexample
7
is the data model or data likelihood.
1 [2], [3], [6]. However, the broader task of learning from point
The data model used by the na¨ıve Bayes (NB) classifier
: patterndataismoreappropriatelyposedasaMultipleInstance
v [14, pp. 718] imposes a conditional independence assumption
Learning(MIL)problem[9],[10],sincemultipleinstancedata
i
X or ‘bags’ are indeed point patterns. According to the recent among the features so that
r review article [9], there are three paradigms for multiple in- m
a (cid:89)
stance classification, namely Instance-Space (IS), Embedded- p(X |y)= p (x |y) (2)
f i
Space (EM), and Bag-Space (BS). These paradigms differ i=1
in the way they exploit data at the local level (individual
where p (x | y) is the conditional probability of the i-th
data points within each bag) or at the global level (the bags f i
feature given class y (see also [15, pp. 82–89], [16, pp. 380–
themselves as data points). IS is the only paradigm exploiting
381]).
data at the local level. At the global level, the ES paradigm
In novelty detection or semi-supervised anomaly detection
maps all point patterns to vectors of fixed dimension, which
[17],thedatalikelihoodplaysanevenmoreimportantrole.In
are then processed by standard classifiers for vectors. On the
this approach, data are ranked according to a data likelihood
(learned from normal data), and data points with likelihoods
DinhPhunggratefullyacknowledgessupportfromtheAirForceOfficeof
ScientificResearchunderawardnumberFA2386-16-1-4138. lower than a threshold are considered as anomalies [18].
Consider the following example on anomalous patterns of Apart from the inconsistency with unit of measurement,
daily fallen apples. The apples land on the ground indepen- suchpointpatternlikelihoodalsosuffersfromanotherproblem
dently from each other, and the probability distribution of the associated with cardinality. Let us revisit the fallen apples
landing positions of the apples is shown in Fig. 1 (the thick example,howevertoeliminatetheeffectoftheunitmismatch,
solid line). The number of apples and their landing positions this time we restrict ourselves to a finite number of landing
for each day are recorded and we are interested in detecting positions, by discretizing the interval [−1m,1m] into 201
anomalous behavior of the daily apple landing pattern. points {−100,...,100} and round the landing positions to the
nearest of these points (Fig. 2). Thus, instead of a probability
Apple tree densityofthelandingpositionsontheinterval[−1m,1m]we
now have a (unitless) probability mass function (pmf) on the
p (pdf of landing positions)
f
discrete set {−100,...,100}, see Fig. 2.
p (pmf of landing positions)
f

−1
1 m 0.6 m−1 0.01 06
−1 −1 0.006
0.6 m 0.2 m 0.0
-1 -0.4 0 0.4 0.8 1 -100 -40 0 40 100
L𝑥𝑥a3nding positions𝑥𝑥 (2meter) 𝑥𝑥1 Landing positions
Figure1. Distributionoflandingpositions.Positionx1=0.8mis3times Figure2. Distributionofdiscretelandingpositions.
less likely than x2 = 0.4m and x3 = −0.4m which are equally likely.
Credit:clipartbest.com(appletreeclipart)
Fig. 3a shows 4 ‘normal’ patterns of fallen apples, each
with about 10 locations i.i.d. from the pmf of Fig. 2. Two
Supposethatonday1weobservedoneapplelandingatx ,
1 new observations X(1) and X(2) whose features are also i.i.d.
and on day 2 we observed two apples Clarendditi: nclgipaarttbexst2.coamn (dappxle3 ,tree cflripoamrt) the same pmf are shown in Fig. 3b.
which of the patterns on these two days is more likely to be
ananomaly?Todetectanomaliesinthissetting,itisnaturalto
ranktheobservedpatternsinorderoftheirlikelihoods.Inthis
illustration, we follow [2], [3], [6] and use the NB likelihood1
-100 -50 0 50 100 -100 -50 0 50 100
(2) which gives -100 -50 0 50 100 -100 -50 0 50 100
p(x )=p (x )=0.2,
1 f 1
-100 -50 0 50 100 -100 -50 0 50 100
p(x ,x )=p (x )p (x )=0.36. -100 -50 0 50 100 -100 -50 0 50 100
2 3 f 2 f 3
(a)Normaldata
where p is pdf of landing positions shown in Fig. 1.
f
Since p(x ) < p(x ,x ), the pattern observed on day 1 is
1 2 3
morelikelytobeananomalythanthepatternobservedonday 𝑋𝑋((11)) 𝑋𝑋((22))
-100 -50 𝑋𝑋0 50 100 -100 -50 𝑋𝑋0 50 100
2. However, if we measure distance in centimeters, then -100 -50 0 50 100 -100 -50 0 50 100
(b)Newobservations.Notethat,byNBlikelihood,wehavep(X(1))≈
p(x )=0.002>p(x ,x )=0.000036, 2×10−23 andp(X(2))=0.009
1 2 3
Figure3. Examplesofnormaldataandanomaly.
and hence the pattern observed on day 2 is more likely to
be an anomaly than that of day 1. The likelihood (2) yields Since X(2) has only 1 feature whereas the ‘normal’ obser-
contradictory results on the same scenario with different units
vations each has around 10 features, it is intuitively obvious
of measurement! that X(2) is anomalous. However, its likelihood is much
Note that in the above analysis, the units of the likelihoods higher than that of X(1) – a normal datum (0.009 versus
wereoverlookedbecausep(x1)ismeasuredinunitsofm−1or 2×10−23).Thiscounterintuitivebehaviorcannotbeattributed
cm−1 while p(x ,x ) is measured in units of m−2 or cm−2.
2 3 to the measurement unit inconsistency because the pmf of the
The observed inconsistency arises from the incompatibility
features is unitless.
in the unit of measurement in the likelihoods p(x ) and
1
p(x ,x ), i.e., we are not “comparing apples with apples”.
2 3
In general, the unit of the likelihood (2) depends on number III. MODELSFORPOINTPATTERNDATA
of features in X, i.e. the cardinality of X. Hence, comparing
The likelihood (2) was used in the above discussions to
the likelihood (2) of different observations is not meaningful,
illustratediscrepancieswithmeasurementunitandcardinality.
unless they have the same number of features or the features
However, these discrepancies arise even in the full joint
are intrinsically unitless.
likelihood (1). In this section, we propose models for point
patterndatausingrandomfiniteset,whichcouldaddressthese
1Forcompactness,theconditiononthenormalclassisomitted,i.e.,p(X)
isusedinsteadofp(X|y=‘normal’). issues.
A. Random Finite Set p(X)=p (|X|)|X|![Up ]X (5)
c f
Pointpatternscanbemodeledasrandomfinitesets(RFSs),
where p is a probability density on X, referred to as the
f
or simple finite point processes. Point process theory, in feature density, and hX (cid:44) (cid:81) h(x), with h∅ = 1 by
general,isconcernedwithabstractrandomcountingmeasures. x∈X
convention, is the finite-set exponential notation.
RFSs are geometrically more intuitive and thus better suited
When p is a Poisson distribution we have the celebrated
for the type of discussions in this article. The likelihood of c
Poisson point process (aka, Poisson RFS)
a point pattern of discrete features is straightforward since
this is simply the product of the cardinality distribution and p(X)=λ|X|e−λ[Upf]X (6)
the joint probability of the features given the cardinality. The
where λ is the mean cardinality. The Poisson model is com-
difficultiesariseincontinuousfeaturespaces.Inthiswork,we
pletelydeterminedbytheintensityfunctionu=λp [12],[21],
only consider continuous feature spaces. f
[22].NotethatthePoissoncardinalitydistributionisdescribed
Let F(X) denote the space of finite subsets of a space X.
by a single non-negative number λ, hence there is only one
Arandomfiniteset(RFS)X ofX isarandomvariabletaking
degree of freedom in the choice of cardinality distribution for
valuesinF(X)[19],[11],[12],[20],[13].Inessence,anRFS
the Poisson model.
is a finite-set-valued random variable that is random in the
Giventhetrainingdata,akeytaskinlearningistocompute
number of elements, as well as the values of the elements.
estimatesoftheunderlyingparametersofthemodel.Learning
An RFS X can be completely specified by a discrete (or
the general model (4) is computationally intensive. The iid-
categorical) distribution that characterizes the cardinality |X|,
clustermodel(5),ontheotherhand,providesagoodtrade-off
andafamilyofsymmetricjointdistributionsthatcharacterizes
between tractability and flexibility.
the distribution of the points (or features) of X, conditional
on the cardinality.
C. Maximum Likelihood Estimation
Analogous to random vectors, the probability density of
an RFS (if it exists) is essential in the modeling of point This subsection presents a solution to learning the param-
pattern data. The probability density p : F(X) → [0,∞) of eters of an iid-cluster RFS model using maximum likelihood
an RFS is the Radon-Nikodym derivative of its probability (ML) estimation.
distribution relative to the dominating measure µ, defined for Given a finite list of observations Z(1),...,Z(N) ∈Z and a
each (measurable) T ⊆ F(X), by [19], [12], [21], [22]: parametrized probability density f(·|θ) on Z, we denote
µ(T)= (cid:32) N (cid:33)
θˆ(f;Z(1),...,Z(N))(cid:44)argmax (cid:89)f(Z(n) |θ) (7)
(cid:88)∞ 1 (cid:90) 1 ({x ,...,x })d(x ,...,x ) (3) θ n=1
m!Um T 1 m 1 m
m=0 IfthedatapointsZ(1),...,Z(N) arei.i.d.accordingtof(·|θ),
where U is the unit of hyper-volume in X, and 1T(·) is the then θˆ(f;Z(1),...,Z(N)) is indeed the maximum likelihood
indicator function for T. The measure µ is the unnormalized estimate (MLE) of θ.
distribution of a Poisson point process with unit intensity Since an iid-cluster RFS is uniquely determined by its
u = 1/U when X is bounded. Note that µ is unitless and cardinality and feature distributions, we consider cardinality
consequently the probability density p is also unitless. andfeaturedistributionsparametrizedbyp (·|θ )andp (·|θ ),
c c f f
In general the probability density of an RFS, with respect i.e.
to µ, evaluated at X ={x1,...,xm} can be written as p(X |θ ,θ )=p (|X||θ )|X|!U|X|[p (·|θ )]X (8)
c f c c f f
p(X)=p (m)m!Umf (x ,...,x ), (4)
c m 1 m Learning the underlying parameters of an iid-cluster model
where pc(m) = Pr(|X| = m) is the cardinality distribution, amountstoestimatingθ =(θc,θf)fromtrainingdata.Further-
and fm(x1,...,xm) is a symmetric joint probability density more, the MLE of the iid-cluster model parameters separates
of the points x1,...,xm given the cardinality, see [23, p. 27] intotheMLEofthecardinalitydistributionparametersθc and
((Eqs. (1.5), (1.6), and (1.7)), [12], [21]. MLEofthefeaturedensityparametersθ .Thisisstatedmore
f
concisely in the following Proposition.
B. Likelihoods for point pattern data
Proposition 1. Let X(1),...,X(N) be N i.i.d. realizations
Instead of a random vector X, we propose to model each
of an iid-cluster RFS with parametrized cardinality distribu-
point pattern as an RFS X. A general form for the likelihood
tion p (·|θ ) and feature density p (·|θ ). Then the MLE of
c c f f
of X is given by (4), which can capture the cardinality
(θ ,θ ), is given by
c f
information as well as the dependence between the features.
(cid:16) (cid:17)
The RFS data model also avoids the unit of measurement θˆ = θˆ p ;|X(1)|,...,|X(N)| (9)
c c
inconsistency since the probability density with respect to µ
(cid:16) (cid:17)
is unitless. θˆ = θˆ p ;(cid:93)N X(n) (10)
f f n=1
Imposing the ‘na¨ıve’ conditional independence assumption
among the features on the model in (4) reduces to the iid- where (cid:93)N X(n) is the disjoint union of X(1),...,X(N).
n=1
cluster RFS model [11] Proof. Using (8), we have
N
(cid:89) to classify new observations. The results are bench-marked
p(X(n) |θ ,θ )
c f against the NB model. The first experiment uses simulated
n=1
data so as to illustrate the benefit of cardinality information
N
= (cid:89)p (|X(n)||θ )|X(n)|!U|X(n)| (cid:89) p (x|θ ) whenthefeaturesbetweentheclassesaresimilarlydistributed.
c c f f
The second experiment uses real-world data from the Texture
n=1 x∈X(n)
(cid:32) N (cid:33) (cid:32) N (cid:33) images dataset [25].
= (cid:89)p (|X(n)||θ ) · (cid:89)|X(n)|!U|X(n)| 1) Classification with simulated data: In this experiment,
c c
data are simulated from an underlying model consisting of
n=1 n=1
  three clusters C , C and C . A datum from C is a finite
N 1 2 3 i
(cid:89) (cid:89) set X whose cardinality is Poisson distributed with mean λ ,
· pf(x|θf) (11) i
andwhosefeaturesarei.i.d.froma2-DGaussianN(·;µ ,Σ )
n=1x∈X(n) i i
where
Hence, to maximize the likelihood we simply maximize the λ =6, µ =[1,2]T, Σ =diag[20,40],
1 1 1
first and last bracketed terms in (11) separately. This is
λ =15, µ =[2,3]T, Σ =diag[60,20],
2 2 2
achieved with (9) and (10). QED.
λ =30, µ =[2,2]T, Σ =diag[30,30].
3 3 3
Observed from Proposition 1 that the MLE of the feature
density parameters is identical to that used in NB. For exam- Fig.4plotsthefeaturesandcardinalitiesofdatasampledfrom
ple, if the feature density is a Gaussian N (µ,Σ), then the these models.
parameters of its ML estimates are:
1
µˆ = (cid:80)N (cid:80) x, (12)
N n=1 x∈X(n)
1
Σˆ = (cid:80)N (cid:80) (x−µˆ)(x−µˆ)T . (13)
N n=1 x∈X(n)
Consequently, the iid-cluster model requires only one addi-
tional task of computing the MLE of the cardinality distribu-
tion parameters, which is relatively inexpensive.
For a categorical cardinality distribution, i.e. θ =
c
(p ,...,p ), the MLE of the cardinality distribution is given
1 K
by Figure4. SimulateddatafortheexperimentinsectionIII-D1.Left:Features
pˆ = 1 (cid:80)N δ (|X(n)|). (14) ofthedata.Right:Cardinalityhistogramofthedata.
k N n=1 k
BoththeNBandRFSmodelsaretrainedviaMLestimation
Since there are K parameters p ,...,p , we require a suf-
1 K on a fully observed training dataset set consisting of 900 data
ficiently large dataset (significantly larger than K). For a
points(300percluster).ForNB,thedatamodelsarethree2-D
smalldataset,acardinalitydistributionwithasmallnumberof
Gaussians (one for each cluster). For the RFS models, we use
parameters should be used to avoid over-fitting, e.g. Poisson,
three Poisson RFSs with 2-D Gaussian feature distributions
i.e. θ =(λ) where its MLE is given by
c (which are in fact the same as the Gaussians learned from the
1
λˆ = (cid:80)N |X(n)|. (15) NB models).
N n=1
Aftertraining,weevaluatethelearnedmodelsbyclassifying
Conceptually, Proposition 1 can be extended to the general
a test set consisting of 1500 data points (500 per cluster, also
RFSmodel(4),whichrelaxesthena¨ıveindependenceassump-
simulated from the described model). The evaluation are run
tion. The MLE of the cardinality distribution parameters is
10timeswith10differenttestsets,andtheaverageaccuracies2
computed as for the iid-cluster RFS model. However, for the
areshowninFig.5a.ObservedthattheRFSmodelcanexploit
feature distribution, instead of θˆ(p ;(cid:93)N X(n)) we need to
f n=1 the cardinality information and delivers better performance
compute the MLE of the parameters for the joint densities,
than NB.
i.e. N
(cid:89)
argmax f (X(n) |θ ) (16)
|X(n)| f
θ n=1
which is far more complex in general. Imposing additional
assumptions such as TAN may provide some simplifications.
Alternativemodelssuchasmixtureofiid-clusterRFSs[24]are
also promising. However, these are topics for future research. (a) Simulateddata (b) Texturedata
Figure5. PerformanceofclassificationbyNBandPoissonRFS.Notethatthe
D. Numerical experiments
error-bars(onthepeakofeachcolumn)arestandarddeviationsofaccuracies,
Thissubsectionpresentstwoclassificationexperimentswith whichshowthatPoissonRFSworksmorestablythanNB.
simulated and real-world data. These experiments involve
learning from training data, and then use the learned models 2Accuracy(cid:44) No.ofcorrectclassifications
No.ofobservationsinthetestset
2) Classification with real data: The second experiment we propose a consistent ranking function for the iid-cluster
involves classification of the two classes “T14 brick1” and model.
“T15 brick2” from Texture images dataset [25]. Fig. 6 show
some example images from these classes. A. Ranking Function
Note that the probability density of an iid-cluster RFS (5)
Class: T14_brick1, image: 8 Class: T15_brick2, image: 6
is a product of pX and a term that depends only on the
f
cardinality p (|X|)|X|!U|X|. Given the cardinality, pX is the
c f
density of the feature distribution relative to the Lebesgue
measure,andhenceisthelikelihoodofX givenitscardinality.
However, p (|X|)|X|!U|X| is proportional to the Radon-
c
Nikodym derivative of the cardinality distribution relative to a
Figure6. Exampleimagesfrom2classes“T15 brick2”and“T14 brick1” Poissondistribution,andthusdoesnotindicatehowlikelythe
inTexturedataset.CirclesmarkdetectedSIFTkeypoints. cardinality of X is. A ranking function (cid:96) that can reconcile
Features are extracted from each image by applying the this problem is:
SIFT algorithm3, followed by Principal Component Analysis (cid:96)(X)∝p (|X|)[Cp ]X (17)
c f
(PCA) to convert the 128-D SIFT features into 2-D features.
Thus each image is compressed into a point pattern of 2-D where C is an unknown constant.
features. Fig. 7 plots the 2-D features of the images in the To determine a suitable C, consider first the special case
Texture images dataset. wherep isuniformonaboundedstatespaceX.Insuchcase,
f
allfinitesetsubsetsofX withthesamecardinalityareequally
likely,andaconsistentrankingfunctionshouldsatisfy(cid:96)(X)∝
p (m), given |X| = m. This condition can be generalized to
c
non-uniform p by replacing (cid:96)(X) with its expected value,
f
given |X|=m
E [(cid:96)(X)||X|=m]∝p (m) (18)
X c
Combining (17) with (18) and using the i.i.d. property of the
features in iid-cluster models yields:
(cid:20) p (cid:21)X
Figure 7. Extracted data from images of the Texture dataset. Left: 2- (cid:96)(X)∝p (|X|) f (19)
D features (after applying PCA to the SIFT features). Right: histogram of c ||pf||22
cardinalitiesoftheextracteddata.
where ||p ||2 = (cid:82) p2(x)dx is the squared L2-norm of p ,
ThemodelparametersarelearnedbyusingMLEonatrain- f 2 f f
which has units of U−1, and hence (cid:96)(X) is unitless.
ingdatasetcontaining30imagesperclass.FortheNBmodel,
we use 2-D Gaussian distributions to model the data; and for
B. Numerical experiments
the RFS model, we use Poisson RFSs with 2-D Gaussian
Inthissectionwecomparetheperformanceoftheproposed
feature distributions. After training, the learned models are
rankingfunction(19)withtheNBlikelihoodandPoissonRFS
evaluated on a test set containing all remaining images of the
likelihood in novelty detection. The threshold is set at the
classes (10 images per class). The performance is evaluated
2nd 10-quantileoftherankingfunctionvalues5 ofthetraining
with4-foldcrossvalidation,andtheaverageresultsareshown
dataset (consisting of only normal data). Observations ranked
inFig.5b.ObservethattheRFSmodeloutperformsNB,since
below this threshold are classified as anomalies.
it can exploit cardinality information of the data.
1) Novelty detection with simulated data: In this experi-
IV. RANKINGOFDATA ment, normal data are samples, having cardinalities between
40 and 60, drawn from a Poisson RFS with mean cardinality
The previous section shows that RFS models avoid the
48 and a 2-D Gaussian feature distribution
unit inconsistency and improve the classification performance
(cid:18) (cid:20) (cid:21) (cid:20) (cid:21)(cid:19)
0 0.06 0.01
by exploiting cardinality information in point pattern data. N ·; , .
0 0.01 0.04
However, using the RFS density as the data ranking function
for novelty detection does not result in a good performance4.
Thesamedatasetcontaining500normaldatapointsisusedto
Due to the non-uniformity of the reference measure, the
train the NB and Poisson RFS models with Gaussian feature
probabilitydensitiesforRFSspresentedintheprevioussection
distributions via MLE (subsection III-C).
donotprovideaconsistentrankingofthedata.Inthissection
5Theperformancedependsonthe(manuallyselected)threshold.Nonethe-
3UsingtheVLFeatlibrary[26]. less, the performance of the proposed ranking function would still be better
4We have done several experiments which are not shown here due to the thantheotherssinceitcanrankthedataconsistentlyasshownintheboxplots
paper’slengthlimitation. oflikelihoodvaluesfortheexperiment.
Three types of anomalies are considered: low-cardinality Bayes model for point patterns. For novelty detection, we
anomaly (cardinality ≤ 10), high-cardinality anomaly (cardi- proposednovelrankingfunctionsbasedonRFSmodels,which
nality ≥80), and feature anomaly which has the same cardi- substantiallyimproveperformance.Ourresultsalsocontribute
nality with normal data, but the Gaussian feature distribution totheBag-Spaceparadigminmultipleinstancelearningwhere
nowhasameanof[1,1]T.Noveltydetectionisperformedona statistical models are not available.
testsetcontaining200normalobservationsand300anomalies
REFERENCES
(100foreachtype).Thetestsarerun10timeswith10different
randomlysampledtestsets.TheaveragedresultsshowninFig. [1] T.Joachims,“Aprobabilisticanalysisoftherocchioalgorithmwithtfidf
fortextcategorization.”DTICDocument,Tech.Rep.,1996.
8aindicatedthattheproposedrankingfunctionyieldssuperior
[2] A.McCallumandK.Nigam,“Acomparisonofeventmodelsfornaive
performance. Bayes text classification,” in AAAI-98 workshop on learning for text
categorization,vol.752. Citeseer,1998,pp.41–48.
[3] G. Csurka, C. Dance, L. Fan, J. Willamowski, and C. Bray, “Visual
categorization with bags of keypoints,” in Workshop on statistical
learningincomputervision,ECCV,2004.
[4] S.Guha,R.Rastogi,andK.Shim,“Rock:Arobustclusteringalgorithm
forcategoricalattributes,”inDataEngineering,1999.Proceedings.,15th
InternationalConferenceon. IEEE,1999,pp.512–521.
[5] Y. Yang, X. Guan, and J. You, “Clope: a fast and effective clustering
(a) Simulateddata (b) Texturedata
algorithm for transactional data,” in Proceedings of the eighth ACM
Figure8. Noveltydetectionresultsby3rankingfunctions:NBlikelihood,
SIGKDD international conference on Knowledge discovery and data
PoissonRFSlikelihood,andproposedrankingfunction.
mining. ACM,2002,pp.682–687.
[6] I.V.Cadez,S.Gaffney,andP.Smyth,“Ageneralprobabilisticframe-
2) Novelty detection with real data: This experiment uses
workforclusteringindividualsandobjects,”inProceedingsofthesixth
the real dataset from the second experiment in subsection ACM SIGKDD international conference on Knowledge discovery and
III-D.Normaldataaretakenfromthe“T14 brick1”classand datamining. ACM,2000,pp.140–149.
[7] T. G. Dietterich, R. H. Lathrop, and T. Lozano-Pe´rez, “Solving the
anomalous test data are taken from the “T15 brick2” class. A
multiple instance problem with axis-parallel rectangles,” Artificial in-
4-fold cross-validation is used: for training we use 30 images telligence,vol.89,no.1,pp.31–71,1997.
from the normal class and for testing we use the remaining [8] F. Minhas and A. Ben-Hur, “Multiple instance learning of calmodulin
binding sites,” Bioinformatics (Oxford, England), vol. 28, no. 18, pp.
images from normal class (10 images) and 10 images from
i416–i422,2012.
abnormal class (different at each time). [9] J. Amores, “Multiple instance classification: Review, taxonomy and
As shown in Fig. 8b, ranking the data using the NB comparativestudy,”ArtificialIntelligence,vol.201,pp.81–105,2013.
[10] J. Foulds and E. Frank, “A review of multi-instance learning assump-
likelihood and Poisson RFS likelihood could not detect any
tions,” The Knowledge Engineering Review, vol. 25, no. 01, pp. 1–25,
anomalies, while the proposed ranking function achieved an 2010.
F1 score6 around 0.91. Moreover, Fig. 9 verified that only the [11] D. J. Daley and D. Vere-Jones, An introduction to the theory of point
processes. Springer,1988,vol.2.
proposedrankingfunctionprovidesaconsistentranking,while
[12] J.MollerandR.P.Waagepetersen,Statisticalinferenceandsimulation
the NB and Poisson RFS likelihoods even rank anomalies forspatialpointprocesses. Chapman&HallCRC,2003.
higher than normal data. [13] R.P.Mahler,Advancesinstatisticalmultisource-multitargetinformation
fusion. ArtechHouse,Inc.,2014.
[14] S. Russell and P. Norvig, Artificial intelligence: A modern approach.
PrenticeHall,2003.
[15] K. P. Murphy, Machine learning: a probabilistic perspective. MIT
press,2012.
[16] C. M. Bishop, Pattern recognition and machine learning. Springer,
2006.
[17] V.J.HodgeandJ.Austin,“Asurveyofoutlierdetectionmethodologies,”
ArtificialIntelligenceReview,vol.22,no.2,pp.85–126,2004.
[18] V.Chandola,A.Banerjee,andV.Kumar,“Anomalydetection:Asurvey,”
ACMComputingSurveys(CSUR),vol.41,no.3,p.15,2009.
[19] D. Stoyan, W. S. Kendall, and J. Mecke, Stochastic geometry and its
Figure 9. Boxplots of likelihoods computed by 3 models, namely NB applications. JohnWiley&Sons,1995.
likelihood, RFS density, and the proposed ranking function, for a fold of [20] R. P. Mahler, Statistical multisource-multitarget information fusion.
Texturedataset.Thesolidlinegoingthrougheachgraphisthethreshold(the ArtechHouse,Inc.,2007.
2nd 10-quantile). [21] B.-N.Vo,S.Singh,andA.Doucet,“Sequentialmontecarlomethodsfor
multitargetfilteringwithrandomfinitesets,”AerospaceandElectronic
Systems,IEEETransactionson,vol.41,no.4,pp.1224–1245,2005.
V. CONCLUSIONS [22] H. G. Hoang, B.-N. Vo, B.-T. Vo, and R. Mahler, “The Cauchy-
Schwarz divergence for Poisson point processes,” IEEE Trans. Inf.
Inthispaper,wehaveintroducedstatisticalmodelsforpoint Theory,vol.61,no.8,pp.4475–4485,2015.
pattern data using Random Finite Set theory. Such models [23] M.vanLieshout,Markovpointprocessesandtheirapplications. Im-
perialCollegePress,2000.
provide the means for developing model-based classification
[24] D.PhungandB.-N.Vo,“Arandomfinitesetmodelfordataclustering,”
and novelty detection for point pattern data. In particular inProc.17thAnnu.Conf.InformationFusion,Salamanca,Spain,2014.
we proposed a maximum likelihood method for learning the [25] S.Lazebnik,C.Schmid,andJ.Ponce,“Asparsetexturerepresentation
using local affine regions,” Pattern Analysis and Machine Intelligence,
parameters of an iid-cluster RFS–the analogue of the na¨ıve
IEEETransactionson,vol.27,no.8,pp.1265–1278,2005.
[26] A.VedaldiandB.Fulkerson,“Vlfeat:Anopenandportablelibraryof
6F1score(cid:44)2× precision×recall computervisionalgorithms,”http://www.vlfeat.org/,2008.
precision+recall