Clustering and classification of analytical data

21 Pages·2000·0.177 MB·English

by  Lavine B.K.

1
CLUSTERINGANDCLASSIFICATIONOFANALYTICALDATA
Clustering and Classification peakinachromatogram.Thus,eachsampleisconsidered
as a point in an n-dimensional measurement space. The
of Analytical Data
dimensionalityofthespacecorrespondstothenumberof
measurementsthatareavailable foreachsample.Abasic
assumption is that the distance between pairs of points in
BarryK.Lavine
this measurement space is inversely related to the degree
ClarksonUniversity,Potsdam,USA of similarity between the corresponding samples. Points
representing samples from one class will cluster in a lim-
ited region of the measurement space distant from the
points corresponding to the other class. Pattern recogni-
1 Introduction 1
tion (i.e. clustering and classification) is a set of methods
2 Principal Component Analysis 2 forinvestigatingdatarepresentedinthismanner,inorder
2.1 Variance-based Coordinate System 3 toassessitsoverallstructure,whichisdefinedastheoverall
2.2 Information Content of Principal relationshipofeachsampletoeveryotherinthedataset.
Components 3
2.3 Case Studies 4
3 Cluster Analysis 5 1 INTRODUCTION
3.1 Hierarchical Clustering 6
3.2 Practical Considerations 8
Sincetheearly1980s,amajorefforthasbeenmadetosub-
3.3 Case Studies 8
stantiallyimprovetheanalyticalmethodologyappliedto
4 Pattern Recognition 9 the study of environmental samples. Instrumental tech-
4.1 k-Nearest Neighbor 12 niques such as gas chromatography, high-performance
4.2 Soft Independent Modeling by Class liquid chromatography (HPLC) and X-ray fluorescence
Analogy 12 spectroscopyhavedramaticallyincreasedthenumber of
4.3 Feature Selection 13 organic and inorganic compounds that can be identified
4.4 Case Studies 13 and quantified, even at trace levels, in the environment.
5 Software 19 This capability, in turn, has allowed scientists to attack
ever more complex problems, such as oil and fuel spill
6 Conclusion 19
identification,buthasalsoledtoaninformation-handling
Abbreviations and Acronyms 19
problem..1/
Related Articles 19 The reason for this problem is that in any monitor-
References 20 ing effort it is necessary to analyze a large number of
samples in order to assess the wide variation in com-
position that an environmental system may possess.
The large number of samples that must be analyzed
Clustering and classification are the major subdivisions
and the number of constituents that must be mea-
ofpatternrecognitiontechniques.Usingthesetechniques,
sured per sample give rise to data sets of enormous
samplescanbeclassifiedaccordingtoaspecificpropertyby
size and complexity. Often, important relationships in
measurementsindirectlyrelatedtothepropertyofinterest
these data sets cannot be uncovered, when the data
(such as the type of fuel responsible for an underground
are examined one variable at a time, because of cor-
spill). An empirical relationship or classification rule
relationsbetweenmeasurementvariables,whichtendto
can be developed from a set of samples for which the
dominate the data and prevent information from being
propertyofinterestandthemeasurementsareknown.The
extracted.
classificationrulecanthenbeusedtopredicttheproperty
Furthermore, the relationships sought in the data
in samples that are not part of the original training set.
often cannot be expressed in quantitative terms, such
The set of samples for which the property of interest and
as the source of a pollutant in the environment. These
measurements is known is called the training set. The set
relationships are better expressed in terms of similarity
ofmeasurementsthatdescribeeachsampleinthedataset
or dissimilarity among groups of multivariate data.
is called a pattern. The determination of the property of
The task that confronts the scientist when investigating
interest by assigning a sample to its respective category is
these sorts of relationships in multivariate data, is
calledrecognition,hencethetermpatternrecognition.
twofold:
For pattern recognition analysis, each sample is rep-
resented as a data vector xD(x ,x ,x ,x,:::,x ), where (cid:15) Canausefulstructurebasedondistinctsamplegroups
1 2 3 j n
component x is a measurement, e.g. the area a of the jth bediscerned?
j
EncyclopediaofAnalyticalChemistry
EditedbyRobertA.Meyers.(cid:211) JohnWiley&SonsLtd,Chichester.ISBN0471976709
2
CHEMOMETRICS
(cid:15) Can a sample be classified into one of these groups 2 PRINCIPALCOMPONENTANALYSIS
forthepredictionofsomeproperty?
PCA is probably the oldest and best known of the
Thefirstquestion is addressedusingprincipalcompo-
techniques used for multivariate analysis. The overall
nentanalysis(PCA).2/ orclusteranalysis,.3/ whereasthe
goalofPCAistoreducethedimensionalityofadataset,
second question is addressed using pattern recognition
whilesimultaneouslyretainingtheinformationpresentin
methods..4/
the data. Dimensionality reduction or data compression
PCA is the most widely used multivariate analysis is possible with PCA because chemical data sets are
technique in science and engineering..5/ It is a method often redundant. That is, chemical data sets are not
for transforming the original measurement variables informationrich.ConsideragaschromatogramofaJP-4
into new variables called principal components. Each fuel (Figure1), which is a mixture of alkanes, alkenes,
principal component is a linear combination of the and aromatics. The gas chromatogram of a JP-4 fuel is
originalmeasurementvariables.Often,onlytwoorthree characterized by a large number of early-eluting peaks,
principal components are necessary to explain all of the which are large in size. There are a few late-eluting
information present in the data. By plotting the data peaks, but their size is small. Clearly, there is a strong
in a coordinate system defined by the two or three negative correlation between the early- and late-eluting
largest principal components, it is possible to identify peaksoftheJP-4fuel.Furthermore,manyofthealkane
keyrelationshipsinthedata,thatis,findsimilaritiesand andalkenepeaksarecorrelated,whichshouldnotcome
differences among objects (such as chromatograms or as a surprise as alkenes are not constituents of crude oil
but instead are formed from alkanes during the refining
spectra)inadataset.
Cluster analysis.6/ is the name given to a set of process.Inaddition,thepropertyofafuelmostlikelyto
bereflectedinahigh resolution gas chromatogram isits
techniques that seek to determine the structural char-
distillationcurve,whichdoesnotrequireall85peaksfor
acteristics of a data set by dividing the data into groups,
characterization.
clusters, or hierarchies. Samples within the same group
Redundancy in data is due to collinearity (i.e. corre-
are more similar to each other than samples in different
lations) among the measurement variables. Collinearity
groups. Cluster analysis is an exploratory data analysis
diminishestheinformationcontentofthedata.Consider
procedure. Hence, it is usually applied to data sets for
asetofsamplescharacterizedbytwomeasurements,X
1
whichthereisnoaprioriknowledgeconcerningtheclass
and X . Figure2 shows a plot of these data in a two-
2
membershipofthesamples.
dimensional measurement space, where the coordinate
Patternrecognition.7/ isanamegiventoasetoftech-
axes(orbasisvectors)ofthismeasurementspacearethe
niquesdevelopedtosolvetheclass-membershipproblem. variables X and X . There appears to be a relationship
1 2
Inatypicalpatternrecognitionstudy,samplesareclassi- between these two measurement variables, which sug-
fiedaccordingtoaspecificpropertyusingmeasurements gests that X and X are correlated, because fixing the
1 2
thatareindirectlyrelatedtothatproperty.Anempirical value of X limits the range of values possible for X .
1 2
relationshiporclassificationruleisdevelopedfromaset Ifthetwomeasurementvariableswereuncorrelated,the
of samples for which the property of interest and the enclosed rectangle in Figure2 would be fully populated
measurements are known. The classification rule is then bythedatapoints.Becauseinformationisdefinedasthe
usedtopredictthispropertyinsamplesthatarenotpart scatterofpointsinameasurementspace,itisevidentthat
oftheoriginaltrainingset.Thepropertyinquestionmay correlationsbetweenthemeasurementvariablesdecrease
be the type of fuel responsible for a spill, and the mea- the information content of this space. The data points,
surementsaretheareasofselectedgaschromatographic whicharerestrictedtoasmallregionofthemeasurement
(GC) peaks. Classification is synonymous with pattern
recognition,andscientistshaveturnedtoitandPCAand
clusteranalysistoanalyzethelargedatasetstypicallygen-
erated in monitoring studies that employ computerized
instrumentation.
This article explores the techniques of PCA, cluster
analysis, and classification. The procedures that must be
implemented toapplythesetechniques torealproblems
2 4 6 8 10 12
are also enumerated. Special emphasis is placed on
the application of these techniques to problems in Figure1 Ahigh-resolutioncapillarycolumngaschromatogram
environmentalanalysis. ofaJP-4fuel.
3
CLUSTERINGANDCLASSIFICATIONOFANALYTICALDATA
X
2 r
C D
o
Second principal
o
component
o o
First principal
o
o component
o o
o
A B o
o o Third principoal
o o component
o oo
X1 o
o
t
Figure2 Seventeen hypothetical samples projected onto a
c
two-dimensional measurement space defined by the measure-
mentvariablesX andX .Thevertices,A,B,C,andD,ofthe
1 2
rectangle represent the smallest and largest values of X and
1
X2.(AdaptedfromMandel..8/) Figure4 Principal component axesdevelopedfrom the mea-
surementvariablesr,c,andt.(Reproducedbypermissionfrom
Brown.5/courtesyofSocietyofAppliedSpectroscopy.)
spaceduetocorrelationsamongthevariables,couldeven
reside in a subspace if the measurement variables are
highly correlated. This is shown in Figure3. Here X3 is Theprincipalcomponentsofthedatadefinethevariance-
perfectly correlated with X1 and X2 because X1 plus X2 basedaxesofthisnewcoordinatesystem.Thelargestor
equals X3.Hence, thesevensamplepoints lieinaplane first principal component is formed by determining the
even though each data point has three measurements directionoflargestvariationintheoriginalmeasurement
associatedwithit. space and modeling it with a line fitted by linear least
squares (Figure4) that passes through the center of the
2.1 Variance-basedCoordinateSystem data. The second largest principal component lies in the
directionofnextlargestvariation–itpassesthroughthe
Variables that have a great deal of redundancy or are
centerofthedataandisorthogonaltothefirstprincipal
highlycorrelatedaresaidtobecollinear.Highcollinearity
component. The third largest principal component lies
betweenvariablesisastrongindicationthatanewsetof
in the direction of next largest variation–it also passes
basisvectorscanbefoundthatwillbebetteratconveying
through the center of the data, it is orthogonal to the
theinformationcontentpresentindatathanaxesdefined
first and second principal components, and so forth.
by the original measurement variables. The new basis
Each principal component describes a different source
set that is linked to variation in the data can be used to
ofinformationbecauseeachdefinesadifferentdirection
developanewcoordinatesystemfordisplayingthedata.
ofscatterorvarianceinthedata.(Thescatterofthedata
points in the measurement space is a direct measure of
X1 X2 X3 thedata’svariance.)Hence,theorthogonalityconstraint
0.5 0.5 1.0
imposed by the mathematics of PCA ensures that each
1.9 0.7 2.6
variance-basedaxiswillbeindependent.
2.0 2.0 4.0
X =
0.3 1.8 2.1
1.9 1.7 3.6
2.2 InformationContentofPrincipalComponents
1.2 0.2 1.4
1.9 0.9 2.8
Onemeasureoftheamountofinformationconveyedby
each principal component is the variance of the data
explained by the principal component. The variance
explained by each principal component is expressed
Figure3 Inthecaseofstronglycorrelatedvariables,thedata
in terms of its eigenvalue. For this reason, principal
points may reside in a subspace of the original measurement
components are usually arranged in order of decreasing
space. (Adapted from Multivariate Pattern Recognition in
Chemometrics..4/Copyright1992,withpermissionfromElsevier eigenvalues or waning information content. The most
Science.) informativeprincipalcomponentisthefirstandtheleast
4
CHEMOMETRICS
informativeisthelast.Themaximumnumberofprincipal new coordinate system is the center of the data, and
components that can be extracted from the data is the the coordinate axes of the new system are the principal
smaller of either the number of samples or number of components of the data. Employing this new coordinate
measurementsinthedataset,asthisnumberdefinesthe system,wecanuncoverrelationshipspresentinthedata,
largestnumberofindependentvariablesinthedata. that is, find distinct samples subgroups within the data.
If the data are collected with due care, one would Thissectionshows,bywayoftwopublishedstudies,how
expect that only the first few principal components principal components can be used to discern similarities
would convey information about the signal, as most anddifferencesamongsamplewithinadataset.
of the information in the data should be about the
effect or property of interest being studied. However,
2.3.1 TroodosDataSet
the situation is not always this straightforward. Each
principalcomponentdescribessomeamountofsignaland In the first study, 143 rock samples collected in the
some amount of noise in the data because of accidental Troodos region of Cyprus were analyzed by X-ray
correlationbetweensignalandnoise.Thelargerprincipal fluorescence spectroscopy for 10 metal oxides, which
components primarilydescribesignal variation, whereas containedinformationabouttheformationoftheserocks.
the smaller principal components essentially describe If the formation of the entire Troodos region occurred
noise. When smaller principal components are deleted, at the same time, one would expect all of the rocks to
noise is being discarded from the data, but so is a small be similar in composition. However, if there are distinct
amount of signal. However, the reduction in noisemore subgroupsinthedata,otherconclusionsmayhavetobe
than compensates for the biased representation of the drawn about the formation of the Troodos region. This
signal that results from discarding principal components study.9/ was initiated to settle a controversy about the
thatcontainasmallamountofsignalbutalargeamount geologicalhistoryofCyprus.
ofnoise.Plottingthedatainacoordinatesystemdefined Figure5 shows a plot of the two largest principal
by the two or three largest principal components often components of the 143 rock samples. (The original
providesmorethanenoughinformationabouttheoverall Troodos data set was modified for the purpose of this
structureofthedata.Thisapproachtodescribingadata principalcomponentmappingexercise.)Samples65and
set in terms of important and unimportant variation is 66 appear to be outliers in the plot as they are distant
knownassoftmodelinginlatentvariables. fromtheothersamples.Asageneralrule,outliersshould
PCA takes advantage of the fact that a large amount deletedbecauseoftheleast-squarespropertyofprincipal
of data is usually generated in monitoring studies components. In other words, a sample that is distant
when sophisticated chemical instrumentation, which is from the other points in the measurement space can
commonly under computer control, is used. The data pull the principal components towards it and away from
haveagreatdealofredundancyandthereforeagreatdeal the direction of maximum variance. Figure6 shows the
of collinearity. Because the measurement variables are results of a principal component mapping experiment
correlated,85peakgaschromatogramsdonotnecessarily
require85independentaxestodefinethepositionofthe
10 PC2 Scores
sample points. Utilizing PCA, the original measurement
variables that constitute a correlated axis system can 8
be converted into a system that removes correlation by
6
forcing the new axes to be independent and orthogonal.
This requirement greatly simplifies the data because 4
the correlations present in the data often allow us to
use fewer axes to describe the sample points. Hence, 2
the gas chromatograms of a set of JP-4 and Jet-A fuel
0
samples may reside in a subspace of the 85-dimensional
measurement space. A plot of the two or three largest - 2
principalcomponentsofthedatacanhelpustovisualize
- 4
therelativepositionoftheJet-AandJP-4fuelsamplesin PC1
- 5 0 5 10 15 20 25
thissubspace.
Figure5 Aprincipalcomponentmapofthe143rocksamples.
2.3 CaseStudies Samples65 and 66 are outliers. (The original data set was
modifiedforthisprincipalcomponentmappingexercise.)The
With PCA, we are able to plot the data in a new principal component map was generated using the program
coordinate system based on variance. The origin of the UNSCRAMBLER.
5
CLUSTERINGANDCLASSIFICATIONOFANALYTICALDATA
1150 PC2 Scores onent 12 643555673419515664620854494744556302 402333327435933389314724226202332820422251
mp 0 58 54 17
co –1 12 1911
5 d 105 141318
n 7
eco –2 3216 4 15
0 S –3 9
8 16
–4 –3 –2 –1 0 1 2 3 4
–5 First component
PC1
–5 0 5 10 15
Figure8 A plot of the two largest principal components for
the63Indianartifactsdevelopedfromtheconcentrationdataof
Figure6 A principal component map of the Troodos rock
10metals.The principalcomponent map wasgeneratedusing
samples with samples65 and 66 removed. Samples129 and
theprogramSCAN.
130 appear as outliers in the plot. (The original data set was
modifiedforthisprincipalcomponent mappingexercise.)The
principal component map was generated using the program 2.3.2 ObsidianDataSet
UNSCRAMBLER.
The second study.10/ also involves X-ray fluorescence
data. Sixty-three Indian artifacts (such as jewelry,
with samples 65 and 66 removed from the data. It is
weapons, and tools) made from volcanic glass, were
evident from the plot that samples 129 and 130 are also
collected from four quarries in the San Francisco Bay
outliers. Figure7 summarizes the results of a principal
area. (Samples1–10 are from quarry1, samples11–19
component mapping experiment with samples 65, 66,
arefromquarry2,samples20–42arefromquarry3,and
129, and 130 removed. Although samples 19 and 20 are
samples43–63 are from quarry4.) Because the compo-
probably outliers and are also candidates for removal,
sition of volcanic glass is characteristic of the site and
it is evident from the principal component plot that the
tends to be homogeneous, it is reasonable to assume
rock samples can be divided into two groups, which
that it should be possible to trace theseartifacts to their
would suggest that other conclusions should be drawn
original source material. In this study, the investigators
about the geological history of the Troodos region. The
attempted to do this by analyzing the 63 glass samples
clustering of the rocks samples into two distinct groups
for 10 elements: Fe, Ti, Ba, Ca, K, Mn, Rb, Sr, Y, and
was not apparent until the four outliers were removed
Zn.Next,aPCAwasperformedonthedata(63artifacts
fromthedata.
with 10 features per artifact). The goal was to identify
the overall trends present in the data. Figure8 shows a
plotofthetwolargestprincipalcomponentsofthedata.
4 PC2 Scores
Fromtheprincipalcomponentmap,itisevidentthatthe
63Indianartifactscanbedividedintofourgroups,which
2 correspond to the quarry sites from which the artifacts
were collected. Evidently, the artifacts in each quarry
0 were made from the same source material. This result
is significant because it provides the archaeologists with
- 2 important information about the migration patterns and
tradingroutesoftheIndiansinthisregion.Furtherdetails
abouttheobsidiandatacanbefoundelsewhere..10/
- 4
- 6
PC1 3 CLUSTERANALYSIS
- 4 - 2 0 2 4 6
Cluster analysis is a popular technique whose basic
Figure7 A principal component map of the Troodos rock
objective is to discover sample groupings within data.
samples with samples65, 66, 129, and 130 removed. The
principal component map was generated using the program The technique is encountered in many fields, such
UNSCRAMBLER. as biology, geology, and geochemistry, under such
6
CHEMOMETRICS
namesasunsupervisedpatternrecognitionandnumerical (Figure9). However, there is a problem with using the
taxonomy. Clustering methods are divided into three Euclideandistance,whichistheso-calledscalingeffect.It
categories, hierarchical, object-functional, and graph arisesfrom inadvertent weighting of the variables in the
theoretical. The focus here is on hierarchical methods, analysis that can occur due to differences in magnitude
astheyarethemostpopular. amongthemeasurementvariables.Forexample,consider
For cluster analysis, each sample is treated as a point adatasetwhereeachsampleisdescribedbytwovariables:
inann-dimensionalmeasurementspace.Thecoordinate the concentration of Na and the concentration of K as
axesofthisspacearedefinedbythemeasurementsused measured by atomic flame emission spectroscopy. The
tocharacterizethesamples.Clusteranalysisassessesthe concentration of Na varies from 50 to 500ppm, whereas
similarity between samples by measuring the distances the concentration of K in the same samples varies from
between the points in the measurement space. Samples 5 to 50ppm. A 10% change in the Na concentration
that are similar will lie close to one another, whereas will have a greater effect on Euclidean distance than a
dissimilar samples are distant from each other. The 10%changeinKconcentration.Theinfluenceofvariable
choiceofthedistancemetrictoexpresssimilaritybetween scaling on the Euclidean distance can be mitigated by
samplesinadatasetdependsonthetypeofmeasurement autoscaling the data, which involves standardizing the
variablesused. measurementvariables,sothateachvariablehasamean
Typically, three types of variables–categorical, ordi- ofzeroandastandarddeviationof1(Equation1):
nal, and continuous–are used to characterize chemical
x (cid:0)m
samples. Categorical variables denote the assignment of x D i;orig i;orig .1/
i;standardized
s
a sample to a specific category. Each category is repre- i;orig
sentedbyanumber,suchas1,2,3,etc.Ordinalvariables where x is the original measurement variable i,
i;orig
arecategoricalvariables,inwhichthecategoriesfollowa m is the mean of the original measurement variable
i;orig
logicalprogressionororder,suchas1,2,and3denoting i, and s is the standard deviation of the original
i;orig
low,middle,andhigh,respectively.However,continuous measurement variable i. Thus, a 10% change in K
variables are quantitative. The difference between two concentration has the same effect on the Euclidean
values for a continuous variable has a precise meaning. distance as a 10% change in Na concentration when the
If a continuous variable assumes the values1, 2, and 3, dataisautoscaled.Clearly,autoscalingensuresthateach
the difference between the values3 and 2 will have the measurementvariablehasanequalweightintheanalysis.
samemeaningasthedifferencebetweenthevalues2and Forclusteranalysis,itisbesttoautoscalethedata,because
1,becausetheyareequal. similarityisdirectlydeterminedbyamajorityvoteofthe
Measurement variables are usually continuous. For measurementvariables.
continuous variables, the Euclidean distance is the
best choice for the distance metric, because interpoint
3.1 HierarchicalClustering
distancesbetweenthesamplescanbecomputeddirectly
Clustering methods attempt to find clusters of patterns
(i.e. data points) in the measurement space, hence
x1 the term cluster analysis. Although several clustering
algorithmsexist,e.g.K-means,K-median,Patrick-Jarvis,
FCV (fuzzy clustering varieties), hierarchical clustering
is by far the most widely used clustering method. The
starting point for a hierarchical clustering experiment is
k 0
the similarity matrix which is formed by first computing
the distances between all pairs of points in the data set.
D Each distance is then converted into a similarity value
kl
(Equation2):
l
d
s D1(cid:0) ik .2/
ik
d
x max
2
Dkl = j (cid:229)n= 1(Xkj - Xij)2 wbehtewreeensiksa(mwphliechs ivaanridesk,frdoimk i0s tthoe1E)uicslidtheeansidmisitlaanrictey
betweensamplesiandk,andd isthedistancebetween
max
thetwomostdissimilarsamples(i.e.thelargestdistance)
Figure9 Euclidean distance between two data points in a
two-dimensional measurement space defined by the measure- in the data set. The similarity values are organized in
mentvariablesx andx .(Reprintedfromref.3.) the form of a table or matrix. The similarity matrix is
1 2
7
CLUSTERINGANDCLASSIFICATIONOFANALYTICALDATA
thenscannedforthelargestvalue,whichcorrespondsto linkageassessesthesimilaritybycomputingthedistances
themostsimilarpointpair.Thetwosamplesconstituting between all point pairs where a member of each pair
the point pair are combined to form a new point, which belongs to the cluster. The average of these distances is
is located midway between the two original points. The used to compute the similarity between the data point
rows and columns corresponding to the old data points andthecluster.
arethenremovedfromthematrix.Thesimilaritymatrix To illustrate hierarchical clustering, consider the data
for the data set is then recomputed. In other words, shown in Table1. (The example shown here is an
the matrix is updated to include information about the adaptation of the exercise described in Chapter6 of
similaritybetweenthenewpointandeveryotherpointin reference4.)FiveHPLCcolumns werecharacterizedby
thedataset.Thenewnearestpointpairisidentified,and thecapacityfactorvaluesobtainedfromthreesubstances,
combinedtoformasinglepoint.Thisprocessisrepeated which served as retention probes. To perform single-
untilallpointshavebeenlinked. linkage hierarchical clustering on this chromatographic
There are a variety of ways to compute the distances data,itisnecessarytofirstcomputethesimilaritymatrix
betweendatapointsandclustersinhierarchicalclustering forthedata,givenasTable2.
(Figure10).Thesingle-linkagemethodassessessimilarity The similarity matrix (Table2) is then scanned for
between a point and a cluster of points by measuring the largest value, which corresponds to the two HPLC
the distance to the closest point in the cluster. The columns that are most similar. An examination of
completelinkagemethodassessessimilaritybymeasuring the similarity matrix suggests that chromatographic
thedistancetothefarthestpointinthecluster.Average columnsA and B with a score of 0.79 are the most
similar.Hence,chromatographiccolumnsAandBshould
becombinedtoformanewpoint.Therowsandcolumns
2
corresponding to the two original points (A and B) are
3 1 5
removedfromthesimilaritymatrix.Thesimilaritymatrix
Single linkage
for the data set is then updated to include information
Distance = d
4 15 about the similarity between the new point and every
otherpoint(C,D,andE)inthedataset.Inthisstudy,the
2 investigatorschosetousethesingle-linkagecriterionfor
3 1 5 assessingthesimilaritybetweenadatapointandapoint
Complete linkage cluster, see Table3. (Using the single-linkage method,
thesimilaritybetweenpointDandtheclusterconsisting
4
of columnsA and B is the larger of the values of 0.69
Distance = d
35 and0.17,seeTable2.Forcompletelinkage,thesimilarity
2
betweenthisclusterandpointDisthesmallerofthetwo
3 1 5 Average linkage values.)
The updated similarity matrix is then scanned for the
largest value; the new nearest pair is combined to form
4 d + d + d + d a single point, which is points D and E. The rows and
Distance = 15 25 35 45
4 columns corresponding to points D and E are deleted
Figure10 The distance between a data cluster and a point
using single linkage, complete linkage, and average linkage. Table2 Similaritymatrix
(ReproducedbypermissionfromLavine.20/courtesyofMarcel
Columns A B C D E
Dekker,Inc.)
A 1.00 0.79 0.58 0.69 0.61
B 0.79 1.00 0.36 0.17 0.34
Table1 HPLCdataset C 0.58 0.36 1.00 0.51 0.72
D 0.69 0.17 0.51 1.00 0.75
Column k k k E 0.61 0.34 0.72 0.75 1.00
1 2 3
A 0.31 17.8 3
B 0.10 9.30 3 Table3 Updatedsimilaritymatrix
C 0.11 21.5 1
Columns A,B C D E
D 0.58 22.0 2
E 0.50 16.0 1 A,B 1.00 0.58 0.69 0.61
C 0.58 1.00 0.51 0.72
(Reprinted from Multivariate Pattern Re-
D 0.69 0.51 1.00 0.75
cognition in Chemometrics..4/ Copyright
E 0.61 0.72 0.75 1.00
1992,bypermissionofElsevierScience.)
8
CHEMOMETRICS
Table4 Updatedsimilaritymatrix of a proposed partitioning of the data. Hence, clusters
are defined intuitively, depending on the context of the
Columns A,B C D,E
problem, not mathematically, which limits the utility
A,B 1.00 0.58 0.61
of this technique. Clearly, prior knowledge about the
C 0.58 1.00 0.72
problem is essential when using these methods. The
D,E 0.61 0.72 1.00
criterionfordeterminingthethresholdvalueforsimilarity
is often subjective and depends to a large degree on
Table5 Updatedsimilarity
thenatureoftheprobleminvestigated–forexample,the
matrix
goalsofthestudy,thenumberofclusterssought,previous
Columns A,B C,D,E experience,andcommonsense.
A,B 1.00 0.69 Allclusteringproceduresyieldthesameresultsfordata
C,D,E 0.69 1.00 setswithwell-separatedclusters.However,theresultswill
differwhentheclustersoverlap.Thatiswhyitisagood
idea to use at least two different clustering algorithms,
from the similarity matrix. The similarity matrix for the
suchassingleandcompletelinkage,whenstudyingadata
datasetisthenupdated(Table4)toincludeinformation
set. If the dendograms are in agreement, then a strong
about thesimilarity betweenthe new point (D+ E)and
case can be made for partitioning the data into distinct
everyotherpointinthedataset.Thisprocessisrepeated
groups as suggested by the dendograms. If the cluster
(Table5)untilallpointsaremergedintoasinglecluster.
memberships differ, the data should be further investi-
The results of a hierarchical clustering study are
gated using average linkage or PCA. The results from
usually displayed as a dendogram, which is a tree-
average linkage or PCA can be used to gauge whether
shaped map of the intersample distances in the data
thesingleorfarthestlinkagesolutionisthebetterone.
set. The dendogram shows the merging of samples
All hierarchical clustering techniques suffer from so-
into clusters at various stages of the analysis and the
calledspacedistortingeffects.Forexample,single-linkage
similaritiesatwhichtheclustersmerge,withtheclustering
favorstheformationoflargelinearclustersinsteadofthe
displayed hierarchically. The dendogram for the single-
usual elliptical or spherical clusters. As a result, poorly
linkageanalysisoftheHPLCdataisshowninFigure11.
separated clusters are often chained together. However,
Interpretationoftheresultsisintuitive,whichisthemajor
completelinkagefavorstheformationofsmallspherical
reasonforthepopularityofthesemethods.
clusters.Becauseofthesespace-distortingeffects,hierar-
chicalclusteringmethodsshouldbeusedintandemwith
3.2 PracticalConsiderations
PCAtodetectclustersinmultivariatedatasets.
A major problem in hierarchical clustering (or cluster All hierarchical methods will always partition data,
analysis for that matter) is defining a cluster. Contrary even randomly generated data, into distinct groups or
to many published reports, there is no cluster validity clusters.Hence,itisimportanttoascertainthesignificance
measure that can serve as an indicator of the quality level of the similarity value selected by the user. For
this task, a simple three-step procedure is proposed.
First, a random data set is generated with the same
0.0
correlation structure, the same number of samples, and
the same number of measurements as the real data set
0.2
that is currently being investigated. Second, the same
clustering technique(s) is applied to the random data.
0.4
Third, the similarity value, which generates the same
number of clusters as identified in the real data set, is
0.6 determined from the dendogram of the random data. If
the similarity value is substantially larger for the real
0.8 dataset,thelikelihoodofhavinginadvertentlyexploited
random variation in the data to achieve clustering is
1.0 probablyinsignificant.
1 2 3 4 5
Figure11 Single-linkage dendogram of the HPLC data set. 3.3 CaseStudies
There are two groups of HPLC columns:1, 2 and 3,
Hierarchical clustering methods attempt to uncover the
4, 5. (Reprinted from Multivariate Pattern Recognition in
Chemometrics..4/Copyright1992,withpermissionfromElsevier intrinsic structure of a multivariate data set without
Science.) makingaprioriassumptionsaboutthedata.Thissection,
9
CLUSTERINGANDCLASSIFICATIONOFANALYTICALDATA
0.00
0.33
y
t
ri
a
mil
Si
0.67
1.00
142936507819732485625874097314012562903861451708992864351372603
1 11111111122232322233424323323334445454455545556456566
Observations
Figure12 Asingle-linkagedendogramoftheobsidiandataset.ThedendogramwasgeneratedusingtheprogramSCAN.
shows, by way of two published studies, how clustering thecompoundsinthedataset.Shouldwethenacceptthe
methodscanbeusedtofindclustersofpointsindata. salesman’sclaimthatanelectronicnosecansniffodor?
Inordertovalidatethesalesman’sclaim,theodordata
was analyzed using single and complete linkage hierar-
3.3.1 ObsidianDataSet
chicalclustering. Figures13and 14show dendograms of
The first study is the obsidian data set, discussed in the odor data. It is evident from the dendograms that
section2.3.2. A principal component map of the data dividing the 41 compounds into three categories based
(Figure8) revealed four distinct clusters, which cor-
onodortypecannotbejustified.(Althoughthedatacan
respond to the sites from which these artifacts were be divided into three clusters by complete linkage at a
obtained. To confirm the four-cluster hypothesis, the
0.56 similarity, the cluster memberships cannot be cor-
investigatorsalsoanalyzedtheirdatausingsingle-linkage
related to compound odor–why? Cluster1 consists of
analysis. The resulting dendogram (Figure12) indicated
samples41, 21, 19, 16, 13, 39, 31, 36, 29, 8, 34, and32.
thatitisreasonabletodividetheglasssamplesintofour
Cluster2consistsofsamples38,5,26,14,24,15,37,6,33,
categories based on the quarry sites from which these
35, 20, 30, and7. Cluster3 consists of samples40, 17, 4,
artifactswereobtained.(Atsimilarity0.40,samples1–10
22,2,25,10,28,18,27,23,12,11,1,9,and3.)Evidently,
formacluster,samples11–19formanothercluster,sam-
the six electronic measurements from the nose do not
ples20–42formathirdcluster,andsamples43–63form
containsufficientdiscriminatoryinformationtoforcethe
the fourth cluster.) Because the dendogram and princi-
compoundstoclusteronthebasisofodortype.Therefore,
palcomponentmapofthedataareinstrongagreement,
the salesman’s claim about the efficacy of the proposed
theinvestigatorsdecidedtopartitiontheirdataintofour
artificialnoseshouldnotbeacceptedatfacevalue.
groupsbasedonthequarrylabelsofthesamples.
3.3.2 ArtificialNoseDataSet
4 PATTERNRECOGNITION
Thesecondstudyinvolvesdatafromanartificialnose..11/
Asalesmanclaimsthatanelectronicnosecansuccessfully So far, only exploratory data analysis techniques, i.e.
sniffodor.Thesalesmanobtaineddatafromtheliterature cluster analysis and PCA, have been discussed. These
to support his claim. Of the 41 compounds in the techniquesattempttoanalyzedatawithoutdirectlyusing
data set, compounds 1–21 are etheral, compounds information about the class assignment of the samples.
22–32 are pungent, and compounds 33–41 are minty. AlthoughclusteranalysisandPCAarepowerfulmethods
Each compound was characterized by six electronic for uncovering relationships in large multivariate data
measurements.Usingaback-propagationneuralnetwork sets,theyarenotsufficientfordevelopingaclassification
algorithm,thesalesmanwasabletocorrectlyclassifyallof rule that can accurately predict the class-membership of
10
CHEMOMETRICS
0.0
0.2
0.4
0.6
0.8
1.0
11969169 42507 30 8 6454027 5088371 32
42113332833323633735211242142212122119311
ppppppppppppppppppppppppppppppppppppppppp
mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
sssssssssssssssssssssssssssssssssssssssss
Figure13 Asingle-linkagedendogramofthenosedataset.ThedendogramwasgeneratedusingtheprogramPirouette.
an unknown sample. In this section, pattern recognition fingerprinting,.13–15/ spectroscopic imaging,.16–18/ and
techniques will be discussed. These techniques were datainterpretation..19–21/
originally developed to categorize a sample on the basis Patternrecognitiontechniquesfallintooneoftwocate-
ofregularities inobserveddata. Thefirstapplications of gories:nonparametricdiscriminants,andsimilarity-based
pattern recognition to chemistry were studies involving classifiers. Nonparametric discriminants,.22–24/ such as
low-resolutionmassspectrometry..12/ Sincethen,pattern neural networks, attempt to divide a data space into
recognition techniques have been applied to a wide different regions. In the simplest case, that of a binary
variety of chemical problems, such as chromatographic classifier, the data space is divided into two regions.