Robust Correlation Analyses: False Positive and Power Validation Using a New Open Source Matlab Toolbox.

4.3 MB·English

by  Pernet, Cyril R.

METHODSARTICLE
published:10January2013
doi:10.3389/fpsyg.2012.00606
Robust correlation analyses: false positive and power
validation using a new open source Matlab toolbox
CyrilR.Pernet1*,RandWilcox2 andGuillaumeA.Rousselet3
1BrainResearchImagingCenter,DivisionofClinicalNeurosciences,UniversityofEdinburgh,Edinburgh,UK
2DepartmentofPsychology,UniversityofSouthernCalifornia,LosAngeles,CA,USA
3CentreforCognitiveNeuroimaging,InstituteofNeuroscienceandPsychology,CollegeofMedical,VeterinaryandLifeSciences,UniversityofGlasgow,Glasgow,
UK
Editedby: Pearson’s correlation measures the strength of the association between two variables.
HolmesFinch,BallStateUniversity, The technique is, however, restricted to linear associations and is overly sensitive to out-
USA
liers.Indeed,asingleoutliercanresultinahighlyinaccuratesummaryofthedata.Yet,it
Reviewedby:
remains the most commonly used measure of association in psychology research. Here
JillS.Budden,NationalCouncilof
StateBoardsofNursing,USA wedescribeafreeMatlab(R)basedtoolbox(http://sourceforge.net/projects/robustcorrtool/)
MichaelSmithson,AustralianNational that computes robust measures of association between two or more random variables:
University,Australia thepercentage-bendcorrelationandskipped-correlations.Afterillustratinghowtousethe
*Correspondence: toolbox, we show that robust methods, where outliers are down weighted or removed
CyrilR.Pernet,BrainResearch
andaccountedforinsignificancetesting,providebetterestimatesofthetrueassociation
ImagingCenter,WesternGeneral
Hospital,CreweRoad,Edinburgh, with accurate false positive control and without loss of power.The different correlation
EH42XU,UK. methods were tested with normal data and normal data contaminated with marginal or
e-mail:[email protected] bivariate outliers.We report estimates of effect size, false positive rate and power, and
adviseonwhichtechniquetousedependingonthedataathand.
Keywords:robuststatistics,correlation,power,outliers,MATLAB
INTRODUCTION affectedbythemagnitudeof theslopearoundwhichpointsare
Robuststatisticalprocedureshavebeendevelopedsincethe1960s clustered,bycurvature,bythemagnitudeoftheresiduals,bythe
(Tukey,1960;Huber,1964) to solve problems inherent in using restrictionofrange,andbyheteroscedasticity.Ourtoolboxcom-
classicparametricmethodswhenassumptionsareviolated(Erceg- putesrobustalternatives:thepercentage-bendcorrelation(Wilcox,
HurnandMirosevich,2008).Althoughmanyscientistsareaware 1994)andskipped-correlations(Wilcox,2004).Thesealternatives
ofthesetechniques,andoftheirsuperiorityinmanycases,robust have a practical value relative to the standard Pearson’s correla-
statistics are not widely used or even part of the standard cur- tionbecausetheyestimatelinearrelationshipsandoftenprovide
riculum. There are two reasons for this. First,no single method betterestimatesofthetruerelationshipbetweenvariables(Rous-
isoptimalinallsituations.Althoughleastsquaresisatechnique seletandPernet,2012).Thepercentage-bendcorrelationprotects
easytocomputeinmanysituations,itisoftendisastrousandinap- againstmarginaloutlierswithouttakingintoaccounttheoverall
propriate(Wilcox,2001)becauseassumptionsareoftennotmet structureofthedata.Importantly,itestimateslinearassociations,
(e.g.,Micceri,1989);leavingustohavetochooseamongmultiple but does not estimate Pearson’s: the results are not comparable
robustalternatives.Second,developersofstatisticalmethodstend across the [−1,+1] range. Skipped-correlations protect against
to provide code that is not sufficiently user-friendly. As a con- bivariate outliers by taking into account the overall structure of
sequence,robust techniques remain underused and do not find thedata,andPearson’sskippedcorrelationisadirectreflectionof
theirwayintocommercialsoftwarepackages(Stromberg,2004). Pearson’sr.
Here,wepresentafreeMatlabtoolboxtoperformrobustcorre-
lation analyses (http://sourceforge.net/projects/robustcorrtool/). TOOLBOXFEATURES
Thetoolboxcontainsseveralcorrelationtechniquesdescribedin Alongsidethecomputationsofcorrelations,thetoolboxincludes
Wilcox (2012a). These techniques can also be found in separate tools for visualization and basic assumption checking. The
Rfunctions(RDevelopmentCoreTeam,2011).Inaddition,the corr_normplot.mfunctionprovides,inonefigure,ascatterplotof
toolboxprovidesgraphicaloutputsandtestsofassumptions. thedata,themarginal(normalized)histogramswiththematch-
Generally,acorrelationreferstoanyofabroadclassofstatis- ing Gaussian curves,and the bivariate histogram (Figure 1,left
tical relationships involving dependence. Correlation also refers column).Thejoint_density.m functionplotsbothameshof the
to a broad class of statistical measures aimed at characterizing joint density and its isocontour. Although the joint density is
the strength of the association between two variables. Among similar to the bivariate histogram,it provides a better visualiza-
these latter measures, Pearson’s correlation is the most widely tion of the bivariate space when there are many observations.
used technique,despite its lack of robustness (Wilcox,2012a,b). Visualization is indeed the first step before computing any cor-
Indeed,Pearson’scorrelationisoverlysensitivetooutliers;itisalso relation: in some extreme situations,as in the case of split data
www.frontiersin.org January2013|Volume3|Article606|1
Pernetetal. Robustcorrelationtoolbox
FIGURE1|VisualizationoftheAnscombe’squartet.Eachpairis (green)orinX andY (black);column3showsbivariateoutliers(red),
illustratedbyascatterplotandwithunivariateandbivariatehistograms withthebestlinefittedtotheremainingpoints.Histograms(right
(leftcolumn).Outliersdetectedusingthebox-plotruleareplottedin column)showthebootstrappedvariancedifferences.Verticalredlines
thetwomiddlecolumns:column2showsunivariateoutliersinY indicate95%CIs.
clouds,acorrelationanalysiswouldbeworthless(seee.g.,Figure correlationisonlymeaningfulforlinearparametricmodelsesti-
1E in Rousselet and Pernet,2012). Tests of correlations are sen- matedvialeastsquares,whilstSpearman’scorrelationdealswith
sitive to different features of the data. For instance, Pearson’s monotonicassociationsinamoreflexiblemanner.Bothtestsare
FrontiersinPsychology|QuantitativePsychologyandMeasurement January2013|Volume3|Article606|2
Pernetetal. Robustcorrelationtoolbox
howeversensitivetoheteroscedasticity(WilcoxandMuska,2001). correlationonlyprotectsagainstoutliersassociatedwiththemar-
The toolbox thus provides tools to compute conditional means ginal distributions. Under normality, the percentage-bend and
and variances (conditional.m) and to test variance homogeneity Pearson’s correlations have very similar values, but these values
based on a percentile bootstrap with adjustment for small sam- candiffermarkedlyassoonasthereisdeviationfromnormality
ples (variance_homogeneity.m). The function outputs the 95% (Wilcox,1994).
confidence intervals (abbreviated CIs in the rest of the paper) The toolbox also computes percentile bootstrap 95% CIs for
andthehistogramofthebootstrappedestimates(Figure1,right eachcorrelation.ForPearson’s,Spearman’s,andpercentage-bend
column). In addition, because skewness can cause large devi- correlations,pairsofobservationsareresampledwithreplacement
ation in correlation estimates, we included the Henze–Zirkler and their correlation values obtained. For skipped-correlations,
testformultivariatenormality(HZmvntest.m).Thisfunctionwas the data after outlier removal are resampled, before computing
implemented by Trujillo-Ortiz et al. (2007) and is distributed correlationvalues1.Correlationvaluesarethensortedandthe2.5
underDSBlicensewiththetoolbox.Finally,univariateandbivari- and97.5percentilesobtainedtoyielda95%CI.CIsprovidean
ateoutlierdetectioncanbeperformedusingseveraltechniques: alternativewaytotestthenullhypothesis.IftheCIencompasses0,
box-plot rule,MAD-median rule,S-outliers (detect_outliers.m – thenthenullhypothesisofindependencecannotberejected.This
Figure1,middlecolumns;Appendix).Thetoolboxalsocomputes isofparticularinterestwhenacorrelationisdeclaredsignificant
Pearson’s (Pearson.m), Spearman’s (Spearman.m), percentage- (e.g., p-value<0.05), because the t-test assumes independence
bend (bendcorr.m – Wilcox, 1994), and skipped-correlations betweenvariables,whichimplieshomoscedasticity.Ifthereishet-
(skipped_correlation.m–Wilcox,2004)withtheir95%percentile eroscedasticity,thet-testusesanincorrectestimateofthestandard
bootstrapCIs. error. The significance of a correlation can therefore be largely
affectedbyheteroscedasticityeventhoughvariablesarenottruly
METHODS correlated.Thetoolboxthusalsoprovidesarejectionofthenull
CORRELATIONMEASURES hypothesis based the percentile bootstrap CI, because it is less
Weillustratetheuseof thetoolboxwiththeAnscombe’s(1973) sensitivetoheteroscedasticitythanthetraditionalt-test.
quartet (Figures 1 and 2). For each pair of variables, standard
Pearson’sandSpearman’scorrelationswerecomputedwiththeir MONTE-CARLOSIMULATIONS:FALSEPOSITIVES,EFFECTSIZES,AND
skipped-correlationcounterparts,aswellasthe20%percentage- POWER
bendcorrelation. Toassessthesensitivityof thedifferentcorrelationmethods,we
Tocomputeskipped-correlations,firstweestimatetherobust ranseveralsimulationsinwhichwerecordedtheactualcorrela-
center of the data cloud. Because a single outlier can result tionvalue(effectsize)andthenumberoftimesthenullhypothesis
in the bivariate mean giving a poor reflection of the typ- of independencewasrejected(falsepositiverateandpower).In
ical response, one relies here on the minimum covariance the first simulation,a parent bivariate normal (N∼0,1) distri-
determinant (MCD) estimator, which is a robust estima- bution of 10 million data points was generated (Figure 3, left
tor of multivariate location and scatter (Rousseeuw, 1984; column).ForoneMonte-Carlorun,500pairsofobservationswere
Rousseeuw and Van Drissen, 1999; Hubert et al., 2008). The randomlyselectedfromtheparentdistribution.Usingthese500
skipped_correlation.m function computes the MCD by calling pairs,Pearson’s,Spearman’s,20%bendandskipped-correlations
the LIBRA toolbox (Verboten and Hubert,2005 – free access at werecomputedforsamplesizesn=10,20,30,40,50,60,80,100,
http://wis.kuleuven.be/stat/robust/LIBRA/LIBRA-home), which 150, 200, 250, 300, 400, and 500. The procedure was replicated
is distributed with the correlation toolbox under an academic 10,000times(i.e.,10,000independentsamplesof500pairswere
public license. Second,outliers are identified using a projection takenfromtheparentpopulation).Thewholeprocesswasthen
technique:datapointsareorthogonallyprojectedontolinesjoin- repeated for parent populations in which the correlation values
ingeachdatapointtotherobustestimateoflocationandoutliers rangedfrom0to1withstepsof 0.1.TogenerateGaussiandata
amongprojecteddatapointsaredetectedusingthebox-plotrule, with outliers,we generated one million data points from a par-
whichreliesontheinterquartilerange(Friggeetal.,1989;Carling, entbivariatenormaldistributionwithacorrelationvaluethatwas
2000).Finally,Pearson’sandSpearman’scorrelationsandassoci- the negative of that in the first population. The center of this
atedt-valuesarecomputedontheremainingdata.Theempirical secondpopulationwaspositionedsuchthatobservationswould
t-values are compared to a critical t-value determined via sim- beeithermarginaloutliersforonevariable(bivariatemean=[6,
ulations(Wilcox,2012a,b).Theusualcriticalvalueistechnically 0], Figure 3, middle column) or both (bivariate mean=[6, 6],
unsoundandshouldnotbeusedbecauseitdoesnottakeoutlier Figure3,rightcolumn–inthiscasethusalsobivariateoutliers).
removalintoconsideration;thecriticalvaluesimplementedinthe For each sample size, 10% of data were substituted by outliers
toolboxensuregoodcontrolofthetypeIerrorrate. takenatrandomfromtheoutlierpopulation:1outlieroutof10,
Tocomputethepercentage-bendcorrelation,aspecifiedper- 2outliersoutof20,andsoon.
centageof marginalobservationsdeviatingfromthemedianare To investigate effect sizes, we first tested if the correlations
down weighted. Pearson’s correlation is then computed on the differed from the true population value. Differences between
transformed data. A skipped correlation is a robust generaliza-
tionof Pearson’srbymeasuringthestrengthof thelinearasso-
1Therearepossibleissuesaboutthedependenceamongorderstatisticsbyremoving
ciation, ignoring outliers detected by taking into account the
outliersandbootstrappingratherthanbootstrappingandre-runthewholeskipped
overall structure of the data. In contrast, the percentage-bend correlation.However,atthemomentitisunclearwhichmethodisbest.
www.frontiersin.org January2013|Volume3|Article606|3
Pernetetal. Robustcorrelationtoolbox
observed correlation values (r) and the true one (ρ) were com- outliers.Finally,testsofvariancehomogeneityrevealedthatvari-
puted, and Bonferroni adjusted percentile CIs were obtained ancesdifferedsignificantlyinpairs1and2,butnotinpairs3and4
(95% CI adjusted for the 14 sample sizes=99.9964% CI). If (Figure1,rightcolumn).Heteroscedasticity,ratherthantrueasso-
0 was not included in the 99.9964% CI, the method did not ciation,couldthushavecausedsignificantcorrelationsforpairs1
estimatethetruecorrelationvalue.Second,wecompared(i)Pear- or2(Wilcox,1991;WilcoxandMuska,2001).Incomparison,Lev-
son’s correlation against Spearman’s, a 20% bend, and skipped ene’stestsfailedtorejectthenullhypothesisofhomoscedasticity
Pearson’s correlations, and (ii) Spearman’s correlation against forallpairs[pair1F(1,20)=3.5,p=0.07;pair2F(1,20)=3.39,
skipped Spearman’s correlation. A percentile bootstrap on the p=0.08;pair 3 F(1,20)=4.15,p=0.055;pair 4 F(1,20)=0.17,
mediandifferenceswascomputedandadjustedforthe14sample p=0.68].ThisisexplainedbyLevene’stestlackofpower:thetest
sizes(αsetto0.05/14=0.36%):theresultsfromtwocorrelations isbasedonthedistancebetweeneachpointfromthemean,which
differed significantly if the CI of median differences did not bydefinitionisaffectedbyoutliers.
containzero. As designed by Anscombe, Pearson’s correlation is fooled by
Toevaluatethefalsepositiverateandpower,theaveragenum- outliers and, for each pair, a significant correlation of r=0.81
beroftimesthenullhypothesiswasrejectedwascomputed.The is observed (Table 1; Figure 2). Importantly, bootstrap CIs are
differentcorrelationtechniqueswerethencomparedforeachsam- alsosensitivetooutliersandsuggestsignificantcorrelationstoo.
plesizebasedontheirbinomialdistributions(accept/rejectH0) Spearman’s correlations performed slightly better, showing no
using a method for discrete cases with adjustment for multiple associationinpair4.Inaddition,thebootstrapCIinpair2shows
comparisons(Kulinskayaetal.,2010). noevidenceforasignificantcorrelation,suggestingthattheobser-
vations are not linearly related but show dependence. The 20%
RESULTS percentage-bendcorrelationgivesbetterresultsthanPearson’sor
ILLUSTRATIONWITHTHEANSCOMBE’SQUARTET Spearman’scorrelations.Fornormaldata(pair1),itperformssim-
AsputforwardbyAnscombe(1973),plottingthedataisanimpor- ilarlytoPearson’scorrelation.Foranon-linearrelationship(pair
tantpartofanystatisticalanalysis(Figure1,leftcolumn).Forthe 2),like Spearman,the 20% percentage-bend correlation returns
readernotawareofthisdataset,itisimportanttoknowthatfor significantresultsbutthebootstrapCIdoesnot.Withaunivariate
eachof thefourpairsof variablesX andY,themeanof X is9, outlier(pair3),itreturnstheexactcorrelation.Finally,itshows
thevarianceofX is11,themeanofY is7.5,thevarianceofY is no significant results for pair 4. Here the bootstrapped CI sug-
4.12,and Pearson’s correlation between X andY is always 0.81. gests a significant result, which is explained by the use of valid
Despitetheseidenticalfirstorderstatisticalpropertiesandiden- resamples(i.e.,resamplescannotbecomposedofauniquevalue)
ticalcorrelationvalues,thenatureoftherelationshipsbetweenX tocomputeCIsinouralgorithm,thatis,foreachbootstrap,the
andY differswidely.Forpair1,inspectionofthescatterplotand singleoutlierinY wasalwayspresent.Inspectionofthedataplot
distributionssuggestsclosetonormallydistributeddatawithno neverthelessrevealsthatthebootstrapdidnotperformwell.This
obvious outlier. Pair 2 shows a non-linear and non-monotonic againillustratesthatthebootstrap,onit’sown,doesnotprotect
relationshipanddataarenotnormallydistributed.Pair3shows againstoutliers,althoughitcanattenuatetheireffect.Theskipped
astrictlinearrelationshipand1marginaloutlier.Finally,pair4 correlationreturnedthesameresultsasPearson’sandSpearman’s
showsnorelationshipand1bivariateoutlier. correlationsbecausethebox-plotruledidnotdetectthebivariate
The Henze–Zirkler test for multivariate normality confirmed outliersinpairs1,2,and3.Theskippedcorrelationfailedtopro-
visual inspection:only pair 1 is normally distributed (HZ=0.1, videanyoutputforpair4,because,oncetheoutlierisremoved,the
p=0.99),whilst the other pairs deviate from the bivariate nor- remainingpointsarealignedwiththesameX valueanditisthus
mal distribution (pair 2 HZ=0.6, p=0.036; pair 3 HZ=1.04, impossibletocomputeanycorrelation.Wewouldindeedexpect
p=0.002; pair 4 HZ=1.06, p=0.002). The outlier detection robustanalysesnottofindanyassociationforsuchdata.
function implemented in the toolbox relies on three methods:
the box-plot rule, as used in the skipped correlation function, MONTE-CARLOSIMULATIONS
the median absolute deviation (MAD)-median rule (Hall and Figure3illustratesthepopulationsusedinthesimulations.The
Welsh, 1985), and the S-estimator deviation (Rousseeuw and populationinthetopleftsubplothadaPearson’scorrelationof
Croux, 1993). Results from the box-plot rule show no univari- 0.5.Itisimportanttoseethatoutliersinthebivariatespace(illus-
ateorbivariateoutliersinpairs1and2,oneunivariateoutlierpair tratedinred)canbeobservedeventhoughunivariatedistributions
3,andoneunivariateandsimultaneouslybivariateoutlierinpair are perfectly normal (case 2). Outliers can be important for the
4(Figure1,middlecolumns).Forpair1,othermethodsgavethe processunderstudy,butgiventhegoalofcharacterizingthebulk
sameresult.Forpair2,boththeMAD-medianruleandS-outlier of the data, they can result in misleading conclusions. As illus-
methods identified the first two points as univariate outliers in tratedatthebottomof Figure3,outlierscanbepresentevenin
Y.Inaddition,theMAD-medianruleidentifiedthefirstandlast datafromanormalpopulation,becausethesampleitself might
points as bivariate outliers, whereas the S-outlier method iden- notbenormal.
tified the first and the last two points as bivariate outliers. This
illustrates the difficulty of spotting bivariate outliers because of Zero-correlationandfalsepositiveerrorrate
the trade off between specificity (true negatives) and sensitivity Gaussiandata. Zero-correlationwaswellestimatedbyallmeth-
(truepositives–Appendix).Forpairs3and4,theMAD-median ods: all correlation values were close to 0 and the 99.99%
ruleandtheS-outliermethodalsoflaggedtheextremepointsas CIs of all methods included 0 (Figure 4). Comparisons of
FrontiersinPsychology|QuantitativePsychologyandMeasurement January2013|Volume3|Article606|4
Pernetetal. Robustcorrelationtoolbox
FIGURE2|Correlationresults.FromlefttorightareillustratedPearson’s, differsfromtheothersbecauserankeddataareplotted.Forthe20%bend
Spearman’s,20%bend,andPearson’sskipped-correlationswiththe95% correlation,redindicatesdatabentinX,greeninY andblackinboth.No
bootstrappedCIsaspinkshadedareas.ThescaleforSpearman’scorrelations skippedcorrelationisreturnedforpair4.
methods showed no significant differences between Pearson’s differencesforn=10–100(p=0)anddidnotdifferforn>100
and Spearman’s (0.1<p<0.8) and the 20% percentage-bend (0.01<p<0.59).Similarly,thestandardandskippedSpearman’s
(0.24<p<0.99)correlations.Pearson’scorrelationsandskipped correlationsdifferedsignificantlyforn=10–100(p=0)anddid
Pearson’s correlations showed small (∼0.001) but significant not differ for n>100 (0.05<p<0.69). The false positive rate
www.frontiersin.org January2013|Volume3|Article606|5
Pernetetal. Robustcorrelationtoolbox
FIGURE3|Populationsusedinthesimulations.Top:populationswitheffectsizesof0.5.Middle:marginalhistogramsforthesepopulations.Bottom:
examplesofdrawswithsamplesizesn=10,50,250,and500.Reddotsmarkbivariateoutliersidentifiedusingthebox-plotruleonprojectdata.
FrontiersinPsychology|QuantitativePsychologyandMeasurement January2013|Volume3|Article606|6
Pernetetal. Robustcorrelationtoolbox
Table1|Correlationresultswiththeir95%CIsfortheAnscombe’squartet.
Pair1 Pair2 Pair3 Pair4
Pearson r=0.81,p=0.002 r=0.81,p=0.002 r=0.81,p=0.002 r=0.81,p=0.002
h=1,CI[0.59,0.95] h=1,CI[0.48,0.96] h=1,CI[0.71,1] h=1,CI[0.75,0.95]
Spearman r=0.81,p=0.002 r=0.69,p=0.01 r=0.99,p∼0 r=0.5,p=0.1
h=1,CI[0.49,0.97] h=0,CI[−0.009,1] h=1,CI[0.99,1] h=1,CI[0.5,0.8]
20%bend r=0.81,p=0.002 r=0.8,p=0.0029 r=1,p=0 r=0.22,p=0.49
h=1,CI[0.43,0.96] h=0,CI[−0.06,0.99] h=1,CI[0.89,1] h=1,CI[0.2,0.79]
SkippedPearson r=0.81,h=1 r=0.81,h=1 r=0.81,h=1 r=NaN,h=0
h=1,CI[0.54,0.95] h=1,CI[0.51,0.96] h=1,CI[0.74,1] h=0,CI[NaNNaN]
SkippedSpearman r=0.81,h=1 r=0.69,h=0 r=0.99,h=1 r=NaN,h=0
h=1,CI[0.4,0.97] h=1,CI[0,1] h=1,CI[0.9,1] h=0,CI[NaNNaN]
FIGURE4|EffectsizesandfalsepositiveerrorratesforGaussiandata rateforPearsons’(blue),skippedPearson’s(cyan),Spearman’s(red),skipped
withzero-correlation.Fromlefttorightaredisplayed:themeancorrelation Spearman’s(magenta),and20%bend(green)correlationsforeachtypeof
values;the99.99%CIs(i.e.,correctedforthe14samplesizes)ofthedistance simulation(Gaussianonly,withunivariateoutliers,andwithbivariateoutliers).
tothezero-correlationinthesimulatedGaussianpopulation;thefalsepositive TheY-axisscalesaredifferentfordatawithbivariateoutliers.
was well controlled by all methods, with values close to the and for skipped Spearman’s correlation 4.1% (min 3.2, max
5% nominal level. Across sample sizes, the average false posi- 6). Comparison of the binomial distributions (significant/non-
tive error rate for Pearson’s correlation was 4.9% (min 4.7,max significant results) found that Pearson’s correlations did not
5.2), for Spearman’s correlation 4.9% (min 4.5, max 5.6), for differ significantly from Spearman’s (0.19<p<0.98) and from
the 20% percentage-bend correlation 4.9% (min 4.6, max 5.4), the 20% percentage-bend (0.23<p<0.88) correlations. From
for the skipped Pearson’s correlation 4.4% (min 3.6, max 5.6), n=10–100, the false positive rate was not different between
www.frontiersin.org January2013|Volume3|Article606|7
Pernetetal. Robustcorrelationtoolbox
standard- and skipped-correlations (Pearson 0.004<p<0.8; thetruecorrelationsallencompassed0.Comparisonsofmethods
Spearman 0.001<p<0.9). However,the false positive rates did neverthelessrevealeddifferences,withPearson’scorrelationbeing
differ for larger sample sizes (Pearson 0.001<p<0.009; Spear- the best method of all. Compared to Spearman’s correlation,
man0.001<p<0.002)suchthatskipped-correlationsweremore Pearson correlation was significantly higher (closer to the true
conservative. value) for 0.1<ρ<0.9 (p=0), with differences from +0.004
to +0.03. The same pattern was observed when compared to
Marginal outlier data. Again, zero-correlation was well esti- the 20% percentage-bend correlation (p=0),except for ρ=0.1
mated by all methods as all correlation values were close to 0. and n=10 (p=0.18), with differences from +0.001 to +0.02.
The99.99%CIsofallmethodsincluded0(Figure4).Compari- WhencomparedtoskippedPearson’scorrelation,significantdif-
sonof methodsshowednosignificantdifferencesbetweenPear- ferences were observed from ρ>6, n>400 to ρ=0.9, n>100
son’sandSpearman’s(0.15<p<0.95),the20%percentage-bend (0<p<0.002 – differences ranging from +0.006 to +0.001).
(0.08<p<0.95),ortheskippedPearson’s(0.03<p<0.99)cor- For smaller correlation values and sample sizes there were no
relations.NosignificantdifferenceswereobservedbetweenSpear- significant differences (0.1<p<1). A similar pattern of results
man’sandskippedSpearman’scorrelations(0.05<p<0.84).The wasobservedwhencomparingSpearman’scorrelationtoskipped
false positive rate was well controlled by Pearson’s correlation Spearman’scorrelation.Significantdifferenceswereobservedfrom
(averagefalsepositiveerrorrate4.8%,min4.7,max5),Spearman’s ρ>3,n>400toρ=0.9,n=80(0<p<0.002–from−0.001to
correlation (4.9%, min 4.5, max 5.3), and the 20% percentage- +0.002).Forsmallercorrelationvaluesandsamplesizestherewere
bendcorrelation(4.9%,min4.6,max5.1).Skipped-correlations nosignificantdifferences(0.1<p<1).Forallcomparisons,there
wereslightlyconservativewithanaveragefalsepositiveerrorrate, werenosignificantdifferenceswhenρ(thetruecorrelationvalue)
for the skipped Pearson’s correlation, of 3.4% (min 3, max 4) wasequalto1.
and,forskippedSpearman’scorrelation,of3.3%(min3,max4). Poweranalysesshowedsimilartrendsforalltechniques,with
ComparisonofthebinomialdistributionsrevealedthatPearson’s maximumpowerforPearson’scorrelationsandminimumpower
correlationdidnotdifferfromSpearman’s(0.07<p<0.98)and forskippedSpearman’scorrelations.Ingeneral,powerincreased
the 20% percentage-bend (0.27<p<0.99) correlations. How- up to 100% as a function of the sample size except for r=0.1.
ever,forn>20,thefalsepositiveratesweresignificantlysmaller Comparison between methods revealed significantly stronger
for skipped-correlations (Pearson 0.001<p<0.004; Spearman power for Pearson’s correlation compared to Spearman’s corre-
0.001<p<0.002). lation (max +10%, 0.001<p<0.003), from high correlations
and small sample sizes (ρ>0.3, n<150), to low correlations
Bivariateoutlierdata. Onlyskippedcorrelationmethodsesti- and large sample sizes (ρ<0.2, n>250). For small correla-
mated well zero-correlation. On average, Pearson’s correlation tion values and small sample sizes or large correlation values
was r=0.77 (min 0.76, max 0.079) and the 99.99 CIs never and large sample sizes, the two methods had similar power
included 0. Spearman and the 20% percentage-bend correla- (0.004<p<0.99).Thesameresults(withtheexceptionof6com-
tions showed similar results with averaged correlations of 0.269 parisons out of 126) were observed when comparing Pearson’s
(min 0.268, max 0.273) and 0.275 (min 0.25, max 0.282), and correlationstothe20%percentage-bendcorrelation(maxdiffer-
CIsincluded0forn=10–60only.Incontrast,skippedPearson’s ence +6.4%). Power comparison between Pearson’s correlation
andSpearman’scorrelationswerecloseto0withaveragecorrela- and skipped Pearson’s correlation showed significant differences
tionsof0.017(min0.014,max0.026)and0.011(min0.008,max (max difference +23% 0.003<p<0.003) for all effect sizes as
0.02).TheirCIsalwaysincluded0.Pearson’scorrelationestimates a function of the sample size. Pearson’s correlation was more
weresignificantlylargerthanestimatesfromSpearman’s(p=0), powerful than skipped Pearson’s correlation at increasing sam-
20%percentage-bend(p=0),andskippedPearson’scorrelations ple sizes as r decreased (ρ<1, n=30; ρ<0.9, n=40; ρ<0.8,
(p=0).Similarly,Spearman’scorrelationsweresignificantlylarger n=50;ρ<0.7,n=60;ρ<0.6,n=60;ρ<0.5,n=100;ρ<0.4,
than their skipped counterparts (p=0). The false positive error n=150; ρ<0.3, n=250; ρ<0.2, n=300); however, for large
ratewasclosetoorequalto100%forPearson’scorrelations.For effectsizesandlargesamplesizes,thetwotechniquesdidnotdiffer
Spearman’sandthe20%percentage-bendcorrelation,itincreased (0.01<p<0.99).Thesameresults(withtheexceptionof4com-
from8.53%forn=10to100%forn=250.Incontrast,thefalse parisonsoutof126)wereobservedwhencomparingSpearman’s
positiveerrorrateofskipped-correlationsstayedclosetothenom- correlationstotheskippedSpearmancorrelation(maxdifference
inal level of 5% (average rate for skipped Pearson’s correlations +19%).
4.3%,min3.7,max5.1;averagerateforskippedSpearman’scorre-
lations4%,min3.5,max5.2).Comparisonofthebinomialdistri- Marginaloutlierdata. Effect sizes for Gaussian data contami-
butionsrevealedthatPearson’scorrelationsdifferedsignificantly natedby10%ofmarginaloutliers(Figure6)showedthatPearson’s
fromalloftheothermethods(p=0.001),exceptSpearman’sand and Spearman’s correlations estimated poorly the true correla-
the20%percentage-bendcorrelationsforn>300,wheretheyalso tions, whereas skipped estimators always estimated properly ρ
provided100%offalsepositives. (all 99.99% CIs included 0). Pearson’s correlations underesti-
mated ρ most of time – the 99.99% CIs included 0 for only
Effectsizesandpower 30% of cases: for ρ=0.1; ρ=0.2, n<250; ρ=0.3, n<100;
Gaussian data. Effect sizes for Gaussian data (Figure 5) were ρ=0.6 n<60; ρ=0.5, n<40; ρ=0.6, n<30; ρ=0.7 and 0.8,
wellcapturedbyallmethods:the99.99%CIsofthedifferenceto n<20.Thisshowsthatthemoretheoutliersdeviatedfromthe
FrontiersinPsychology|QuantitativePsychologyandMeasurement January2013|Volume3|Article606|8
Pernetetal. Robustcorrelationtoolbox
FIGURE5|EffectsizesandpowerforGaussiandata.Fromlefttorightare population;thepowerforPearson’s(blue),skippedPearson’s(cyan),
displayed:themeancorrelationvalues;the99.99%CIs(i.e.,correctedforthe Spearman’s(red),skippedSpearman’s(magenta),and20%bend(green)
14samplesizes)ofthedistancetothecorrelationinthesimulatedGaussian correlationsforeacheffectsize(fromtopr=0.1tobottomr=1).
www.frontiersin.org January2013|Volume3|Article606|9
Pernetetal. Robustcorrelationtoolbox
FIGURE6|EffectsizesandpowerforGaussiandatacontaminatedby contaminatedbyunivariateoutliers;thepowerforPearson’s(blue),skipped
10%ofmarginaloutliers.Fromlefttorightaredisplayed:themean Pearson’s(cyan),Spearman’s(red),skippedSpearman’s(magenta),and20%
correlationvalues;the99.99%CIs(i.e.,correctedforthe14samplesizes)of bend(green)correlationsforeacheffectsize(fromtopr=0.1tobottom
thedistancetothecorrelationinthesimulatedGaussianpopulation r=1).Incolumnone,thescalesofthemeancorrelationvaluesdiffer.
FrontiersinPsychology|QuantitativePsychologyandMeasurement January2013|Volume3|Article606|10