Table Of ContentStatisticalScience
2007,Vol.22,No.3,401–406
DOI:10.1214/07-STS239
(cid:13)c InstituteofMathematicalStatistics,2007
Multivariate Meta-Analysis: Contributions
of Ingram Olkin
Betsy Jane Becker
8
0 Abstract. Theresearchonmeta-analysis andparticularly multivariate
0 meta-analysis hasbeengreatly influencedbythework ofIngramOlkin.
2
This paper documents Olkin’s contributions by way of citation counts
n and outlines several areas of contribution by Olkin and his academic
a
descendants. An academic family tree is provided.
J
8 Key words and phrases: Meta-analysis, multivariate.
2
]
E 0. INTRODUCTION univariate case—one endpoint per study. Such end-
M points can be represented by correlations, mean dif-
Much of the research on statistical methods for
ferences, proportions, odds ratios (or log odds) and
t. meta-analysis in the last three decades has been in-
a even observed probabilities.
fluenced by Ingram Olkin, either through his direct
t
s Thefirstof Olkin’s contributions tometa-analysis
contributions or through the work of his students
[ (Hedges and Olkin, 1980) examined the intuitively
andtheiracademicdescendants.Weindicatetheex-
1 appealing vote-counting methods used in many re-
tent of this influence and present a tree of Olkin’s
v search syntheses and traditional literature reviews.
7 academic descendants who have made, or are mak-
Votecountingentailscountingthenumberofstudies
0 ing, contributions to research in meta-analysis. We
2 that have statistically significant results in support
then consider the outcome metrics that have been
4 of, and counter to, a particular hypothesis, as well
. used in the multivariate meta-analysis context and
1 as those with nonsignificant results. The category
brieflyreviewkey resultsforeach metric,thusshow-
0 with the most votes (or more than some specific
8 ing Olkin’s seminal influence on this important sub-
proportion of votes) “wins” and the set of all re-
0 field of meta-analysis.
: sults is then characterized as supporting that view
v
(e.g., if half of the studies have significant tests in
i 1. OLKIN’S INFLUENCE ON META-ANALYSIS
X
favor of a hypothesis, the studies are viewed as sup-
r Meta-analysis is a set of methods for combining porting the hypothesis). Hedges and Olkin showed
a
and analyzing results from series of related stud- that the statistical properties of this approach were
ies. Glass (1976) coined the term “meta-analysis,” problematic—in that more evidence can lead to
but the idea of summarizing study results is much poorer decisions.
older, with references dating to the turn of the last Since then Olkin has authored or co-authored
century (e.g., Pearson, 1904). Much of the litera- 39 more articles or book chapters and one
ture on methods for meta-analysis deals with the book on meta-analysis. The influence of his
work is shown by the fact that these documents
Betsy Jane Becker is Professor of Measurement and have generated over 5600 citations. (Based on
Statistics, College of Education, Florida State searches of the Web of Science at
University, Tallahassec, Florida 32306, USA e-mail: http://80isi4.isiknowledge.com.proxy.lib.fsu.edu/
[email protected] using the author names “Olkin I*,” “Hedges L*,”
“Gleser L*” and “Sampson A*.”) His book Statis-
This is an electronic reprint of the original article
tical Methods for Meta-analysis with Larry Hedges
published by the Institute of Mathematical Statistics in
Statistical Science, 2007, Vol. 22, No. 3, 401–406. This (Hedges and Olkin, 1985) is something of a cita-
reprint differs from the original in pagination and tion classic, having been cited at least 3270 times.
typographic detail. However, Olkin’sarticles andbookchapters arealso
1
2
B. J. BECKER
highly cited, with the number of citations per work 2. CONTRIBUTIONS TO META-ANALYSIS
OF OLKIN’S ACADEMIC DESCENDANTS
rangingfrom0to916withameancountof62.5cita-
tions (SD=157.4) and a median count of 20.5 cita- BesidesOlkin’sowncontributionstometa-analysis,
tionsperarticle.[Asistypicalofcitation counts,the individuals that he has mentored and trained have
also made many contributions to this literature—
distribution of citation counts per article is highly
some writing dissertations on meta-analysis topics.
skewed (skewness coefficient = 4.8), suggesting that
(ApologiesaremadetoanystudentsofOlkinandhis
the median citation count per article is the more descendants who have inadvertently been omitted
appropriate measure of central tendency.] The ma- from this analysis.) All first-generation descendants
were students at StanfordUniversity, though not all
jority of this work is collaborative—30 of these pa-
earned degrees in the Department of Statistics. In
pers are co-authored, with the mean number of co-
addition, students of those students are considered,
authors across all 39 documents being 2.74 (SD= and so on, through several generations of Olkin aca-
3.5). As might be expected from the recipient of the demic “descendants.”
These individuals are displayed in Figure 1, the
Elizabeth L. Scott Award from the Committee of
Olkinmeta-analytic family tree. Theyears shownin
PresidentsofStatisticalSocieties(in1998),overhalf
the figure are the graduation dates for each person;
(17) of Olkin’s 30 co-authored works were written dissertations concerning meta-analysis methods are
with at least one female co-author. included in the reference list as well. The tree shows
Fig. 1. The Olkin meta-analytic family tree.
3
MULTIVARIATEMETA-ANALYSIS
on the bottom-most branches three former students exhibit subtle dependencies because of common in-
of Olkin who wrote dissertations on meta-analysis. strumentation, treatments and the like, their out-
They are Hedges (1980), Holmgren (1989) and Yen comesdonothaveacorrelationstructurethatiseas-
(1997). In addition, three other former students of ily characterized. Hedges and Olkin (1985) firstpre-
Olkinareshown.Gleser,PerlmanandSampsoneach sented methods for dealing with multivariate data
have contributed to the literature on meta-analysis in meta-analysis. Their Chapter 10 dealt with stan-
or synthesis of results, though none wrote a disser- dardized mean differences that are dependent be-
tation on the topic. Relevant works include Gleser cause p (dependent)responsevariables are observed
and Olkin (1994, 1996), Koziol and Perlman (1978) within each primary study.
and Olkin and Sampson (1998), among others. We denote the results as T , where i indexes the
ij
The next set of leaves shows students of Olkin’s study and j the outcome. Across studies we may
students—perhaps we can call these Olkin’s meta- have
analytic “grandchildren.” Here are listed seven who
T11 ... T1p
wrotedissertations on meta-analytic methods.Abu- T21 ... T2p
Libdeh (1984), Becker (1985), Champney (1983), . .
. .
Konstantopoulos (2003), Pigott (1992) and Zhang . .
(1993)weredissertationswrittenbystudentsofHed- Ti1 ... Tip
ges, and Sylvester (2001) and Sezer (2006) were dis- .. ..
. .
sertations directed by Gleser. Two other students of
Tk1 ... Tkp
Hedges (Vevea and Friedman) have contributed to
for k studies and upto p outcome indices. The p de-
the literature on meta-analytic methods after com-
pendent indices arise when p response variables are
pleting a dissertation using meta-analytic methods
observed, when contrasts are dependent (e.g., com-
or on another topic (e.g., Friedman, 1989, 2000;
Hedges and Vevea, 1996, 1998). mon controls, multiple proportions), when multiple
Finally we reach the currentends of the branches. indices involve each response variable (e.g., correla-
Six additional students are listed who worked with tion matrices), and when multivariate analyses ap-
Becker on meta-analytic methods (Chang, 1992; pear within a primary study. The possible metrics
Chiu, 1999; Cho, 2000; Fahrbach, 2001; Schram, include multivariate standardized mean differences,
1996;Wu,2006)andtwowhowerestudentsofVevea correlations and proportions (or odds ratios). Each
and who have either written dissertations on meta- such metric will be considered in turn.
analysis methods (Hafdahl, 2001) or contributed to
themeta-analyticliterature(VeveaandWoods,2005; 4. MULTIVARIATE STANDARDIZED MEAN
Woods et al., 2002) while writing a dissertation on DIFFERENCES
a different topic. We can be assured that others will
This metric may be the most thoroughly inves-
follow.
tigated of all those for which multivariate analy-
ses have been proposed. Gleser and Olkin (1994)
3. OVERVIEW OF MULTIVARIATE
dealt with multiple treatment studies and multiple
META-ANALYSIS
endpoint studies for standardized mean differences.
We next turn to the topic of multivariate meta- Some studies combine both of these multivariate as-
analysis and explore Olkin’s fundamental contribu- pects.Evidencethatmultivariateeffect-sizedataare
tions to this domain. (See Becker, 2000, and van common is found in the fact that Gleser and Olkin
Houwelingen et al., 2002, for overviews of the topic (1994)hasbeencitedover100times,infieldssuchas
of multivariate meta-analysis.) Multivariate meta- psychology, education, medicine, ecology and crim-
analysis occurs when more than one (dependent) inal justice. Similarly, an early paper by Rauden-
outcome is measured in a study. This can occur bush, Becker and Kalaian (1988) dealt with multi-
when subjects are measured on several outcomes or variate standardized-mean-difference data.
atseveraltimepoints(multipleendpointstudies),or
4.1 Multiple Treatment Studies
whenstudyindicesarecomputedusingsharedtreat-
mentorcontrolgroups(multipletreatmentstudies). Multiple treatment studies are illustrated here
Thesecases donot typically includestudies with re- with an example of studies with a common control
sults for multiple samples. While such samples may group. Further elaborations of this scenario (e.g.,
4
B. J. BECKER
with three or more treatment groups or multiple relevantreferencesincludeArends,VokoandStijnen
controlgroups)leadtomoreoutcomes,buttheprin- (2003) and Nam, Mengersen and Garthwaite (2003)
ciples underlying these methods can be illustrated which concern analyses of multiple log-odds ratios.
with this simplest scenario. Additional forthcoming work will undoubtedly ad-
Supposeastudyhastwotreatmentgroups,T1 and dress this issue.
T2, and one control group C. Then if we define X¯A
to represent the mean of group A and S to be the 6. MULTIVARIATE CORRELATIONS AND
pooled within-groups standard deviation across all SLOPES
groups, we can compute
The topic of synthesis of correlation matrices has
T1=(X¯T1 −X¯C)/S and T2=(X¯T2 −X¯C)/S seen increasing activity in the past few years. This
increase in interest is likely related to the increas-
for each study. If we index these outcomes as Ti1 ingly complex models investigated in primary re-
and Ti2 with i for the ith study, we will have search, at least in the social sciences. Researchers
T11 T12 want to be able to statistically model the effects of
T21 T22 multiple predictors as well as to control for poten-
.. .. tial confounding variables, and this is done by in-
. .
cluding such variables in complex models. Results
Ti1 Ti2 of such techniques as structural equation modeling,
.. ..
. . factor analysis and multiple regression have often
Tk1 Tk2 been omitted from meta-analyses because of a lack
ofmethodsforsynthesizingindicesfromtheseanaly-
whichhasamultivariatestructure.GleserandOlkin
ses.While Olkin has not contributed directly to this
(1994) gave two formulas for Cov(T ,T ) for mul-
ij ij′
area of synthesis methods, his work is fundamental
tiple treatment studies. More recent work by Cook
because most of the analyses proposed to date are
(2004) presents a formula tailored to small-sample
asymptoticandrelyonthelarge-sampledistribution
cases.
theory presented by Olkin and Siotani in 1967.
4.2 Multiple Endpoint Studies Themultivariateworkinthisrealmofmeta-analysis
has involved the synthesis of correlation matrices,
Gleser and Olkin (1994) also cover dependence of
and the use of those summaries in further modeling
standardized mean differences due to multiple re-
of linear models, structural equation models, and
sponse variables (expanding on Hedges and Olkin,
even factor analysis (G. Becker, 1996). B. Becker
1985). If we define T to represent an effect size for
ij
andhercollaborators (B.Becker,1992,1995;Becker
outcome measure j (j=1 to p) in study i, we have
and Fahrbach, 1994; Becker and Schram, 1994) be-
Tij =(Y¯iTj −Y¯iCj )/Sij gan this stream of work by presenting methods for
the synthesis of correlation matrices, specifically es-
for i = 1 to k studies and j = 1 to p measures.
timates of mean matrices under fixed- and random-
This was labeled the multiple endpoint design. The
effects models and tests of the homogeneity of the
effect-size data structure is identical to that shown
seriesofmatrices underreview.Atroughlythesame
above but the covariances between the multiple ef-
time, applications of like methods appeared in the
fects from each study differ from those in the mul-
personnelpsychologyliterature(e.g.,Schmidt,Hunter
tiple treatment case.
andOuterbridge,1986).Beckeralsopresentedmeth-
ods for estimating linear models based on the mean
5. MULTIVARIATE PROPORTIONS
correlationmatricesandtestingcomponentsofthose
Lesshasbeenpublishedonthemultivariate meta- composite models. Others have pursued this work
analysis of proportions. One contribution is Gleser andinvestigatedtheuseofmeanmatriceswithstruc-
and Olkin’s (2000) chapter on multiple treatment tural equation modeling software (e.g., Cheung and
studies with outcomes expressed as two-by-two ta- Chan,2005;FurlowandBeretvas,2005).Allofthese
bles. Gleser and Olkin present large-sample gener- works rely on the fundamental result derived by
alized least squares methods for dealing with risk OlkinandSiotani(1976,page238) ofthecovariance
differences, log odds ratios, and arcsine transformed among correlations from a single sample. Specifi-
proportions from multiple treatment studies. Other cally, the large-sample covariance, σ , between
ist,iuv
5
MULTIVARIATEMETA-ANALYSIS
population correlations ρist and ρiuv within study i Becker, B. J. (1995). Correction to “Using results from
is replicated studies to estimate linear models.” J. Educa-
tional and Behavioral Statistics 20 100–102.
2 2 2 2
σ =[0.5ρ ρ (ρ +ρ +ρ +ρ ) Becker, B. J. (2000). Multivariate meta-analysis. In Hand-
rist,riuv ist iuv isu isv itu itv
book of Applied Multivariate Statistics and Mathematical
+ρisuρitv+ρisvρitu Modeling (H. E. A. Tinsley and S. D. Brown, eds.) 499–
−(ρ ρ ρ +ρ ρ ρ 525. Academic Press, San Diego, CA. MR1823752
ist isu isv its itu itv Becker, B. J. and Fahrbach, K. (1994). A comparison of
+ρ ρ ρ +ρ ρ ρ )]/n , approaches to the synthesis of correlation matrices. Paper
ius iut iuv ivs ivt ivu i
presented at the annual meeting of the American Educa-
wheren isthesamplesizeinstudy iand s,t,u and tional Research Association, New Orleans, LA.
i
v index the variables within study i that are corre- Becker, B. J. and Schram, C. M. (1994). Examining ex-
planatorymodelsthroughresearchsynthesis.InTheHand-
lated. That is, ρ is the correlation between vari-
ist book of Research Synthesis (H. M. Cooper and L. V.
ables X and X within studyi.This resultwas also
s t Hedges,eds.)357–381.RussellSageFoundation,NewYork.
used by Hafdahl (2001) who examined exploratory Becker,G.(1996).Themeta-analysisoffactoranalyses:An
factor analysis methods based on synthesized ma- illustration based on the cumulation of correlation matri-
trices, and papers by Olkin and other collaborators ces. Psychological Methods 1 341–353.
Champney, T. F. (1983). Adjustments for selec-
(e.g., Olkin and Finn, 1976, 1990; Olkin and Saner,
tion/publication bias in quantitative research. Ph.D.
2001) also rely on this fundamental result.
dissertation, Univ.Chicago.
Chang,L.(1992).Apoweranalysisofthetestofhomogene-
7. CONCLUSION ity in effect-size meta-analysis. Ph.D. dissertation, Michi-
gan StateUniv.
It is safe to say that much of the work on meta- Cheung, M. W. L. and Chan, W. (2005). Meta-analytic
analysis, and especially multivariate issues in meta- structural equation modeling: A two-stage approach. Psy-
analysis, has its genesis in the contributions of In- chological Methods 10 40–64.
Chiu, C. W.-T. (1999). Scoring performance assessments
gram Olkin. The review of research in this paper
based on judgements: Utilizing meta-analysis to estimate
shows the significant impact of Olkin’s work. The
variance components in generalizability theory for unbal-
familytreeillustratesthatcontributionsfromOlkin’s
ancedsituations.Ph.D.dissertation,MichiganStateUniv.
academic descendants are numerous and will con- Cho, J. M.(2000).Hypothesesandlimitingdistributionsfor
tinue to be forthcoming. p-value summaries to compare studies in meta-analysis.
Ph.D. dissertation, Michigan StateUniv.
Cook,S.R.(2004).Anoteontestingforhomogeneityamong
ACKNOWLEDGMENTS
effect sizes sharing a common controlgroup. Psychological
Methods 9 446–452.
This work was supported by National Science
Fahrbach, K. R. (2001). An investigation of methods for
FoundationGrantsREC-0335656andREC-0634013.
mixed-modelmeta-analysisinthepresenceofmissingdata.
Parts of this work were presented in the sympo- Ph.D. dissertation, Michigan StateUniv.
sium “Multivariate Analysis: In Celebration of In- Friedman, L. (1989). Mathematics and the gender gap: A
gramOlkin’s80thBirthday,”attheJointStatistical meta-analysisofrecentstudiesonsexdifferencesinmathe-
maticaltasks.Review ofEducational Research 59185–213.
Meetings, Toronto, Canada, August 2004.
Friedman, L. (2000). Estimators of random effects variance
components in meta-analysis. J. Educational and Behav-
ioral Statistics 25 1–12.
REFERENCES
Furlow, C. F. and Beretvas, S. N. (2005). Meta-analytic
Abu-Libdeh,O.I.(1984).Strengthofassociationinthesim- methods of pooling correlation matrices for structural
ple general linear model: A comparative study of Hays’ equationmodelingunderdifferentpatternsofmissingdata.
omega-squared. Ph.D. dissertation, Univ.Chicago. Psychological Methods 10 227–254.
Arends, L. R., Vokoˇ, Z. and Stijnen, T. (2003). Com- Glass, G. V. (1976). Primary,secondary,and meta-analysis
bining multiple outcome measures in a meta-analysis: An of research. Educational Researcher 5 3–8.
application. Statistics in Medicine 22 1335–1353. Gleser,L.J.andOlkin,I.(1994).Stochasticallydependent
Becker, B. J. (1985). Applying tests of combined signifi- effect sizes. In The Handbook of Research Synthesis (H.
cance:Hypothesesandpowerconsiderations.Ph.D.disser- M. Cooper and L. V. Hedges, eds.) 339–355. Russell Sage
tation, Univ.Chicago. Foundation,New York.
Becker, B. J. (1992). Using results from replicated studies Gleser, L. J. and Olkin, I. (1996). Models for estimating
toestimatelinearmodels.J.EducationalStatistics 17341– the number of unpublished studies. Statistics in Medicine
362. 15 2493–2507.
6
B. J. BECKER
Gleser, L. J. and Olkin, I. (2000). Meta-analysis for 2×2 and Statistics Chapter 16 (S. Ikeda, ed.) 235–251. Shinko
tableswithmultipletreatmentgroups.InMeta-Analysisin Tsusho, Tokyo. MR0603847
MedicineandHealthPolicy (D.A.BerryandD.K.Stangl, Pearson, K.(1904).Reportoncertainentericfeverinocula-
eds.) 339–355. Dekker,New York. tions. British Medical Journal 2 1243–1246.
Hafdahl, A. R. (2001). Multivariate meta-analysis for Pigott, T. D. (1992). The application of normal maximum
exploratory factor analytic research. Ph.D. dissertation, likelihoodmethodstomissingdatainmeta-analysis.Ph.D.
Univ.North Carolina at Chapel Hill.
dissertation, Univ.Chicago.
Hedges, L. V.(1980).Combiningtheresultsofexperiments Raudenbush, S. W., Becker, B. J. and Kalaian, H.
using different scales of measurement. Ph.D. dissertation,
(1988). Modeling multivariate effect sizes. Psychological
Stanford Univ. Bulletin 103 111–120.
Hedges,L.V.andOlkin,I.(1980).Vote-countingmethods
Schmidt, F., Hunter, J. E. and Outerbridge, A. N.
in research synthesis. Psychological Bulletin 88 359–369.
(1986). Impactofjob experienceandability onjob knowl-
Hedges, L. V. and Olkin, I. (1985). Statistical Methods for
edge,work sample performance and supervisory ratingsof
Meta-Analysis. AcademicPress,Orlando,FL.MR0798597
job performance. J. Applied Psychology 71 432–439.
Hedges, L. V. and Vevea, J. L. (1996). Estimating effect
Schram, C. M. (1996). A comparison of methods for cor-
size under publication bias: Small sample properties and
robustness of a random effects selection model. J. Educa- recting multivariate data for attenuation with application
tional and Behavioral Statistics 21 299–332. to synthesizing correlation matrices. Ph.D. dissertation,
Hedges,L.V.andVevea,J.L.(1998).Fixed-andrandom- Michigan StateUniv.
effects models in meta-analysis. Psychological Methods 3 Sezer, A. (2006). Reporting uncertainty by spline function
486–504. approximation of log-likelihood. Ph.D. dissertation, Univ.
Holmgren,E.B.(1989).Utilizingp–pplotsinmeta-analysis Pittsburgh.
as general measures of treatment effects. Ph.D. disserta- Sylvester,M.A.,II.(2001).Estimationofacommonmean
tion, Stanford Univ. through a series of similar interlaboratory experiments.
Konstantopoulos, S.(2003).Variancecomponentsestima- Ph.D. dissertation, Univ.Pittsburgh.
tioninmeta-analysisemployingmaximumlikelihoodmeth- van Houwelingen, H. C., Arends, L. R. and Stijnen,
ods. Ph.D. dissertation, Univ.Chicago. T. (2002). Tutorial in biostatistics. Advanced methods in
Koziol, J. A. and Perlman, M. D. (1978). Combining in- meta-analysis:Multivariateapproachandmeta-regression.
dependent chi-squared tests. J. Amer. Statist. Assoc. 73 Statistics in Medicine 21 589–624.
753–763.
Vevea,J. L.andWoods, C. M.(2005).Publicationbiasin
Nam, I. S., Mengersen, K. and Garthwaite, P. (2003).
researchsynthesis:Sensitivityanalysisusingaprioriweight
Multivariatemeta-analysis.StatisticsinMedicine222309– functions. Psychological Methods 10 428–443.
2333.
Woods, C. M., Vevea, J. L., Chambless, D. L. and
Olkin, I. and Finn, J. D. (1976). Correlations redux. Psy-
chological Bulletin 118 155–164. Bayen, U. J. (2002). Are compulsive checkers impaired
in memory? A meta-analytic review. Clinical Psychology:
Olkin, I.andFinn, J. D.(1990).Testingcorrelatedcorrela-
tions. Psychological Bulletin 108 330–333. Science and Practice 9 353–366.
Olkin, I. and Sampson, A. (1998). Comparison of meta- Wu, M. J. (2006). Methods of meta-analyzing regression
analysis versus analysis of variance of individual patient studies: Applications of generalized least squares and fac-
data. Biometrics 54 317–322. MR1626809 toredlikelihoods.Ph.D.dissertation,MichiganStateUniv.
Olkin,I.andSaner,H.(2001).Approximationsfortrimmed Yen, J. H.(1997). Robustestimation ofeffectsizes inmeta-
Fisherproceduresinresearchsynthesis.StatisticalMethods analysis. Ph.D. dissertation, Stanford Univ.
in Medical Research 10 267–276. Zhang,J.(1993).Statisticalsignificancepublicationbias,its
Olkin,I.andSiotani,M.(1976).Asymptoticdistributionof determinationandstatisticaladjustmentsinmeta-analysis.
functions of a correlation matrix. In Essays in Probability Ph.D. dissertation, Univ.Chicago.
