Table Of ContentJorge González • Marie Wiberg
Applying
Test Equating Methods
Using R
123
JorgeGonzález MarieWiberg
FacultyofMathematics DepartmentofStatistics
PontificiaUniversidadCatólicadeChile UmeåSchoolofBusinessandEconomics
Santiago,Chile UmeåUniversity
Umeå,Sweden
ISSN2367-170X ISSN2367-1718 (electronic)
MethodologyofEducationalMeasurementandAssessment
ISBN978-3-319-51822-0 ISBN978-3-319-51824-4 (eBook)
DOI10.1007/978-3-319-51824-4
LibraryofCongressControlNumber:2017932073
©SpringerInternationalPublishingAG2017
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Foreword
Thisbooksignalsthematurityofthemethodologies andtechnology thatsurround
test score equating and the acceptance of these methodologies and of the
open-sourcesoftwarebythetestingindustry’sexpertsandacademia.
Testequatingisastatisticalprocedureforadjustingthetestformdifferencesin
standardized testing. Test equating methodologies originated in testing companies
many decades ago. Due to the pragmatic operational perspective, many of these
methodologies had lacked rigorous theoretical background and research for many
years. Then in the 1980s, test score equating started to receive the appropriate
attention from researchers and academia (Holland and Rubin 1982). Several
notable journal articles and books were written at that time, and the field was
brought forward significantly. The availability of personal and fast computers
also facilitated the implementation of the established equating methods. For
example, psychometricians and technology specialists at Educational Testing
Service developed GENASYS, a very comprehensive, modular, and sophisticated
software system to support the large-scale application of test score analysis and
equating with all the available methodologies at that time. The first edition of the
Kolen and Brennan (1995) book became a landmark in equating. A second wave
oftheoreticalinterestandthedevelopmentofnewmethodstookplacearound2005
(KolenandBrennan2004;vonDavieretal.2004;Doransetal.2007).
Since2005,therehasbeenanoticeableshiftintestequating.First,therehasbeen
anincreasinginterestinanduseoftestequatingallovertheworld.Second,asitwas
illustrated in the edited volume of von Davier (2011a), there has been a renewed
interestinsearchingforbettermethodologiesthatimprovethescoreconversionin
termsofbias(theobserved-scoreequatingframework,variouskernelmethods,local
equating), in terms of error (kernel equating, equating with exponential families),
in terms of hypothesis testing (Lagrange multiplier tests for item response theory
(IRT) methods, Wald test for kernel methods), in terms of approximations for
small sample equating, and in terms of monitoring the quality control of equating
results (using time series approaches); third, there has been an increased interest
in equating by university professors in statistics and educational measurement;
and more importantly, there has been a significant software development using
different software environments, from SAS to R. Moreover, the rapid changes
in the technology made the upgrade of proprietary software challenging; the
society became more appreciative of the open-source environment that facilitates
transparency,research,andcommunitywork.
Thisvolumeisremarkablefromseveralperspectives.First,itisaperfectmirror
of the changes in test equating during recent years: it includes new equating
methodsalongsidetheolderones,anditprovidescodeinRforusers.Theauthors
of this volume have both an academic background (one works for the Pontificia
Universidad Católica de Chile, in Santiago de Chile, and one works for Umeå
University in Sweden) and a practitioner background (one is a consultant for the
operationaltestsfromthemeasurementcenterMIDEUCandtheotherconsultsfor
theoperational large-scale SwedishSAT).Bothauthors developed orcodeveloped
R packages for conducting various equating methods. Last but not least, one is
from Chile and one is from Sweden. Again, the technology makes the world
interconnectedandsupportscollaborationslikethese!
The structure of this volume follows the process of operationalizing various
test equating methods. The volume starts by locating equating in the realm of
statistical models, which was also attempted by von Davier (2011b). Then the
volumecontinuesbybrieflypresentingthetheorybehindthemethods(smoothingof
discretetestscoredata,linearinterpolationequating,severalkernelmethods,local
andIRTequating)andbyillustratingtheapplicationofthesemethodswithsetsof
real data and with various R packages, including those developed by the authors
themselves.
Forthoseofuswhoworkoneducationalassessments,therearestillmanyopen
challenges. Measurement accuracy for all groups of test takers across the range
of scores is hard to ensure in operational testing. With the increased shifts in
demographics,wehadtoprovidemoresophisticatedandcomplex,yettransparent,
typesoftests.Nevertheless,aswetestmorediversegroupsofpeople,thechallenge
of providing accurate scores increases. For example, much of the research on
alternativekernels,inparticularonadaptivekernels,hasbeenmotivatedbytheneed
to analyze data from heterogeneous populations. Another way to look at fairness
is through the lenses of transparency of the process: Will the transparency of the
open-sourcesoftwareaidthefairnessarguments?
Another area that remains of interest for future research is how to ensure that
the computations with the open-source software environment are well tested and
error-freeforlarge-scalehigh-stakestestsadministeredovertime.Inthefuture,we
willalsoneedtoprovideopen-sourcecodeformethodsforqualitycontrol.
Asmanyofushavestartedtoworkonsimulation-andgame-basedassessments,
newchallengesoccurred.Tomeetthesechallenges,weneedtoredefinetheconcept
of test equating and the notion of adjusting for difficulty across test forms. While
wedonothaveasolutionyet,itisquiteprobablethatthesolutionwillusesomeof
thesenewerapproachestotestequatingandopen-sourcesoftware.
A final challenge relates to improved communication about the practical
consequences of addressing equating issues as an ongoing activity in order to
ensure meaningful scores. We need to explain the limitations of the methods and
the cost of being able to make truthful claims about score accuracy for all groups
over time. The authors of this book examine theoretical and practical issues that
cut across different types of methodological and operational situations and open
the doors to hands-on teaching and learning and to more theoretical research on
equatingmethodsconductedbystudentsandotherresearchers.
Princeton,USA AlinaA.vonDavier
December2016
References
Dorans,N.J.,Pommerich,M.,&Holland,P.W.(2007).Linkingandaligningscoresandscales.
NewYork:Springer.
Holland,P.,&Rubin,D.(1982).Testequating.NewYork:AcademicPress.
Kolen, M., & Brennan, R. (1995). Test equating: Methods and practices (1st ed.). New York:
Springer.
Kolen,M.,&Brennan,R.(2004).Testequating,scaling,andlinking:Methodsandpractices(2nd
ed.).NewYork:Springer.
von Davier, A. (2011a). Statistical models for test equating, scaling, and linking. New York:
Springer.
vonDavier,A.(2011b).Astatisticalperspectiveonequatingtestscores.InA.vonDavier(Ed.),
Statisticalmodelsfortestequating,scaling,andlinking(Vol.1,pp.1–17).NewYork:Springer.
vonDavier,A.A.,Holland,P.,&Thayer,D.(2004).Thekernelmethodoftestequating.NewYork:
Springer.
Preface
This book provides an overview of test equating methods with a focus on how to
implementthemusingR(RCoreTeam2016).
TheideaforwritingthisbookemergedinApril2014whenwebothattendedthe
NCME conference in Philadelphia. There are many reasons to go to conferences
including presenting your research, meeting other researchers, working on
collaborative projects, and getting new ideas. For us, writing a book on equating
with applications using R was a natural step as we had both been involved in the
development of two different R packages, kequate (Andersson et al. 2013) and
SNSequate (González 2014), and we have noticed that a number of software and
R packages that can be used when equating test scores have emerged in recent
years. Weeks (2010) (and references therein) offers a list that includes EQUATE,
ST, mcmequate, IpLink, POLYST, STUIRT, POLYEQUATE, IRTEQ, irtoys,
and MischPsycho. These software packages are mainly designed for conducting
item response theory (IRT) linking rather than equating, with the exception of
POLYEQUATEandIRTEQthatimplementIRTequating.Thelistcanbeenlarged
by including CIPE, Equating Recipes, RAGE-RGEQUATE, Equating Error,
PIE, and LEGS (Kolen and Brennan 2014), the KE software (ETS 2011), the
SAS/IML macro for log-linear smoothing (Moses and von Davier 2011), and the
Rpackagesplink(Weeks2010),lordif(Choietal.2011),equate(Albano2016),
andequateIRT(Battauz2015).
Twoquestionsofinteresttouswerewhetheralloftheavailableequatingmethods
had been implemented in these packages and if we could replicate known results
usingallofthesedifferentpackages.Whenwestartedwritingthisbook,neitherof
us had used all of these packages, but we got to learn many things and got many
ideas for new research along the way. By using an open-source software, a new
generation can easily start their research where the previous generation left off.
Thus, we expect many new research papers within the equating field in the next
fewyears.
People in the test equating field might wonder what is new and unique in this
equatingbookcomparedwithotherssuchastheworksofvonDavieretal.(2004)
and Kolen and Brennan (2014). This is the first equating book that provides all
of the necessary tools to perform equating regardless of which framework the
equatingbelongsto.Thismeansthatnomatterifitisatraditionalequatingmethod,
an IRT equating method, a local equating method, or a kernel equating method,
this book provides the essential tools both in terms of theory and computational
implementation. The goal is to give the readers a pamphlet of as many different
equatingmethodsaspossibleandtoprovidethemwithhands-onexamplesofhowto
performtheanalysesbythemselves.Auniquefeatureofthisbookisthatitprovides
R codes to implement all of the described equating methods. R has become an
essentialtoolforresearchinmanyfieldsincludingpsychometrics,andouraimisto
ensurethatequatingmethodsarenottheexceptionofthistendency.
Theintendedaudienceforthisbookisverybroad.Itincludestestconstructors,
researchersinequating,graduatestudentsbothinstatisticsandpsychometrics,and
allofthosewhohaveaninterestinlearninghowtoperformequatingwithdifferent
methods.Oneoftheaimsofthebookisalsotomakeiteasierforbeginnerstoget
intotheresearchfieldofequating.
Organizationofthisbook.Thefirstchaptergivesasummaryofourviewofthe
equating transformation as a statistical estimator. It then moves on to the practical
concernsinChap.2inwhichthedatausedinthebookaredescribed.Thischapter
alsoexplainshowthedataneedtobepreparedandstructuredinordertobeutilized
withthedifferentRpackagesusedinthebook.Chapter3containsdescriptionsand
illustrations of the traditional equating methods, while Chap. 4 goes step by step
throughthekernelequatingframework.Chapter5describesandillustratesdifferent
IRTequatingmethods,andChap.6containsmaterialregardingsomeoftheexisting
local equating methods. Finally, in Chap.7, we illustrate how to replicate some
recent developments in test equating research. In all chapters, examples using R
codeareprovided.AlluseddatasetsandRscriptsaswellasadditionalexamples
not appearing in the book are available from the book’s Web page at http://www.
mat.uc.cl/~jorge.gonzalez/EquatingRbook.
Jorge’s Acknowledgments. Writing this book has been possible thanks to many
peoplewhohavebeenimportantinmycareer.AlthoughmyPh.D.dissertationwas
not related to equating, I was lucky to be part of the psychometric research group
at K.U. Leuven and to have Paul de Boeck and Francis Tuerlinckx as supervisors.
OnceIgraduated,IwashiredbytheMeasurementCenterMIDEUCwhereIlearned
many things related to educational measurement, equating being among them. I
thank Jorge Manzi, director of the Measurement Center MIDE UC, for trusting
me and allowing me to be part of the center. I have also been very lucky to meet
wonderful colleagues and friends from whom I have learned many of the things I
now know. Thanks to Alina von Davier for always being receptive and willing to
collaborate on projects. Thanks to Ernesto San Martín for always being there to
discussthemostinterestingproblemsinstatisticsandespeciallyinpsychometrics.
Iwouldalsoliketothankallofmystudentsforworkingwithmeontheirmaster’s
thesesonequating.Iwouldespeciallyliketothankmyparents,LuisandIris,who
havealwaysgivenwhateverisnecessaryformywell-being.Thankstomybrother,
Carlos, for being my eternal best friend whom I can always trust. For all the time
thatIhavestolenfromyouandforallofyourunderstanding,thanksFannyforbeing
my partner for all these years. Lastly, for being the non-equatable ones, thanks to
mychildrenJavieraandRenato.Thisbookisdedicatedtobothofyou.
Marie’sAcknowledgments.IwouldliketothankallofthosewhomIhaveworked
withwithintheequatingresearchfield.Ihavelearnedmanydifferentthingsfromall
ofyou.Afewpersonsshouldbementioned.IamthankfultoWimvanderLinden,
whogotmeinterestedintheequatingresearchfieldin2006.IamgratefultoAlina
vonDavierwhointroducedmetokernelequatingandallofthenewideaswehave
workedwithandwillworkwithinthefuture.IamalsothankfultoBjörnAndersson,
who programmed the majority of kequate and continues to improve the package
withhisownresearchideas.IwouldalsoliketothankmyparentsBrittandUlf,for
alwaysbeingthereforme.Finally,Iamgratefultomyfamily–myhusbandJonas
andmytwodaughtersEmelieandHanna–fortheirloveandsupportandforbeing
thebestintheworld.
Santiago,Chile JorgeGonzález
Umeå,Sweden MarieWiberg
October2016
References
Albano,A.D.(2016).equate:AnRpackageforobserved-scorelinkingandequating.Journalof
StatisticalSoftware,74(8),1–36.
Andersson,B.,Bränberg,K.,&Wiberg,M.(2013).Performingthekernelmethodoftestequating
withthepackagekequate.JournalofStatisticalSoftware,55(6),1–25.
Battauz,M.(2015).equateIRT:AnRpackageforIRTtestequating.JournalofStatisticalSoftware,
68(7),1–22.
Choi, S., Gibbons, L., & Crane, P. (2011). lordif: An R package for detecting differential item
functioningusingiterativehybridordinallogisticregression/itemresponsetheoryandmonte
carlosimulations.JournalofStatisticalSoftware,39(8),1–30.
ETS(2011).KE-software(Version3)[Computersoftware].
González,J.(2014).SNSequate:Standardandnonstandardstatisticalmodelsandmethodsfortest
equating.JournalofStatisticalSoftware,59(7),1–30.
Kolen,M.,&Brennan,R.(2014).Testequating,scaling,andlinking:Methodsandpractices(3rd
ed.).NewYork:Springer.
Moses, T., & von Davier, A. (2011). A SAS IML macro for loglinear smoothing. Applied
PsychologicalMeasurement,35(3),250–251.
RCoreTeam(2016).R:Alanguageandenvironmentforstatisticalcomputing.Vienna,Austria:R
FoundationforStatisticalComputing.
vonDavier,A.A.,Holland,P.,&Thayer,D.(2004).Thekernelmethodoftestequating.NewYork:
Springer.
Weeks,J.P.(2010).plink:AnRpackageforlinkingmixed-formattestsusingIRT-basedmethods.
JournalofStatisticalSoftware,35(12),1–33.
Contents
1 GeneralEquatingTheoryBackground.................................... 1
1.1 Introduction............................................................ 1
1.1.1 AConceptualDescriptionofEquating...................... 2
1.1.2 AStatisticalModelViewofEquating....................... 2
1.2 StatisticalModels ..................................................... 3
1.2.1 GeneralDefinition,Notation,andExamples................ 3
1.2.2 TypesofStatisticalModels .................................. 4
1.2.3 MathematicalStatisticsFormulationofthe
EquatingProblem ............................................ 6
1.2.4 MathematicalFormoftheEquatingTransformation....... 7
1.2.5 Continuization................................................ 8
1.2.6 RequirementsforComparabilityofScores ................. 9
1.2.7 AssessingtheUncertaintyofEquatingResults............. 9
1.3 CollectingDatainEquating .......................................... 10
1.3.1 DataCollectionDesignsinEquating........................ 11
1.4 SomeExamplesofEquatingTransformations....................... 13
1.4.1 TheEquipercentileEquatingFunction...................... 13
1.4.2 TheLinearEquatingFunction............................... 14
1.4.3 TheKernelEquatingFunction............................... 14
1.5 RPackagesThatAreUsedinThisBook............................ 15
1.6 SummaryandOverviewoftheBook................................. 15
References..................................................................... 16
2 PreparingScoreDistributions.............................................. 19
2.1 Data.................................................................... 19
2.1.1 DatafromKolenandBrennan(2014)....................... 19
2.1.2 DatafromvonDavieretal.(2004).......................... 20
2.1.3 TheADMAdmissionsTestData ............................. 20
2.1.4 TheSEPATestData.......................................... 21
2.2 PreparingtheScoreData.............................................. 21
2.2.1 FunctionstoCreateScoreFrequencyDistributions ........ 22
2.2.2 ScoreDataintheEGDesign ................................ 22
2.2.3 ScoreDataintheSGDesign ................................ 27
2.2.4 ScoreDataintheNEATDesign............................. 30
2.3 PresmoothingtheScoreDistributions................................ 33
2.3.1 PolynomialLog-LinearModelsforPresmoothing ......... 33
2.3.2 PolynomialLog-LinearSmoothinginequate.............. 35
2.3.3 Examples ..................................................... 36
2.3.4 ChoosingtheBestLog-LinearModel....................... 38
2.4 UsingOtherArguments,PackagesandFunctions................... 41
2.5 Summary .............................................................. 42
References..................................................................... 42
3 TraditionalEquatingMethods ............................................. 43
3.1 Equipercentile,Linear,andMeanEquatingTransformations....... 43
3.2 AssumptionsintheDifferentDesigns ............................... 44
3.2.1 AssumptionsinEG,SG,andCBDesigns .................. 44
3.2.2 AssumptionsintheNEATDesign........................... 45
3.3 TraditionalEquatingMethodsfortheEG,SGandCBDesigns.... 46
3.4 TraditionalEquatingMethodsfortheNEATDesign................ 46
3.4.1 LinearEquatingMethodsfortheNEATDesign............ 47
3.4.2 EquipercentileEquatingMethodsfortheNEATDesign... 50
3.5 ExampleswiththeequateFunction ............................... 52
3.5.1 TheequateFunction....................................... 52
3.5.2 ExamplesUndertheEGandSGDesigns................... 53
3.5.3 ExamplesUndertheNEATDesign ......................... 60
3.5.4 ExamplesUsingtheADMDataUnderthe
NEATDesign................................................. 63
3.6 AdditionalFeaturesinequate........................................ 63
3.7 PerformingTraditionalEquatingMethodswithSNSequate ....... 64
3.8 ComparingTraditionalTestEquatingMethods...................... 65
3.8.1 BootstrapStandardErrorsofEquating...................... 65
3.8.2 BiasandRMSE .............................................. 66
3.8.3 ExamplesUsingequate...................................... 67
3.8.4 Additional Example: A Comparison of
TraditionalEquatingMethods............................... 68
3.9 Summary .............................................................. 71
References..................................................................... 71
4 KernelEquating.............................................................. 73
4.1 AQuickOverviewofKernelEquating .............................. 73
4.2 Step1:Presmoothing ................................................. 74
4.2.1 PresmoothingwithSNSequate.............................. 74
4.2.2 Presmoothingwithkequate ................................. 81
4.2.3 AssessingLog-LinearModelFit ............................ 86
