Table Of ContentMethodology of Educational Measurement and Assessment
Jorge González
Marie Wiberg
Applying
Test Equating
Methods
Using R
Methodology of Educational Measurement
and Assessment
Serieseditors
Bernard Veldkamp, Research Center for Examinations and Certification (RCEC),
UniversityofTwente,Enschede,TheNetherlands
MatthiasvonDavier,NationalBoardofMedicalExaminers(NBME),Philadelphia,
USA1
1ThisworkwasconductedwhileM.vonDavierwasemployedwithEducationalTestingService.
This new book series collates key contributions to a fast-developing field of
educationresearch.Itisaninternationalforumfortheoreticalandempiricalstudies
exploring new and existing methods of collecting, analyzing, and reporting data
from educational measurements and assessments. Covering a high-profile topic
from multiple viewpoints, it aims to foster a broader understanding of fresh
developments as innovative software tools and new concepts such as competency
models and skills diagnosis continue to gain traction in educational institutions
around the world. Methodology of Educational Measurement and Assessment
offers readers reliable critical evaluations, reviews and comparisons of existing
methodologies alongside authoritative analysis and commentary on new and
emergingapproaches.Itwillshowcaseempiricalresearchonapplications,examine
issues such as reliability, validity, and comparability, and help keep readers up to
speedondevelopmentsinstatisticalmodelingapproaches.Thefullypeer-reviewed
publications in the series cover measurement and assessment at all levels of
educationandfeatureworkbyacademicsandeducationprofessionalsfromaround
the world. Providing an authoritative central clearing-house for research in a core
sector in education, the series forms a major contribution to the international
literature.
Moreinformationaboutthisseriesathttp://www.springer.com/series/13206
Jorge González • Marie Wiberg
Applying Test Equating
Methods
Using R
123
JorgeGonzález MarieWiberg
FacultyofMathematics DepartmentofStatistics
PontificiaUniversidadCatólicadeChile UmeåSchoolofBusinessandEconomics
Santiago,Chile UmeåUniversity
Umeå,Sweden
ISSN2367-170X ISSN2367-1718 (electronic)
MethodologyofEducationalMeasurementandAssessment
ISBN978-3-319-51822-0 ISBN978-3-319-51824-4 (eBook)
DOI10.1007/978-3-319-51824-4
LibraryofCongressControlNumber:2017932073
©SpringerInternationalPublishingAG2017
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
ToJavieraandRenato
–J.G.
ToEmelieandHanna
–M.W.
Foreword
Thisbooksignalsthematurityofthemethodologies andtechnology thatsurround
test score equating and the acceptance of these methodologies and of the
open-sourcesoftwarebythetestingindustry’sexpertsandacademia.
Testequatingisastatisticalprocedureforadjustingthetestformdifferencesin
standardized testing. Test equating methodologies originated in testing companies
many decades ago. Due to the pragmatic operational perspective, many of these
methodologies had lacked rigorous theoretical background and research for many
years. Then in the 1980s, test score equating started to receive the appropriate
attention from researchers and academia (Holland and Rubin 1982). Several
notable journal articles and books were written at that time, and the field was
brought forward significantly. The availability of personal and fast computers
also facilitated the implementation of the established equating methods. For
example, psychometricians and technology specialists at Educational Testing
Service developed GENASYS, a very comprehensive, modular, and sophisticated
software system to support the large-scale application of test score analysis and
equating with all the available methodologies at that time. The first edition of the
Kolen and Brennan (1995) book became a landmark in equating. A second wave
oftheoreticalinterestandthedevelopmentofnewmethodstookplacearound2005
(KolenandBrennan2004;vonDavieretal.2004;Doransetal.2007).
Since2005,therehasbeenanoticeableshiftintestequating.First,therehasbeen
anincreasinginterestinanduseoftestequatingallovertheworld.Second,asitwas
illustrated in the edited volume of von Davier (2011a), there has been a renewed
interestinsearchingforbettermethodologiesthatimprovethescoreconversionin
termsofbias(theobserved-scoreequatingframework,variouskernelmethods,local
equating), in terms of error (kernel equating, equating with exponential families),
in terms of hypothesis testing (Lagrange multiplier tests for item response theory
(IRT) methods, Wald test for kernel methods), in terms of approximations for
small sample equating, and in terms of monitoring the quality control of equating
results (using time series approaches); third, there has been an increased interest
in equating by university professors in statistics and educational measurement;
and more importantly, there has been a significant software development using
vii
viii Foreword
different software environments, from SAS to R. Moreover, the rapid changes
in the technology made the upgrade of proprietary software challenging; the
society became more appreciative of the open-source environment that facilitates
transparency,research,andcommunitywork.
Thisvolumeisremarkablefromseveralperspectives.First,itisaperfectmirror
of the changes in test equating during recent years: it includes new equating
methodsalongsidetheolderones,anditprovidescodeinRforusers.Theauthors
of this volume have both an academic background (one works for the Pontificia
Universidad Católica de Chile, in Santiago de Chile, and one works for Umeå
University in Sweden) and a practitioner background (one is a consultant for the
operationaltestsfromthemeasurementcenterMIDEUCandtheotherconsultsfor
theoperational large-scale SwedishSAT).Bothauthors developed orcodeveloped
R packages for conducting various equating methods. Last but not least, one is
from Chile and one is from Sweden. Again, the technology makes the world
interconnectedandsupportscollaborationslikethese!
The structure of this volume follows the process of operationalizing various
test equating methods. The volume starts by locating equating in the realm of
statistical models, which was also attempted by von Davier (2011b). Then the
volumecontinuesbybrieflypresentingthetheorybehindthemethods(smoothingof
discretetestscoredata,linearinterpolationequating,severalkernelmethods,local
andIRTequating)andbyillustratingtheapplicationofthesemethodswithsetsof
real data and with various R packages, including those developed by the authors
themselves.
Forthoseofuswhoworkoneducationalassessments,therearestillmanyopen
challenges. Measurement accuracy for all groups of test takers across the range
of scores is hard to ensure in operational testing. With the increased shifts in
demographics,wehadtoprovidemoresophisticatedandcomplex,yettransparent,
typesoftests.Nevertheless,aswetestmorediversegroupsofpeople,thechallenge
of providing accurate scores increases. For example, much of the research on
alternativekernels,inparticularonadaptivekernels,hasbeenmotivatedbytheneed
to analyze data from heterogeneous populations. Another way to look at fairness
is through the lenses of transparency of the process: Will the transparency of the
open-sourcesoftwareaidthefairnessarguments?
Another area that remains of interest for future research is how to ensure that
the computations with the open-source software environment are well tested and
error-freeforlarge-scalehigh-stakestestsadministeredovertime.Inthefuture,we
willalsoneedtoprovideopen-sourcecodeformethodsforqualitycontrol.
Asmanyofushavestartedtoworkonsimulation-andgame-basedassessments,
newchallengesoccurred.Tomeetthesechallenges,weneedtoredefinetheconcept
of test equating and the notion of adjusting for difficulty across test forms. While
wedonothaveasolutionyet,itisquiteprobablethatthesolutionwillusesomeof
thesenewerapproachestotestequatingandopen-sourcesoftware.
A final challenge relates to improved communication about the practical
consequences of addressing equating issues as an ongoing activity in order to
ensure meaningful scores. We need to explain the limitations of the methods and
Foreword ix
the cost of being able to make truthful claims about score accuracy for all groups
over time. The authors of this book examine theoretical and practical issues that
cut across different types of methodological and operational situations and open
the doors to hands-on teaching and learning and to more theoretical research on
equatingmethodsconductedbystudentsandotherresearchers.
Princeton,USA AlinaA.vonDavier
December2016
References
Dorans,N.J.,Pommerich,M.,&Holland,P.W.(2007).Linkingandaligningscoresandscales.
NewYork:Springer.
Holland,P.,&Rubin,D.(1982).Testequating.NewYork:AcademicPress.
Kolen, M., & Brennan, R. (1995). Test equating: Methods and practices (1st ed.). New York:
Springer.
Kolen,M.,&Brennan,R.(2004).Testequating,scaling,andlinking:Methodsandpractices(2nd
ed.).NewYork:Springer.
von Davier, A. (2011a). Statistical models for test equating, scaling, and linking. New York:
Springer.
vonDavier,A.(2011b).Astatisticalperspectiveonequatingtestscores.InA.vonDavier(Ed.),
Statisticalmodelsfortestequating,scaling,andlinking(Vol.1,pp.1–17).NewYork:Springer.
vonDavier,A.A.,Holland,P.,&Thayer,D.(2004).Thekernelmethodoftestequating.NewYork:
Springer.
Preface
This book provides an overview of test equating methods with a focus on how to
implementthemusingR(RCoreTeam2016).
TheideaforwritingthisbookemergedinApril2014whenwebothattendedthe
NCME conference in Philadelphia. There are many reasons to go to conferences
including presenting your research, meeting other researchers, working on
collaborative projects, and getting new ideas. For us, writing a book on equating
with applications using R was a natural step as we had both been involved in the
development of two different R packages, kequate (Andersson et al. 2013) and
SNSequate (González 2014), and we have noticed that a number of software and
R packages that can be used when equating test scores have emerged in recent
years. Weeks (2010) (and references therein) offers a list that includes EQUATE,
ST, mcmequate, IpLink, POLYST, STUIRT, POLYEQUATE, IRTEQ, irtoys,
and MischPsycho. These software packages are mainly designed for conducting
item response theory (IRT) linking rather than equating, with the exception of
POLYEQUATEandIRTEQthatimplementIRTequating.Thelistcanbeenlarged
by including CIPE, Equating Recipes, RAGE-RGEQUATE, Equating Error,
PIE, and LEGS (Kolen and Brennan 2014), the KE software (ETS 2011), the
SAS/IML macro for log-linear smoothing (Moses and von Davier 2011), and the
Rpackagesplink(Weeks2010),lordif(Choietal.2011),equate(Albano2016),
andequateIRT(Battauz2015).
Twoquestionsofinteresttouswerewhetheralloftheavailableequatingmethods
had been implemented in these packages and if we could replicate known results
usingallofthesedifferentpackages.Whenwestartedwritingthisbook,neitherof
us had used all of these packages, but we got to learn many things and got many
ideas for new research along the way. By using an open-source software, a new
generation can easily start their research where the previous generation left off.
Thus, we expect many new research papers within the equating field in the next
fewyears.
People in the test equating field might wonder what is new and unique in this
equatingbookcomparedwithotherssuchastheworksofvonDavieretal.(2004)
and Kolen and Brennan (2014). This is the first equating book that provides all
xi
