Table Of ContentJorge Manzi
María Rosa García
Sandy Taut Editors
Validity
of Educational
Assessments in
Chile and Latin
America
Validity of Educational Assessments in Chile
and Latin America
· ·
Jorge Manzi María Rosa García Sandy Taut
Editors
Validity of Educational
Assessments in Chile
and Latin America
Editors
JorgeManzi MaríaRosaGarcía
PontificiaUniversidadCatólicadeChile PontificiaUniversidadCatólicadeChile
Santiago,Chile Santiago,Chile
SandyTaut
MinistryofEducation
Gunzenhausen,Germany
ISBN978-3-030-78389-1 ISBN978-3-030-78390-7 (eBook)
https://doi.org/10.1007/978-3-030-78390-7
JointlypublishedwithEdicionesUC
TranslationfromtheSpanishlanguageedition:ValidezDeEvaluacionesEducacionalesEnChileYLati-
noaméricabyJorgeManzi,MaríaRosaGarcíaandSandyTaut,©EdicionesUniversidadCatólicade
Chile,2019.OriginalPublicationISBN978-956-14-2471-5.Allrightsreserved.
PreviousEdition:TranslationfromtheSpanishlanguageedition:Validezdeevaluacioneseducacionales
enChileyLatinoaméricabyJorgeManzi,etal.,©EdicionesUC2019.PublishedbyEdicionesUC.All
RightsReserved.
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SwitzerlandAG2021
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsofreprinting,reuseofillustrations,
recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor
informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublishers,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishersnortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublishersremainneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Contents
1 Introduction ................................................. 1
JorgeManzi,MaríaRosaGarcía,andSandyTaut
2 Is Validation a Luxury or an Indispensable Asset
forEducationalAssessmentSystems? ........................... 17
SandyTautandSiugminLay
PartI ValidityofStudentLearningAssessments
3 HowtoEnsuretheValidityofNationalLearningAssessments?
PriorityCriteriaforLatinAmericaandtheCaribbean ........... 37
MaríaJoséRamírezandGilbertA.Valverde
4 Contemporary Practices in the Curricular Validation
of National Learning Assessments in Latin America:
AComparativeStudyofCasesfromChile,Mexico,andPeru ...... 69
GilbertA.ValverdeandMaríaJoséRamírez
5 LearningProgressAssessmentSystem,SEPA:EvidenceofIts
ReliabilityandValidity ........................................ 93
AndreaAbarzúaandJohanaContreras
6 Validation Processes of the National Assessment
of Educational Achievements—Aristas: The Experience
oftheINEEd(Uruguay) ....................................... 117
CarmenHaretcheandMarianoPalamidessi
7 TheValidityandSocialLegitimacyoftheChileanNational
Assessment of Learning Outcomes (SIMCE): The Role
ofEvidenceandDeliberation ................................... 145
LorenaMeckesandMaríaAngélicaMena
v
vi Contents
PartII ValidityofInternationalAssessments
8 TestComparabilityandMeasurementValidityinEducational
Assessment ................................................... 173
JorgeGonzálezandRenéGempp
9 MeasurementofFactorInvarianceinLarge-ScaleTests ........... 205
VíctorPedrero
10 InvarianceofSocioeconomicStatusScalesinInternational
Studies ....................................................... 235
ErnestoTreviño,AndrésSandoval-Hernández,DanielMiranda,
DavidRutkowski,andTylerMatta
11 Measuring Attitudes Towards Gender Equality
inInternationalTests:ImplicationsforTheirValidity ............ 259
JuanCarlosCastillo,DanielMiranda,andAngélicaBonilla
PartIII ValidityofAdmissionsandCertificationAssessments
12 Validity of Assessment Systems for Admissions
andCertification .............................................. 283
MaríaVerónicaSantelices
13 IstheSABER11thTestValidasaCriterionforAdmission
toColombianUniversities? ..................................... 307
JuliánP.Mariño,AdrianaMolina,andYadiraGómez
14 ValidityEvidenceoftheUniversityAdmissionTestsinChile:
PruebadeSelecciónUniversitaria(PSU) ........................ 331
JorgeManziandDiegoCarrasco
15 Validity of the Single National Examination of Medical
Knowledge(Eunacom) ......................................... 353
BeltránMena
PartIV ValidityofTeacherEvaluations
16 TeacherEvaluationwithMultipleIndicators:Conceptual
andMethodologicalConsiderationsRegardingValidity ........... 373
JoséFelipeMartínezandMaríaPazFernández
17 How Valid Are the Content Knowledge and Pedagogical
ContentKnowledgeTestResultsoftheTeacherProfessional
DevelopmentSysteminChile? .................................. 395
EdgarValencia,MarthaKluttig,andBeatrizRodríguez
Contents vii
18 The Portfolio in the National Teacher Evaluation
System in Chile: Collecting Evidence of Validity as Part
oftheInstrumentConstructionProcess ......................... 431
DavidTorresIrribarraandÁlvaroZapata
Chapter 1
Introduction
JorgeManzi,MaríaRosaGarcía,andSandyTaut
1.1 ABookonValidity
EducationalassessmentshavehadastrongdevelopmentinLatinAmericainrecent
years.Almostallcountriesintheregionregularlyassesstheirstudents’achievement.
Severalalsodosoforselectionpurposes—forexample,accesstohighereducation.
Itisalsoincreasinglycommonthedevelopmentofassessmentsthataddressteacher
performance.Moreover,internationalassessmentshaveshowngreatimpact,which
hasledtoasustainedincreaseinthenumberofcountriesintheregionthatpartici-
pateinandusePISA,TIMMSandERCEresults.Theresultsoftheseassessments,
in addition to the direct use assigned to them according to the declared purposes
(training,diagnosis,selection,certificationandpromotion),conformafundamental
antecedent for judging the state of progress of education systems, as well as for
makingcomparisonsbetweengroups(forexample,menversuswomenandsocioe-
conomicandregionalgroups),andforestimatingtrendsineducationalachievement
overtime.Asaresult,assessmentshavebecomecentraltoolsforeducationalpolicy,
andstronglyinfluenceopinionsthattheeliteandcitizenshaveabouttheeducational
system.
Inthiscontext,assessmentsintheregionhavereachedacertainlevelofmaturity
as have the development of technical capacities to design, implement and use the
resultsofassessments.Atthesametime,theyhavebeengivenincreasinglybroader
usesthathavegreaterimpactattheindividual,institutionalandnationallevels.Thus,
B
J.Manzi·M.R.García( )
PontificiaUniversidadCatólicadeChile,Santiago,Chile
e-mail:[email protected]
J.Manzi
e-mail:[email protected]
S.Taut
MinistryofEducationofBavaria,Munich,Germany
e-mail:[email protected]
©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2021 1
J.Manzietal.(eds.),ValidityofEducationalAssessmentsinChileandLatinAmerica,
https://doi.org/10.1007/978-3-030-78390-7_1
2 J.Manzietal.
thefundamentalquestionthisbookseekstoansweris:Istheinformationprovided
bythesetestsandassessmentprogramsofsufficientqualitytomakeinterpretations
andproposedusesvaluableanddefensible?
Itisknownthatassessmentsshouldnotbeusedintheeducationalsphereifthere
are no studies to support the validity of the interpretation of the scores, the use of
suchtests,aswellasthepossibilityofmakingcomparisonsoftheirscoresovertime
andbetweengroups.However,thereisaconsensusthat,inspiteoftherecognition
giventovalidity,relativelyfeweducationalassessmentprogramshaveaccumulated
sufficientevidencetosupporttheirinterpretationsanduses.
The problem is that without validity evidence, it is not feasible to know how
the scores or trends resulting from those assessments should be interpreted, nor is
it possible to support the uses of those scores, even if they are explicitly reported
orembodiedinalegalbody.Inshort,theabsenceofvalidityevidencerepresentsa
seriousthreattoassessments,compromisingtheirvalueaswellastheirpoliticaland
technicalviability.
WithintheframeworkoftheStandardsforEducationalandPsychologicalTesting
revision,publishedin2014(AmericanEducationalResearchAssociation[AERA]
et al., 2014), this book addresses conceptual, methodological and applied aspects
related to the efforts made in several countries to validate different types of large-
scaleeducationalassessments.Theseeducationalassessmentsrangefromtraditional
teststhatassessstudentlearningachievementtorecentassessmentsofnon-cognitive
aspects, teacher assessments, as well as certification and selection tests. Addition-
ally,thebookcompilestheexperienceofvaliditystudiesregardingthemaininterna-
tionalprogramspresentinLatinAmerica(PISA,TIMSS,ERCE,ICCS).Finally,it
revealsthechallengesthatmustbeconsideredwhenassessmentsareusedtocompare
countries,groupsorachievementtrendsovertime.
Thisbookmakesauniquecontributioninourregionduetoseveralreasons.One
isthefactthatthisbookreflectsamorecontemporaryviewofvalidity.According
to the Standards, validity refers to the amount of evidence that exists to support
particularinterpretationsforthespecificproposedusesoftestscores.Thus,itisnot
about establishing validity of an assessment in a general way, since what needs to
bevalidatedistheinterpretationanduseofthescoresinagivencontextandfora
specifiedpopulation.
Consequently, as stated by Kane (2006), validation is a process that involves
accumulating evidence that allows us to articulate (or refute) a validity argument.
Validationbeginsbyexplicitlystatingtheintendedinterpretationsanduses.Inthis
context,validityisabroadandflexibleconcept.Thebookillustratesthisaspectby
includingchaptersthatmakeaconceptualcontributiontovalidity,asdescribedlater
on,andotherchaptersthatreviewnotonlyempiricalevidenceoftestscoresbutalso
theprocessofdevelopmentofassessmentandtheirconsequencesshowingthebroad
meaningofvalidityasitiscurrentlyunderstood.
A second contribution of this book is based on the fact that validity has been
recognized as a requirement that should be asumed by those who decide to create
and use assessment systems. The Standards are clear on this issue and state that
test developers and those who decisions based on test results are responsible for
1 Introduction 3
conductingvalidationstudies.Therefore,itisessentialtotakevalidityintoaccount
fromthemomentadecisionismadeinordertodevelopanassessmentprogram.This
impliesthatresourceshavetobeallocatedtoprovidesufficientevidencetosupport
orrefutetheinterpretationanduseoftheinstrumentsinvolved.Thisrequirementis
especiallyrelevantinthecaseofassessmentsthathavehighimpactforindividuals,
groups,orinstitutionsbasedontheresultsofeducationalassessments.
Thirdly,thisbookillustrateshowdifferentcountrieshaveaddressedthecollection
ofvalidityevidenceinvariouseducationalassessmentsystems.Inthisway,thisbook
seekstoreaffirmtheneedfordecision-makerstoachievesufficientawarenessonthis
matter, so that development and use of assessments, especially those with highest
stakes,havesufficientsupportforthedecisionsthataremadebasedonthem.
Wewillreviewthebook’smaincontributionsinmoredetailbelow,considering
(i)conceptualcontributions;(ii)technicalcontributions;and(iii)chaptersthatillus-
tratevalidationprocessesbasedonmultipleorspecificevidence.Thus,thedifferent
chapterswillbecoveredaccordingtothecontributiontheymake.Thispresentation
doesnotfollowthebook’sTableofContentsorder.
1.2 ConceptualContributions
The book contains chapters that constitute important conceptual contributions
regardingvalidityinparticularspheres.Thechapterof TautandLay,IsValidation
a Luxury or an Indispensable Asset for Educational Assessment Systems? Initi-
atesthisvolumewithaconceptualdiscussiononvalidityasitreviewsthehistoryof
theconcept—consideringthecontributionsofmainreferents:Messick(1989,1994,
1995),Cronbach(1989)andKane(1992)—andestablishesthecurrentdefinitionof
thisconstructaccordingtotheinternationalStandardsforEducationalandPsycho-
logicalTesting(AERAetal.,2014).BasedontheStandards,itsummarizesthefive
areasinwhichitispossibletocollectvalidityevidenceonassessmentsandarebased
on (i) test content; (ii) cognitive processes; (iii) internal structure; (iv) relations to
other variables; and (v) test consequences. Regarding this last source of evidence,
theauthorsdiscussthecontroversythathasexistedamongthosewhoarguethatthe
studyoftheconsequencesofassessmentsshouldnotbepartoftheconceptofvalidity
(Borsboom et al., 2004; Cizek, 2012; Maguire et al., 1994; Popham, 1997; Wiley,
1991), and those who, in contrast, consider it relevant to study both intended and
unintended consequences as part of validity, especially in high-stakes assessments
for examinees (Kane, 2013; Lane & Stone, 2002; Lane et al., 1998; Linn, 1997;
Messick,1989,1995;Shepard,1997).
This chapter’s contribution also relies on explaining how these different pieces
of evidence should be integrated into a line of argumentation on validity, and the
differentactorsthatneedtobeinvolvedintheprocess.Finally,theauthorsanalyze
the political dimension of validation, pointing out the main obstacles that need to
be tackled when research, policy and practice converge in the development of a
validationprogram.
