Table Of ContentMethodology of Educational Measurement and Assessment
Débora B. Maehler
Beatrice Rammstedt Editors
Large-Scale
Cognitive
Assessment
Analyzing PIAAC Data
Methodology of Educational Measurement
and Assessment
SeriesEditors
BernardVeldkamp,ResearchCenterforExaminationsandCertification(RCEC),
UniversityofTwente,Enschede,TheNetherlands
MatthiasvonDavier,NationalBoardofMedicalExaminers(NBME),Philadelphia,
USA
EditorialBoardMembers
ClausH.Carstensen,UniversityofBamberg,Bamberg,Germany
Hua-HuaChang,PurdueUniversity,WestLafayette,USA
HongJiao,UniversityofMaryland,CollegePark,USA
DavidKaplan,UniversityofWisconsin-Madison,Madison,USA
JonathanTemplin,TheUniversityofIowa,IowaCity,USA
AndriesvanderArk,ResInstofChildDevt&Education,UniversityofAmsterdam,
Amsterdam,TheNetherlands
This book series collates key contributions to a fast-developing field of education
research.Itisaninternationalforumfortheoreticalandempiricalstudiesexploring
new and existing methods of collecting, analyzing, and reporting data from
educational measurements and assessments. Covering a high-profile topic from
multipleviewpoints,itaimstofosterabroaderunderstandingoffreshdevelopments
asinnovativesoftwaretoolsandnewconceptssuchascompetencymodelsandskills
diagnosis continue to gain traction in educational institutions around the world.
Methodology of Educational Measurement and Assessment offers readers reliable
critical evaluations, reviews and comparisons of existing methodologies alongside
authoritative analysis and commentary on new and emerging approaches. It will
showcase empirical research on applications, examine issues such as reliability,
validity, and comparability, and help keep readers up tospeed on developments in
statistical modeling approaches. The fully peer-reviewed publications in the series
cover measurement and assessment at all levels of education and feature work
by academics and education professionals from around the world. Providing an
authoritative central clearing-house for research in a core sector in education, the
seriesformsamajorcontributiontotheinternationalliterature.
Moreinformationaboutthisseriesathttp://www.springer.com/series/13206
Débora B. Maehler • Beatrice Rammstedt
Editors
Large-Scale Cognitive
Assessment
Analyzing PIAAC Data
Editors
DéboraB.Maehler BeatriceRammstedt
SurveyDesignandMethodology SurveyDesignandMethodology
GESIS–LeibnizInstitutefortheSocial GESIS–LeibnizInstitutefortheSocial
Sciences Sciences
Mannheim,Germany Mannheim,Germany
ISSN2367-170X ISSN2367-1718 (electronic)
MethodologyofEducationalMeasurementandAssessment
ISBN978-3-030-47514-7 ISBN978-3-030-47515-4 (eBook)
https://doi.org/10.1007/978-3-030-47515-4
©TheEditor(s)(ifapplicable)andTheAuthor(s)2020.Thisbookisanopenaccesspublication.
Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate
credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicateifchangesweremade.
The images or other third party material in this book are included in the book’s Creative Commons
license,unlessindicatedotherwiseinacreditlinetothematerial.Ifmaterialisnotincludedinthebook’s
CreativeCommonslicenseandyourintendeduseisnotpermittedbystatutoryregulationorexceedsthe
permitteduse,youwillneedtoobtainpermissiondirectlyfromthecopyrightholder.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG.
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Contents
1 Large-ScaleAssessmentinEducation:AnalysingPIAACData....... 1
DéboraB.MaehlerandBeatriceRammstedt
2 DesignandKeyFeaturesofthePIAACSurveyofAdults ............. 7
IrwinKirsch,KentaroYamamoto,andLaleKhorramdel
3 Plausible Values: Principles of Item Response Theory
andMultipleImputations ................................................. 27
Lale Khorramdel, Matthias von Davier, Eugenio Gonzalez, and
KentaroYamamoto
4 Adult Cognitive and Non-cognitive Skills: An Overview
ofExistingPIAACData.................................................... 49
DéboraB.MaehlerandIngoKonradt
5 AnalysingPIAACDatawiththeInternationalDataExplorer
(IDE) ......................................................................... 93
EmilyPawlowskiandJalehSoroui
6 AnalysingPIAACDatawiththeIDBAnalyzer(SPSSandSAS) ..... 117
AndresSandoval-HernandezandDiegoCarrasco
7 AnalysingPIAACDatawithStata........................................ 149
FrançoisKeslair
8 AnalysingPIAACDatawithStructuralEquationModelling
inMplus ..................................................................... 165
RonnyScherer
9 UsingEdSurveytoAnalysePIAACData................................ 209
PaulBailey,MichaelLee,TrangNguyen,andTingZhang
v
vi Contents
10 AnalysingLogFileDatafromPIAAC ................................... 239
FrankGoldhammer,CarolinHahnel,andUlfKroehne
11 Linking PIAAC Data to Individual Administrative Data:
InsightsfromaGermanPilotProject.................................... 271
JessicaDaikeler,BrittaGauly,andMatthiasRosenthal
Reviewers
AlexanderRobizsch(LeibnizInstituteforScienceandMathematicsEducation/IPN,
Germany)
Anna-LenaKeute(StatisticsNorway,Norway)
AnoukZabal(GESIS−LeibnizInstitutefortheSocialSciences,Germany)
BrittaGauly(GESIS−LeibnizInstitutefortheSocialSciences,Germany)
DéboraB.Maehler(GESIS−LeibnizInstitutefortheSocialSciences,Germany)
EugenioGonzales(EducationalTestingService/ETS,US)
IsabelleThony(StatisticsCanada,Canada)
Jamis He Jia (Leibniz Institute for Research and Information in Education/DIPF,
Germany)
JanHeisig(BerlinSocialScienceCenter/WBZ,Germany)
Jan-PhillipKolb(GESIS−LeibnizInstitutefortheSocialSciences,Germany)
JensBender(GESIS−LeibnizInstitutefortheSocialSciences,Germany)
JuliaGorges(MarburgUniversity,Germany)
LottaLarsson(StatisticsSweden,Sweden)
ManuelReif(StatisticAustria,Austria)
NataschaMassing(GESIS−LeibnizInstitutefortheSocialSciences,Germany)
PaulSatherley(MinistryofEducation,NewZealand)
Ralph Carstensen (International Association for the Evaluation of Educational
Achievement/IEA,Germany)
SaidaMamedova(AmericanInstitutesforResearch/AIR,US)
SanjaKapidzic(GESIS−LeibnizInstitutefortheSocialSciences,Germany)
SilkeMartin(GESIS−LeibnizInstitutefortheSocialSciences,Germany)
SilkeSchneider(GESIS−LeibnizInstitutefortheSocialSciences,Germany)
SimonWiederhold(CatholicUniversityofEichstätt-Ingolstadt,Germany)
TobiasKoch(PsychologischeHochschuleBerlin,Germany)
Valentina Gualtieri (Istituto Nazionale per l’Analisi delle Politiche Pub-
bliche/INAPP,Italy)
vii
Chapter 1
Large-Scale Assessment in Education:
Analysing PIAAC Data
DéboraB.MaehlerandBeatriceRammstedt
Abstract This methodological book aims to summarise existing techniques for
analysing data resulting from the Programme for the International Assessment of
Adult Competencies (PIAAC). The present chapter provides an overview of the
programme,outliningitsgoal,theparticipatingcountries,thesurveycycles,andthe
research questions that can be addressed with the available data. In addition, the
structureofthetextbookisdescribed.
1.1 Large-ScaleAssessmentinEducation: AnalysingPIAAC
Data
To actively participate in modern society, skills such as literacy and numeracy are
ofutmostimportance.Essentialinformationisusuallyprovidedinwrittenformat—
for example, in manuals, memos, and medication package inserts. To adequately
processandreacttosuchinformation,individualsrequiresufficientskills.
In2008,theOrganisationforEconomicCo-operationandDevelopment(OECD)
initiated the Programme for the International Assessment of Adult Competencies
(PIAAC). PIAAC aims to assess in an internationally comparable way basic adult
skillssuchasliteracy,numeracy,andproblemsolving.Theseskillsareconsideredto
beessentialforsuccessfulparticipationinmodernsocietyandtobeafoundationfor
developingnumerousother,morespecific,skillsandcompetencies(OECD2013).
PIAAC provides information about the skill levels of the adult population in
the participating countries and the extent to which countries differ in terms of
these skills. Moreover, and in particular, PIAAC provides information on factors
associated with the acquisition, maintenance, and outcomes of these skills. Thus,
D.B.Maehler((cid:2))·B.Rammstedt
GESIS–LeibnizInstitutefortheSocialSciences,Mannheim,Germany
e-mail:[email protected]
©TheAuthor(s)2020 1
D.B.Maehler,B.Rammstedt(eds.),Large-ScaleCognitiveAssessment,
MethodologyofEducationalMeasurementandAssessment,
https://doi.org/10.1007/978-3-030-47515-4_1
2 D.B.MaehlerandB.Rammstedt
it sheds light on effects of these basic skills on social and, in particular, economic
participation.
Like the OECD Programme for International Student Assessment (PISA),
PIAAC is designed as a cross-sectional study to be repeated at regular intervals.
The first cycle of PIAAC started in 2008 and comprised three rounds, in which a
totalof38countriesparticipated.ThesecondcycleofPIAACwaslaunchedin2018
andwilllikelycover33countries(seeTable1.1).
The OECD consistently pursues an open science strategy regarding the data
resultingfromPIAAC.Todate,morethan60PIAACdatasetshavebeenpublished
worldwide(seeChap.4inthisvolume);thefirstdatawerereleasedin2013(OECD
2013). These datasets have been widely accessed and used by an interdisciplinary
researchcommunity(foranoverview,seeMaehleretal.2020).Furthermore,there
arealargeandincreasingnumberofPIAAC-basedpublications.1
As in the case of other international large-scale assessments (Rutkowski et al.
2014), analyses with the PIAAC data are very challenging for users due to the
complex data structure (e.g. plausible values computed by imputation, complex
sampling). To ensure the quality and significance of the data analyses, users
requireinstructioninthecorrecthandlingofthedata.Thismethodologicaltextbook
therefore aims to summarise existing techniques for analysing PIAAC data. It
provides a standardised approach to successfully implement these data analyses.
The present volume provides examples and tools for the analysis of PIAAC data
using different statistical approaches and software and also offers perspectives
from various disciplines. The textbook is designed for use by researchers and
students from diverse fields (e.g. educational research, economics, sociology, and
psychology)whoareinterestedinworkingwithlarge-scaleeducationalassessment
data.
This methodological textbook covers the following topics: (1) background
information on PIAAC that is required for the analyses—for example, the design
ofPIAACandtheavailabledatasets;(2)the(web)toolsavailablefortheanalysisof
PIAACdata,particularlythepublicusefiles;and(3)theanalysisofcross-sectional
PIAACdatawithmultidisciplinarymethods(e.g.StataorR)usingpublicusefiles
orscientificusefiles.
Thenextthreechaptersprovidebackgroundinformationthatservesasabasisfor
working with PIAAC data. Chapter 2 summarises the core features of the PIAAC
survey design and briefly addresses sampling and data collection. In addition,
it provides an overview of the background questionnaire and the competence
domains assessedinPIAAC.The chapter concludes withadiscussion ofpotential
improvementstofuturePIAACcycles.
Chapter 3 introduces item response theory and the principles of multiple
imputations. It describes plausible values and explains how they can be used to
1TherelevantliteraturecanbefoundviathebibliographicsearchonthehomepageofthePIAAC
ResearchDataCenter(PIAACRDC;seealsoMaehler,etal.2020)atGESIS.
