Table Of ContentStatistics for Social and Behavioral
Sciences
Series Editor
Stephen E. Fienberg
Carnegie Mellon University Dept. Statistics
Pittsburgh
Pennsylvania
USA
StatisticsforSocialandBehavioralSciences(SSBS)includesmonographsand
advancedtextbooks relating to education, psychology,sociology,political sci-
ence, public policy, and law.
More information about this series at http://www.springer.com/series/3463
Russell G. Almond • Robert J. Mislevy
Linda S. Steinberg • Duanli Yan
David M. Williamson
Bayesian Networks in
Educational Assessment
2123
Russell G. Almond Duanli Yan
Florida State University Educational Testing Service
Tallahassee Princeton
Florida New Jersey
USA USA
Robert J. Mislevy David M. Williamson
Educational Testing Service Educational Testing Service
Princeton Princeton
New Jersey New Jersey
USA USA
Linda S.Steinberg
Pennington
New Jersey
USA
StatisticsforSocialandBehavioralSciences
ISBN 978-1-4939-2124-9 ISBN 978-1-4939-2125-6 (eBook)
DOI 10.1007/978-1-4939-2125-6
LibraryofCongressControlNumber:2014958291
SpringerNewYorkHeidelbergDordrechtLondon
(cid:2)c SpringerScience+Business MediaNewYork2015
This work is subject to copyright. All rights are reserved by the Publisher, whether the
whole or part of the material is concerned, specifically the rights of translation, reprint-
ing, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any
otherphysicalway,andtransmissionorinformationstorageandretrieval,electronicadap-
tation,computersoftware,orbysimilarordissimilarmethodologynowknownorhereafter
developed.
Theuseofgeneral descriptivenames, registerednames,trademarks,servicemarks,etc. in
thispublicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnames
areexemptfromtherelevantprotectivelawsandregulationsandthereforefreeforgeneral
use.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinforma-
tioninthisbookarebelievedtobetrueandaccurateatthedateofpublication.Neitherthe
publisher nor the authors or the editors give a warranty, express or implied, with respect
tothematerialcontained hereinorforanyerrorsoromissionsthatmayhavebeenmade.
Printedonacid-freepaper
SpringerispartofSpringerScience+Business Media(www.springer.com)
Dedication
Forward into future times we go
Over boulders standing in out way
Rolling them aside because we know
Others follow in our steps one day
Under deepest earth the gems are found
Reaching skyward ’till we grasp the heights
Climbing up to where the view surrounds
Hidden valleys offer new delights
Inch by inch and yard by yard until
Luck brings us to the hidden vale
Desiring a place to rest yet still
Returning home now to tell the tale
Ever knowing when that day does come
New hands will take up work left undone
Acknowledgements
Bayesian Inference in Educational Assessments (BNinEA) is the direct issue
of two projects, and the descendant, cousin, or sibling of many more. We are
grateful for all we have learned in these experiences from many collaborators
and supporters over the years.
The first direct ancestor is the series of workshops we have presented at
the annual meeting of the National Council on Measurement in Education
(NCME) almost every year since 2001. We are grateful to NCME for this
opportunity and to ETS for support in developing the materials they have
granteduspermissiontouse.Workshopparticipantswillrecognizemanycon-
cepts, algorithms, figures, and hands-on exercises from these sessions.
Its second direct ancestor is the Portal project at Educational Testing
Service. It was here that Linda, Russell, and Bob fleshed out the evidence-
centered design (ECD) assessment framework, implemented it in an object
model and design system, and carried out the applications using Bayes nets.
WearegratefultoHenryBraunandDrewGitomer,successivevice-presidents
of Research, and Len Swanson, head of New Product Development, for sup-
portingPortal.OurcollaboratorsincludedBrianBerenbach,MarjorieBiddle,
Lou DiBello, Howie Chernik, Eddie Herskovits, Cara Cahallan Laitusis, Jan
Lukas, Alexander Matukhin, and Peggy Redman.
Biomass (Chaps. 14 and 15) was a ground-up demonstration of Portal,
ECD design, standards-based science assessment, web-delivered interactive
testing, with automated scoring of inquiry investigations and Bayes net mea-
surement models. The Biomass team included Andy Baird, Frank Jenkins,
our subject matter lead Ann Kindfield, and Deniz Senturk. Our subject mat-
ter consultants wereScott Kight,Sue Johnson,GordonMendenhall, Cathryn
Rubin, and Dirk Vanderklein.
The Networking Performance Skill System (NetPASS) project was an
online performance-basedassessmentactivityfordesigningandtroubleshoot-
ing computernetworks.Itwasdevelopedincollaborationwiththe CiscoNet-
working Academy Program,and led by John Behrens of Cisco. NetPASS fea-
tured principled design of proficiency, task, and evidence models and a Bayes
VIII Acknowledgements
netpsychometricmodelusingthemethodsdescribedinBNinEA.Teammem-
bersincludedMalcolmBauer,SarahDeMark,MichaelFaron,BillFrey,Dennis
Frezzo, Tara Jennings, Peggy Redman, Perry Reinert, and Ken Stanley. The
NetPASS prototype was foundational for the Packet Tracer simulation sys-
tem and Aspire game environment that Cisco subsequently developed, and
millions of Cisco Network Academy (CNA) students around the world have
used operationally to learn beginning network engineering skills.
TheDISCscoringenginewasamodularBayes-net-basedevidenceaccumu-
lation package developed for the Dental Interactive Simulations Corporation
(DISC), by the Chauncey Group International, ETS, and the DISC Scor-
ing Team: Barry Wohlgemuth, DISC President and Project Director; Lynn
Johnson,ProjectManager;GeneKramer;andfivecoredentalhygienistmem-
bers,PhyllisBeemsterboer,RDH,CherylCameron,RDH,JD,AnnEshenaur,
RDH,KarenFulton,RDH,andLynnRay,RDH.JayBreyerwastheChauncey
Grouplead,andwasinstrumentalinconductingtheexpert–novicestudiesand
constructing proficiency, task, and evidence models.
Adaptive Content with Evidence-basedDiagnosis (ACED) was the brain-
childofValerieJ.Shute.IthadalargenumberofcontributorsincludingLarry
Casey,EdithAuroraGraf,EricHansen,WaverlyHester,SteveLandau,Peggy
Redman, Jody Underwood, and Diego Zapata-Rivera. ACED development
and data collection were sponsored by National Science Foundation Grant
No. 3013202. The complete ACED models and data are available online; see
the Appendix for details.
Bob’sinitialforaysintoapplyingBayesiannetworksineducationalassess-
ment were supported in part by grants from the Office of Naval Research
(ONR) andfromthe National Center forResearchonEvaluation, Standards,
and Student Testing (CRESST) at the University of California at Los Ange-
les. We are grateful to Charles Davis, Project Officer of ONR’s Model-Based
Measurement program and Eva Baker, Director of CRESST, for their sup-
port. Much of the work we draw on here appears in ONR and CRESST
research reports. The findings and opinions expressed in BNinEA, however,
do notreflectthe positionsorpolicies ofONR, the National Institute onStu-
dent Achievement, Curriculum, and Assessment, the Office of Educational
Research and Improvement, or the U.S. Department of Education.
Some of the ideas in this book are based on Russell’s previous work on
the Graphical-Beliefproject.ThankstoDougMartinatStatSci forspon-
soring that project as well as the NASA Small Business Innovation Research
(SBIR) program for supporting initial development. David Madigan made a
number of contributions to that work, particularly pointing out the impor-
tance of the weight of evidence. Graphical-Belief is based on the earlier
work on Belief while Russell was still a student at Harvard. Art Dempster
and Augustine Kong both provided valuable advise for that work. The work
of Glenn Shafer and David Schum in thinking about the representation of
evidence has beenveryuseful aswell. Those contributionsaredocumented in
Russell’s earlier book.
Acknowledgements IX
Along with the NCME training sessions and working on NetPASS and
DISC, David completed his doctoral dissertationon model criticism in Bayes
nets in assessment at Fordham University under John Walsh, with Russell
and Bob as advisors.
Hydrive was an intelligent tutoring system for helping trainees learn to
troubleshoot the hydraulics subsystems of the F-15 aircraft. Drew Gitomer
was the Principal Investigator and Linda was the Project Manager. The
projectwassupportedbyArmstrongLaboratoriesofthe USAirForce,under
the Project Officer Sherrie Gott. Design approaches developed in Hydrive
were extended and formalized in ECD. Bob and Duanli worked with Drew
and Linda to create and test an offline Bayes net scoring model for Hydrive.
RussellandBobuseddraftsofBNinEAinclassesatFloridaStateUniver-
sity (FSU) and the University of Maryland, respectively. We received much
helpful feedback from students to clarify our ideas and sharpen our presen-
tations. Students at Maryland providing editorial and substantive contribu-
tions included YounyoungChoi, Roy Levy, Junhui Liu, Michelle Riconscente,
and Daisy Wise Rutstein. Students at FSU, providing feedback and advice,
included MengyaoCui, Yuhua Guo, Yoon JeonKim, Xinya Liang, Zhongtian
Lin, SicongLiu, Umit Tokac,Gertrudes Velasquez,HaiyanWu, andYanXia.
Kikumi Tatsuoka has been a visionary pioneer in the field of cognitive
assessment, whose research is a foundation upon which our work and that
many others in the assessment and psychometric communities builds. We
are grateful for her permission to use her mixed-number subtraction data in
Chaps. 6 and 11.
Brent Boerlage, of Norsys Software Corp., has supported the book in a
numberofways.Firstandforemost,hehasmadethestudentversionofNetica
available for free, which has been exceedingly useful in our classes and online
training. Second, he has offered general encouragement for the project and
offered to add some of our networks to his growing Bayes net library.
Manyimprovementstoadraftofthebookresultedfromrigorousattention
from the ETS review process. We thank Kim Fryer, the manager of editing
services inthe ResearchandDevelopment divisionatETS, Associate Editors
Dan Eignor and Shelby Haberman, and the reviewers of individual chapters:
Malcolm Bauer, Jianbin Fu, Aurora Graf, Shelby Haberman, Yue Jia, Feifei
Li, Johnny Lin, Ru Lu, Frank Rijmen, Zhan Shu, Sandip Sinharay, Lawrence
Smith, Matthias von Davier, and Diego Zapata-Rivera.
We thank ETS for their continuing support for BNinEA and the various
projects noted above as well as support through Bob’s position as Frederic
M. Lord Chair in Measurement and Statistics under Senior Vice-President
for Research, Ida Lawrence. We thank ETS for permission to use the figures
and tables they own and their assistance in securing permission for the rest,
through Juana Betancourt, Stella Devries, and Katie Faherty.
Wearegratefulalsotocolleagueswhohaveprovidedsupportinmoregen-
eralandpervasivewaysovertheyears,includingJohnMarkAgosta,Malcolm
Bauer,BetsyBecker,JohnBehrens,JudyGoldsmith,GenevaHaertel,Sidney
X Acknowledgements
Irvine,KathrynLaskey,RoyLevy,BobLissitz,JohnMazzeo,AnnNicholson,
Val Shute, and Howard Wainer.
It has taken longer than it probably should have to complete Bayesian
Networks in Educational Assessment. For their continuing encouragement
and support, we are indebted to our editors at Springer: John Kimmel, who
brought us in, and Jon Gurstelle and Hannah Bracken, who led us out.
