On the merits of, and myths about, international assessments

22 Pages·2015·0.19 MB·English

Quality Assurance in Education
On the merits of, and myths about, international assessments
Oren Pizmony-Levy James Harvey William H. Schmidt Richard Noonan Laura Engel Michael J. Feuer
Henry Braun Carla Santorno Iris C. Rotberg Paul Ash Madhabi Chatterji Judith Torney-Purta
Article information:
To cite this document:
Oren Pizmony-Levy James Harvey William H. Schmidt Richard Noonan Laura Engel Michael J. Feuer
Henry Braun Carla Santorno Iris C. Rotberg Paul Ash Madhabi Chatterji Judith Torney-Purta , (2014),"On
T) the merits of, and myths about, international assessments", Quality Assurance in Education, Vol. 22 Iss 4
P pp. 319 - 338
15 ( Permanent link to this document:
0
2 http://dx.doi.org/10.1108/QAE-07-2014-0035
April Downloaded on: 02 April 2015, At: 13:38 (PT)
2
0 References: this document contains references to 38 other documents.
8
3:3 To copy this document: [email protected]
At 1 The fulltext of this document has been downloaded 85 times since 2014*
y
sit Users who downloaded this article also downloaded:
er
v
ni Edmund W. Gordon, Michael V. McGill, Deanna Iceman Sands, Kelley M. Kalinich, James W. Pellegrino,
U
a Madhabi Chatterji, (2014),"Bringing formative classroom assessment to schools and making it count",
bi Quality Assurance in Education, Vol. 22 Iss 4 pp. 339-352 http://dx.doi.org/10.1108/QAE-07-2014-0034
m
u
ol W. James Popham, David C. Berliner, Neal M. Kingston, Susan H. Fuhrman, Steven M. Ladd, Jeffrey
C
of Charbonneau, Madhabi Chatterji, (2014),"Can today’s standardized achievement tests yield instructionally
e useful data?: Challenges, promises and the state of the art", Quality Assurance in Education, Vol. 22 Iss 4
g
olle pp. 303-318 http://dx.doi.org/10.1108/QAE-07-2014-0033
C
s Meiko Lin, Erin Bumgarner, Madhabi Chatterji, (2014),"Understanding validity issues in international large
er
h scale assessments", Quality Assurance in Education, Vol. 22 Iss 1 pp. 31-41 http://dx.doi.org/10.1108/
c
ea QAE-12-2013-0050
T
y
b
d
e
d
a
nlo Access to this document was granted through an Emerald subscription provided by 226991 []
w
Do For Authors
If you would like to write for this, or any other Emerald publication, then please use our Emerald for
Authors service information about how to choose which publication to write for and submission guidelines
are available for all. Please visit www.emeraldinsight.com/authors for more information.
About Emerald www.emeraldinsight.com
Emerald is a global publisher linking research and practice to the benefit of society. The company
manages a portfolio of more than 290 journals and over 2,350 books and book series volumes, as well as
providing an extensive range of online products and additional customer resources and services.
Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the Committee
on Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archive
preservation.
*Related content and download information correct at time of download.
T)
P
5 (
1
0
2
pril
A
2
0
8
3
3:
1
At
y
sit
er
v
ni
U
a
bi
m
u
ol
C
of
e
g
e
oll
C
s
er
h
c
a
e
T
y
b
d
e
d
a
o
nl
w
o
D
Thecurrentissueandfulltextarchiveofthisjournalisavailableat
www.emeraldinsight.com/0968-4883.htm
On the merits of, and myths International
assessments
about, international assessments
Oren Pizmony-Levy, James Harvey, William H. Schmidt,
Richard Noonan, Laura Engel, Michael J. Feuer, Henry Braun,
319
Carla Santorno, Iris C. Rotberg, Paul Ash,
Madhabi Chatterji and Judith Torney-Purta
(Authoraffiliationscanbefoundattheendofthearticle)
T)
P
5 (
01 Abstract
2
April lPimuritpaotisoens–oTfhIinstepranpaetironparelsLeanrtgsea-ScmaoledeArastseedssdmisecnuts(sIiLoSnAo)nprpoogprualmars,mcliascriofnycinepgtihoonws,IbLeSnAefirtessaunltds
02 couldbemoreappropriatelyinterpretedandusedinpublicpolicycontextsintheUSAandelsewherein
38 theworld.
13: Design/methodology/approach–Tobringkeyissues,points-of-viewandrecommendationsonthe
At themetolight,themethodusedisa“moderatedpolicydiscussion”.Ninecommentarieswereinvitedto
sity representvoicesofleadingILSAscholars/researchersandmeasurementexperts,juxtaposedagainstviews
er ofprominentleadersofeducationsystemsintheUSAthatparticipateinILSAprograms.Thediscussionis
v
ni excerptedfromarecentblogpublishedbyEducationWeek.Itismoderatedwithintroductoryremarksfrom
U
a theguesteditorandconcludingrecommendationsfromanILSAresearcherwhodidnotparticipateinthe
bi originalblog.Referencesandauthorbiographiesarepresentedattheendofthearticle.
m
olu Findings–Together,thecommentariesaddresshistorical,methodological,socio-politicalandpolicy
C issuessurroundingILSAprogramsvis-a`-visthemajorgoalsofeducationandlargersocietalconcerns.
of Authorsofferrecommendationsforimprovingtheinternationalstudiesthemselvesandformaking
e
eg reports more transparent for educators and the public to facilitate greater understanding of their
oll purposes,meaningsandpolicyimplications.
C
s Originality/value–Whenassessmentpoliciesareimplementedfromthetopdown,asisoftenthecase
er
h with ILSA program participation, educators and leaders in school systems tend to be left out of the
c
ea conversation.Thisarticleisintendedtofosteraproductivetwo-waydialogueamongkeyILSAactorsthat
T
y canserveasastepping-stonetomoreconcertedpolicyactionswithinandacrossnationaleducationsystems.
b
ed Keywords Accountability,TIMSS,Assessmentpolicy,Internationalassessments,PISA,
d
a Educationalquality,ILSA,Nationalcompetitiveness
o
wnl PapertypeTechnicalpaper
o
D
Introduction
MadhabiChatterji,GuestEditor,TeachersCollege,ColumbiaUniversity[2]
Front-pageheadlines and editorial sections of newspapers around the world today
grab our attention frequently by announcing the latest results from various
International Large-Scale Assessment (ILSA) programs. The media are quick to
QualityAssuranceinEducation
Vol.22No.4,2014
pp.319-338
© Assessment and Evaluation Research Initiative at Teachers College, Columbia University EmeraldGroupPublishingLimited
0968-4883
(AERI@TC)andtheNationalSuperintendentsRoundtable,USA(2014) DOI10.1108/QAE-07-2014-0035
QAE highlightthestandingsofparticularnationsonILSAscores,commentingontheir
22,4 implicationsforglobalcompetitiveness.Provocativeandsatiricalheadlineslikethe
following,“ChineseThird-GradersFallingbehindUSHighSchoolStudentsinMath,
Science” (The Onion, 2013; www.theonion.com/articles/report-chinese-thirdgraders-
falling-behind-us-high,31464/),appearedafterthepublicationofthe2012Programmefor
InternationalStudentAssessment(PISA)reportsinDecember2013.
320 World-wide attention notwithstanding, ILSA reports tend to be miscast by the
media, and are often over-interpreted and over-generalized in public and policy
discussions(Backhoff,2013;Feuer,2013;Laurie,2013;Plisko,2013;Wagemaker,2013in
Chatterji, 2013). There is ambivalence among educators, researchers and the public
T) aboutILSAprogramsandtheirfindings,especiallywhenrecentresultslikethoseofthe
5 (P PISA in Shanghai lead to instant international economic comparisons. Some of the
1 critiques are well-founded. Others are not. There are myths surrounding what ILSA
0
2
April raelspoormtserciatnstaonIdLcSaAnnportotgerlalmuss,thsaotmneeeodfwtohbicehdniseceudstsoedbeanudndeveresncqorueeds,tiwonheilde.oTthheerrseaarree
02 clarifiedorqualified.
38 ILSAprogramsaretechnicallycomplexandmulti-layeredresearchendeavors.Inthe
3:
1 typical case, the public sees ILSA rankings of countries based on average student
y At achievement,measuredviasample-surveytestingofstudents,focusingonsubjectslike
sit math,readingandscience.Theassessmentsonwhichtheserankingsarebasedarenot
er
v thesimpleteststhatmostmembersofthepublicmayenvision,wherescoresconsistof
ni
U percentagesofitemsansweredcorrectly.ThequestionsincludedintheILSAtests,their
a
bi scoringmechanismsandfinalscaleproperties,thesamplesofstudentstestedandways
m
u in which reports present differences between countries are complex and difficult to
ol
C explaininlaylanguage.Furthermore,manyparticipants–includingresearchers,public
e of educatorsandpolicymakers–donotunderstandeitherthehistoricalforcesthatbrought
g
e ILSAsintoexistence,northepoliticalandeducationalfactorsthatareinfluencingtheir
oll
C rapidexpansionandsustenanceinternationally(currently,participatingnationsareat
s
er 65forPISA).
h
ac Thereareotherquestions,aswell.HowareILSAprogramsimpactingstudents,
e
y T teachersandleadersineducationsystems,andwhataretheirrepercussionsinother
b
d sectors of society? How might we transcend the media “hype” and steer ILSA
e
ad programsintomoreproductiveinternationalconversationsoneducationalpolicy?
o
nl Toanswersuchquestions,thismoderatedpolicydiscussioninQAEistitled:On
w
o the Merits of, and Myths about, International Assessments. From their respective
D
contextsandexpertvantagepointsintheUSA,theparticipantsbringtothetable
extensive knowledge and direct experience with the “ILSA phenomenon”.
Perspectives include those of ILSA researchers, policy scholars and measurement
experts: Oren Pizmony-Levy, William Schmidt, Laura Engel and Michael J. Feuer,
Iris C. Rotberg and Henry Braun, in order of presentation; and school district
superintendents and education leaders: James Harvey, Richard Noonan, Carla
Santorno and Paul Ash, also in order of appearance. The discussion is excerpted
fromarecentblogpublishedinEducationWeek,co-facilitatedbyJamesHarveyof
the National Superintendents Roundtable and myself (http://blogs.edweek.org/
edweek/assessing_the_assessments). To conclude, Judith Torney-Purta provides
anoutsideILSAresearcher’sthoughtsandrecommendations.
Back to the future on international assessments International
OrenPizmony-Levy,TeachersCollege,ColumbiaUniversity[1]
assessments
FascinatedbytheimmensegrowthandvisibilityofILSAprograms,Ispentthepastfew
years exploring the socio-historical roots of ILSAs through archival materials and
interviewswithkey-informants(Pizmony-Levy,2013).Here,Idiscusstwomajorchanges
that took place in the world of ILSAs over the past 50 years, which allow us to better 321
understandtheILSAphenomenonandperhaps,theintendedandunintendedconsequences
ofILSAprograms.
Firstchange:ashiftinownershipfromresearcherstogovernments.TheInternational
AssociationfortheEvaluationofEducationalAchievement(IEA),establishedin1958,
PT) wasthefirstorganizationtoconductILSAs.TheIEAemergedfromaworkinggroupof
5 ( scholarsundertheauspicesoftheUnitedNationsEducational,ScientificandCultural
1
20 Organization(UNESCO)InstituteforEducationinHamburg,Germany.Keyfiguresin
pril the group included Professor Benjamin Bloom (University of Chicago), Professor
A
2 TorstenHusén(StockholmUniversity)andProfessorArthurFoshay(TeachersCollege,
8 0 Columbia University). For these scholars, comparing educational systems using
3
3: large-scalequantitativedatawasdrivenbyintellectualobjectives,forexample:
1
At Ifcustomandlawdefinewhatiseducationallyallowablewithinanation,theeducationalsystems
y
sit beyondone’snationalboundariessuggestwhatiseducationallypossible(Foshay,1962,p.7).
er
niv TheIEAGeneralAssembly(GA)istheorganization’sgoverningbody.Formanyyears,a
U majority of the countries were represented at the GA by individuals affiliated with an
a
bi academic or research institute, while the rest of the countries were represented by
m
u individuals affiliated with governmental agencies. Since the mid-1990s, that pattern has
ol
C beenreversed.In1986,theproportionofrepresentativesfromgovernmentalagencieswas
e of 43.3 per cent; this figure jumped to 59.6 per cent in 1998 and to 73.4 per cent in 2012.
g
e Representativesfromacademicandresearchinstitutescorrespondinglydeclinedfrom56.7
Coll to40.4percentandthento26.6percent.Inaninterview,ahigh-rankingofficialattheIEA
ers commented:
h
c
ea Whenthefirstmathstudystarted[1964],thesewereresearcherswhobecameinterestedin
T
y exploiting(orunderstanding)thevariancebetweencountries.Andthen,morecountriesjoined,
b
d because they thought that would be a good thing. Now, it became much of a more
e
d governmentalthing,notsomucharesearchthing.Governments,misguidedly,Ithink,jumped
a
nlo in on the bandwagon to see who is better or worse than Americans (excerpted from
ow Pizmony-Levy,2013).
D
This transformation is also evident in other organizations that conduct ILSAs.
Throughoutitssomewhatshorterhistory,thePISAhasbeenrunbytheOrganisation
forEconomicCo-operationandDevelopment(OECD),whichisaninter-governmental
organization. Moreover, the Governing Board of PISA has been populated by
individualsaffiliatedwithgovernmentalagencies.
Second change: from research to policy […] and educational quality and
accountability.Intheearlydecades,theleadingrationaleforoperatingILSAprograms
was formulated in terms of basic educational research. Indeed, in reading the IEA
missionstatementfrom1968,wecanfindastrongemphasisonscholarship:
[The]IEAisaninternational,non-profit-making,scientificassociation[…]whoseprincipal
aims are: (a) to undertake educational research on an international scale; (b) to promote
QAE researchaimedatexaminingeducationalproblemscommontomanycountries,inorderto
providefactswhichcanhelpintheultimateimprovementofeducationalsystems[…](IEA,
22,4
1968,p.1).
At that time, the IEA had a diverse portfolio of studies on topics such as literature
education, English and French as foreign languages, classroom environment, civic
educationandcomputersineducation.TheIEApublicationsofthattimeavoidedthe
322 “horserace”discourseamongcountriesandresultswerepresentedinalphabeticalorder
ofcountries.
Theaimistodevelopasystematicstudyofeducationaloutcomesintheschoolsystemsofthe
cooperatingcountries.Thequestionwewishtoaskisnot“ArethechildrenofcountryXbetter
T) educated than those of country Y?” To us this seemed a false question begging all the
P
5 ( importantissuesweneedtostudy(MinutesofTheIEAProject,17-22October1960,p.9).
1
20 Inthepasttwodecades,however,thisresearch-orientedrationalewasreplacedbya
pril morepolicy-orientedrationale,onethatismorelinkedtoaccountability,educational
A quality indicators and, sadly, to international competition. The current version of
2
8 0 theIEA’smissionstatementemphasizestheprovisionof“internationalbenchmarks
3:3 toassistpolicymakers”,andthecollectionof“high-qualitydata”forfacilitatingthe
1
At implementation of accountability policies in education. The portfolio of topics
y covered by ILSAs is less diverse and tends to focus on the basics: mathematics,
ersit science and reading. Finally, publication of ILSAs reinforces the “horse race”
niv discourse by presenting results in “league” tables and “report cards” of ranked
U
a countries that improved and declined in student achievement, without describing
bi thetechnicallimitationsofsuchinferencesforthosewhoarenon-specialists.
m
olu Wheredowegofromhere?Somerecommendations.Thetwointer-relatedchangesin
C theworldofILSAsareimportantbecausetheyunmaskthelinkbetweenpolitics(inabroad
of
e sense) and practices of ILSA programs. We should aim to bring back research and
g
olle researcherstothefrontlinesofILSAprograms.Indeed,therearesomepreliminarysigns
C that this is happening. For example, since 2004 the IEA has organized a biennial
s
er InternationalResearchConference(IRC)thatisintendedtocreateaforumforscholarsfrom
h
ac differentcountrieswhoareinterestedinfurtherexploringILSAprogramdata.The5thIEA
e
T IRC was held in Singapore in 2013 with over 140 participants (www.unescobkk.org/
y
d b education/news/article/5th-iea-international-research-conference-irc-2013/).
de AnotherexampleistheOECD’sThomasJ.AlexanderFellowshipforILSAscholars
a
nlo (www.oecd.org/edu/thomasjalexanderfellowship.htm).Thisprogramisfundedbythe
w
o OpenSocietyFoundationthatsupportsinnovativeanalysisofdatafromPISAandother
D
OECD-sponsoredILSAs.
Catnip[3] for politicians: international assessments
JamesHarvey,NationalSuperintendentsRoundtable,USA[1]
As a boy in Ireland, I attended Mass every Sunday, during which priests with their
backstothecongregationmumbledinincomprehensibleLatin.Noneofusknewwhatit
meant,butwewereassuredthesemysteriesweregoodforthesoul.
In education today, ILSA measurement experts occupy the exalted status of Irish
priests.Withtheirfigurativebacktotheschools,theseprelatesgenuflectatthealtarof
ItemResponseTheoryandmumbleconfidentlyamongthemselvesinalanguageknown
as“psychometrics”.Nooneintheschoolcongregationunderstandsawordofit,butwe International
areassuredthesemysteriesarebasedonscience.
assessments
LackoftransparencyinILSAs.Itisatroublingsituation.Assessmentresultsarebut
thetipofaniceberg(Harvey,2014).Alltheassumptionsthatproducethenumbersthe
public sees are buried beneath the waterline. Discussion between the ILSA
measurementexpertswhodeveloptheassessments,ILSAresearcherswhoanalyzeand
preparereportsandfront-lineeducatorswhomustactonthemisalmostnon-existent.It 323
isabrokenconversation.
Pizmony-Levy’s review (this issue) of the history of the governance of ILSAs
introducesanewelementthatmakesthesituationevenmorecomplex.Researchersonce
dominated the governance mechanisms of ILSAs. That is not true today. Nearly
T)
P three-quartersofthemembersofthepolicygovernancebodyoftheIEA,forexample,
5 ( arefromgovernmentorthebureaucracy.Itisasafebetthatmanyofthesepeopledonot
1
0
2 possess even a rudimentary understanding of the complexities of ILSA reports. It is
pril “voodoothatwedosowell”,asthearcanaofILSApsychometricswasaptlydescribed
A
2 by Jakob Wandall of Denmark at a 2012 conference hosted by the Assessment and
0
8 Evaluation Research Initiative at Teachers College, Columbia University (Wandall,
3
3: 2013).
1
At The ILSA horse race. Pizmony-Levy’s history (this issue) reminds me of Torsten
sity Husén’s1982testimonybeforetheNationalCommissiononExcellenceinEducation,of
er whichIwasthedeputydirector(Husén,1982).Husén,oneofthetoweringfiguresinthe
v
ni earlyhistoryofeducationalresearchonILSAsinEuropeandofIEA,madeapleatothe
U
a commission: Do not interpret ILSA reports as a “horse race” between nations. ILSAs
bi
m werenotdevelopedbybookiestohandicaphorsesbutbyresearcherstohelpclarifythe
u
ol goalseachnationpursuedforitselfthroughitsschools.Unfortunately,thepolitically
C
of appointed members of the excellence commission could not resist the “catnip” of
ge ranking nations by mean results. In its startling 1983 report, A Nation at Risk, the
e
oll commissioners thundered: “On 19 [international] academic tests, American students
C
s wereneverfirstorsecondand,incomparisonwithotherindustrializednations,were
her lastseventimes”(NationalCommissionofExcellenceinEducation,1983).
c
ea What would Husén make of where we are today? It is not just that the horse-race
T
y mentality of mean scores on ILSAs dominates the conversation, domestically and
b
d internationally.Itisalsothatbureaucratsinbothindividualnationsandinternational
e
d
a associationshavedemonstratedremarkableskillandcunninginprovidingmoreofthe
o
wnl horse-racerhetoricpoliticianscrave,whichservesascatnip.And,truthbetold,inthe
Do quietoftheiruniversityandthink-tankoffices,manymeasurementexpertsandILSA
researchers acknowledge that, although troubled by some directions in assessment
today,theyfeelpressuretofollowthecrowdandthemoneyavailabletoexploreschool
systemswithapparentlyunsatisfactoryresults.
Lookingahead.So,wheredoesthisleaveus?Severalthingsmightbesaid.First,no
responsible educator denies the need for national/local assessments, international
assessments or of accountability policies in education. But, accountability for
appropriateuseofILSAreportsneedstoruninbothdirections.Itisclearthatteachers
andschooladministratorsaretodayatthemercyofhowpublicofficialsinterpretILSA
results. This puts a special onus on ILSA leaders, researchers and measurement
specialiststoliveuptothebestpracticesoftheprofession.Theyshouldnotbeinthe
businessofproviding“catnip”forpoliticians.
QAE Second,itistimeforanotherlookatthegovernancestructuresofILSAassessment
programs. The dominance of bureaucrats and politicians on ILSA governing bodies
22,4
shouldgivewaytomorebalancedrepresentationofmeasurementexperts,researchers
andpracticingeducators.Finally,weshouldopenlyacknowledgethatwhatlies“below
the waterline” at present is simply incomprehensible to policymakers and the lay
audience. It is time to clarify communications among these sectors (including
324 researchers, measurement experts, politicians and educators). After all, even in the
churches of Ireland, priests long ago abandoned Latin and turned their faces to the
congregation.
T) What can PISA and TIMSS tell us?
P
5 ( WilliamH.Schmidt,MichiganStateUniversity[1]
1
0
2
April Gcormowpeintigtivaewneasrsehnaesssmaodfetthheerecsruucltisalofrinolteernoaftioednualcaastisoenssminenatsacomuanjtorryp’subelciconevomenitc.
02 TheTrendsinInternationalMathandScienceStudy(TIMSS)andthePISAhavebeen
38 usedasfodderforpoliticalandpolicydebates;inparticularstressingthatthemediocre
3:
1 performanceofstudentsfromtheUSAinmathematicsonthesetestsisevidencethat
At significantchangesarerequiredintheAmericaneducationalsystem.Unfortunately,a
y
sit lotofdiscussionoftheTIMSSandPISAresultsinvolveshastygeneralizationsbasedon
ver country-levelaveragesandanobsessivefocusoninternationalrankings.Fartoooften
ni
U we find simplistic imitation of a high-ranking country’s educational policies, or
bia sweepingassertionsthatUSA’sperformanceisdeterminedbystudentpovertyorthe
m
u falsebeliefthathigher-performingcountriesonlytesttheirbrighteststudents.
Col WhyILSAreportsposeinterpretativechallenges.TheTIMSSandPISAtestsareoften
of confused with one another, partly because they happen to have similar scales with
e
g averagesaround500.Yet,thereareimportantdifferencesbetweenthetwo.
e
oll The different rankings of the USA on the TIMSS and PISA are in part due to the
C
s different countries that take each test. There are fewer rich countries that take the
er
h TIMSS,whichmakestheUSAlookabitbetter.Buttheteststhemselveshavedifferent
c
a
Te content. TIMSS assesses mathematical knowledge, but PISA assesses mathematical
by literacy(howmathisapplied).Althoughmostcountries’performanceissimilaronboth
ed tests,thereareimportantexceptionswhereacountrydoesquitewellontheTIMSSbut
d
oa poorlyonthePISA,orviceversa(Mullisetal.,2012;OECD,2013).
nl
w Finally,itisimportanttorememberthattheinternationalaverageisthemeanacross
o
D countries,notthemeanofallstudentsineverycountry.Thelatterisabitlowerbecause
of the below-average performance of some larger countries. However, we should be
payingalotlessattentiontocountryaveragesandrankingsanyway.Itisamistaketo
define a country’s students based on a single number. The reality is much more
complicated.
Infact,mostofthevariationinstudentperformanceonboththeTIMSSandPISA
iswithincountries,notacrossthem(OECD,2003).SimplybecauseJapan’sstudents
have a higher average test score does not mean that every Japanese student does
better than every American student. In both nations, there is a wide variation in
student outcomes. These variations exist between schools and also within schools.
Weareusedtotalkingabout“goodschools”and“badschools”asifeverystudent
attending them is performing at the same level. Yet the TIMSS and PISA
demonstratethatthisjustisnottrue.ResearchbasedontheTIMSSsuggeststhere International
is also a wide variation in performance across classrooms (Schmidt et al., 1999).
assessments
Unfortunately,PISAdoesnotsamplebyclassroom,obscuringthecontributionsof
between-classroomvariationtoinequality.
Lessonsforthefuture.Thekeylessontobedrawnfrominternationalassessmentsis
thatthesystemofeducationinagivennation–thepackageofeducationalpolicies–hasa
major impact both on the average performance of students and the extent of inequality 325
amongstudents.Studentpovertyisanimportantcontributortobothofthese,butUSA’s
performancecannotbeattributedsolelytothenumberordistributionofpooranddisadvantaged
students(SousaandAmour,2010).Somecountriesdoamuchbetterjobatmitigatingtheeffects
ofstudents’poverty,andpovertycannotexplainwhyevenaffluentstudentstrailtheirpeersin
PT) othercountries.Othernationshavemuchgreaterequalityineducationaloutcomes,andwe
5 ( shouldstudytheirapproachesandadaptsomeofthemtoourowncircumstances.
1
20 There is a great deal to learn from international assessments like the PISA and
pril TIMSS.Theoverallperformanceofacountrycangiveusaclueastonationsonwhich
A
2 wecouldfocusourattention.However,wemusttakecareinhowweinterprettheresults
8 0 ofthesetestsifwearetoavoiddrawingmisleadingconclusions.
3
3:
1
At On international assessments (and any assessments): less is more
y
sit RichardNoonan,Wallingford-SwarthmoreSchoolDistrict,Pennsylvania[1]
er
v
ni
U Myschooldistrict,inasuburbofthecityofPhiladelphiaintheUSA,recentlycompleted
bia an ambitious strategic planning project in collaboration with the University of
m
u Pennsylvania’s Penn Center for Educational Excellence (www.gse.upenn.edu/pcel).
Col When surveyed, parents, residents, students, staff and representatives from higher
of educationuniversallyproclaimedastheirleadrecommendationthatthedistrictshould
e
g relaxtheinvasivegripthatstandardizedtestingprogramscurrentlyholdoncurriculum,
e
oll teaching and learning in our schools. The periodic publication and clamor about
C
s international assessment results typically strengthens that hold of testing on our
er
h schools.PoliticiansofallstripesjumpontheresultstopilloryAmericaneducatorsand
c
a
Te double-downonmorestandardizedtesting.
by Implications of ILSA differences. If we are going to talk intelligently about these
ed international tests, let us start with the ways in which the different ILSA programs
d
oa differ.PISAandTIMSSdonotprovideasingle,universalstandardofqualityregarding
nl
w studentachievementbywhichschoolsaroundtheworldcanbejudged.PISA,which
o
D shiftssubjectareafocusperadministration,aimstoassessstudents’abilityto“apply
knowledge”, while TIMSS, which looks only at math and science performance in
selectedgrades,focusesonmoretraditionallypresented,curriculum-basedknowledge
andskills.ThesamecountrycandowellonPISA,butnotsowellonTIMSS.
Finland, broadly held-up as an educational model to the world because of
consistentlyhighILSArankings,scoresrelativelylesswellonTIMSSthanitdoeson
PISA.Intheareaofmath,PISAvaluesmoreofaconstructivistviewofknowledge,while
TIMSSassessesthetraditionallysequencedconceptsandskillsinadiscipline.TIMSS
mirrorsmorecloselythestructureoftestingintheNationalAssessmentofEducational
Progress(NAEP)modelintheUSA,wheretheperformanceatselectedgradesinGrades
kindergarten-12isexamined(http://nces.ed.gov/nationsreportcard/studies),whilePISA
doesnotseemtohaveanyAmericancounterpart.Thepointisthatthecontent,structure
QAE and emphases of these two ILSA programs differ. So, their reports do not reveal the
samethings,andwouldnotnecessarilyleadustothesameconclusions.
22,4
WhatcanwelearnfromILSAresults.This fact is not sufficient reason for us to
dismissexaminingandlearningfromwhatinternationaltestresultsdoreveal.We
know that NAEP performance is widely variable across the states (http://nces.
ed.gov/nationsreportcard/studies/statemapping/). TIMSS results enable us to
326 benchmarktheperformanceofstatesinrelationtowholecountriesaroundtheglobe.
NAEPtypicallypointstoMassachusettsasthestateachievementleaderintheUSA;yet
TIMSSresultsshowanumberofcountriesoutperformingMassachusettsinmathand
science.
TIMSS results can offer fresh perspectives from which we can gauge our relative
PT) successinadvancingselectedachievementgoals.Girlsinselectedcountriesaroundthe
5 ( worldoutperformtheUSAonTIMSStests.Wecanexaminewhatthosecountriesare
1
20 doing to broaden the range of strategies our own schools could use to achieve this
pril importantgoal.Mydistrict,likeothers,hasbeentakingsuccessfulstepstoboostthe
A
2 participationlevelofgirlsinthemostadvancedmathandsciencecourses.
8 0 As Schmidt (this issue) points out, there is arguably one fair conclusion that both
3
3: internationalassessmentsleadto,whichisthatmanyothercountrieshavebeenmore
1
At successful in establishing unifying national educational goals. We continue to make
y slow progress, via implementation of reforms related to the Common Core State
ersit Standards (www.corestandards.org), toward a goal that many other industrialized
v
ni countriesaddresseddecadesago.ThestateofIndiana’sdecisiontopull-out(inorder,as
U
a onepoliticianputit,toestablishcurriculum“byHoosiers,forHoosiers”)showsjusthow
mbi steep the challenge is, given the strong resistance to releasing curriculum from strict
olu localcontrol.Still,weneedtopersistifourstudentsaretobewell-preparedtoperform
C
of atthehighereducationlevelinanincreasinglyglobalizedemploymentmarketplace.
e Advantagestolesstestinginschools.Wecouldgainagreatdealbypursuinga“less
g
olle is more” testing approach in our schools. That is, reduce the scope and span of
C standardized testing programs while ensuring that any assessment we implement
s
er providesmeaningfulresults.Itisalmostunfathomabletomethatstateswouldsignup
h
ac to participate in regularly scheduled PISA or TIMSS testing programs on top of the
e
y T expansivetestingregimenwealreadyhaveinplace.Yetwehavetocreatespace,inwhat
d b istodayanoverscheduledregimenoftestingofallkinds,forinternationalassessments.
e
d Participatingregularlyininternationaleducationassessmentscanprovideuswith
a
o
nl valuablebenchmarksandinsights.Wecancreatethatspacebyreturningtothepre-No
w
o ChildLeftBehind(NCLB,2001)eraofstateproficiencytestinglimitedtoonegradeper
D
schoollevel,peryear.Doingsowouldprovidethoseofusatthelocalschooldistrictlevel
with the breathing room needed for a more genuine perspective on schooling and
studentprogress,withlessfreneticattentiontodatagathering.
Five myths about international large scale assessments (ILSAs)
LauraEngelandMichaelJ.Feuer,TheGeorgeWashingtonUniversity[1]
Withtherecentreleaseofthe2012PISAresults,weareonceagainremindedaboutthe
extenttowhichILSAshavegrippedtheworldofeducation.ILSAs,consistofadiverse
set of assessments, ranging across math and science, reading, civic and citizenship
education,teachereducation,andothers(foracompletelistandhistoryofILSAs,seethe