ERIC ED529178: Where You Come from or Where You Go? Distinguishing between School Quality and the Effectiveness of Teacher Preparation Program Graduates. Working Paper 63

1.4 MB·English

by  ERIC

NATIONAL
CENTER for ANALYSIS of LONGITUDINAL DATA in EDUCATION RESEARCH
TRACKING EVERY STUDENT’S LEARNING EVERY YEAR
A program of research by the American Institutes for Research with Duke University, Northwestern University,
Stanford University, University of Missouri-Columbia, University of Texas at Dallas, and University of Washington
Where You Come From
or Where You Go?
Distinguishing Between School
Quality and the Effectiveness of
Teacher Preparation Program
Graduates

K ATA M I H A LY ,
D A N I E L M c C A F F E RY ,
T I M R . S A S S ,
A N D J . R . L O C K W O O D
WORKING PAPER 63 • JANUARY 2012
Where You Come From or Where You Go?
Distinguishing Between School Quality and the
Effectiveness of Teacher Preparation Program Graduates

i
Contents

Acknowledgements........................................................................................................................... iii
Abstract ............................................................................................................................................ iv
Introduction ...................................................................................................................................... 1
Review of Previous Studies of Preparation Programs and Student Outcomes ...................................... 5
Data for the Current Study ................................................................................................................. 7
Value Added Model .......................................................................................................................... 9
School Fixed Effects Specification – Feasibility and Sustainability ...................................................... 10
Regional Clustering ............................................................................................................. 11
Connectivity of Preparation Programs .................................................................................. 11
Plausibility of Homogeneity Assumption ............................................................................... 12
Results ............................................................................................................................................ 14
Value Added Models ............................................................................................................ 14
Variance Inflation ................................................................................................................ 17
Discussion ....................................................................................................................................... 18
References ...................................................................................................................................... 22
Tables and Figures .......................................................................................................................... 24

ii
Acknowledgements

Kata Mihaly is an Associate Economist and the RAND Corporation. Daniel McCaffrey is a senior statistician at
the RAND Corporation, where he holds the PNC Chair in Policy Analysis. Tim Sass is a professor in the Andrew
Young School of Policy Studies at Georgia State University and a member of the CALDER Florida team. J.R.
Lockwood is a senior statistician at the RAND Corporation. They thank Dan Goldhaber and two anonymous
referees at Education Finance and Policy for useful comments and suggestions.

This research was supported by an IES grant through a supplement to the National Center on Performance
Incentives. The authors are grateful to the Florida Department of Education for providing the data.

CALDER working papers have not gone through final formal review and should be cited as working papers.
They are intended to encourage discussion and suggestions for revision before final publication.

The views expressed are those of the authors and should not be attributed to the American Institutes for
Research, its trustees, or any of the funders or supporting organizations mentioned herein. Any errors are
attributable to the authors.

CALDER • American Institutes for Research
1000 Thomas Jefferson Street N.W., Washington, D.C. 20007
202-403-5796 • www.caldercenter.org
iii
Where You Come From or Where You Go? Distingushing Between School Quality and the
Effectiveness of Teacher Preparation Program Graduates
Kata Mihaly, Daniel McCaffery, Tim R. Sass, and J. R. Lockwood
CALDER Working Paper No. 63
January 2012

Abstract

In this paper we consider the challenges involved in evaluating teacher preparation programs when
controlling for school contextual bias. Including school fixed effects in the achievement models used to
estimate preparation program effects controls for school environment by relying on differences among
student outcomes within the same schools to identify the program effects. However, identification of
preparation program effects using school fixed effects requires teachers from different programs to teach in
the same school. Even if program effects are identified, the precision of the estimated effects will depend on
the degree to which graduates from different programs overlap across schools. In addition, if the connections
between preparation programs result from the overlap of atypical graduates or from graduates teaching in
atypical school environments, use of school effects could produce bias. Using statewide data from Florida, we
show that teachers tend to teach in schools near the programs in which they received their training, but
there is still sufficient overlap across schools to identify preparation program effects. We show that the
ranking of preparation programs varies significantly depending on whether or not school environment is
taken into account via school fixed effects. We find that schools and teachers that are integral to connecting
preparation programs are atypical, with disproportionately high percentages of Hispanic teachers and
students compared to the state averages. Finally, we find significant variance inflation in the estimated
program effects when controlling for school fixed effects, and that the size of the variance inflation factor
depends crucially on the length of the window used to compare graduates teaching in the same schools.
iv
Introduc(cid:415)on
OnFebruary17,2009,PresidentObamasignedintolawtheAmericanRecoveryandReinvestmentActof
2009. Thishistoriclegisla(cid:415)onincluded$4.35billionfortheRacetotheTopFund(RTTT),acompe(cid:415)(cid:415)vegrant
program designed to reward States that are demonstra(cid:415)ng success in raising student achievement scores
and developing effec(cid:415)ve teachers and principals. The selec(cid:415)on criteria included a provision on improving
the effec(cid:415)veness of teacher and principal prepara(cid:415)on programs. Specifically, it awarded points to states
basedon``(t)heextenttowhichtheStatehasahigh-qualityplanandambi(cid:415)ousyetachievableannualtargets
tolinkstudentachievementandstudentgrowthdatatothestudents￿teachersandprincipals,tolinkthis
informa(cid:415)ontothein-Stateprogramswherethoseteachersandprincipalswerepreparedforcreden(cid:415)aling,
andtopubliclyreportthedataforeachcreden(cid:415)alingprogramintheState''(USDOE(2009)).
Inaddi(cid:415)on,inSeptember2011,theDepartmentofEduca(cid:415)onreleasedtheObamaAdministra(cid:415)on'splan
forteachereduca(cid:415)onreformandimprovement(USDOE(2011)). Thiscomprehensiveagendadescribesthe
disbursement of federal money in three areas: ins(cid:415)tu(cid:415)onal repor(cid:415)ng and state accountability, reform fi-
nancingofstudentspreparingtobecometeachers,andtargetedsupporttoins(cid:415)tu(cid:415)onsthatprepareteach-
ersfromadiversebackground. Stateswillbeprovidedfundstoiden(cid:415)fytop-(cid:415)erandlowperformingteacher
prepara(cid:415)onprogramsbasedonthreeoutcomemeasures: studentlearninggrowth,jobplacementandre-
ten(cid:415)on,andcustomersa(cid:415)sfac(cid:415)onsurveyresults.
A persistent and unresolved concern with the value added modeling that is proposed for evalua(cid:415)ng
teacherprepara(cid:415)onprogramsistheexistenceofcontextualeffectsoftheschoolswheretheteachersteach.1
Becauseteachersfromapar(cid:415)cularprepara(cid:415)onprogramarehiredinmorethanoneschool,thegrowthin
studentachievementassociatedwiththeprepara(cid:415)onprogramwillcomefromvarioussources.(Boydetal.
2008). Inaddi(cid:415)on,newteachersarenotrandomlydistributedacrossschoolswithinthestate. Forexample,
thereisanecdotalevidencefromotherstatesthatschoolstendtohireteachersfromlocalprepara(cid:415)onpro-
grams,sugges(cid:415)ngthatthereisageographicclusteringofprogramgraduates. If,inaddi(cid:415)ontogeographic
preferences in hiring decisions, student ability is not evenly distributed across schools, then failing to ac-
1Fortheremainderofthisar(cid:415)clewereferto``prepara(cid:415)onprograms''astheins(cid:415)tu(cid:415)onsthattrain(andcer(cid:415)fy)teachers,and
``schools''astheins(cid:415)tu(cid:415)onswheretheyteacha(cid:332)ergradua(cid:415)on.
1
countforschoolcontextualfactorscouldbiasprepara(cid:415)onprogrames(cid:415)mates. Inthispaperwefocusonthe
feasibilityandimplica(cid:415)onsofcontrollingforschoolcontextualfactorswhencomparingteacherprepara(cid:415)on
programs.2
Policymakersmaywishtoremovethedifferencesinschoolswhencomparingteacherprepara(cid:415)onpro-
gramsusingstudentgrowthmeasures. Onemethodtoovercomeobserved differencesinschoolsistoin-
cludeschoolcharacteris(cid:415)csinthevalueaddedmodel. Analterna(cid:415)vespecifica(cid:415)onofthevalueaddedmod-
elsthatcanovercomeunobserveddifferencesinschoolcontextistoincludeschoolfixedeffects. Thisway,
comparisonsamongteachersfromdifferentprogramsaremadewithinschools. Schoolfixedeffectsmaybe
desirableinprepara(cid:415)onprogrammodelsbecausetheycontrolforunobservedteacherqualitythatispoten-
(cid:415)allycorrelatedwithschoolquality. However,itisimportanttounderstandwhethertheinclusionofschool
fixed effects is feasible in this se(cid:427)ng, the sensi(cid:415)vity of the es(cid:415)mates to assump(cid:415)ons underlying for fixed
effects,andwhattheirinclusionimpliesabouttheprecisionoftheprepara(cid:415)onprogrames(cid:415)matesandthe
resul(cid:415)ngrankingsofprepara(cid:415)onprogrameffec(cid:415)veness.
When fixed effects are included in a regression, a primary concern is whether these coefficients are
iden(cid:415)fied. Prepara(cid:415)onprogramsnotdirectlysharingteachersinschoolscans(cid:415)llbecomparedindirectly,as
longasthereissomelinkagewithteachersfromotherprogramsthatteachinthesameschool. However,
if prepara(cid:415)on program graduates are not sufficiently mixed across schools, this type of es(cid:415)ma(cid:415)on is not
feasible.
Iden(cid:415)fica(cid:415)ondependsthe(cid:415)mehorizonofthedatabeingusedtoes(cid:415)mateprogrameffects. Inthesim-
plestcase,across-sec(cid:415)onofrecentgraduatesandtheschoolstheyendupteachinginmaybeused,which
could provide single-year es(cid:415)mates of program effects. This ensures that programs are being compared
based on graduates teaching in the same school at the same point in (cid:415)me. However, this also limits the
(cid:415)esbetweenprograms,asmanyschoolsmaynothaverecentgraduatesfrommul(cid:415)pleprogramsteaching
thereduringanyoneschoolyear. Alterna(cid:415)vely,onecanemployamul(cid:415)-yearwindowofsuccessivecohorts
ofgraduatesandes(cid:415)mateaverageprogrameffectsoveralonger(cid:415)mehorizon. Increasingthelengthofthe
2Animplicitassump(cid:415)oninthisexerciseisthatteacherprepara(cid:415)onprogramscanbevalidlycomparedbasedontheperformance
oftheteacherstheytrain.Therearenumerousconcernswiththistypeofcomparison,includingselec(cid:415)onofteachersintoandout
ofprograms,selec(cid:415)onofprogramgraduatesintoteachingposi(cid:415)onswithinthestate,andhowteacherperformanceismeasured.
TheseissuesareaddressedintheDiscussionsec(cid:415)onbelow.
2
window increases both connec(cid:415)vity of prepara(cid:415)on programs and the power to discern among them, but
requires(cid:415)meinvarianceofmodelparameters.
Even when the (cid:415)me horizon of the data permit the inclusion of school fixed effects in the model, the
extenttowhichthees(cid:415)ma(cid:415)onreliesontheindirectlinkagesofprepara(cid:415)onprogramsneedstobeconsid-
ered. Theinclusionofschoolfixedeffectsassumeshomogeneityofeffects, namelythattheteachersand
schoolswhichcreate(cid:415)esamongtheprepara(cid:415)onprogramsdonothavedifferenteffectsthanotherteachers
orschoolsinthestate. Thelargertherelianceonindirectlinkages,themoresensi(cid:415)vearetheassump(cid:415)ons
regardingthehomogeneityofeffects. Inaddi(cid:415)on,indirectlinkagescanmakees(cid:415)matesimprecise,withthe
poten(cid:415)alforsignificantvarianceinfla(cid:415)on. Tounderstandtheimplica(cid:415)onsofthehomogeneityassump(cid:415)on
weusetoolsfromsocialnetworkanalysistoiden(cid:415)fythekeyteachersandschoolscrea(cid:415)ngdirectlinksinour
prepara(cid:415)onprogram/schoolnetworkandweconsiderwhethertheseteachersandschoolsarerepresenta-
(cid:415)veofthestate.
Anotherconsidera(cid:415)onforevalua(cid:415)ngprepara(cid:415)onprogrameffec(cid:415)venessisthesampleofteacherstoin-
cludeintheanalysis. Inordertoseparatetheeffectoftheprepara(cid:415)onprogramfromotherfactors,itmaybe
desirabletorestrictthesampletorecentgraduatesoftheprepara(cid:415)onprogram. However,includingschool
fixedeffectswithonlyinexperiencedteacherscangreatlyreducethesampleusedtoes(cid:415)matetheprogram
effects, which can result in variance infla(cid:415)on of program effects. While including experienced teachers in
the modeling can help make the analysis feasible and may be more desirable from a policy perspec(cid:415)ve,
thisspecifica(cid:415)onmayfalselyimplythattheprepara(cid:415)onprogrameffectisconstantforalllevelsofteacher
experience.
This paper uses a case study of elementary school teachers and their prepara(cid:415)on programs from the
stateofFloridain2000-2004toexplorethefeasibility,underlyingassump(cid:415)ons,varianceinfla(cid:415)on,andsam-
plingchoiceimplica(cid:415)onsofcontrollingforschoolcontextinthees(cid:415)ma(cid:415)onofprepara(cid:415)onprogrameffects.
Weexaminewhethertheschoolfixedeffectparametersareiden(cid:415)fiedandthedifferenceintheprecisionof
theprogrames(cid:415)matesunderdifferentmodelingchoices. Wealsoconsiderwhetherprogrames(cid:415)mateswith
schoolfixedeffectsarebiasedduetoviola(cid:415)onsoftheassump(cid:415)onsunderlyingthefixedeffectspecifica(cid:415)on
andtheimplica(cid:415)onsofrestric(cid:415)ngtheteachersampletoinexperiencedteachers. Wethenes(cid:415)matestudent
3
growthachievementmodelswithandwithoutcontrollingforschoolfixedeffects. Usingthees(cid:415)matedpro-
grameffects,weranktheprepara(cid:415)onprogramsinorderofeffec(cid:415)veness,andexaminethesensi(cid:415)vityofthe
rankingstothemodelingchoices.
Our findings indicatethatwhile thereis some regionalclusteringof programgraduates, newteachers
frommanyprogramsarehiredbyschoolsacrossthestateofFlorida. Therefore,schoolfixedeffectscanbe
includedinthestudentachievementmodelaslongasthreeormoreyearsofdataareusedinthees(cid:415)ma-
(cid:415)on. However,wefindevidencethattheschoolsandteachersthatareintegraltoconnec(cid:415)ngprepara(cid:415)on
programsaredifferentfromtheaveragewithinthestate,withadispropor(cid:415)onatelylargerHispanicandim-
migrantpopula(cid:415)onsinschoolsandmoreHispanicteachers. Thesedifferencesintheschoolsandteachers
thatiden(cid:415)fythees(cid:415)mateschallengetheplausibilityofthehomogeneityassump(cid:415)onrequiredbythefixed
effectses(cid:415)ma(cid:415)on.
Importantlyforpolicymakers,wefindthattherankingsofprepara(cid:415)onprogramseffec(cid:415)venessaresensi-
(cid:415)vetotheinclusionofschoolfixedeffects. Whencomparingtherankingquar(cid:415)lesofprepara(cid:415)onprograms
with and without school fixed effects, we find significant changes to the programs that are ranked in the
topandbo(cid:425)omquar(cid:415)lesunderdifferentspecifica(cid:415)ons. Forexample,regardlessofoursamplerestric(cid:415)ons,
wefindatleastoneprepara(cid:415)onprogramthatmovesfromthebo(cid:425)omquar(cid:415)leofrankingswithoutschool
fixed effects to the top quar(cid:415)le of rankings with school fixed effects. The quar(cid:415)le rankings of prepara(cid:415)on
programs are more stable across the specifica(cid:415)ons for low performing programs as compared to top-(cid:415)er
programs.
Finally,wefindthatincludingschoolfixedeffectsresultsinlesspreciseprepara(cid:415)onprogrames(cid:415)mates.
Even with a five-year window there is significant variance infla(cid:415)on due to the inclusion of school fixed ef-
fects. The variance infla(cid:415)on grows rapidly as we shorten the window for es(cid:415)ma(cid:415)on to one or two years,
primarilybecausemanymoregraduatesteachinschoolswithgraduatesfromasingleprogramandthusdo
notcontributetoprogrames(cid:415)matesinmodelswithschoolfixedeffects.
Based on these results, we argue that states will need to choose amongst three op(cid:415)ons for modeling
prepara(cid:415)on program effec(cid:415)veness, each with its own drawbacks. The first op(cid:415)on is to es(cid:415)mate models
without school fixed effects and make conclusions about prepara(cid:415)on programs that may be sensi(cid:415)ve to
4
themodel'suntestableassump(cid:415)onofnoschoolcontextualeffects. Alterna(cid:415)vely,statesmightconsideran
approachthatincludesobservableschoolcharacteris(cid:415)csratherthanschoolfixedeffects. Finally,statescould
choosetoes(cid:415)matemodelswithschoolfixedeffectsandpossiblyrelyonasmallandatypicalsetofschools
and teachers to iden(cid:415)fy the models which yield much less precise es(cid:415)mates. It is unclear which of these
threeapproacheswillyieldes(cid:415)mateswiththesmallestmeansquareerrorsandtheleastbias. Statesmay
needtodescribetheuncertaintyofthemodeltheyemploy,butthiscouldweakentheu(cid:415)lityofes(cid:415)mates.
Withoutclearevidencefororagainstcontextualeffectsandthesensi(cid:415)vityofconclusionsaboutprograms
likewefoundinFlorida,statesmayneedtoreconsiderifthisapproachalonecanprovideusefulinforma(cid:415)on
aboutprepara(cid:415)onprograms.
Theremainderofthepaperisorganizedasfollows. First,wereviewpreviousstudieswhichhavecom-
paredteacherprepara(cid:415)onprogramsonthebasisoftheoutcomesofthepublicelementaryandsecondary
studentstaughtbytheirgraduates. Second,wepresentthevalueaddedmodelandtheexplora(cid:415)onofthe
dataregardingthefeasibilityandsuitabilityoftheschoolfixedeffectes(cid:415)ma(cid:415)on. Next,wepresenttheprepa-
ra(cid:415)onprogrameffec(cid:415)venesses(cid:415)matesunderalterna(cid:415)vemodelspecifica(cid:415)ons,andfinallyweconcludewith
asummaryanddiscussionofourfindings.
Review of Previous Studies of Prepara(cid:415)on Programs and Student Outcomes
Dueinlargemeasuretoextensivedatarequirements, thereareonlyahandfulofexis(cid:415)ngstudiesthat
have a(cid:425)empted to link value-added measures of teacher performance to the prepara(cid:415)on programs the
teachers graduated from. These include studies of teachers in five states: New York, Florida, Louisiana,
Kentucky, and Texas. These studies have dealt with the problem of school contextual effects in different
ways. IntheirstudyofNewYorkCitypublicschoolteachers,Boyd,etal. (2008),includeschoolfixedeffects
intheirmodel. Theydonotdiscusstheimplica(cid:415)onsofthischoiceintermsoftheoverlapofprogramgrad-
uatesinschools, ortheimpactofschoolfixedeffectsontheprecisionoftheires(cid:415)matedprogrameffects.
They find considerable varia(cid:415)on in teacher value-added across prepara(cid:415)on programs but do not provide
standarderrorsoftheseeffects.
Sass(2008)andKukla-Acevedo,Stream,sandToma(2009)alsoincludeschoolfixedeffectsintheachieve-
5