Does an aptitude test affect socioeconomic and gender gaps in

64 Pages·2014·0.51 MB·English

Does an aptitude test affect socioeconomic and gender
gaps in attendance at an elite university?

Jake Anders

Department of Quantitative Social Science
Working Paper No. 14-07
April 2014
Disclaimer

Any opinions expressed here are those of the author(s) and not those of the Institute of Education.
Research published in this series may include views on policy, but the institute itself takes no
institutional policy positions.

DoQSS Workings Papers often represent preliminary work and are circulated to encourage discussion.
Citation of such a paper should account for its provisional character. A revised version may be
available directly from the author.

Department of Quantitative Social Science, Institute of Education, University of London
20 Bedford Way, London WC1H 0AL, UK
Does an aptitude test affect socioeconomic and gender gaps in
attendance at an elite university?

Jake Anders1

Abstract
The increasing use of aptitude tests as part of the admissions processes at elite English
universities potentially has significant implications for fair access to these institutions. I attempt
to isolate the impact of the introduction of one such test on the proportion of successful
applicants by school type (as a proxy for socioeconomic status) and by gender using a
difference in differences approach and administrative data from the University of Oxford. The
introduction of the test coincided with the implementation of a guideline number of interviews
per available place, significantly reducing the proportion of applicants offered an interview (by
14 percentage points) and, hence, increasing the proportion of interviewees offered places (by
3.6 percentage points). By gender, I find some evidence that these changes may be having
differing effects at different stages of the admissions process, but not on each group's overall
chances of securing an offer. I do not find any evidence that the policy has negative side effects
on the chances of applicants from less advantaged socioeconomic backgrounds at any stage of
the process.

JEL classification: I23, I 24

Keywords: Higher Education, Aptitude Test, Gender, Socioeconomic Gradient, Difference in
Differences.

1 Department of Quantitative Social Science, Institute of Education, University of London
([email protected])

Acknowledgements

Thanks to my supervisors Lorraine Dearden and John Micklewright for their patient support. Thanks
also to John Jerrim for helpful feedback and Claire Crawford for providing analysis to validate the
school-area IDACI measure. Many thanks to Ian Crawford and Margaret Stevens of the Department of
Economics, University of Oxford for clarifications and advice. Likewise to Elizabeth Frazer and
members of the Admissions Office at New College, Oxford for their assistance with this project. I am
very grateful to the University of Oxford Admissions Office for permission to use the data.
Nevertheless, these individuals and organisations are not responsible for any errors made in the author's
use of the data. Any views expressed are the author's own and do not necessarily reflect those of New
College, Oxford or the University of Oxford. I gratefully acknowledge my studentship funded by the
IoE as part of the ESRC NCRM ADMIN node.

3
1 Introduc(cid:415)on
Theincreasinguseofap(cid:415)tudetestsaspartoftheadmissionsprocessesateliteuniversi(cid:415)esin
Britain poten(cid:415)ally has significant implica(cid:415)ons for fair access to these ins(cid:415)tu(cid:415)ons. While the
inten(cid:415)onistoimprovetheefficiencyoftheprocess,makingiteasiertoselectindividualswith
a be(cid:425)er ‘ap(cid:415)tude’ ¹ for their university course, is this efficiency gain traded off against other
aims of the admissions process? In par(cid:415)cular, previous research suggests there are reasons
tothinkap(cid:415)tudetes(cid:415)ngmayhavesideeffectsonthepropor(cid:415)onofapplicantsfromdifferent
socioeconomicbackgrounds(Rothstein,2002)anddifferentgenders(Tannenbaum,2012)who
getaplace.
Toexplainthisconcern,letustaketheexampleoffairaccessbysocioeconomicstatus. There
are at least two poten(cid:415)al reasons that the introduc(cid:415)on of an ap(cid:415)tude test could result in a
smaller intake of those from lower socioeconomic backgrounds. First, the outcomes of the
testcouldreflectskillsacquiredinpreviouseduca(cid:415)on,henceskewingthedistribu(cid:415)onofthose
offered a place towards those who received certain kinds of schooling, or reflect training to
thetest,bothofwhichmightbeofconcern(Stringer,2008). Alterna(cid:415)vely,itcouldreflectgen-
uinedifferencesinap(cid:415)tudefortheuniversity’sdegreeprogrammeacrossthesocioeconomic
spectrum. However, there are also reasons to see the possibility of the opposite effect as a
result of the introduc(cid:415)on of an ap(cid:415)tude test, with more offers of places made to those from
lessadvantagedbackgrounds. Ifmoreweightisgiventoap(cid:415)tudetestresultsoverandabove
school examina(cid:415)on results then this could help overcome bias in those indicators caused by
schooling rather than underlying ability. This paper aims to iden(cid:415)fy which, if either, of these
effectsseemstodominateandhenceunderstandthewiderimpactofusingap(cid:415)tudetestsas
aselec(cid:415)ontool.
In2007-2008,theUniversityofOxford,aneliteBri(cid:415)shuniversity,introducedanap(cid:415)tudetest
as part of the admissions process for Economics-related subjects. I use administra(cid:415)ve data
fromtheUniversity’sadmissionssystem,coveringallundergraduateapplica(cid:415)ons,toes(cid:415)mate
the differen(cid:415)al impact of the introduc(cid:415)on of this test on applicants by their socioeconomic
backgroundsandtheirgender. Iemployadifferenceindifferencesframework: thisa(cid:425)empts
tocontrolforanygeneraltrendsusingthoseseeninsubjectswheretheap(cid:415)tudetestwasnot
¹‘Ap(cid:415)tude’istakenbroadlyasameasureofpoten(cid:415)ala(cid:425)ainment,asagainstpriora(cid:425)ainmentsuchasmeasured
byALevelsorGCSEs,orinnateability.
4
introduced,henceisola(cid:415)ngtheimpactduetothepolicychange.
Thepaperproceedsasfollows. InSec(cid:415)on2,Isurveytheliteratureonaccesstoeliteuniversi-
(cid:415)es,iden(cid:415)fyimportantdetailsabouttheuseofap(cid:415)tudetestsinuniversityadmissions,andlay
out the research ques(cid:415)ons for this paper. I then detail the admissions process at the Univer-
sityinSec(cid:415)on3anddescribethedatausedinthisworkinSec(cid:415)on 4. Sec(cid:415)on5describesthe
changes in admissions during this period and iden(cid:415)fies the par(cid:415)cular features of the change
in policy. It then lays out the empirical strategy for iden(cid:415)fying the changes in outcomes that
seemtobeassociatedwithitsintroduc(cid:415)onandpresentssimplees(cid:415)matesofimpact. Iextend
this using regression analysis, describing my models in Sec(cid:415)on 6 and presen(cid:415)ng the results
in Sec(cid:415)on 7. I consider an alterna(cid:415)ve way of looking at the results in Sec(cid:415)on 8 and conduct
variousrobustnesschecksinSec(cid:415)on9,beforeconcludinginSec(cid:415)on10.
2 Previous research and research ques(cid:415)ons
Whytakeaninterestintheadmissionsprocessesofeliteuniversi(cid:415)es,andtheintroduc(cid:415)onof
anap(cid:415)tudetestinpar(cid:415)cular? Iconsidertheseques(cid:415)onsinturn.
Giventhehigherwagepremiumsgraduatesfromeliteuniversi(cid:415)esseemtocommand(Cheva-
lierandConlon,2003),fairaccesstotheseins(cid:415)tu(cid:415)onsisimportanttofutureequalityofoppor-
tunity. Furthermore,wecannotnecessarilyrelyoninsightsaboutfairaccesstoalluniversi(cid:415)es
tounderstandinequali(cid:415)esateliteuniversi(cid:415)es; Pallaisarguesthat“itisen(cid:415)relyplausiblethat
barriers to enrollment at the most selec(cid:415)ve ins(cid:415)tu(cid:415)ons are somewhat different than at the
marginofenrollment”(PallaisandTurner,2008,p.132)andassuchthecorrectpolicyresponse
maywellbedifferent.
The current UK government’s belief is that “progress over the last few years in securing fair
accesstothemostselec(cid:415)veuniversi(cid:415)eshasbeeninadequate,andthatmuchmoredetermined
ac(cid:415)onnowneedstobetaken”(Wille(cid:425)s,2011). PreviousresearchfromboththeUKandtheUS
hashighlightedconcernabouttheequalityofopportunityinaccesstoeliteHigherEduca(cid:415)on
ins(cid:415)tu(cid:415)ons. In Anders (2012a) I showed that, among young English people who do a(cid:425)end
university,thosefromthebo(cid:425)omincomequin(cid:415)legrouparealmost20percentagepointsless
likelytoa(cid:425)endaRussellGroupins(cid:415)tu(cid:415)on(agroupofeliteUKuniversi(cid:415)es)thanthosefromthe
topincomequin(cid:415)legroup. Similarly,analysisbyBoliver(2013)highlightedthatRussellGroup
5
applicantsfromstateschoolsarelesslikelytoreceiveoffersofadmissionfromRussellGroup
universi(cid:415)es in comparison with their equivalently qualified peers from private schools. Such
concernsalsoexistintheUS:“Lessthan11percentoffirst–yearstudentsmatricula(cid:415)ngat20
highlyselec(cid:415)veins(cid:415)tu(cid:415)onswerefromthebo(cid:425)omincomequar(cid:415)leoftheincomedistribu(cid:415)on”
(PallaisandTurner,2006,p.357).
Specifically regarding the University of Oxford, Bha(cid:425)acharya et al. (2012) use administra(cid:415)ve
datafromoneundergraduateprogrammetoes(cid:415)matetheexpectedperformanceofthemarginal
admi(cid:425)ed candidate by sex and school type, arguing that in an academically fair process this
threshold for admission would be equal between such groups. However, they es(cid:415)mate that
the expected performance of the marginal candidate from an independent school is approx-
imately 0.3 standard devia(cid:415)ons higher than their state school counterpart. Similarly, the ex-
pected performance of the marginally admi(cid:425)ed male candidate is about 0.6 standard devia-
(cid:415)onshigherthantheirfemalecounterpart.
Ap(cid:415)tudetes(cid:415)nghasbecomeamuchmoreimportantissueinrecentyears. Asmorestudents
havebeguntoreachtheupperboundofperformanceinALevels(examina(cid:415)onstakenbymost
EnglishstudentsaimingforentrytoHigherEduca(cid:415)on,usuallyatage18)ithasbecomeharder
foruniversi(cid:415)estodifferen(cid:415)atebetweenpoten(cid:415)alstudentsatthetopendoftheabilitydistri-
bu(cid:415)on². Thishasledtoanincreasinguseofap(cid:415)tudetestsamongeliteins(cid:415)tu(cid:415)ons, including
theBioMedicalAp(cid:415)tudeTestandUnitedKingdomClinicalAp(cid:415)tudeTestforadmissiontomedi-
calcoursesatmanyuniversi(cid:415)es;thePhysicsAp(cid:415)tudeTest,attheUniversityofOxford;and,the
focus of this paper, the Thinking Skills Assessment at the University of Oxford, the University
of Cambridge and University College London (Admissions Tes(cid:415)ng Service, 2013b);. However,
animportantques(cid:415)oniswhetherthisresponseisasensiblecourseofac(cid:415)on,especiallyinthe
lightoftheinequali(cid:415)esdiscussedabove.
Ifweseeap(cid:415)tudeasameasureofpoten(cid:415)alabilityinagivenfield,thenap(cid:415)tudetestsshouldbe
effec(cid:415)veatpredic(cid:415)ngtheperformanceofcandidatesoncetheyreachuniversityandshoulddo
so without being biased by candidates’ other characteris(cid:415)cs. Unfortunately, McDonald et al.
(2001b)findli(cid:425)leevidencethattheScholas(cid:415)cAp(cid:415)tudeTest(SAT)predictsa(cid:425)ainmentonceat
collegeintheUSanybe(cid:425)erthanhighschoolrecordalone. Thesefindingswerereplicatedina
pilot study in Britain (McDonald et al., 2001a). A more recent Department of Business, Inno-
²Thisanalysiscoverstheperiodbeforetheintroduc(cid:415)onofthenewA*gradeforA-Levels,whichhasamelio-
ratedthisproblemtosomeextent.
6
va(cid:415)onandSkills(BIS)reportcomestosimilarconclusions,arguingthattheSATdoesnotpro-
vide significantly more informa(cid:415)on on applicants’ likely performance at undergraduate level,
rela(cid:415)ve to a baseline of GCSE (English school examina(cid:415)ons taken at the end of compulsory
educa(cid:415)on)a(cid:425)ainmentscores(Kirkupetal.,2010,p.20).
On the ques(cid:415)on of bias in ap(cid:415)tude test scores, the fact that “low-income students not only
are less likely to take college placement tests but also tend to have lower scores on these
exams”(PallaisandTurner,2008,p.135)suggests,onthefaceofit,thatap(cid:415)tudetes(cid:415)ngcould
cause more harm than good. In addi(cid:415)on, Pallais and Turner (2008) note that the “gap [in
ap(cid:415)tude tests between low and high income students] is par(cid:415)cularly marked at the top of
the distribu(cid:415)on from which elite colleges and universi(cid:415)es are likely to draw students”, which
meansthat,evenifap(cid:415)tudetes(cid:415)ngbecomescommonplaceamongHEins(cid:415)tu(cid:415)onsofallkinds,
itseffectsremainpar(cid:415)cularlyper(cid:415)nenttoeliteuniversi(cid:415)es.
There have long been concerns about gender differences on performance in ap(cid:415)tude tes(cid:415)ng
intheUS(LinnandHyde,1989)and,whilefindingdifferencesinscoresbysocioeconomicsta-
tusorgenderdoesnotnecessarilyimplybias(Zwick,2007,p.20),McDonaldetal.(2001b)do
iden(cid:415)fy specific evidence of biases in the SAT, in the US, with “consistent evidence that [it]
under-predicts female a(cid:425)ainment” once they get to university and more mixed evidence on
bias by ethnic groups. Similarly, Wikström and Wikström (2014) present evidence from Swe-
denthat,onaverage,femalesperformworsethanmalesintheSweSAT(ana(cid:415)onaluniversity
admissionstest),whiletheoppositeistrueinmeasuresbasedontheirperformanceatschool.
Tannenbaum(2012)arguesthatonereasonforthesefindingsisdifferinggenderstylesintest
taking,analysinginpar(cid:415)culartheSATanddifferinga(cid:427)tudestorisk.
AlthoughtheseanalysescannotbeextrapolatedtotheThinkingSkillsAssessment,noanalysis
thatIamawareofevaluateswhetheritspredic(cid:415)vepowerissignificantlyhigherthanabaseline
ofschoolexamina(cid:415)onresults,norwhetherthereisevidenceofbiasinitsassessments. There-
searchthathasbeendonespecificallyintotheThinkingSkillsAssessmenthasbeenrestricted
tosimpleanalysisofpredic(cid:415)vevaliditywithnobaseline. ResearchbyCambridgeAssessment
(thedevelopersandadministratorsofthetest)soughttoexaminetheextenttowhichtheTSA
could predictfuture academic performance (Emery, 2006; Emery et al., 2006). This was con-
ducted using data from the University of Cambridge courses Computer Science, Economics,
EngineeringandNaturalScienceusingstudentswhotooktheTSAin2003. Asisstandardprac-
7
(cid:415)ceinevalua(cid:415)ngthepredic(cid:415)vevalidityofselec(cid:415)ontests,thisinvolvedcalcula(cid:415)ngcorrela(cid:415)ons
between TSA score and subsequent academic outcomes. In par(cid:415)cular, the research finds a
correla(cid:415)on between higher marks in the TSA and higher marks in first year university exami-
na(cid:415)ons;strongsimilari(cid:415)esinthecandidatesthatwouldberejectedbyalowTSAcutoffscore
and those rejected under the present selec(cid:415)on system; and higher mean TSA scores among
thosegaininghigherdegreeclassifica(cid:415)onmarksinthesameexamina(cid:415)ons.
Theauthorsalsostatethatthecorrela(cid:415)ons,some(butnotall)ofwhicharesta(cid:415)s(cid:415)callysignif-
icant,arelikelytobeanunderes(cid:415)mateofthetruepredic(cid:415)vepowersincetheydonotinclude
those who were unsuccessful in ge(cid:427)ng a place at the university. However, there are poten-
(cid:415)al problems in some of the analysis done because of the data they were able to work with.
Ratherthan havinganydatawherethe TSAwasadministeredbut not used forselec(cid:415)on, the
TSAwasalreadyinuseintheselec(cid:415)onprocess(Emeryetal.,2006,p.13). Thismeansthatcare
shouldbetakenininterpreta(cid:415)on,especiallyofthedistribu(cid:415)onssugges(cid:415)ngsimilaritybetween
those who would be rejected by a TSA cut off and those rejected by the original selec(cid:415)on
methods.
With rather mixed evidence on predic(cid:415)ve validity, we should also consider the wider conse-
quences of introducing an ap(cid:415)tude test. McDonald et al. (2001b, p.53) highlight the impor-
tance of this, and draws on the concept of ‘consequen(cid:415)al validity’ (Messick, 1989, p.8). This
refers to the wider consequences of introducing the test on other aspects of the admissions
process. In this context, we might expect to see a reduced focus on the other informa(cid:415)on
about a candidate that an admissions tutor has: use of ap(cid:415)tude tes(cid:415)ng may reduce focus
on a candidate’s examina(cid:415)ons results. This might have posi(cid:415)ve consequences, given known
socioeconomic gradients in a(cid:425)ainment in such exams. However, that is only the case if the
alterna(cid:415)veprovidesafairerassessmentofcandidates’ability.
‘Consequen(cid:415)alvalidity’alsoreferstoresponsestotheuseofap(cid:415)tudetes(cid:415)ngoutsidethead-
missions process itself. For example, Wilmouth (1991) argues that students might spend in-
creased(cid:415)mepreparingforap(cid:415)tudetestsandlessontheiracademicstudies(citedinMcDonald
etal.,2001b,p.54). Thiscouldhaveanega(cid:415)veknock-oneffectonindividuals’academica(cid:425)ain-
ment, both in the short term and on their a(cid:425)ainment at university. Similarly, Geiser (2008)
arguesthattheeduca(cid:415)onsystemshouldrewardindividualswhoworkhardthroughouttheir
schoolcareers, a(cid:425)aininghighlyasa result; ap(cid:415)tudetes(cid:415)ngmayincen(cid:415)visebrightindividuals
8
toworklesshardatachievinghighlevelsofa(cid:425)ainment,iftheybelievetheycanbesuccessful
in gaining access to higher educa(cid:415)on simply by doing well on a test supposedly designed to
assessinnateskills.
Thispapercontributestotheliteraturebyprovidingevidenceontheconsequencesofap(cid:415)tude
tes(cid:415)ng for applicants to an elite Bri(cid:415)sh university. Given concerns about bias in scores on
ap(cid:415)tude tests (Zwick, 2007, p.20) I pay par(cid:415)cular a(cid:425)en(cid:415)on to these issues, with the paper’s
researchques(cid:415)onsasfollows:
1. Does use of the TSA have an effect on the propor(cid:415)on of applicants called to interview,
thepropor(cid:415)onofapplicantsofferedaplace,orthepropor(cid:415)onofintervieweesoffereda
place?
2. Dotheseimpactsaffecthighsocioeconomicstatusapplicantsdifferentlytolowsocioe-
conomicstatusapplicants?
3. Dotheseimpactsaffectfemaleapplicantsdifferentlytomaleapplicants?
3 The admissions process
Unlike at some Bri(cid:415)sh universi(cid:415)es, the admissions process at the University of Oxford con-
sists of more than one stage, with a shortlist of candidates invited to interview before final
admissions decisions are taken. I show the basic form of the admissions process graphically
inFigure1, highligh(cid:415)ngthreekeydecisionpointsthatmakeuptheprocess. First, individuals
choosewhethertoapplytoOxford;second,theUniversitychooseswhichapplicantstocallto
interview³;andthird,theUniversitychooseswhethertoofferintervieweesaplace. SinceIam
usingadministra(cid:415)vedatafromtheUniversity(whichIwilldescribefurtherinSec(cid:415)on4),Ican
analysethela(cid:425)ertwodecisionpoints.
Referringbacktotheideaof‘consequen(cid:415)alvalidity’ofusinganap(cid:415)tudetest,andthepoten(cid:415)al
for wider societal effects of its introduc(cid:415)on, an important part of the story is the impact of
theintroduc(cid:415)onoftheTSAonwhoappliestoEconomicscoursesattheUniversityofOxford.
³Star(cid:415)ngin2009,theUniversityhasstartedtointroduceuseofcontextualdatainselec(cid:415)ontointerivewacross
allsubjects.Qualifiedapplicantswithvariouscombina(cid:415)onsof‘flags’(indica(cid:415)ngmorechallengingcircumstances
basedonprioreduca(cid:415)onandarea-basedmeasures)arestronglyrecommendforinterview(UniversityofOxford,
2014).
9
Figure1: Simplifiedmodeloftheadmissionsprocess
1: Make an Application 2: Call to Interview 3: Offer a Place
Eligible Applicants Interviewees Offer
Don't Apply Not Interviewed No Offer
Unfortunately,thedataavailabledonotallowforthepropor(cid:415)onofyoungpeoplewhochoose
to apply to be modelled since poten(cid:415)al applicants are not observed by the university. In any
case,thedenominatorisratherpoorlydefined. Dowereallywanttoconsiderthepropor(cid:415)on
ofallyoungpeopleofthisagewhoapply,orrestricta(cid:425)en(cid:415)ontosubsetof‘eligible’applicant?
Ifthela(cid:425)er,whoshouldweregardasaneligibleapplicant? However,withoutaddressingthis
ma(cid:425)er we might be ignoring significant effects of the policy change. I return to this issue in
Sec(cid:415)on8.
Thusfar,Ihavedescribedthedecisiontocallcandidatestointerview,andwhetherul(cid:415)mately
toofferthemaplace,asbeingmadeby‘theUniversity’. However,tounderstandwhoactually
makes the decisions it is important to understand the unusual way admissions are organised
at the University of Oxford. The University is made up of more than 30 different, fairly au-
tonomous,‘colleges’. Muchundergraduateteachingoccurswithinthesecolleges,ratherthan
at university level, although students at all colleges, on the same course, study towards the
same degree examina(cid:415)ons. It is usually one or more of the members of staff who undertake
this undergraduate teaching within a college who decide which applicants to invite to inter-
view and, subsequently, which to offer places to. For this purpose, they are referred to as
‘admissionstutors’.
Acollege’sadmissionstutors’decisionoverwhethertoadmitanindividualisfinal: University
departmentscannotoverrulecollegedecisions. Mostapplica(cid:415)onsforundergraduatecourses
are made to colleges. However, some individuals do make open applica(cid:415)ons (which are not
to any par(cid:415)cular college); these are allocated to a college with a lower applica(cid:415)ons to places
ra(cid:415)oandthenproceedonthesamebasis.
10