The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socio ...

26 Pages·2017·2.41 MB·English

McManusetal.BMCMedicine 2013, 11:244
http://www.biomedcentral.com/1741-7015/11/244
RESEARCH ARTICLE Open Access
The UKCAT-12 study: educational attainment,
aptitude test performance, demographic and
socio-economic contextual factors as predictors
of first year outcome in a cross-sectional
collaborative study of 12 UK medical schools
I C McManus1,2*, Chris Dewberry3, Sandra Nicholson4 and Jonathan S Dowell5
Abstract
Background: Most UK medical schools use aptitude tests during student selection, but large-scalestudies of
predictive validity are rare.This study assesses theUnited Kingdom Clinical Aptitude Test (UKCAT), and itsfour
sub-scales, along with measures of educationalattainment,individualand contextual socio-economic background
factors, as predictors of performance inthe first year of medical school training.
Methods: A prospective study of 4,811 students in 12 UKmedical schools taking the UKCAT from 2006 to 2008 as
a part of themedical school application, for whom first year medical school examination results were availablein
2008 to 2010.
Results: UKCAT scores and educational attainment measures(General Certificate of Education (GCE): A-levels, and
so on; or Scottish Qualifications Authority (SQA): Scottish Highers, and so on)were significant predictorsof
outcome. UKCAT predicted outcome better infemalestudents than male students, and better inmaturethan
non-mature students.Incremental validity ofUKCAT taking educational attainment intoaccount was significant,
butsmall. Medicalschoolperformance was also affected by sex (male students performing less well), ethnicity
(non-White students performing less well), and a contextualmeasure of secondary schooling, students from
secondary schools with greater average attainment atA-level (irrespective of public or private sector) performing
less well. Multilevel modeling showed no differences between medical schools inpredictive ability ofthevarious
measures. UKCAT sub-scales predicted similarly, except thatVerbal Reasoning correlated positively with
performance onTheory examinations, butnegatively with Skills assessments.
(Continuedonnextpage)
*Correspondence:[email protected]
1UCLMedicalSchool,UniversityCollegeLondon,GowerStreet,London
WC1E6BT,UK
2ResearchDepartmentofClinical,EducationalandHealthPsychology,
DivisionofPsychologyandLanguageSciences,UniversityCollegeLondon,
GowerStreet,LondonWC1E6BT,UK
Fulllistofauthorinformationisavailableattheendofthearticle
©2013McManusetal.;licenseeBioMedCentralLtd.ThisisanopenaccessarticledistributedunderthetermsoftheCreative
CommonsAttributionLicense(http://creativecommons.org/licenses/by/2.0),whichpermitsunrestricteduse,distribution,and
reproductioninanymedium,providedtheoriginalworkisproperlycited.
McManusetal.BMCMedicine 2013, 11:244 Page2of25
http://www.biomedcentral.com/1741-7015/11/244
(Continuedfrompreviouspage)
Conclusions: This collaborative study in12 medical schools shows the power of large-scale studies ofmedical
education for answering previously unanswerablebut important questions about medical student selection,
education and training. UKCAT has predictivevalidityas a predictor of medical school outcome, particularly in
mature applicants to medical school. UKCAT offers small but significant incremental validitywhich is operationally
valuable where medical schoolsare making selection decisions based on incomplete measures ofeducational
attainment. The study confirms the validity of using allthe existing measuresof educational attainment in full at
thetime of selection decision-making. Contextual measures provide littleadditional predictive value, except
that students from high attaining secondary schools perform less well, aneffectpreviously shown for UK
universities ingeneral.
Keywords: Medical student selection, Educational attainment, Aptitude tests, UKCAT, Socio-economic factors,
Contextual measures
Background at medical school than students of equal ability attend-
Formanyyearstheprimarycriterionusedtoselectmed- ingselectiveor privatesector secondaryschools.Arising
ical school applicants world-wide has been measures of from such concerns has also been a growing awareness
educational attainment. In the UK, General Certificate of the paucity of large-scale, longitudinal studies which
of Education (GCE) A-levels, for those educated in have examined performance in medical school in rela-
England,WalesandNorthernIreland,andScottishQuali- tion to a wide range of measures collected during selec-
ficationsAuthority(SQA)Highersformostofthoseed- tion. Without such studies it is difficult to assess the
ucated in Scotland, have been the norm. A-levels have validity and defensibility of the processes currently used
been shown to be valid predictors of outcome, both toselectmedicalstudents.
duringthemedicalcourse[1]andlaterinmedicalcareers In response to these challenges, most UK medical
[2], and for university education in general [3], and schoolsinrecent years have used aptitude testsasa sup-
more generally in Europe, secondary school grades are plementary selection technique. An aptitude test usually
predictive of university performance [4]. Educational at- consists of a series of multiple choice sub-tests. Sub-
tainment is also important in medical student selection tests are developed to assess specific aptitudes thought
in many countries, such as Australia, Denmark, Iran, to be relevant for performance at medical school (for
The Netherlands [5], New Zealand and Thailand [6]. example, numerical ability and verbal ability). Unlike
However, in recent years the continued reliance on such secondary school examinations, which measure attain-
attainment measures as the sole or principal basis for ment in relation to a particular discipline (for example,
medical student selection has been questioned for three chemistry), aptitude tests are specifically designed to
reasons. First, A-levels and Scottish Highers, which are measure intellectualandbehavioral capacity, andthepo-
typicallytakeninthefinalyearofsecondaryschool,have tential to perform well in a particular role. Furthermore,
over the past two decades shown continual increases in aptitude tests offer considerable operational advantages:
grades attained (so-called,‘grade inflation’). The result is they can be taken in standardized settings by anyone,
that a large proportion of UK applicants to medical whatever their cultural or educational background, at
schools now achieve the highest grades (a ceiling effect), geographical locations all over the world; they can assess
so that discriminating between them for the purpose of people against items for which the difficulty level has
selection is increasingly problematic. Second, selection been pre-established; they can be completed in a short
on the basis of A-level and Scottish Higher grades may time (typically less than three hours); and candidates’
be construed as unfair, because students who have performance on the test can be computed immediately.
attended selective schools, including independent sec- Although aptitude tests are used in medical selection
ondary schools charging high fees, tend to obtain higher in many countries including Australia [7], Chile [8],
grades than others, although this in part may reflect dif- Germany [9,10], Pakistan [11], Italy [12], Mexico [13],
ferentintakeprofiles.Third,thereareconcernsthataca- Switzerland [14], the United States [15,16] and the UK
demic assessment alone may not necessarily select the [17,18],researchontheextenttowhichtheysuccessfully
candidates that possess the behavioral or non-cognitive predict medical performance is patchy. Given the high-
attributes thought desirable in medical students and stakes nature of medical selection it is clearly important
doctors. As a consequence, able but economically or so- to establish the predictive validity of aptitude tests, and
cially disadvantaged children attending comprehensive the extent to which they add value to more traditional
secondary schools might be less likely to obtain a place approaches to selection, such as previous educational
McManusetal.BMCMedicine 2013, 11:244 Page3of25
http://www.biomedcentral.com/1741-7015/11/244
attainment, using datasets large enough to provide sub- current and accepted methods of selection, which at
stantial statistical power. Here we report the results of present for medicine are primarily achievement tests
the first large-scale analysis of the predictive and incre- taken in secondary education, coupled in many cases
mental validity of UK Clinical Aptitude Test (UKCAT), with interviews. That is particularly important as tests
the aptitude test used in the selection of medical stu- such as the American Medical College Admission Test
dentsinmostUKmedicalschools. (MCAT),whichhave both aptitude andattainment com-
Aptitude tests can be used as a stand-alone selection ponents, typically find that most of the prediction is due
device or, more typically, to supplement the existing in- to the attainment component, rather than the aptitude
formation on which selection decisions are made, such component[32].
as a candidate’s secondary school attainment, Univer- Given the widespread adoption of UKCAT for medical
sities and Colleges Admissions Service (UCAS) personal schoolselectionsinceitsintroductionin2006[33],amore
statements and medical school interview performance. comprehensiveexaminationoftheextenttowhichthetest
Because these tests can be specifically designed to differ- cansuccessfullypredictperformanceandwidenparticipa-
entiate between highly able candidates, and to do so in tion is required. In this article we address this issue by
relation to the particular characteristics required in the examining the relationships among multiple predictors of
medical profession, they can potentially discriminate ap- medical school performance (including UKCAT, A levels
propriately between those with equally good attainment andScottishHighers,andabroadrangeofcontextualand
at A Level and Scottish Highers. In addition, by measur- socio-cultural measures, including selective schooling), in
ing the extent to which candidates’ possess aptitudes relation to the first year medical school performance of
relevant tothe medical profession, rather than theirlevel 4,811studentsstudyingat12EnglishandScottishmedical
of school-related educational attainment, aptitude tests schoolsinthree cohortswho tooktheUKCATin2006to
mayhelp towidenparticipation inmedicine[19]. 2008, entered medical school in 2007 to 2009, and com-
The two main aptitude tests currently in operation in pletedtheirfirstyearin2008to2010.
the United Kingdom are the UK Clinical Aptitude Test Although the primary impetus for the present study
(UKCAT) [20], which is the principle interest of the was to evaluate UKCAT in the context of medical stu-
present study, and the Biomedical Admissions Test dent selection, the UKCAT-12 study can also be used to
(BMAT) [21,22]. For admission in 2013, UKCAT is be- address a wider set of important issues. As yet there has
ing used by 26 UK medical schools, and BMAT by 4 UK beennolarge-scale,prospectivestudyofmedical student
medical schools. If aptitude tests are to be of added performance drawing on a wide range of measures
value in addressing the problem of ceiling effects in edu- which might predict that performance (including de-
cational qualifications, they must predict the future per- tailed socio-economic background measures) across a
formance of medical school candidates over and above substantial sample of medical schools. UKCAT-12 pro-
that provided by A-levels and Scottish Highers [23]. In vides exactly that, giving not only a platform from which
addition,iftheyaretowidenparticipationinthemedical to ask many questions about the nature of medical stu-
profession, it is desirable that scores on UKCAT and dent selection and education and the assessment of the
BMAT should be less strongly associated with selective effects of a large number of different background mea-
secondaryschoolingthanAlevels orScottish Highers. sures, but also allowing a determination of the extent to
At present, both UKCAT and BMAT are somewhat which different measures might have different predictive
controversial [23,24], mainly because of concerns about values in different medical schools. Although, therefore,
how well they predict performance at medical school. a prime interest of the present study is to evaluate
However, there is also concern over the cost deterring UKCAT, it also represents the first, long-term, large-
poorer applicants and the effects of coaching [25], which scale study of medical student training in the UK. Im-
in general can have an effect size of about .26 [26]. To portant features of the present analysis are that the
date there have been four studies ofthe predictive ability sampleislarge(nearly5,000students),it isdiverseand
of UKCAT [17,27-29], all of which are moderately small representative of a range of medical schools (12 medical
(Ns = 292, 307, 204 and 146), and have widely varying schools taking part), it is extended over time (the data
conclusions, from a study suggesting the test provides beingcollectedacrossseveralyears),andthereisa‘hard’
nosignificantprediction[30]toclaimsofsignificantpre- outcomemeasureintheformofmedicalschoolexamin-
dictive ability [17]. Additionally, one study [31] found no ation results on a continuous scale. That means the
relationship between UKCAT scores and scores on ad- current study has high statistical power, and also makes
missions interviews. An important consideration in de- itpossibletocomparemedicalschoolsinordertoassess
termining the validity of any aptitude test is that as well the degree to which the conclusions can be generalized
as demonstrating predictive validity in its own right, it acrossmedicalschools.Thus,theaccumulatingdatabase
should also show incremental validity when used with associated with the UKCAT provides an important
McManusetal.BMCMedicine 2013, 11:244 Page4of25
http://www.biomedcentral.com/1741-7015/11/244
opportunity not only to assess the effectiveness of the entered medical school in 2007, 2008 or 2009, and for
UKCAT, but also to assess the influence of a far broader whom outcome measures were available at the end of
range of issues concerning how educational, demo- their first academic year. For those cohorts, UKCAT
graphic and social factors influence medical school out- was used in selection by 23, 25 and 26 medical schools.
come, including those assessed with the ‘contextual The secondary datasets contained data on a range of
measures’ which will soon be available for routine use other measures, including prior educational achieve-
duringselection. ment, socioeconomic background, and so on. Many of
the secondary measures are missing, either for structural
Aimsoftheanalysis reasons (for example, some socio-economic measures
The present analysis takes into account the aims which are only available for England; A-levels were not avail-
UKCATset for itself [20,34], as well as various previous able for mature entrants; and so on) and others were
studies of aptitude tests (and the criticisms of those also sporadically missing, probably mostly at random.
studies).Ittherefore looksat: Some secondary measures do not describe individual
students, but instead are contextual variables, describ-
◦Thepredictive validity ofA-levelsand Scottish ing not the students themselves, but features of the edu-
Highersforperformance inthe firstyearofmedical cational and socioeconomic environment in which the
schoolstudies. students lived prior to joining medical school (for ex-
◦Thepredictive validity ofUKCATforperformance in ample,aggregatedmeasuresoftheattainmentofthesec-
thefirstyearofmedical schoolstudies. ondary school attended, socio-economic measures of the
◦Theincrementalvalidity ofUKCAToverand above local community where the student lived, and so on).
existing measuresofeducational attainment,both Contextual measures need to be treated with care, but
General CertificatesofSecondaryEducation(GCSEs)/ have been included not only for their sociological inter-
AS-levels/A-levelsand ScottishHighers/Advanced est, but also because similar measures are now provided
Highers. routinely by UCAS, and have been shown to relate to
◦Thespecific predictiveability, with andwithout achievement at the BMAT aptitude test [36]. Table 1
taking educationalattainment intoaccount,ofthefour summarizes the many measures which were in the ana-
subscalesofUKCAT. lysis, and more detailed information can also be found
◦An assessment ofwhether ‘Theory ‘and‘Skills’ in the UKCAT-12 Technical Report [34]. The measures
measuresatmedical schoolare predicted differentlyby can bedividedinto sixbroadcategories.
educational attainmentand UKCATaptitude measures. Variables are indicated by their SPSS variable names
◦An assessment ofwhetherthe predictive validity of (inbold)toreduce ambiguity.
anyofthemeasures isdifferent inthe 12medical
schools thathavetakenpart inthestudy. 1. MEDICAL SCHOOLOUTCOMEDATA.Medical
◦Theroleofdemographicand socio-economicfactors schoolsprovidedinformation onoveralloutcome on
inmoderating anyofthefindings. afour-point scale(passed allexams atfirstattempt;
passed afterre-sitting exams;repeatingthefirstyear;
Itshouldbenotedthatthepresentstudyisrestrictedto and leavingthe course),which we called
medical school entrants and, therefore, it cannot look OutcomeFirstYear4pt.Average percentagemarks
more generally at how social and other factors relate to onassessmentswerealsoavailableformost students
educational attainment and UKCAT performance in the (OverallMark),and formanystudentsseparate
entire set of medical school applicants (rather than en- markswerealsoavailablefor ‘theory’and ‘skills’
trants). The analysis also considers only simple predictor- assessments(TheoryMarkandSkillsMark;see
outcome correlations, and makes no attempt to calculate Table1).Theoverall,theory and skillsmarkswere
construct validity, taking into account the unreliability of allbasedonmarksattained atthefirstattempt.
predictor and outcome measures, restriction of range due 2. PRIOREDUCATIONAL ATTAINMENT,AND SO
to selection, and the right-censorship of predictor vari- ON.Informationonprior educationalattainment
ables such as A-level scores. All of that is considered in a wasprovidedbyUCAS,consistingofScottish
separate paper, which carries out a meta-regression of HigherandAdvancedHigherresultsforstudents
construct validity not only in the UKCAT-12 study, but from Scotland (collectively SQAqualifications),and
alsoinfiveothercohortstudies[35]. A-level,AS-leveland GCSE resultsforother
students(collectively GCEqualifications).
Methods Educational qualificationsarealways complexto
The primary dataset for the UKCAT-12 study consists analyze,becausedifferentcandidatestake different
of the 4,811 students in three separate cohorts, who examinationswith different structuresand grading
McManusetal.BMCMedicine 2013, 11:244 Page5of25
http://www.biomedcentral.com/1741-7015/11/244
Table1Summaryofvariablesintheanalysis
Category Variablesusedintheanalysis Notesandcomments
Medicalschool Outcomefirstyear4pt Medicalschooldataonstudentperformanceintheirfirst
outcomedata academicyearforthethreecohorts.Notallschoolsprovided
Overallmark
dataforallcohorts-11,11and9schoolsprovidingdataforthe
Theorymark 2007to2009cohorts,for1,661,1,710and1,440students.Inthe
samecohorts,UKCATwasusedforselectionby23,25and26
Skillsmark
medicalschools.Theoverallnumberofstudentsfromthe12
schoolsvariedfrom87to945(median=335,mean=401,SD=
243).Medicalschoolswereaskedtoprovideseveralitemsof
informationoneachstudent,althoughnotallschoolsprovided
allinformation.DatawerecollectedbytheUKCATConsortium
Office,andnotbytheresearchers.Measuresusedwereas
follows:OutcomeFirstYear4pt:Outcomeofthefirstyearona
four-pointscale(Passedallexamsatfirstattempt;passedafter
re-sittingexams;repeatingthefirstyear;andleavingthecourse);
OverallMark,TheoryMark,andSkillsMark:Averagedpercentage
marksinmedicalschoolassessments.OverallMark,basedonall
assessments,wasavailablefor4,510students,oneschool
providingonlyOutcomeFirstYear4pt,andoccasionalstudents
elsewherenothavingpercentagemarks;ineachcaseaproxy
OverallMarkwascalculatedasanormalscore,usingSPSS’sRank
Cases/NormalScorescommand.Separatemarkswerealso
availablefor‘Theory’and‘Skills’assessments,thedefinitionof
theoryandskillsbeinglefttomedicalschools.TheoryMarkand
SkillsMarkwereavailablefor2,075and3,184students.Because
percentagemarksarenotnecessarilycomparableacrossschools,
OverallMark,TheoryMarkandSkillsMarkwerestandardizedto
ameanofzeroandSDofonewithinmedicalschoolsandcohorts.
Prioreducational Alevel_number_total WewilldescribetheanalysisofA-levelsinsomedetail.Other
achievement examinationsshowminorvariationsfromtheanalysisofA-levels
Alevel_number_totalAlevel_Totalbest
whichwewillthendescribe.
Alevel_TotalPoints A(Advanced)levels.ScoredasA=10,B=8,C=6,D=4,E=2,
Else=0.A*gradesatAlevelwerenotawardedduringthestudy
Alevels_Taken_1_or_more_Biology
period.Measureswereonlycalculatedforstudentswiththreeor
Alevels_Taken_1_or_more_Chemistry moreA-levels,othersbeingsetasmissing.Fourteenmeasures
separatemeasureswereobtained,describedfurtherinthe
Alevels_Taken_1_or_more_Physics
TechnicalReport[34].GeneralStudieswasnotcountedinthe
Alevels_Taken_1_or_more_Maths overalltotals,meansandsoon,butwasanalyzedseparately,as
itsstatusisunclear.Themeasures(withtheirnamesinbold),
Alevels_highest_Biology were:Alevel_number_total:Numberofnon-GeneralStudies
A-levels,ofthe2,764entrants,41.8%had4ormore;
Alevels_highest_Chemistry
Alevel_Totalbest:SumofthethreehighestA-levelgrades,which
Alevels_highest_Physics was73.0%ofstudents,wasthemaximumscoreof30(thatis,
AAA),with21.3%scoring28(AAB),5.0%scoring26(ABB/AAC),
Alevels_highest_Maths
0.6%scoring24(BBBorequivalent),andfourcandidatesscoring
Alevels_Taken_1_or_more_NonScience 20,16,16and10.Alevel_TotalPoints:Totalpointsachievedbya
studentforallofA-levels,whichforthosetakingthreeA-levels
Alevels_Taken_1_or_more_GeneralStudies
wasthesameasthepreviousmeasure;
Alevels_highest_GeneralStusdie Alevels_Taken_1_or_more_Biology,
Alevels_Taken_1_or_more_Chemistry,
Inadditionequivalentvariablesforother
Alevels_Taken_1_or_more_Physics,and
qualificationsarenamedinsimilarwaysbut Alevels_Taken_1_or_more_Maths):aseriesof‘dummy
withAlevel…replacedbyASlevel…, variables’,scoredas1ifthesubjecthadbeentakenand0ifit
GCSE…,SQAhigher…,SQAhigherPlus… hadnot.95.7%,99.1%,24.8%and63.3%ofAlevelstudentshadA
andSQAadvHigherPlus….
levelsinBiology,Chemistry,PhysicsandMath.
EducationalAttainmentGCE Alevels_highest_Biology,Alevels_highest_Chemistry,
Alevels_highest_Physics,andAlevels_highest_Maths:Highest
EducationalAttainmentSQA gradeattainedbyastudentonBiology,Chemistry,Physicsand
Mathsubjects;exceptforMath,studentsmostlyhadtakenonly
EducationalAttainment
oneexamineachcategory;
zEducationalAttainmentGCE Alevels_Taken_1_or_more_NonSciencewasa1/0dummy
zEducationalAttainmentSQA’ variableindicatingthatastudenthadA-level(s)otherthaninthe
coresciencesofBiology,Chemistry,PhysicsorMath(orGeneral
SQAorGCE Studies).Atotalof49.9%ofthestudentshadatleastonenon-
scienceA-level;Alevels_Taken_1_or_more_GeneralStudies:A
1/0dummyvariableindicatingwhetherastudenthadtaken
GeneralStudiesA-level;26.0%haddoneso;Alevels_highest_
McManusetal.BMCMedicine 2013, 11:244 Page6of25
http://www.biomedcentral.com/1741-7015/11/244
Table1Summaryofvariablesintheanalysis(Continued)
GeneralStudies:ForstudentstakingGeneralStudies,thehighest
gradeattained,46.9%havinganAgrade;
AS(AdvancedSubsidiary)levels.Variablesaresimilartothosefor
A-levelsexceptthattheyarenamedASlevel…ratherthan
ALevel…ScoredasforAlevels(A=10,B=8,C=6,D=4,E=2,
Else=0).Measuresare
similarexceptthatstudentshadtohavetakenatleastfourAS-levels,
andtotalswereforthebestfourAS-levelsachieved.Forreasons
whicharenotclear,fewerstudentshad4+AS-levels(n=1,877)
thanhad3+A-levels(n=2,764).AS-levelgradesshowedmore
variabilitythanA-levels,only56.3%ofstudentsscoringamaximum
40pointsfortheirbestgrades,comparedwith73.0%ofstudents
gaining30pointsfromtheirbestthreeA-levels.
GCSE(GeneralCertificateofEducation).Variablesarebroadlysimilar
tothoseforA-levelsexceptthattheyarenamedGCSE…rather
thanALevel…GCSEresultswereonlyavailableforthe2009entry
cohort.SinglesubjectswerescoredasA*=6,A=5,B=4,C=3,
D=2,E=1,else=0anddoubleScienceandothersubjectswere
scoredasA*A*=12,A*A=11,andsoon,andcountedastwo
GCSEstaken.VeryfewstudentshadeightorfewerGCSEs,and
thereforeoverallscoreswerethereforebasedontheninebest
grades.GCSEscoreswereavailablefor930students,andwere
morevariablethanA-levelsorAS-levels,only16.6%ofstudents
havingthemaximumof54points(equivalentto9A*GCSEs).
Scoreswerecalculatedforthefourindividualcoresciences,and
scorewerealsocalculatedforCombinedScience(takenby32.8%
ofstudents).GCSE_Number_NonScience_Exams:Becauseall
studentshadtakenseveralnon-sciencesubjects,thisvariablewas
thenumberofnon-sciencesubjectstaken.
ScottishHighers.MeasuresarebroadlysimilartothoseforA-levels,
exceptthatnamesbeginSQAhigher…Gradeswerescoredas
A=10,B=8,C=6andD=4.Studentswereonlyincludedwho
hadfiveormoregradesatHighers,thefivehighestbeingsummed.
OtherdifferencesfromA-levelsarethatthereisnoGeneralStudies
component,andalmostallstudentswilltakeanon-scienceHigher.
ResultsforScottishHigherswereavailablefor769students,72.4%
gainingamaximumscoreof50pointsbasedonbestfivegrades.
‘ScottishHighersPlus’.Thisisaconstructionofourown,reflecting
thefactthatalthoughScottishHighersarescoredbyUCASand
bymostScottishuniversitiesasA,B,CandD,theUCASgradesare
actuallyA1,A2,B3,B4,C5,C6andD7.Theseresults,withtwo
bandsateachgrade,aretreatedasmeaningfulbymanyEnglish
universities(althoughnot,itwouldseem,Scottishuniversities),and
thereforewealsoscoredHighersonabasisofA1=10,A2=9,
B3=8,B4=7,C5=6,C6=5andD7=4.Wehavenamedthisas
‘ScottishHighersPlus’,andvariablenamesbeginSQAhigherPlus….
Theseresultshaveawiderrangeofscores,only19.9%ofstudents
gainingthemaximum50points.
ScottishAdvancedHighers.VariablenamesbeginSQAadvHigherPlus….
ManyScottishuniversitiesseemnottorequireAdvancedHighers,
anargumentagainsttheirusebeingthatonlyselectiveschools
havetheresourcesorprovidethepossibilityofstudyingAdvanced
Highers,andhencethereareconcernsaboutwideningaccess.We
note,however,thatinthisgroupofstudents,of478applyingfrom
thestatesector,93.1%hadoneormoreAdvancedHighers,
comparedwith81.8%of237nonstatesectorentrants.Overall,573
studentsinthepresentsurveyhadatleasttwoAdvancedHighers
(thatis,74.5%ofthe769studentswithHighers),andafurther108
hadoneAdvancedHigher.We,therefore,alsoassessedthepredictive
valueofAdvancedHighers.Scoringwasasfor“ScottishHighersPlus”
(thatis,A1=10,A2=9,B3=8,B4=7,C5=6,C6=5andD7=4),
withscorescalculatedforindividualcoresciencesubjects,along
withhighestoverallscoreattained.22%ofthe694studentswithat
leastoneAdvancedHigherhadamaximumof10pointsontheir
bestAdvancedHigher,and22.6%had7orfewerpoints.
McManusetal.BMCMedicine 2013, 11:244 Page7of25
http://www.biomedcentral.com/1741-7015/11/244
Table1Summaryofvariablesintheanalysis(Continued)
Overallmeasuresofeducationalattainment.Asdescribedinthetext,
anoverallmeasureofeducationalattainmentwascalculatedfor
eachstudent,EducationalAttainmentGCEor
EducationalAttainmentSQAforGCEandSQAassessments.These
variableswerebasedonasetofeightortenmeasuresrespectively,
withmissingvaluesreplacedbytheEMalgorithm,andthenthe
firstprinciplecomponentextracted.Asinglevariable,
EducationalAttainmentwascreatedwhichwasthezscoreof
eitherEducationalAttainmentGCEorEducationalAttainmentSQA,
whicheverwasnotmissing.Becausethepresentanalysisisinterested
inmeasureswithinmedicalschools,EducationalAttainmentGCE
andEducationalAttainmentSQAwerealsostandardizedtohavea
meanofzeroandSDofonewithineachmedicalschoolcohort,to
producethevariableszEducationalAttainmentGCEand
zEducationalAttainmentSQA.Wealsousedadummyvariable,
SQAorGCE,toindicatewhetherentrantshadtakenScottishorother
qualifications.NotethatinthepaperonConstructValidity[35]the
unstandardizedmeasureswereused,inorderthatinformationon
applicantsaswellasentrantscouldbeonacommonscale.
UKCATmeasures zUKCATtotal DatawereprovidedbytheUKCATconsortium,withsomeadditional
measurescalculatedbyHICinDundee.Theoverallmeasureof
zUKCATabstractReasoning
performancewasthetotalscore,UKCATtotal,andtherewerealso
zUKCATdecisionAnalysis scoresonthefoursubscalesUKCATabstractReasoning,
UKCATdecisionAnalysis,UKCATquantitativeReasoning,and
zUKCATquantitativeReasoning
UKCATverbalReasoning.Eachofthemeasureswasalso
zUKCATverbalReasoning standardisedasaz-scorewithinmedicalschoolsandcohorts,to
givezUKCATtotal,withthefoursubscalesbeing
UKCATskipped zUKCATabstractReasoning,zUKCATdecisionAnalysis,
UKCATdayOfTakingPctileRank zUKCATquantitativeReasoning,andzUKCATverbalReasoning,
TherewasalsoinformationonthedateoftakingUKCAT,the
UKCATexamSeriesCode variableUKCATdayOfTakingPctileRankgivingrelativedateof
takingthetestwithincohorts,lowscoresindicatingearlytakersof
UKCATcandPerSchool
thetest.Notallcandidatesansweredallquestions,inmostcases
probablybecausetheyranoutoftime,andasaresultonaverage
hadlowerscoresthaniftheyhadguessedatitems,themeasure
UKCATskippedgivingtheoverallnumberofskippeditems,which
hadamedianof4,only25.9%ofcandidatesansweringallitems.
Somecandidateswereallowedextratimebecauseofspecialneeds,
whichisindicatedbythevariableUKCATexamSeriesCode;on
averagethesecandidateshadhigheroverallscoresthanother
candidates.
IntheiranalysesofBMAT[36],Emeryetal.reportedthatcandidates
fromschoolswithmoreextensiveexperienceofthetestperformed
somewhatdifferently,andthereforeacontextualvariable,
UKCATcandPerSchool,wasprovidedbyHICwhichcountedthe
numberofcandidatestakingUKCATinastudent’sschoolsince
thetest’sinception.
Schoolingmeasures SelectiveSchool Someinformationonschooling,includingschoolcodes,was
availablefromUCAS,andtheschoolcodescouldalsobelinked
DFESshrunkVA
intocontextualdataavailablefromtheDepartmentforEducation
DFES.AvePointStudent (DfE;formerlyDFES)atKeyStage5fortheacademicyear2010(file
createdMay2011),forschoolsinEngland.Themergingofthetwo
DFES.AvePointScore:
datasetswascarriedoutbyHIC.Schooltypewasavailablefrom
twoseparatesources,UCASandDFES.InUCAS’sdata,of4,811
students,69hadmissinginformation,360wereinUCAS’s‘Unknown’
category,219were‘ApplyOnlineUK’,and86were‘Other’.Of4,077
studentsforwhominformationwasavailable,1,941(47.6%)were
classifiedascomingfromSelectiveSchools(‘GrammarSchool’or
‘IndependentSchool’),and2,136(52.4%)fromnon-SelectiveSchools
(‘ComprehensiveSchool’,‘Further/Highereducation’,‘SixthForm
Centre’and‘SixthFormCollege’).TheDFESdatabasealsohada
measureofSelectiveSchooling,withinformationon2,830
individualsavailable,ofwhom1,387(49.0%)attendedselective
schools.TheoverlapoftheUCASandDFESclassificationswasgood,
butnotperfect.Ourfinalmeasure,entitledSelectiveSchoolhada
valueof1ifeitherUCASorDFESdatasuggestedaschoolwas
selective,andotherwisewas0.Altogetherofthe4,811individuals
inthePrimaryDatabase,informationwasavailableforoneorboth
McManusetal.BMCMedicine 2013, 11:244 Page8of25
http://www.biomedcentral.com/1741-7015/11/244
Table1Summaryofvariablesintheanalysis(Continued)
sourcesin4,114cases,ofwhom1,986(48.3%)hadevidenceof
havingattendedaselectiveschool.
Contextualschoolmeasures.TheDfEdatahadatotalof22
contextualmeasuresonschools.Afterarangeofpreliminary,
exploratoryanalysesweconfinedtheanalysestothreevariables:
DFESshrunkVA,whichisameasureofvalueaddedbetweenKey
stages4and5,andwasavailablefortheschoolsof2,561students;
DFES.AvePointStudent,whichisameasureoftheaveragepoints
gainedbyeachstudentataschoolacrossallofthatschool’s
examinationentries,andwasavailablefortheschoolsof2,586
students;andDFES.AvePointScore,whichisasimilarmeasure
tothepreviousoneexceptthattheaverageisatthelevelof
examinationentries(ratherthanstudents),andwasavailablefor
theschoolsof2,582students.
Demographicmeasures UK Nationalitywasbasedontheonlineinformationprovidedwhen
studentstookUKCAT;of4,811students,4,598(95.6%)wereUK
UCAS.male
nationals,176(3.7%)wereEU/EEAnationalsand37(0.8%)were
CAND.Age fromoutsidetheEU/EEA;thebinaryvariablewascalledUK.
CAND.AgeGT21 SexwasbasedoninformationprovidedbyUCAS;of4,811students,
2,081(43.3%)weremaleand2,730(56.7%)werefemale.The
CAND.Age30plus
variablewascalledUCAS.male,scoring1=maleand0=female.
UCAS.Ethnic2. AgewasbasedonstatedageinyearswhentakingtheUKCATtest,
andrangedfrom17to45(mode=18,mean=19.55,SD=2.84).
Agewasmissingin45cases,28.9%ofstudentswereaged21+,
and1.3%wereaged30+.ThevariablewascalledCAND.Age.
Additional0/1variableswerecreatedtoindicatewhether
candidateswere21orolderor30orolder(CAND.AgeGT21,
CAND.Age30plus).
Ethnicitywasbasedonthestandard23categoriesintheUCAS
coding.Ethnicitywasmissingin69cases,for214wascodedas
Unknown,andfor192wascodedas‘Notgiven’.Onasimplified
sixcategorybasistherewere3,057White,577Indiansub-continent,
223OtherAsian,92Black,140Mixedand60Other.Forsimplicity,
andasinmanyotherstudies[37])wegroupedstudentsasWhite
(n=3,057,73.7%)andNon-White(n=1,092,26.3%),inavariable
calledUCAS.Ethnic2.
Socio-economicmeasures CAND.NSSEC Socio-economicclassification(SEC),variableCAND.NSSEC,was
basedontheonlineinformationprovidedbystudentstaking
IMDOverallQualityDecile
UKCAT,whocompletedtheabbreviatedversionoftheself-coded
IMD1IncomeDecile(withtwosubscales) questionnaire(NS-SEC)providedbyUKNationalStatistics2.SECwas
calculatedseparatelyforeachparent(ifprovided),andthehigher
IMD2EmploymentDecile
SECused.Of4,091individualswithusableinformation,3,740(91.4%)
IMD3HealthDisabilitySkillsDecile wereinSECgroup1,105(2.6%)ingroup2,146(3.6%)ingroup3,
38(0.9%)ingroup4,and62(1.5%)ingroup5,wheregroup1has
thehigheststatus.
IMD4EducationDecile(withtwosubscales), Socio-economiccontextualmeasures.ForapplicantslivinginEngland,
postcodesforplaceofresidencewereusedtolinktosmall-area
IMD5HousingAndServicesDecile(withtwo
censusstatisticscollectedaspartofTheEnglishIndicesof
subscales)
Deprivation[38]andwhichgenerateaseriesofIndicesofMultiple
IMD6CrimeDecile, Deprivation(IMD).Foreaseofanalysis,HICconvertedthemeasures
todeciles,lowscoresindicatinggreaterdeprivation.
IMD7LivingEnvironmentDecile(withtwo
IMDOverallQualityDecileprovidesanoverallsingleindicatorof
sub-scales).
deprivation.Inadditionthereare15moredetailedscalesand
subscales,whosenamesaremoderatelyself-explanatory:
IMD1IncomeDecile(withtwosubscales),IMD2EmploymentDecile,
IMD3HealthDisabilitySkillsDecile,IMD4EducationDecile(withtwo
subscales),IMD5HousingAndServicesDecile(withtwosubscales),
IMD6CrimeDecile,andIMD7LivingEnvironmentDecile(withtwo
sub-scales).Notethatalthoughthesescalesaredescribedinterms
ofdeprivation,theyarescoredas1=highdeprivationand10=
lowdeprivation,andthereforearerenamedas‘Quality’sothat
higherscoresindicateahigherqualityonthemeasure.
McManusetal.BMCMedicine 2013, 11:244 Page9of25
http://www.biomedcentral.com/1741-7015/11/244
schemes,andcandidateshave chosen tostudy componentextracted(TRTable4).Since the
different subjects.Four medical schools werefrom principle componentsforthetwoanalyseswere
Scotlandand eight fromthe restoftheUK,entrants bothon standardizedscales,they couldbe combined
tothe former mostly, butnotentirely,taking toprovidean optimal summarymeasureof
ScottishHighersrather thanA-levels.Noeasy EducationalAttainmentforthe majority of
solutionispossibleforthedifficultproblem of students.Educationalattainment measuresdiffer
equatingthetwodifferent setsofresults[37],and betweenmedicalschoolsandbetweencohorts,but
we havefollowedtheapproachofTiffinetal.[19]in because themaininterest inthisstudy isprediction
convertingScottishHighersandA-levelstoz-scores, within medical schools,wehavestandardized
whichcanthenbe combined. TheTechnicalReport EducationalAttainmentwithincohortsandmedical
[34]describesanextensivesetofpreliminary schools,resultinginthe variable we call
analysesofthewiderangeofdifferentmeasuresof zEducationalAttainment.It shouldbenoted that
attainment(seeTable1).Briefly,each ofthe 42 educational qualificationswereonlyavailablein
derivedscoresforA-levels,AS-levelsand GCSEs most casesfornon-maturestudents(age lessthan
wascorrelated withOverallScore(TRTable1a,b). 21).Althoughstatisticallyoptimal,andhencegood
Multipleregressionssuggestedthatonlya subsetof forassessingunderlyingprocessesusingasmuch
eight measures(TRTable2)showedindependent informationaspossible,werealizethatzEducational
predictionsofoutcome.Missing values forthese Attainmentdoesnotreflectthecurrentselection
eight measureswerereplacedbyexpectation- processes,andthereforewealsoreportresultsfor
minimization(EM)imputation,theresulting8x8 themoreconventionalmeasuresofthreebestA-
correlationmatrixfactoranalyzed,thefirstprinciple levels,fourbestAS-levels,ninebestGCSEs,fivebest
componentextracted, (which hasameanof zero ScottishHighers,fivebestScottish“HighersPlus”
and SDofone),and scoresonthatwere usedasan (whichincludesfinerdefinitionofbandswithin
optimal summarymeasure ofattainmentatA-level, grades),andthebestScottishAdvancedHigher.
AS-leveland GCSE.A similar processwascarried 3. UKCATSCORES.ThemainmeasuresfromUKCAT
outforthe 51derived measuresofHighers,‘Highers werethe scoresonthe cognitivetests,thetotal
Plus’and Advanced Highers,each ofwhich was score,UKCATtotal,andthe scoresonthefour
correlated withOverallScore(TRTable3).Ten subtests,UKCATabstractReasoning,
independentpredictorswere found,missing values UKCATdecisionAnalysis,
replacedbyimputation,and thefirstprinciple UKCATquantitativeReasoningand
Table2CorrelationsofUKCATsub-scoreswithoutcomes
Abstract Decision Quantitative Verbal OverallMark SkillsMark TheoryMark
reasoning analysis reasoning reasoning
Abstractreasoning(zUKCATabstractReasoning) 1 .196*** .190*** .114*** .080*** .053** .052*
(4,811) (4,811) (4,811) (4,811) (3,184) (2,075)
Decisionanalysis(zUKCATdecisionAnalysis) .196*** 1 .156*** .146*** .090*** .056*** .077***
(4,811) (4,811) (4,811) (4,811) (3,184) (2,075)
Quantitativereasoning .190*** .156*** 1 .213*** .076*** .044* .079***
(zUKCATquantitativeReasoning)
(4,811) (4,811) (4,811) (4,811) (3,184) (2,075)
Verbalreasoning(zUKCATverbalReasoning) .114*** .146*** .213*** 1 .115*** .028 .177***
(4,811) (4,811) (4,811) (4,811) (3,184) (2,075)
TotalUKCATscore(zUKCATtotal) .604*** .655*** .583*** .591*** .148*** .075*** .160***
(4,811) (4,811) (4,811) (4,811) (4,811) (3,184) (2,075)
Educationalattainment(zEducationalAttainment) .144*** .131*** .133*** .087*** .362*** .210*** .351***
(3,432) (3,432) (3,432) (3,432) (3,432) (2,240) (1,407)
ThreebestA-levels(Alevels_TotalBest) .123*** .121*** .127*** .062** .177*** .096*** .248***
(2,764) (2,764) (2,764) (2,764) (2,764) (2,000) (1,250)
Fivebesthighers(SQAhighers_TotalBest) .083* .129*** .202*** .070 .003 .027 .074
(769) (769) (769) (769) (769) (298) (199)
CorrelationsoftheUKCATsubscaleswitheachother,withUKCATtotalscore,andwithpriorEducationalAttainment(threebestA-levels,fivebestHighersand
zEducationalAttainment),andmedicalschoolperformance(Overall,andSkillsandTheoryassessments);***P<.001;**P<.01;*P<.05.
Table3SimplePearsoncorrelationsofkeymeasureswitharangeofdemogErdaupchaticio,nsaclhAototali,nsmoecniatlan3d/5UbKeCstAAT-lepvreolcs/eHsisghmeresasUuKrCeAsTtotalscore Overallmedicalschool http://w McMan
(zEducationalAttainment) (Alevels_TotalBest (zUKCATtotal) score(OverallScore) w u
w s
SQAhighers_TotalBest) .b et
Demographic UKnational(UK) .008 .000/-.042 .060*** -.007 iom al.
measures e BM
(3,432) (2,764/769) (4,811) (4,811) dc C
e M
Male(UCAS.Male) (-3.0,43372*) (.022,766/4.0/57869) (.046,7142**)* -(4.0,73492*)* ntral.c edicin
o e
AAggeedd2310++((CCaanndd..AAggeeG30Tp2l1u)s) nn//aa nn//aa -(-4..00,2763606*)** .(-04.08,070366**)* m/1741-702013, 11:2
14
(4,766) (4,766) 5/14
1
Ethnic2(non-White)(UCAS.Ethnic2) -.053** -.062**/-.033 -.141*** -.142*** /2
4
4
(3,221) (2,549/766) (4,149) (4,149)
Schoolmeasures Selectiveschooling(SelectiveSchool) .051** .038*/.120*** .075*** -.101***
(3,432) (2,764/769) (4,811) (4,811)
DfESvalueaddedKS5(DFESshrunkVA) -.012 .012/n/a -.014 -.049*
(2,092) (2,119) (2,561) (2,561)
DfESaveragepointsperstudent(DFES.AVEPointStudent) .085*** .127***/n/a .097*** -.065***
(2,114) (2,141) (2,586) (2,586)
DfESaveragepointsperexamentry(DFES.AvePointScore) .111*** .101***/n/a .044* -.111***
(2,109) (2,136) (2,582) (2,582)
Socialbackground Socio-economicclassification(SEC)(1=High5=Low) -.058* -.084***/-.046 -.056*** -.011
(CAND.NSSEC)
(2,939) (2,356/675) (4,091) (4,091)
Overalldeprivationdecile(1=high,10=lowdeprivation) .079*** .076***/n/a .113*** .032
(IMDOverallQualityDecile)
(2,275) (2,307) (3,074) (3,074)
Incomedeprivationdecile(IMD1IncomeDecile) .078*** .083***/n/a .125*** .039*
(2,275) (2,307) (3,074) (3,074)
Employmentdeprivationdecile(IMD2EmploymentDecile) .063** .073***/n/a .109*** .008
(2,275) (2,307) (3,074) (3,074)
Healthdisabilitydecile(IMD3HealthDisabilitySkillsDecile) .055** .048*/n/a .098*** .016
(2,275) (2,307) (3,074) (3,074)
P
Educationdeprivationdecile(IMD4EducationDecile) .056** .064**/n/a .083*** -.019 a
g
e
(2,275) (2,307) (3,074) (3,074) 1
0
o
f
2
5