Table Of ContentSmitsetal.BMCMedicalResearchMethodology2012,12:4
http://www.biomedcentral.com/1471-2288/12/4
RESEARCH ARTICLE Open Access
A proof of principle for using adaptive testing
in routine Outcome Monitoring: the efficiency
of the Mood and Anxiety Symptoms
Questionnaire -Anhedonic Depression CAT
Niels Smits1*, Frans G Zitman2,3, Pim Cuijpers1, Margien E den Hollander-Gijsman2 and Ingrid VE Carlier2
Abstract
Background: In Routine Outcome Monitoring (ROM) there is a high demand for short assessments. Computerized
Adaptive Testing (CAT) is a promising method for efficient assessment. In this article, the efficiency of a CAT
version of the Mood and Anxiety Symptom Questionnaire, - Anhedonic Depression scale (MASQ-AD) for use in
ROM was scrutinized in a simulation study.
Methods: The responses of a large sample of patients (N = 3,597) obtained through ROM were used. The
psychometric evaluation showed that the items met the requirements for CAT. In the simulations, CATs with
several measurement precision requirements were run on the item responses as if they had been collected
adaptively.
Results: CATs employing only a small number of items gave results which, both in terms of depression
measurement and criterion validity, were only marginally different from the results of a full MASQ-AD assessment.
Conclusions: It was concluded that CAT improved the efficiency of the MASQ-AD questionnaire very much. The
strengths and limitations of the application of CAT in ROM are discussed.
Background administration of a test or questionnaire via the compu-
In the clinical field, self report questionnaires are fre- ter, and at its core lie psychometric models. Each item
quently used to assess mental health, and there is a high is dynamically selected from a pool of items and is most
demand for efficient assessments [1-3]. This demand is informative for the responder in question. The CAT
most apparent in mental health institutes which are stops when a given level of measurement precision is
involved in Routine Outcome Monitoring (ROM, [4,5]), attained, and typically needs substantially less items
a method devised to collect data on the effectiveness of than the full item set. In the last decade, CAT has
treatments in clinical practice. It involves recording the received a lot of attention in the field of quality of life
outcome of treatments through repeated assessments, research. For example, the Patient Reported Outcomes
which allow for monitoring the development of patient Measurement Information System (PROMIS [2]) project
characteristics through time. In this setting, care provi- has as its goal the development of CATs for the mea-
ders and patients only have a very limited amount of surement of physical and mental outcomes which allow
time, and assessments should be short. for monitoring the health-related quality of life of medi-
A very successful methodology on this point is Com- cal patients. CATs have now been developed for depres-
puterized Adaptive Testing (CAT). CAT involves the sion [6,7] and anxiety [8].
Although these CATs are readily available, there are at
least two reasons for clinicians and researchers not
*Correspondence:[email protected] using them in ROM. In the first place, a CAT may not
1DepartmentofClinicalPsychology,FacultyofPsychologyandEducation,
have been developed in the right language. For example,
VUUniversity,Amsterdam,TheNetherlands
Fulllistofauthorinformationisavailableattheendofthearticle
©2012Smitsetal.;licenseeBioMedCentralLtd.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommons
AttributionLicense(http://creativecommons.org/licenses/by/2.0),whichpermitsunrestricteduse,distribution,andreproductionin
anymedium,providedtheoriginalworkisproperlycited.
Smitsetal.BMCMedicalResearchMethodology2012,12:4 Page2of10
http://www.biomedcentral.com/1471-2288/12/4
Dutch researchers wishing to implement the Depres- sufferedfromananxietydisorder,and17%sufferedfrom
sion-CAT [6] need to first translate its items from Ger- a somatoform disorder. In addition, comorbidity among
man into Dutch and then validate them in a relevant thedisorderswasfound:2% ofthe patientshadallthree
Dutch sample, which may simply be too much of an disorders,3%hadbothanxietyandsomatoformdisorder,
effort. Likewise, clinicians may be used to assessing 3% had both depression and somatoform disorder, and
mental health with an existing ROM instrument with 18% suffered from both depression and anxiety; 8, 20,
familiarcontent,andthereforeswitchingtoanalternative and 23% exclusively suffered from somatoform, anxiety
measure may be unappealing. Instead of switching to and depression disorder, respectively, whereas 23% had
existing CATs it may be feasible to convert the familiar nodisorder.
scale into a computer-adaptive version. If clinics have Rivierduinencollaborates withtheDepartmentofPsy-
collected large samples of item scores of their preferred chiatryoftheLeidenUniversityMedicalCenter(LUMC)
scale using ROM, they can use these data to validate its in the development of ROM. At intake, patients were
itemsunderapsychometricmodel,andusetheoutcomes informed that ROM is part of the general policy of Riv-
todevelopanew CAT,allowingformuchshorterfuture ierduinen and LUMC to monitor treatment outcome,
assessments. thatoutcomesaremadeavailable onlytotheirtherapist,
The present paper has two goals. The first is to show andthatthedatawouldbeusedforresearchpurposesin
that the efficiency of the assessment of the Mood and anonymous form. If patients objected to such use, their
AnxietySymptomQuestionnaire-AnhedonicDepression datawouldberemoved.Acomprehensiveprotocolsafe-
scale (MASQ-AD [9]) can be substantially improved by guarded anonymity of the patients and ensured proper
buildinganadaptiveversionofit.Althoughseveralstudies handlingofthedata.Thisprotocol(PsychiatricAcademic
on building CAT versions of existing clinical scales have Registration Leiden database) was available for patients
beenconducted[1,10], thesestudiesfocusedonresearch onrequest,andinformed consentwasnotrequired.The
settings, and not on the assessment of mental health Medical Ethical Committee of the LUMC approved the
patients. Our second goal is therefore to show that such regulations and agreed with this protocol (for more
CATsare an efficient method formonitoringpatients in details,seeDeBeursetal.[4]).
clinicalsettingsaswell.WeusedtheROM-dataofalarge
sample of Dutch patients who filled out the full MASQ- The MASQ
AD as input for a CAT simulation: for each respondent, The MASQ [9,12] is a 90-item self-report questionnaire
theactualresponsesofthefulladministrationweretreated which has three scales: Anhedonic Depression (AD),
asiftheyhadbeencollectedadaptively. Anxious Arousal (AA) and General Distress (GD). AD
Thisarticlehasthefollowingstructure.Inthemethods measures(thelackof)positiveaffect,AAmeasuressymp-
and results sections, the CAT simulation and its out- toms of somatic arousal and GD measures non-specific
comes are described. To ameliorate readability, the eva- symptoms for a depressive or an anxiety disorder. The
luation of the psychometric quality of the MASQ-AD, responderisaskedtoindicateonaLikertscale(0=notat
whichisaprerequisiteforapplyingCATbut-obviously- all, 2 = a bit, 3 = moderately, 4 = much, 5 = very much)
not the main topic of this study, is presented in the howmuchtheyhavefeltorexperiencedthesestatedfeel-
Appendix. Finally, we discuss some benefits and limita- ings or thoughts in the past week including today. The
tionsoftheapplicationofthismethodologyinROMand Dutch adaptation of the MASQ [13] was used. Here, we
mentalhealthresearchingeneral. focusonthedevelopmentofaCATfortheADsub-scale,
whichcontains22items.Thecontentoftheitemsispre-
Methods sentedintheleftpartofTable1.
Participants In the Appendix a psychometric evaluation of the
The sample consisted of 3,597 patients (63% females) MASQ-AD using the current sample is presented. On
from three outpatient centres of Psychiatric Regional the basis of this evaluation it was concluded that the
Mental Health Care Centers Rivierduinen with an aver- item set of the MASQ-AD was a valid input for an
ageageof38.79years(SD=13.22,range17-91).Patients adaptive test.
werereferredtotheseclinicsbytheirgeneralpractitioner
for a potential mood, anxiety or somatoform disorder. The Simulated MASQ-AD CAT
Thediagnosiswasassessedwithastandardizeddiagnos- A CAT algorithm usually consists of five components
tic interview, the Mini International Neuropsychiatric [14,15]. The first component consists of the estimated
Interview (MINI-plus [11]), which was carried out by a parameters of an appropriate Item Response Theory
research nurse (a psychiatric nurse or a psychologist). (IRT) model for the items which form the item ‘pool’.
According to the MINI, 46% of the patients suffered The second component is a method to select new items
fromdepression(4%minor,42%majordepression),43% during the assessment; the third is a method to make a
Smitsetal.BMCMedicalResearchMethodology2012,12:4 Page3of10
http://www.biomedcentral.com/1471-2288/12/4
Table 1Estimated GRMparameters ofthe itemsofthe MASQ-AD (N =3597).
Nr.(MASQNr.) Item ItemParameterEstimates
a b b b b
1 2 3 4
1(1) Feltcheerful(+) 2.41 -2.50 -1.08 -0.08 0.78
2(14) Feltreallyhappy(+) 2.61 -2.52 -1.44 -0.57 0.18
3(18) Feltoptimistic(+) 2.37 -2.74 -1.47 -0.60 0.40
4(21) Feltreallybored(-) 0.98 -0.40 0.81 1.86 3.58
5(23) FeltlikeIwashavingalotoffun(+) 2.75 -2.77 -1.51 -0.69 0.08
6(26) Feltwithdrawnfromotherpeople(-) 1.18 -0.46 0.74 1.44 2.81
7(27) Seemedtomovequicklyandeasily(+) 1.41 -3.05 -1.33 -0.26 0.68
8(30) Lookedforwardtothingswithenjoyment(+) 2.55 -2.11 -0.98 -0.25 0.61
9(33) Feltlikenothingwasveryenjoyable(-) 2.08 -0.59 0.26 0.80 1.77
10(35) FeltlikeIhadaccomplishedalot(+) 1.74 -3.49 -2.03 -1.07 -0.20
11(36) FeltlikeIhadalotofinterestingthingstodo(+) 1.94 -3.34 -1.86 -0.98 -0.17
12(39) Feltlikeittookextraefforttogetstarted(-) 1.08 -1.72 -0.40 0.34 1.93
13(40) FeltlikeIhadalottolookforwardto(+) 2.21 -2.65 -1.43 -0.62 0.27
14(44) Feltliketherewasn’tanythinginterestingorfuntodo(-) 1.38 -0.72 0.33 1.10 2.39
15(49) Wasproudofmyself(+) 2.06 -2.84 -1.68 -0.77 0.29
16(53) Feltunattractive(-) 0.90 -0.68 0.57 1.49 2.65
17(58) Feltreally“up”orlively(+) 2.90 -2.72 -1.65 -0.81 -0.20
18(66) Feltreallysloweddown(-) 1.15 -0.75 0.51 1.34 2.75
19(72) FeltlikeIhadalotofenergy(+) 2.01 -3.00 -1.79 -0.84 -0.14
20(78) Felthopefulaboutthefuture(+) 1.96 -2.59 -1.40 -0.53 0.44
21(86) Feltreallygoodaboutmyself(+) 2.63 -2.88 -1.57 -0.69 0.22
22(89) Thoughtaboutdeathorsuicide(-) 1.18 0.86 1.71 2.26 3.34
Note.aisthediscriminationparameter;theb’sarethresholdparameters.
Aplusrepresentsapositivelyandaminusanegativelyformulateditem;positivelyformulateditemswerescore-reversed.
(provisional) estimate of the subject’s scale score after b, parameters which specify the location on θ on which
the administration of each item. Because a provisional one is expected to step from a lower to a higher item
score estimate is unavailable before administrating the category; the number of bs per item is equal to the
first item, a starting level is specified, which is the fourth number of item categories minus one. A more thorough
component. The fifth component of CAT is a rule description of the GRM and the meaning of its para-
which specifies when to stop the assessment. Details meters can be found in other sources [10,19].
associated with the five CAT components of our simula- The estimates of the complete sample (see, Table 1)
tion, are presented in what follows. are evidently the best estimates of the population GRM
To obtain item parameters of the MASQ-AD items, parameters. It would, however, be unfair to use these
the Graded Response Model (GRM [16]) for polytomous estimates in the present CAT simulations. Utilizing the
items, was fit to the data. Although other IRT model same sample to both calibrate the items and to simulate
candidates exist, the GRM is often preferred because (i) the CAT upon, may lead to overfitting [20], giving out-
it has parameters which can be easily interpreted in comes which are too optimistic. To deal with this issue,
terms of the responder behavior [17], and (ii) it is easier two-fold cross validation [21] was performed: the sample
to understand and illustrate to users than the other was randomly split in two equally sized groups. In each
models [18]. Like all IRT models, the GRM assumes of the two sub-samples, the parameters of the GRM
that the responses to the items of a questionnaire are were estimated. Next, each set of estimates was used as
accounted for by a latent construct (often denoted by θ) input for the CAT of respondents in the other sample.
and characteristics of the items. It specifies two types of In other words, for each subject, the simulated CAT
item parameters which quantify the relationship used parameter estimates that were obtained in the sub-
between the latent trait and the item score. Each item sample (s)he did not belong to.
has one discrimination, or a, parameter, which expresses Like inmostotherCATs,the method ofselectingnew
the discriminative power of an item to demarcate differ- itemswasbasedoniteminformation[15,19].Iteminforma-
ences between respondents of similar scores on the tionquantifieswithhowmuchprecisionanitemcanmea-
latent trait. In addition, it has one or more threshold, or sure the latenttraitgiven the locationof the provisional
Smitsetal.BMCMedicalResearchMethodology2012,12:4 Page4of10
http://www.biomedcentral.com/1471-2288/12/4
person estimate [22]. In the simulation study,the CATs focusedonthecorrespondenceincriterion-relatedvalid-
selectedthatnewitemwhichhadthehighestinformation ity [22] between fixed-test and CAT estimates; that is,
attheprovisionalestimateofθ. thesetwotypesofestimateswerecomparedattheirrela-
TwoscoreestimationmethodsareavailableinIRT:Max- tion with three other measures. Two criteria were the
imumLikelihood(ML)andBayesianestimation[19].The total scores of the two other sub-scales of the MASQ:
MLapproachestimatesθasthatvaluewhichhasthehigh- AAandGD.ThevaliditywasdeterminedusingthePear-
est likelihood of bringing forth the responses observed son correlation coefficient. The third criterion was an
[23].Bycontrast,Bayesianestimationuses,inadditionto ‘eithermajororminor’depressionclassificationbasedon
this likelihood, an a priori population distribution of the the MINI diagnosis. The predictive utility for this out-
latentvariable,suchasthestandardnormal.Consequently, comewasexpressedintheAreaUndertheCurve(AUC)
Bayesian estimation can and ML estimationcannot pro- ofthereceiveroperatingcurve,whichisaneffectsizefor
vide an estimate for item response patterns consisting expressing clinical significance [25]. It was presented
exclusivelyofeitherextremelowerorextremehighercate- here to provide clinicians with a quantification of the
gories.Becausesuchresponsepatternswereanticipatedin impactofCATonpredictiveutility.Inthepresentstudy,
thepresentstudy,theestimationofθwasperformedusing AUC may be interpreted as the probability that a ran-
a Bayesian method, called Maximum a Posteriori (MAP domly selected depressed person scores higher on the
[19]), which assumed θ to follow a standard normal MASQ-AD than a randomly selected person without
distribution. depression[26].
In the CAT procedure the starting levelwas set to the
averagevalueofthelatenttrait,whichwas0.Asaconse- Software
quence, the item with the highest information at this LISREL[27]wasusedfortheConfirmatoryFactorAnaly-
initial latentdepressionvalue,Item5,waschosenasthe sis(CFA,see Appendix).Allotheranalyses(see,Appen-
firstitemforallrespondents. dix) were performed using several R [24] libraries. Item
Generally,therearetwotypesofstoppingrulesforter- parameterswere estimatedusing the ltm [28,29] library.
minatingaCAT:eitherafixednumberofitemsadminis- Thepolychoriccorrelationswereobtainedwiththepoly-
tered or a pre-specified level of measurement precision. corlibrary[30].Mokkenscalingwasperformedusingthe
The lattercriterion,whichwasused inthisstudy, ismet mokken library [31,32]. DIF detection was performed
when the subject’s Standard Error (SE) of θ is small using the lordif library [33,34]. The code of the CAT
enough. To illustrate the impact of this rule, the CAT algorithm comprised an alteration of, and additions to
wasrununderseverallevelsofminimallyrequiredSEsof the code of the ltm library, and is obtainable from the
θ(0.2,0.3,0.4,0.5,and0.6). firstauthor.
To simulate the adaptive version of the MASQ-AD, a
CATprogram[10]waswritteninthestatisticalenviron- Results
ment R [24]. To perform a simulation for a particular Characteristics of the CAT
questionnaire, the program needs (a) estimates of its Table 2 presents several features of the CAT procedure
GRM parameters and (b) a data file with scores on its under the different stopping rules. The first row shows,
items from a sample from the population in question as as a reference point, the outcomes of the CAT with no
input. Inthe simulation, for eachresponder in the sam- stopping rule. The first and second column show the
ple, the full set of MASQ-AD item responses was used, average and standard deviation (SD) of the number of
anditemscoreswereselectedfromitand evaluatedasif items administered; the third column presents the per-
theywerebeingcollectedadaptively. centage of respondents for whom all 22 items had to be
administered. Clearly, the higher the required level of
Comparing complete and CAT data measurement precision, the higher the number of items
Evidently, for an adaptive MASQ-AD to be efficient, its administered.
outcomes should be very similar to those of the full The fourth column shows the average SE of the final
assessment. Moreover, the usefulness of the CAT esti- estimate of θs for each level of measurement. The aver-
mate, for example in diagnosing depression, should be ageSEunderthe‘SE(θ)<0.2’stoppingruleishigherthan
similar to that of the fixed questionnaire estimate. the required SE; obviously, this resulted from the 65.3%
Twoanalyseswere performedto determine the extent of the respondents for whom the item pool was
to which CAT assessment were in accordance with the exhaustedbeforethepre-specifiedmeasurementwasmet
fullassessment.Inthefirstplace,CATestimatesoflatent (i.e., SE was higher than desired). The upper panel of
depressionwerecomparedwithestimatesresultingfrom Figure 1 shows the number of items administered as a
the full assessment (i.e., 22 item scores), using Pearson function of the estimated θ in the CAT with the ‘SE(θ)
correlations between the estimates. The other analysis <0.3’ stopping rule. From this figure [Figure 1] it
Smitsetal.BMCMedicalResearchMethodology2012,12:4 Page5of10
http://www.biomedcentral.com/1471-2288/12/4
Table 2Characteristics ofthe CATunder several stopping rules.
Stoppingrule NumberofItemsUsed MeanSE(θ) MarginalReliability CorrelationbetweenCATθandCompleteTestθ
Mean SD %All
None 22.000 0.000 100.0 0.231 0.945a 1.000
SE(θ)<0.2 19.726 3.504 65.3 0.234 0.944 0.999
SE(θ)<0.3 8.644 5.712 11.0 0.298 0.910 0.979
SE(θ)<0.4 4.484 3.849 2.9 0.373 0.850 0.946
SE(θ)<0.5 2.789 1.840 0.3 0.425 0.793 0.917
SE(θ)<0.6 1.505 0.520 0.0 0.521 0.577 0.837
aCoefficientalphaforthefullMASQ-ADwas0.940.
becomesclearthatthe11%ofrespondentsmakingall22 marginalreliability[35]inthefifthcolumnofTable2.In
items were mostly found on the right-hand side of the IRT, there is no single estimate of reliability, because
latent depression scale. This is a result of the test infor- measurement precisionvaries asa functionofthe latent
mation being relatively low in that region (for the three trait.MarginalreliabilityisarathercrudeestimateofIRT
subjectslefttothecenter,thefullsetofitemswasneeded measurementprecisionwhentestinformationispeaked;
because of inconsistent response behavior: they chose thereforeitshouldbekeptinmindthatthecurrentmar-
low categories for easy to endorse items and high cate- ginal reliability estimates are inaccurate, and used for
goriesfordifficulttoendorseitems). illustrativepurposesonly.Nevertheless,itisobviousthat
To provide readers unfamiliar with IRT a sense of the reliabilitydecreasesastheCATusesastoppingrulewith
meaning of the CAT SEs, we present an estimate of ahigherstandarderror.
5
2
5
2
d
e ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● 0
r 2
e ● ●● ●●●●●●
st 0 ●●●●●●●●●●●●●●●●●●
mini 2 ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●● on
Ad ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● 15 mati
s ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● r
r of Item 15 ● ● ● ● ● ●● ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● 10 Test Info
e ● ● ● ●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
mb 10 ● ● ● ● ●●● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
u ● ●● ●● ● ●●● ●●●●● ●●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
N ●●●●● ● ●●●●●●●●●●● ●●●●●●●●●●●●●● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● 5
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
5 ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
−3 −2 −1 0 1 2 3 θ
y
sit
n
e
D
Figure1Theupperpanelshowstherelationshipbetweenthelatentdepression(θ)estimateandthenumberofadministereditems
forstoppingrule‘SE(θ)<0.3’(bluedotsrepresentrespondents).Theblackcurverepresentstestinformationasafunctionofθ.Thelower
panelshowsthekerneldensityestimatesofthedistributionofθinthegroupwith(darkblueline)andwithout(lightblueline)adepression
diagnosisaccordingtotheMINI.
Smitsetal.BMCMedicalResearchMethodology2012,12:4 Page6of10
http://www.biomedcentral.com/1471-2288/12/4
The sixth column presents the correlations between CATsunderfivestoppingrules(requiredstandarderrors
the complete data and CAT θ estimates. These correla- in decreasing steps of 0.10) were performed. Naturally,
tions were quite high. For example, whilst the respon- whenincreasingtherequiredmeasurementprecision,the
dents answered, on average, only 5 of the 22 items in average number of administered items increased. Like-
the ‘SE(θ) <0.4’ CAT, a correlation of 0.946 was found. wise,therelationshipbetweenthelatentdepressionesti-
Self-evidently, the correlations decreased as measure- matesusingthefullandadaptiveassessmentincreasedas
ment precision decreased, with a somewhat lower corre- measurement requirements increased. Moreover, with
lation (0.837) when, on average, administering only 1.5 increasingrequiredmeasurementprecision,thecriterion
item in the ‘SE(θ) <0.6’ CAT. validityofthelatentdepressionestimatewasdecreasingly
attenuatedbymeasurementerror.
Criterion Validity of the CAT In spite of the obvious loss of information as require-
Table 3 shows the relationship between the CAT esti- mentsofmeasurementprecisionwere relaxed,themag-
mates and the three criterion variables. Results were nitudeofthislosswassurprisinglylow.Forexample,the
highly similar for these three measures: the criterion- CATrequiringSEtobeatleast0.4,recordingonaverage
related validity was an increasing function of the only about a fifth part of the itemsper respondent, gave
required measurement precision of the CAT. depression scores that correlated 0.946 with the full
The correlation of the full CAT θ with the MASQ-AA assessment score, and 0.651 with the MASQ-GD scale
total score was 0.460. This correlation diminished as score,whichwasonly marginally smallerthan the origi-
measurement requirements went down. For example, nalconcurrentvalidity(0.739).Inaddition,theCATesti-
when using the ‘SE(θ) <0.5’ rule (average number of matesunderthe‘SE(θ)<0.5’ stoppingrule,recording on
items administered 2.789), the correlation dropped to averageonlyaboutaseventhoftheitemsperrespondent,
0.385. The correlation of the ‘no stopping rule’ CAT θ correlated0.917withtheoriginalscore,andhadanAUC
and the MASQ-GD total score was 0.739 (column two). (0.769)forpredictingdepressionthatwasonlymarginally
This value became smaller as the measurement require- smallerthanthatofthefullassessmentestimates(0.810).
ments decreased. For example, the CAT using the ‘SE(θ) This study, therefore, links up to other studies on CAT
<0.6’ rule (administering on average only 1.5 item) gave [1,6,7,10] in its conclusion that it is a fruitful way of
a correlation of 0.556. increasingtheefficiencyofselfreportquestionnaires.
The last column of Table 3 reports on the CAT’s Beforediscussingwhichstoppingruleisbestforareal
diagnostic accuracy, expressed in AUC, for the depres- MASQ-AD CAT in ROM, it should be stressed that the
sion classification. The diagnostic accuracy of the esti- present outcomes were based on a simulated CAT on
mates of θ based on the ‘no stopping rule’ was high and data that were obtained in a standard assessment.
went down somewhat as measurement requirements Obviously,simulatedandactualadaptiveadministrations
decreased. However, the estimates remained clinically may yield different results concerning item reductions
significant under all stopping rules: all AUCs were because respondents may behave differently in reality.
higher than the value commonly used as a lower bound Therefore, in addition to the present study, an actual
for a large effect [36]. MASQ-AD CAT administration should be studied as
well. Fortunately, others have shown that the outcomes
Discussion and Conclusions ofsimulatedandactualCATadministrationscanbevery
Inthisstudy it wasshownhow CATcansuccessfully be similar[37],whichmayrenderthepresentstudyinstruc-
applied as an efficient method of reducing the length of tive nevertheless. Stopping rules ‘SE(θ) <0.4’ and ‘SE(θ)
theadministrationoftheMASQ-ADinROM.Simulated <0.5’ seem to be the best for real MASQ-AD CAT
Table 3Relationship withexternal criteria (95%confidence intervals between brackets)ofthe CAT estimates under
several stoppingrules.
Stoppingrule MASQ-AA(r) MASQ-GD(r) AnyDepression(auc)
None:Sumscore 0.499(0.475-0.524) 0.784(0.772-0.797) 0.815(0.801-0.829)
None:θˆ 0.460(0.434-0.486) 0.739(0.724-0.754) 0.810(0.795-0.824)
SE(θ)<0.2 0.453(0.427-0.479) 0.730(0.715-0.746) 0.807(0.793-0.822)
SE(θ)<0.3 0.414(0.387-0.442) 0.685(0.668-0.703) 0.794(0.779-0.809)
SE(θ)<0.4 0.396(0.368-0.423) 0.651(0.632-0.670) 0.782(0.767-0.797)
SE(θ)<0.5 0.385(0.358-0.413) 0.627(0.607-0.647) 0.769(0.754-0.784)
SE(θ)<0.6 0.336(0.307-0.365) 0.556(0.533-0.578) 0.746(0.730-0.762)
Note.risthePearsoncorrelation;AUCistheareaundertheROC-curve.
Smitsetal.BMCMedicalResearchMethodology2012,12:4 Page7of10
http://www.biomedcentral.com/1471-2288/12/4
administrationsinpatientpopulations.CATsunderthese freeonlineresearchmanagementtoolsponsoredbyPRO-
rules used a relatively low number of items, showed a MIS,whichenablesmentalhealthworkersandresearchers
substantivecorrelationwiththeoriginallatentdepression to create websites and CATs for their own patients and
estimates, and had a small attenuation in criterion- scales.
relatedvalidity. Inthisstudy wesoughttofindasolutionforthe main
Inadditiontoallowingforanimprovementoftheeffi- disadvantage of ROM: its time consumption [5].
ciencyofMASQ-ADassessment,IRTmodelingprovided Althoughwe restrictedourattentiontothe efficiency of
estimates of test information (see Appendix). This a single scale, the ultimate solution would be to convert
showedthattheinformationpeakedsomewhatbelowthe allscalesusedforassessmentinROMtoaCATversion.
middleofthelatenttraitscale.Figure 1showsthatforθ One could add smart testing designs to this [41-43], for
>1, information is relatively low. In addition, the lower example, allowing for the collection of scores for only
panel shows the estimated distribution of the latent some time pointsforeachsubject.Sucha hybridsystem
depression variable for each of the two MINI groups, wouldleadtoanevenlargerreductionintimeconsump-
separately. It is apparent that the test is informative in tion. It should be noted that efficient assessments have,
theregionwherethesetwogroupsaretobetoldapart;in at least, two other advantages. First, for some patient
the region where the patients suffering from depression groups,suchasseverelydiseasedpatients,administering
prevail(i.e.,ontherighthandside),informationislower. many questionsmay decrease the quality ofthe answers
Consequently,inthatregionsmalldifferencesarenotas given [3];shortassessmentswillleadtodatawithhigher
easily detected as for respondents with lower scores of quality. Second, ROM data can be used by researchers
depression (which caused the MINI depression group for effectivenessstudies[4].In such studies, it is impor-
requiring,onaverage,aboutfouritemsmorethantheno tantthatpatientswhoenteredtreatmentremainintreat-
depression group under the ‘SE(θ) <0.3’ rule). MASQ ment.Becausethewillingnessofpatientstocooperatein
users should ask themselves if the test information pre- researchisknowntodecreaseswiththesizeandnumber
sented complies with their testing goal. For example, ofquestionnaires[44,45],shorterROMassessmentsmay
when using CAT for monitoring the development of lead to lower drop-out rates. We hope that both mental
mental health, and for obtaining reliable change scores, health researchers and practitioners are convinced by
anitembankwithmoreuniformtestinformationmaybe theseoutcomesandwillimplementtheMASQ-ADCAT
preferred. By contrast, in other situations MASQ-AD or develop a CAT for their preferred mental health
assessment may be much more aimed at deciding instrumentsforuseinROM.
whether a respondent scores high on depression or not
(see, [38]). In such cases CAT as described in this study Appendix
seembeasoundchoice.Actually,whenpredictiveutility Psychometric evaluation of the MASQ-AD items
istheonly goal,itmaybe even bettertoadjusttheCAT Tostudy the quality ofthe MASQ-ADitemsas input of
algorithm and use so called clinical decision adaptive an adaptive version, we followed the methodology as
testing[39,40],inwhichitemswiththresholdparameters recommended by the PROMIS project [18]. Attention
around a cut-score are needed. When MASQ-AD users wasdirectedoneffectsizes,notonstatisticalsignificance.
are satisfied with the test information presented in this As most model fit statistics are sensitive to sample size,
study, however, they can take advantage of CAT as a statistically significant outcomes may be trivial (models
methodofveryefficientMASQ-ADassessmentinROM. neverfitthedataperfectly,andeventhetiniestdeviations
AlthoughCATprovides anopportunity toimprove the canbedetectedbyincreasingsamplesize).
efficiencyinpatientassessmentsubstantially,itsimplemen- When building a CAT version of the MASQ-AD, it
tationnotonlyneedsknowledgeofbuildinganitembank should be shown that itsitems comply withseveral psy-
andpsychometricanalysis.Foranactualimplementation,a chometricrequirements.CATreliesuponItemResponse
test delivery system is needed, andthissystem should be Theory (IRT), and before applying an IRT model, it is
flexibleenoughtoincorporatethedynamicnatureofCAT. importanttoevaluateitsmainassumptionsofunidimen-
Evidently, the extent of the investment of switching to sionality, local independence, andmonotonicity[18].To
CAT depends upon the current assessment method. For studyunidimensionality,aConfirmatoryFactorAnalysis
example,whenROM isperformed usingweb-based soft- (CFA) was performed on the polychoric correlation
ware, which is the case at the Rivierduinen Centers, the matrix [46] of the MASQ-AD items. The resulting one
adaptationdoes not have to be so extensive. By contrast, factor CFA model had four out of five fit indices (Com-
forthoseinstitutionsstillusingpaperandpencilquestion- parative Fit Index, Tucker Lewis Index, Standardized
naires, such a system may have to be built from scratch. RootMean Residuals, and average residual correlations)
For such institutions it may be fruitful to register at the whichshowedgoodfit(forrulesofthumb,see[18]);the
Assessment Center (http://www.assessmentcenter.net), a RMSEA was higher (0.097) than what is commonly
Smitsetal.BMCMedicalResearchMethodology2012,12:4 Page8of10
http://www.biomedcentral.com/1471-2288/12/4
requiredforexcellentfit(<0.06).Inaddition,thefirstfac- removing item 18 from the item pool. In both cases, for-
torinaprincipal componentsanalysisonthepolychoric tunately, the parameters of the model hardly changed:
correlationsaccountedfor54%ofthequestionnairevar- the largest change in b parameters was six hundredths,
iance,more thanadequately meetingthe Reckase criter- and the largest difference in a parameters was four
ion of 20% ([47] cited in [48]). In addition, the second hundredths.
factor only explained 9% of the variance which gave a Underthemonotonicity assumption, theprobabilityof
ratioof varianceexplainedof thefirsttothesecondfac- choosing a higher item category should increase or
tor of aboutsix,whichishigherthan the requiredmini- remainconstant,butshouldnotdecreasewithincreasing
mum of four [18]. On the basis of these results we valuesonthedominant dimension.Themonotonicityof
concluded that the MASQ-AD items shared a single the MASQ-ADitems wasstudied using Mokken scaling
commonfactor. [51],whichisbasedonanon-parametricIRTmodel,and
Under the assumption of local independence, there can easily be engaged to visually inspect estimates of
should be no covariance left among the items after con- such probabilities. Mokken scaling showed that the
trolling for the dominant factor. To test local depen- MASQ-AD scale complied with monotonicity to a high
dence, the matrix of residual correlations resulting from extent: all 22 items showed monotone increasing func-
the one factor CFA was studied; coefficients with values tions with increasing sum scores. The accompanying
higher than 0.2 were considered as possibly locally scalability coefficient [51] of the whole scale was 0.49,
dependent [18]. Only one out of the 231 (1/2 × 21 × whichaccordingtorulesofthumb[31]isascaleofmod-
22) item pairs was marked as possibly locally dependent: eratequality.Alloftheitemshadascalabilitycoefficient
items 12 (’Felt like it took extra effort to get started’) thatwashigherthanthelowerboundof0.3.Onthebasis
and 18 (’Felt really slowed down’) had a value of 0.29. It ofthese outcomes itwas concludedthat the MASQ-AD
is apparent that, in addition to being an indicator of itemsmettheassumptionofmonotonicity.
depression, both items are associated with the respon- After these basic assumptions were confirmed, the
dent feeling ‘sluggish’. By contrast, the modification GRMwasfit to the data. Tostudy the model fit of GRM
indices in LISREL [18], were all equal to zero. In addi- on the MASQ-AD items, several analyses were per-
tion, local independence under the GRM was studied formed. First, itemparameterswere estimated and cate-
using Yen’s [49] Q3 statistic. This statistic calculates the goryresponsecurves(CRCs)wereplottedperitemtosee
residual item scores under the GRM (i.e., observed - on what intervals on the latent depression scale each of
expected scores), and correlates these among items. To thefiveitemcategoriesweremostfrequentlychosen(see
assess lack of model fit, we used Cohen’s [50] rules of Figure 2, for the CRCs of item 1). The parameter esti-
thumb for correlation effect sizes: Q3’s between 0.24 matesoftheGRMfortheMASQ-ADitemsareshownin
and 0.36 are moderate deviations, and values of 0.37 Table 1. The second column of the table shows the a
and larger represent large deviations. Four of the 231 (discrimination)parameters;item16hasthelowest,and
item pairs had Q3 values that showed at least a moder- item 17 has the highest strength of association with the
ate deviation of model fit: item pairs 9 and 14, 10 and latent depression variable. The other columns show the
11, 17 and 19, had moderate values, whereas pair 12
and 18 showed a large value (0.39). The outcomes sug-
gested that there may exist a specific relationship Felt cheerful
between some items over and above their association
1.0
with the latent depression score. When considering the
ictoenmtesn1t2o,f1t7h,e1i8teamnsdt1h9isawreasacllosntfairtmemede.nFtsorasesxoacmiapteled, 0.8 1 2 5
wAiDthhfaeselsinogm‘esluimggbiaslha’n.cIteminayitsbeiteamrgudeedsitghnatinthtehaMtAsoSmQe- ability 0.6 3 4
b
aspects, such as feeling sluggish, appear in more items Pro 0.4
than others, which causes them to co-vary after control-
ling for the dominant dimension. Although most of 0.2
these deviations were only moderate, one item pair
(item 12 and 18) showed lack of local independence 0.0
both under the one factor model and GRM, which −4 −2 0 2 4
would qualify one of the items for possible removal. To θ
study the issue in more detail we followed the recom- Figure2Estimatedcategoryresponsecurvesforitem1ofthe
mendations of Reeve et al. [18] and fitted two new MASQ-ADscale.Notethattheitemwasreverse-scored(5=notat
all,4=abit,3=moderately,2=much,1=verymuch).
GRMs, one after removing item 12, and one after
Smitsetal.BMCMedicalResearchMethodology2012,12:4 Page9of10
http://www.biomedcentral.com/1471-2288/12/4
estimates of the threshold parameters. In addition, the concludedthattheitemsetoftheMASQ-ADwasavalid
item and test information [19] plots showed that the inputforanadaptivetest.
MASQ-AD scale wasmost informative near the average
latent depression score in the current population. The
Authordetails
test information(also, see [Figure 1],right-handvertical
1DepartmentofClinicalPsychology,FacultyofPsychologyandEducation,
axis) was roughly bell-shaped, with a peak at θ = -0.50, VUUniversity,Amsterdam,TheNetherlands.2DepartmentofPsychiatry,
andabout60% oftheinformationfor-2<θ<1. Inaddi- LeidenUniversityMedicalCenter,Leiden,TheNetherlands.3MentalHealth
InstitutionRivierduinen,Leiden,TheNetherlands.
tion, on the basis of the estimated item parameters, for
eachpatientalatentdepressionscore(θ)wasestimated. Authors’contributions
Second,wefollowedthesuggestionofReeveetal.[18] NSanalyzedthedataandpreparedthemanuscript.FZ,MH,andIChadfull
in using the G2 statistic [52] to compare observed and accesstotheROMdataandgavesignificantcommentsonthemanuscript.
PCgavesignificantcommentsonthemanuscript.Allauthorshavereadand
expected response frequencies under the estimated IRT approvedthefinalversionofthemanuscript.
model. For a proper use of G2 it is necessary that the
Competinginterests
expected frequencies in the cells are at least 5, however,
Theauthorsdeclarethattheyhavenocompetinginterests.
anditsusewasproblematicbecauseforasmuchas13of
the 22itemsitwasimpossible to obtain expectedvalues Received:11May2011 Accepted:10January2012
Published:10January2012
of at least 5. This was mainly because under the GRM,
respondents with high θ were almost never expected to
References
choose one of the lower item categories. Consequently, 1. GardnerW,ShearK,KelleherKJ,PajerKA,MammenO,BuysseD,FrankE:
G2 was of no use, and therefore for assessing GRM Computerizedadaptivemeasurementofdepression:Asimulationstudy.
BMCPsychiatry2004,4:Article13,[DOI:10.1186/1471-244X-4-13].
modelfitwefocusedonotheroutcomes.
2. CellaD,YountS,RothrockN,GershonR,CookK,ReeveB,AderD,FriesJ,
Third, another assumption of IRT models is that the BruceB,RoseM:ThePatient-ReportedOutcomesMeasurement
itemparametersapply forallrespondents.Ifparameters InformationSystem(PROMIS):ProgressofanNIHRoadmapCooperative
GroupDuringitsFirstTwoYears.MedicalCare2007,45:S3-11.
differbetweengroups,atestissaidtosufferfromDiffer-
3. SmitsN,CuijpersP,BeekmanATF,SmitJH:Reducingthelengthofmental
entialItemFunctioning(DIF,[19]). The consequence of healthinstrumentsthroughstructurallyincompletedesigns.International
DIFisthatrespondentsfromdifferentgroups,whoactu- JournalofMethodsinPsychiatricResearch2007,16:150-160.
4. DeBeursE,denHollander-GijsmanM,VanRoodY,vanDerWeeN,GiltayE,
ally have an identical score on the latent trait, have an
VanNoordenM,VanderLemR,VanFenemaE,ZitmanF:Routine
unequal probability of endorsing an item [53]. Conse- outcomemonitoringintheNetherlands:practicalexperienceswitha
quently,theestimatedlatenttraitscoresmaybedifferent. web-basedstrategyfortheassessmentoftreatmentoutcomeinclinical
practice.ClinicalPsychology&Psychotherapy2011,18:1-12.
Adaptive tests may be more vulnerable to the effects of
5. CarlierIVE,MeuldijkD,VanVlietI,VanFenemaE,VanderWeeN,ZitmanF:
DIF on validity [15,18], because in shorter assessments Routineoutcomemonitoringandfeedbackonphysicalormentalhealth
DIFitemsmay haveahigherimpact.We performedDIF status:evidenceandtheory.JournalofEvaluationinClinicalPractice2010,
[DOI:10.1111/j.1365-2753.2010.01543.x].
analysiswithrespecttoage,gender,anddiagnosticgroup
6. FliegeH,BeckerJ,WalterOB,BjornerJB,KlappBF,RoseM:Development
using ordinal regression methods [54]. As a measure of ofacomputer-adaptivetestfordepression(D-CAT).QualityofLife
effectsizeweusedthechangeinMcFadden’sR2,andfol- Research2005,14:2277-2291.
7. ForkmannT,BoeckerM,NorraC,EberleN,KircherT,SchauerteP,MischkeK,
lowedthesuggestionofusingavalueof0.02[33]asacri-
WesthofenM,GauggelS,WirtzM:DevelopmentofanItemBankforthe
tical value for rejecting the nulhypothesis of no DIF. AssessmentofDepressioninPersonsWithMentalIllnessesandPhysical
Fortunately,noDIFitemsweredetected. DiseasesUsingRaschAnalysis.RehabilitationPsychology2009,54:186-197.
8. WalterOB,BeckerJ,BjornerJB,FliegeH,KlappBF,RoseM:Development
As a final step in assessing GRM model fit, the latent andevaluationofacomputeradaptivetestfor‘Anxiety’(Anxiety-CAT).
depression score was estimated for all respondents, and QualityofLifeResearch2007,16:143-155.
correlated with the traditional MASQ-AD sum score. 9. WatsonD,ClarkLA:TheMoodandAnxietySymptomsQuestionnaireIowa
City:UniversityofIowa:DepartmentofPsychology;1991.
The correlation was 0.98, which was interpreted as a
10. SmitsN,CuijpersP,vanStratenA:ApplyingComputerizedAdaptive
good fit of the GRM to the data as well. TestingtotheCES-Dscale:ASimulationStudy.PsychiatryResearch2011,
Thepsychometricanalysesshowedthattheitemsofthe 188:147-155.
11. SheehanDV,LecrubierY,SheehanKH,AmorimP,JanavsJ,WeillerE,
MASQ-ADscalehadmanystrongfeaturesandonlyafew
HerguetaT,BakerR,DunbarGC:TheMini-InternationalNeuropsychiatric
weaknesses.Theitemsetshowedbothunidimensionality Interview(MINI):Thedevelopmentandvalidationofastructured
and monotonicity and no DIF items were detected. By diagnosticpsychiatricinterviewforDSM-IVandICD-10.JournalofClinical
Psychiatry1998,59(suppl20):22-57.
contrast,twomethodsfordetectinglocaldependenceindi-
12. ClarkLA,WatsonD:Atripartitemodelofanxietyanddepression:
cated that there was a substantial residual covariation Psychometricevidenceandtaxonomicimplications.JournalofAbnormal
betweenonepairofitemswhencontrollingforthelatent Psychology1991,100:316-336.
13. DeBeursE,denHollander-GijsmanM,HelmichS,ZitmanF:Thetripartite
trait score. Fortunately, thisdependence hada negligible
modelforassessingsymptomsofanxietyanddepression:Psychometrics
effectontheparameterestimatesofGRM;thereforeboth oftheDutchversionofthemoodandanxietysymptomsquestionnaire.
items were maintained in the item pool. All in all it was BehaviourResearchandTherapy2007,45(7):1609-1617.
Smitsetal.BMCMedicalResearchMethodology2012,12:4 Page10of10
http://www.biomedcentral.com/1471-2288/12/4
14. WeissDJ,KingsburyGG:ApplicationofComputerizedAdaptiveTestingto 43. SmitsN,VorstHCM:Reducingtestlengththroughstructurally
EducationalProblems.JournalofEducationalMeasurement1984, incompletedesigns:AnIllustration.LearningandIndividualDifferences
21:361-375. 2007,17:25-34.
15. WainerH:ComputerizedAdaptiveTesting:APrimer.2edition.MahwahNJ: 44. DillmanDA,SinclairMD,ClarkJR:Effectsofquestionnairelength,
LawrenceErlbaumAssociates;2000. respondent-friendlydesign,andadifficultquestiononresponserates
16. SamejimaF:Estimationoflatentabilityusingapatternofgraded foroccupant-addressedcensusmailsurveys.PublicOpinionQuarterly
resonses.PsychometrikaMonographSupplementNo.171969. 1993,57:289-304.
17. MellenberghGJ:Conceptualnotesonmodelsfordiscretepolytomous 45. WacholderS,CarrollRJ,PeeD,GailMH:Thepartialquestionnairedesign
itemresponses.AppliedPsychologicalMeasurement1995,19:91-100. forcase-controlstudies.StatisticsinMedicine1994,13:623-634.
18. ReeveBB,HaysRD,BjornerJB,CookKF,CranePK,TeresiJA,ThissenD, 46. BollenK:StructuralEquationswithLatentVariablesNewYork:JohnWiley;
RevickiDA,WeissDJ,HambletonRK,LiuH,GershonR,ReiseSP,LaiJ, 1989.
CellaD:Psychometricevaluationandcalibrationofhealth-relatedquality 47. ReckaseMD:UnifactorLatentTraitModelsAppliedtoMultifactorTests:
oflifeitembanks:PlansforthePatient-ReportedOutcomes ResultsandImplications.JournalofEducationalStatistics1979,4:207-230.
MeasurementInformationSystem(PROMIS).MedicalCare2007,45:S22-31. 48. HambletonRK:PrinciplesandselectedapplicationsofItemResponseTheory.3
19. EmbretsonS,ReiseSP:ItemResponseTheoryforPsychologistsMahwah,NJ: edition.NewYork:AmericanCouncilonEducation;1988,147-200.
LawrenceErlbaum;2000. 49. YenWM:Scalingperformanceassessments:Strategiesformanaging
20. HastieT,TibshiraniR,FriedmanJH:TheElementsofStatisticalLearning:Data localitemdependence.JournalofEducationalMeasurement1993,
mining,inferenceandpredictionNewYork:Springer;2001. 30:187-213.
21. StoneM:Cross-validationchoiceandassessmentofstatistical 50. CohenJ:Statisticalpoweranalysisforthebehavioralsciences.2edition.New
predictions.JournaloftheRoyalStatisticalSocietyB1974,36:111-147. Jersey:LawrenceErlbaumAssociates;1988.
22. McDonaldRP:TestTheory:AUnifiedTreatmentMahwah,NJ:Lawrence 51. MokkenRJ:AtheoryandprocedureofscaleanalysisTheHague:Mouton;
Erlbaum;1999. 1971.
23. ThissenD:MULTILOGUser’sGuideMooresville,IN:ScientificSoftware;1991. 52. McKinleyRL,MillsCN:Acomparisonofseveralgoodness-of-fitstatistics.
24. RDevelopmentCoreTeam:R:Alanguageandenvironmentforstatistical AppliedPsychologicalMeasurement1985,9:49-57.
computingRFoundationforStatisticalComputing,Vienna,Austria;2010 53. HollandPW,WainerH:DifferentialItemFunctioningHillsdale,NJ:Lawrence
[http://www.R-project.org],[ISBN3-900051-07-0]. ErlbaumAssociates,Inc;1993.
25. KraemerH,MorganG,LeechN,GlinerJ,VaskeJ,HarmonR:Measuresof 54. CranePK,GibbonsLE,JolleyL,vanBelleG:DifferentialItemFunctioning
clinicalsignificance.JournaloftheAmericanAcademyofChildand AnalysisWithOrdinalLogisticRegressionTechniques.MedicalCare2006,
AdolescentPsychiatry2003,42(12):1524. 44:S115-23.
26. ZweigMH,CampbellG:Receiver-operatingcharacteristic(ROC)plots:a
fundamentalevaluationtoolinclinicalmedicine.ClinicalChemistry1993, Pre-publicationhistory
39:561-577. Thepre-publicationhistoryforthispapercanbeaccessedhere:
27. JöreskogKG,SörbomD:LISREL8.8forWindowsLincolnwood,IL:Scientific http://www.biomedcentral.com/1471-2288/12/4/prepub
Software;2006.
28. RizopoulosD:ltm:AnRPackageforLatentVariableModelingandItem doi:10.1186/1471-2288-12-4
ResponseTheoryAnalyses.JournalofStatisticalSoftware2006,17(5):1-25. Citethisarticleas:Smitsetal.:Aproofofprincipleforusingadaptive
29. RizopoulosD:TheltmPackage:LatenttraitmodelsunderIRT2007,[Library testinginroutineOutcomeMonitoring:theefficiencyoftheMood
oftheRpackage]. andAnxietySymptomsQuestionnaire-AnhedonicDepressionCAT.
BMCMedicalResearchMethodology201212:4.
30. FoxJ:ThepolycorPackage:PolychoricandPolyserialCorrelations2007,
[LibraryoftheRpackage].
31. VanderArkLA:MokkenscaleanalysisinR.JournalofStatisticalSoftware
2007,20(11):1-19[http://www.jstatsoft.org/v20/i11/paper].
32. VanderArkLA:Package‘mokken’2009,[LibraryoftheRpackage].
33. ChoiSW,GibbonsLE,CranePK:lordif:AnRPackageforDetecting
DifferentialItemFunctioningUsingIterativeHybridOrdinalLogistic
Regression/ItemResponseTheoryandMonteCarloSimulations.Journal
ofStatisticalSoftware2011,39(8):1-30.
34. ChoiSW:Package‘lordif’2011,[LibraryoftheRpackage].
35. GreenBF,BockRD,HumphreysLG,LinnRL,ReckaseMD:Technical
GuidelinesforAssessingComputerizedAdaptiveTests.Journalof
EducationalMeasurement1984,21:347-360.
36. KraemerHC,KupferDJ:Sizeoftreatmenteffectsandtheirimportanceto
clinicalresearchandpractice.BiologicalPsychiatry2006,59:990-996.
37. KocaleventRD,RoseM,BeckerJ,WalterOB,FliegeH,BjornerJB,KleiberD,
KlappBF:Anevaluationofpatient-reportedoutcomesfound
computerizedadaptivetestingwasefficientinassessingstress
perception.JournalofClinicalEpidemiology2009,62:278-287.
38. FinkelmanM,HeY,KimW,LaiA:Stochasticcurtailmentofhealth
questionnaires:Amethodtoreducerespondentburden.Statisticsin Submit your next manuscript to BioMed Central
Medicine2011,30:1989-2004. and take full advantage of:
39. WallerNG,ReiseSP:Computerizedadaptivepersonalityassessment:An
illustrationwiththeabsorptionscale.JournalofPersonalityandSocial
• Convenient online submission
Psychology1989,57:1051-1058.
40. ThompsonN:Itemselectionincomputerizedclassificationtesting. • Thorough peer review
EducationalandPsychologicalMeasurement2009,69(5):778.
• No space constraints or color figure charges
41. FinkelmanM,WeissD,Kim-KangG:ItemSelectionandHypothesisTesting
fortheAdaptiveMeasurementofChange.AppliedPsychological • Immediate publication on acceptance
Measurement2010,34(4):238-254. • Inclusion in PubMed, CAS, Scopus and Google Scholar
42. McArdleJJ,WoodcockRW:Expandingtest-retestdesignstoinclude
• Research which is freely available for redistribution
developmentaltime-lagcomponents.PsychologicalMethods1997,
2:403-435.
Submit your manuscript at
www.biomedcentral.com/submit
