Psychometric Versus Actuarial Interpretation of Intelligence and Related Aptitude Batteries

29 Pages·2013·0.23 MB·English

CHAPTER
4 Psychometric Versus Actuarial
Interpretation of Intelligence and
Related Aptitude Batteries
Gary L. Canivez
Abstract
Interpretation of intelligence tests involves making various inferences about the individual based
on their performance. Because there are many different scores within intelligence tests reflecting
different levels (Full Scale, factors, subtests) and there are many different comparisons provided in
test manuals and the extant literature, there is a multitude of possible inferences. This chapter is
concerned with reviewing the various available scores and comparisons with suggested interpretations
and a review of the empirical investigations of their psychometric fitness (reliability, validity, utility).
Differentiation of psychometric interpretation versus actuarial interpretation methods is presented,
as well as a review of research related to each. Most intelligence test interpretation methods
are considered psychometric in nature, and most lack sufficient reliability, validity, or utility for
individual clinical use; improvements in the clinical assessment of intelligence may result from greater
development and use of actuarial approaches.
Key Words: intelligence test interpretation, reliability, validity, utility, actuarial decision-making,
clinical decision-making
Introduction Standards for Educational and Psychological Testing
Interpretation of intelligence tests involves draw- (American Educational Research Association,
ing inferences about an individual based on scores American Psychological Association, and the
obtained on a particular test. Because contemporary National Council on Measurement in Education
intelligence tests provide many diff erent types of [AERA, APA, NCME], 1999) provides numerous
scores, there is a variety of inferences that could be guidelines for considering the reliability and validity
made about any individual. Furthermore, various of test scores that should be applied to intelligence
intelligence tests are constructed to refl ect diff erent test scores. Such guidelines apply to test authors
theories of intelligence or cognitive abilities, and and publishers, but ultimately it is the test user who
interpretations may also be based on the particular must decide which test scores, comparisons, and
theory upon which the test is based. While a test procedures possess suffi cient evidence of reliability
may be created to refl ect a particular theory, clini- and validity to report and interpret. Test scores that
cians may also apply alternate or competing theories do not possess adequate reliability, validity, and util-
in the interpretation of test scores. Legitimate infer- ity will lead the test user to make inaccurate and
ences about an individual from various intelligence inappropriate inferences about the individual when
test scores or procedures, however, must e ach be interpreting those test scores and comparisons.
supported by reliability, validity, and utility research Such inaccurate and inappropriate inferences may
on the various scores, comparisons, and their uses. well lead to recommendations for classifi cation,
84
diagnosis, or treatment that may also be wrong. scores refl ecting diff erent levels of the test. Such
Weiner (1989) cogently noted that in order to prac- scores include an overall, omnibus Full Scale score;
tice in an ethical manner, psychologists must “(a) several factor-based scores or indexes; subtest scores;
know what their tests can do and (b) act accordingly” ipsative or deviation scores derived from compar-
(p. 829). Numerous ethical standards also concern ing subtest or factor scores to the individual’s overall
the use of tests and measurement procedures in mean score; and theoretically or logically combined
clinical practice (APA, 2010; National Association subtest composite scores.
of School Psychologists [NASP], 2010). Kamphaus et al. (2005, p. 26) referred to a
Interpretation of intelligence tests may involve “second wave” of test interpretation proposed by
description of the individual’s performance, pre- Rapaport, Gil, and Schafer (1945–1946) that ush-
diction of the individual’s performance in related ered in an approach to intelligence test interpretation
areas such as classroom learning or performance still in use by many today. Th e approach advocated
on academic achievement tests, classifi cation or by Rapaport et al. was that of going beyond the
diagnosis of the individual, and informing or rec- overall, omnibus Full Scale IQ and examining the
ommending treatments. Each of these “interpreta- shape of the subtest profi le to provide a descrip-
tions” requires empirical support for appropriate tion of subtest highs and lows for the individual.
use. With respect to individual test scores, it has Th ese strengths and weaknesses were presumed to
been argued that if a score is to be used for indi- refl ect some aspect of psychopathology as well as
vidual decisions or clinical decision-making, reli- consideration for intervention. Wechsler’s examina-
ability indices should meet or exceed .85 (Hills, tion (1944) of diff erences between verbal and per-
1981) or .90 (Aiken, 2000; Guilford & Fruchter, formance scales as well as subtest profi le shape and
1978; Nunnally & Bernstein, 1994; Ponterotto deviations was also included in this second wave
& Ruckdeschel, 2007; Salvia & Ysseldyke, 1988, and refl ected Wechsler’s clinical approach to test
2001). Specifi c inferences about the individual are interpretation. Th is is in contrast to Wechsler’s note
tied more to the various estimates of the validity that subtests are merely diff erent practical estimates
of test scores and their diagnostic utility; however, for measuring general intelligence.
scores that lack suffi cient reliability cannot be valid Criticism of these early interpretation methods
or of diagnostic utility. It is within this framework on empirically based psychometric grounds led
that psychometric and actuarial intelligence test to what Kamphaus et al. (2005) referred to as the
interpretations are examined. “third wave,” wherein application of psychological
measurement methods was used to evaluate various
History of IQ Test Interpretations intelligence test scores and interpretation methods.
Kamphaus, Winsor, Rowe, and Kim (2005) Cohen’s investigation (1959) of the factor struc-
provided a useful description of the history of intel- ture of the Wechsler Intelligence Scale for Children
ligence test interpretation. Th e earliest application (WISC; Wechsler, 1949) was one of the fi rst of its
of intelligence test interpretation was associated kind and provided an empirical means for identifying
with a classifi cation of the individual’s test score subtest association to factors or dimensions under-
(and thus the individual) according to descriptive lying the WISC. While Wechsler assigned WISC
terms based on the overall test score and prediction subtests to the verbal or performance scale based on
of school functioning. Th e earliest classifi cation subtest content, Cohen’s factor analysis empirically
systems contained descriptive terms (viz., “idiots,” assigned subtests to factors based on their shared
“imbeciles,” “morons”; Levine & Marks, 1928) that variance, and resulted in the three-factor structure
are considered pejorative by today’s standards. Each (Verbal Comprehension, Perceptual Organization,
intelligence test presently published also contains and Freedom from Distractability) of the WISC.
descriptive classifi cations for test score ranges, and Cohen was critical of WISC subtest scores based
these are frequently the fi rst “interpretation” made. on the high levels of shared variance and low sub-
Kamphaus et al. also noted that present-day descrip- test specifi city (non-error variance unique to the
tive terms typically refl ect some aspect of score devi- subtest). Subsequently, this three-factor structure
ation from the average range, but each test contains was frequently replicated with other samples and
somewhat diff erent descriptors and may include Cohen’s WISC factor names were retained in both
diff erent score ranges. While the earliest intelli- the second (WISC-R; Wechsler, 1974) and third
gence tests provided one overall test score (i.e., IQ), editions of the WISC (WISC-III; Wechsler, 1991).
present-day intelligence tests contain numerous Kaufman (1979) provided a means for calculating
canivez 85
scores for the three factors to provide more factori- Contemporary intelligence tests are now greatly
ally pure scores than the VIQ and PIQ. infl uenced by theories of intelligence or models of
Kaufman’s signifi cant infl uence on intelligence intelligence measurement in their construction and
test interpretation is refl ected in the hierarchical interpretation. Presently, one of the most infl uential
and sequential process of interpretation and sub- models of intellectual measurement is that of Carroll
test analyses (Kaufman, 1979, 1994; Kaufman (1993, 1995, 1997a, 2003), which proposes that
& Lichtenberger, 2000, 2006) frequently taught intelligence tests measure various intellectual abili-
in graduate programs and used by practitioners ties that are hierarchically ordered. Carroll’s (1993,
(Alfonso, Oakland, LaRocca, & Spanakos, 2000; 2003) three-stratum theory of cognitive abilities
Groth-Marnat, 1997; Kaufman, 1994; Pfeiff er, proposes some 50–60 narrow abilities (Stratum I)
Reddy, Kletzel, Schmelzer, & Boyer, 2000). Levels at the bottom (subtests), 8–10 broad-ability factors
of test interpretation were ordered from the most (Stratum II) in the middle (fi rst-order factors), and
reliable and valid scores (Full Scale IQ and com- the general (“g”) ability factor (Stratum III) at the
posite scores) to the least reliable and valid scores top (second-order factor) and refl ected by the over-
(single subtest scores). Kaufman’s approach contin- all FSIQ or global IQ. Many contemporary intelli-
ued the clinical interpretation method of unique gence tests are constructed to refl ect Carroll’s model
profi le shape (subtest strengths and weaknesses) of intelligence measurement either explicitly or
much like that of Rapaport et al. (1945–1946), but implicitly. Th e American Psychological Association
his profi le interpretation method of comparing an task force study of intelligence (Neisser et al., 1996)
individual’s subtest scores to their own overall mean noted that the hierarchical nature of intelligence
performance was through the identifi cation of sta- measurement was the most widely accepted view,
tistically signifi cant strengths and weaknesses and and this still appears to be true.
consideration of base rates. Th is ipsative approach Another theoretical perspective of intelligence
examines intraindividual diff erences and is an ideo- measurement closely related to, and preceding that
graphic interpretation approach, in contrast to the of Carrol, is the Gf –G c theory of Cattell and Horn
nomothetic interpretation approach (normative (Cattell, 1943; Horn, 1988, 1991; Horn & Cattell,
comparisons) provided by the standard scores of 1966; Horn & Noll, 1997). Cattell and Horn’s
intelligence tests. Sattler (1982, 1988, 1992, 2001, G f –G c theory describes aspects of reasoning abili-
2008) and Sattler and Ryan (2009) also provided ties that allow the individual to solve novel prob-
for similar intelligence test interpretations based on lems (fl uid intelligence [Gf ]) and abilities acquired
a sequential order from the global score to subtest through the individual’s exposure to aspects of their
comparisons. culture such as language and educational experi-
Reliability and validity research highlighting ences (c rystallized intelligence [G c ]). Extension of
major problems with ipsative comparisons and pro- G f –G c theory by Horn (1991) and Horn and Noll
fi le interpretations (discussed in detail later in this (1997) is similar to that of Carroll with some 8–9
chapter) led to what Kamphaus et al. (2005) referred or more broad dimensions but does not include
to as the “fourth wave” where intelligence theory was higher-order g , arguing there was insuffi cient con-
applied to intelligence test construction and inter- struct validity evidence for singular g .
pretations. Th e earliest intelligence tests were seem- Cattell-Horn-Carroll (CHC) theory is an
ingly constructed from a pragmatic perspective and approach wherein researchers melded the work
some would say atheoretical. Th orndike (1990, p. of Cattell and Horn with that of Carroll (Evans,
226) noted Alfred Binet was “theoretically agnostic” Floyd, McGrew, & Leforgee, 2001; Flanagan,
with regard to the Binet-Simon scale. Zachary (1990) 2000; McGrew, 2005), but this is an odd combi-
noted David Wechsler’s minimal explication of theory nation, given that Carroll provided evidence for
in constructing the original Wechsler-Bellevue (WB) higher-order g while Horn argued that singular g
scales in 1939. However, as several have pointed out, did not exist and was a statistical artifact. Carroll’s
when constructing and modifying his tests, Wechsler model, Cattell and Horn’s model, or the combined
used Spearman’s (1904, 1927) theory of general intel- CHC model are often cited as theoretical founda-
ligence (g ) and was also infl uenced by other contem- tions or infl uences in present versions of intelli-
porary intelligence theories (Saklofske, 2008; Tulsky gence tests such as the Wechsler Intelligence Scale
et al., 2003; Zhu & Weiss, 2005). Wechsler’s defi ni- for Children–Fourth Edition (WISC-IV: Wechsler,
tion of i ntelligence (i.e., “global capacity”; Wechsler, 2003), Wechsler Adult Intelligence Scale–Fourth
1939, p. 229) also refl ected Spearman’s g . Edition (WAIS-IV: Wechsler, 2008a), Stanford-Binet
86 psychometric versus actuarial interpretation of intelligence
Intelligence Scales–Fifth Edition (SB-5: Roid, and thus, knowledge of such score interpretation
2003a), Diff erential Ability Scales–Second Edition improvement beyond the overall IQ score is impera-
(DAS-II: Elliott, 2007a), Wide Range Intelligence tive (Brody, 1985). Furthermore, interpretation of
Test (WRIT: Glutting, Adams, & Sheslow, 2000a), scores at every level of the test ignores the fact that
Reynolds Intellectual Assessment Scales (RIAS: scores at the subtest and factor level are themselves
Reynolds & Kamphaus, 2003), Kaufman Adolescent correlated and n ot independent, representing mix-
and Adult Intelligence Test (KAIT: Kaufman & tures of g and subtest variance (Carroll, 1993). Th ese
Kaufman, 1993), and Kaufman Assessment Battery and other critically important interpretation issues
for Children–Second Edition (KABC-II: Kaufman are further explicated throughout this chapter.
& Kaufman, 2004a).
Luria’s neuropsychological theory was the Psychometric Interpretation Methods
foundation for the development of the Kaufman Descriptions of intelligence test interpreta-
Assessment Battery for Children (K-ABC; Kaufman tion methods from stage one through stage four
& Kaufman, 1983) and later, the development of (Kamphaus et al., 2005) are ostensibly p sychomet-
the Planning, Attention, Simultaneous, Successive ric interpretation methods. Interpretations based
(PASS) theory (Das, Naglieri, & Kirby, 1994) which on psychometric grounds are interpretations based
is the foundation for the Cognitive Assessment on various scores provided by the intelligence test
System (CAS: Naglieri & Das, 1997a). While the as well as derived scores or comparisons of diff er-
KABC-II is linked to and can be interpreted accord- ent scores. Th ese scores can be used in descriptive
ing to CHC theory, it is also linked to and can be (level of performance), predictive (estimate of per-
interpreted according to Luria’s theory through its formance on related variable), and classifi catory
dual theoretical foundation (Kaufman & Kaufman, (assignment to diagnostic group) ways. Clinicians
2004a). CAS interpretation follows the PASS the- use test scores and analysis information as well as
ory and is related to Luria’s three functional units test session observations, background information,
of the brain (Naglieri, 1997; Naglieri & Das, 1990, interview information, etc., and make a clinical deci-
1997b). sion, judgement, or inference regarding the indi-
Kamphaus et al. (2005) noted that, in practice, vidual. Such decisions might be considered clinical
clinicians frequently interpret not only the overall decisions rather than a ctuarial (statistical) decisions
Full Scale/Composite intelligence test score, which is because it is the clinician’s j udgement of meaning of
an estimate of Spearman’s g (Spearman 1904, 1927) the score(s) that guides the interpretation (decision)
and the apex of Carroll’s hierarchy (1993), but also rather than strict adherence to a statistically based
interpret factor-based scores a nd ability profi les pro- (formula) interpretation (decision) (Meehl, 1954,
duced from subtest strengths and weaknesses. Th ese 1957; Meehl & Rosen, 1955).
subtest profi le-interpretation systems (Flanagan Intelligence test scores are (or can be) associated
& Kaufman, 2004; Kaufman, 1994; Kaufman & with a hierarchical model similar to Carroll’s (1993,
Lichtenberger, 2000; Sattler, 2001, 2008; Sattler & 2003) Stratum III (omnibus, Full Scale score),
Ryan, 2009) are exceedingly popular in psychology Stratum II (factor-based scores), or Stratum I (sub-
training and clinical practice (Alfonso et al., 2000; test scores). Th ese scores are standardized scores and
Groth-Marnat, 1997; Kaufman, 1994; Pfeiff er refl ect comparison to the normative group appro-
et al., 2000). priate for the individual. Interpretations from the
Despite their popularity, clinicians m ust con- standardized scores are nomothetic and allow for
front the issue of whether there is suffi cient empiri- understanding how the individual performed rela-
cal support for a ll these interpretation methods and tive to others their age in the population. Other
the resulting inferences. While many test publish- derived scores such as ipsative factor score com-
ers, test authors, workshop presenters, and textbook parison, pairwise factor score comparisons, ipsative
authors present all the above methods for interpret- subtest comparisons, and pairwise subtest compari-
ing intelligence tests and argue for their utility, it is sons are rooted in the clinical intelligence test inter-
the test user who must decide which methods have pretation approaches articulated by Rapaport et al.
adequate reliability, validity, and utility for clinical (1945–1946), Kaufman (1979, 1994), Kaufman
application and making inferences and decisions and Lichtenberger (2000, 2006), Sattler (1982,
about the individual they assess. By interpreting all 1988, 1992, 2001, 2008), and Sattler and Ryan
such scores, the clinician is adding (or replacing) (2009) and are ideographic. Th ese are comparisons
interpretations beyond the overall Full Scale score of the individual to himself or herself. Another
canivez 87
means by which clinicians may interpret scores 1995) hierarchical model. Substantial evidence for
from intelligence tests is the combination of various interpretation of global Full Scale scores (g estimates)
subtest scores into composite scores based on some exists (Gottfredson, 2002, 2008; Jensen, 1998;
logical or theoretical connection (Kaufman, 1994; Kubiszyn et al., 2000; Lubinski, 2000; Lubinski &
Kaufman and Lichtenberger, 2000, 2006; Sattler, Humphreys, 1997; Neisser et al., 1996).
2001, 2008; Sattler & Ryan, 2009). Interpretation of Full Scale scores typically
Finally, some have argued that no single intel- begins with a presentation and description of the
ligence test adequately measures all of Carroll’s standard score, percentile rank, and confi dence
(1993, 2003) Stratum II dimensions, or Horn interval (obtained score or estimated true score)
and Noll’s (1997) G f –G c factors, and in order to for the standard score to account for measurement
better assess these broad intelligence dimensions, error, consistent with S tandards for Educational
subtests from diff erent intelligence tests should be and Psychological Testing (AERA, APA, NCME,
combined in what is referred to as “cross-battery 1999). Classifi cation of the Full Scale score within a
assessment” (Flanagan & McGrew, 1997; Flanagan descriptive category (or range of descriptive catego-
& Ortiz, 2001; Flanagan, Ortiz, & Alfonso, 2008; ries around the confi dence interval) refl ecting some
McGrew & Flanagan, 1998). Cross-battery assess- deviation from average is also frequently made (i.e.,
ment is rooted in the aforementioned CHC theory. average, above average, signifi cantly below average).
Such cross-battery assessment is proposed to bet- Th ese normative descriptors are intended to provide
ter account for an individual’s varied broad abili- an illustration of how the individual performed rel-
ties (Stratum II) in cognitive assessment; which in ative to others their age. Th is is necessary given the
turn is used to understand their educational dif- interval level measurement such standardized scores
fi culties, provide better diff erential diagnosis, and represent.
guide interventions. While the cross-battery assess- Another aspect of interpretation of the Full Scale
ment approach attempts to improve assessment of score is an inference as to what to expect from the
cognitive abilities and has intuitive appeal, there individual in regard to their acquisition of academic
are numerous substantial psychometric problems skills (i.e., academic achievement). Th e predictive
cogently pointed out by Glutting, Watkins, and validity of intelligence test Full Scale scores is well
Youngstrom (2003), which have yet to be ade- documented (Bracken & Walker, 1997; Brody, 2002;
quately empirically addressed. Other issues noted Brown, Reynolds, & Whitaker, 1999; Flanagan,
later in this chapter also have implications for such Andrews & Genshaft, 1997; Gottfredson, 2008,
interpretations. Th us, individual clinical use of Konold & Canivez, 2010; Naglieri & Bornstein,
such cross-battery interpretation methods is not 2003) and intelligence is a construct that precedes
recommended. and infl uences the development of academic achieve-
Examination of psychometric interpretation ment because as Jensen (1998) noted “school learn-
methods proceeds in order of the hierarchy and ing itself is g -demanding” (p. 279). Watkins, Lei,
structure of intelligence tests (Carroll, 1993, 1995; and Canivez (2007) also demonstrated that verbal
Neisser et al., 1996). Th is order also parallels the and perceptual measures of intelligence (WISC-III)
psychometric interpretation procedures recom- predicted future academic achievement in reading
mended by Kaufman (Flannagan & Kaufman, and mathematics but reading and mathematics
2004; Kaufman, 1994; Kaufman & Lichtenberger, did not predict future measured intelligence, thus
2002, 2006) and Sattler (Sattler, 2008; Sattler & supporting Jensen’s position of the temporal pre-
Ryan, 2009) as well as contemporary intelligence cedence and infl uence of intelligence on academic
test authors through their test technical and inter- achievement.
pretation manuals. Full Scale scores are also used in making clas-
sifi cation or diagnostic decisions regarding psycho-
Global IQ Score Interpretation/Stratum III pathology, disability, and giftedness. Classifi cation
Th e fi rst level of intelligence test interpretation or diagnosis of mental retardation (MR)/intellec-
involves the reporting and description of the overall, tual disability (ID) requires by defi nition “signifi -
omnibus, global, or Full Scale score; which is mea- cantly subaverage general intellectual functioning”
sured or estimated by two or more subtests within (Public Law 108–446 [IDEIA]; U. S. Department
the scale. Full Scale scores represent an estimate of g of Education, 2006, p. 46756) or “signifi cantly sub-
or Spearman’s general intelligence factor (Spearman, average intellectual functioning” (APA, 2000, p.
1904, 1927), which is the apex of Carroll’s (1993, 49). It is generally agreed that a score 2 S D below
88 psychometric versus actuarial interpretation of intelligence
the population mean would satisfy this criterion. is important as it addresses one of the major sources
Historically, the operational defi nition for classify- of error variance in intelligence tests not addressed
ing specifi c learning disability (SLD), according to by the internal consistency estimate (Hanna,
IDEA prior to IDEIA 2004, involved fi rst a severe Bradley, & Holen, 1981). Hanna et al. also noted
discrepancy between predicted academic achieve- the importance of assessing intelligence test mea-
ment given a certain level of intelligence and actual surement error related to scoring and administra-
academic achievement. Th is criterion was concerned tion errors.
with the u nexpectedly low achievement associated IQ tests have a rich history of accounting for
with the concept of learning disability (Reynolds, meaningful levels of academic achievement vari-
1984). While federal law no longer mandates the ance (Brody, 2002; Naglieri & Bornstein, 2003),
use of the severe discrepancy criterion, states may with average IQ-achievement correlations near
allow local school districts to continue using this .55 across age groups (Neisser et al., 1996; Brody,
criterion in SLD classifi cation (IDEIA 2004) and 2002). Th orndike (1986) also noted approximately
predicted (expected) achievement is based on a per- 85% to 90% of predictable criterion variable vari-
son’s general intellectual ability. DSM-IV-TR (APA, ance is accounted for by a single general score.
2000) specifi es individually measured achievement Among co-normed intelligence and achievement
from a standardized test that “is substantially below tests it is quite common to observe concurrent
that expected given the person’s chronological age, IQ-achievement correlations near .70. It is often
measured intelligence, and age-appropriate educa- said that the most important application of intelli-
tion” (p. 53, p. 54, p. 56) for classifi cation of its SLD gence tests is their ability to forecast student achieve-
type disorders (viz., Reading Disorder, Mathematics ment (Brown, Reynolds, & Whitaker, 1999; Weiss
Disorder, Disorder of Written Expression, respec- & Prifi tera, 1995) and the prediction of school per-
tively). Assessment for intellectual giftedness also has formance with intelligence tests has been a primary
implications for intelligence testing and it is gener- use since the creation of the fi rst Binet-Simon Scale
ally thought that a Full Scale intelligence test score of Intelligence (Binet & Simon, 1905).
2 SD above the population mean would satisfy the
criterion of signifi cantly above average intelligence. Broad Factor/Verbal-Nonverbal Score
Interpretation/Stratum II
global score psychometric support verbal vs. nonverbal assessment
Psychometric support for Full Scale IQ, or related Recognition of literacy issues aff ecting stan-
omnibus scores, is strong and includes the highest dardized intelligence testing was noted in the early
internal consistency estimates, short-term temporal 1900s and infl uenced the creation and use of the
stability, long-term temporal stability, and predictive Army Alpha (verbal) and Army Beta (nonverbal)
validity coeffi cients (Bracken & McCallum, 1998b; tests (Th orndike, 1997), with Yoakum and Yerkes
Canivez & Watkins, 1998; Elliott, 2007b; Glutting, (1920) noting the use of Army Beta for recruits
Adams, & Sheslow, 2000b; Kaufman & Kaufman, who failed Army Alpha to prevent “injustice by
1983, 1993, 2004a, 2004b; Naglieri & Das, 1997b; reason of relative unfamiliarity with English” (p.
Th e Psychological Corporation, 1999; Reynolds & 19). Wechsler (1939) also recognized the need to
Kamphaus, 2003; Roid, 2003b; Wechsler, 2002a, assess intellectual abilities through both verbal a nd
2002b, 2003, 2008b; Wechsler & Naglieri, 2006; nonverbal (performance) means. Th e selection and
Woodcock, McGrew, & Mather, 2001). In stud- aggregation of subtests into composite scores were
ies of long-term temporal stability of WISC-III IQ originally based on subtest task requirements, and,
scores, the Full Scale IQ possessed the highest sta- Wechsler noted, “the subtests are diff erent measures
bility coeffi cient of all scores available and this also of intelligence, not measures of diff erent kinds of
held across a variety of demographic variables (sex, intelligence” (1958, p. 64). For Wechsler, verbal and
age, race/ethnicity, disability) (Canivez & Watkins, performance measures were not diff erent types of
1998, 1999, 2001). Similar results were also reported intelligence, but rather diff erent ways to measure it.
by Krohn and Lamp (1999) for the K-ABC and Verbal and performance measures relate to Carroll’s
Stanford-Binet Intelligence Scale: Fourth Edition (1993) Stratum II dimensions, which he referred to
(SB:FE: Th orndike, Hagen, & Sattler, 1986). Th ese as “fl avors” of g . Authors of contemporary n onverbal
are expected fi ndings according to true score theory, intelligence tests note that it is the method of assess-
and refl ect the power of aggregate scores that con- ment that is nonverbal rather than the cognitive
tain less error variance. Assessing temporal stability process involved in solving the tasks (Bracken &
canivez 89
McCallum, 1998a, 1998b; Naglieri, 2003a, 2003b; Herman, 1984) for due caution for such individual
Wechsler & Naglieri, 2006). Naglieri wrote, “the use of VIQ-PIQ diff erences by attending to base
term n onverbal refers to the content of the test, not rates in the population. Furthermore, it is one thing
a type of ability” (2003a, p. 2). Th us, the content or to assess an individual with known brain injury and
demands of subtests and composites may diff er but lesion, observe VIQ-PIQ diff erences of a large mag-
they still measure general intelligence. Spearman nitude, and infer the likely cause for the VIQ-PIQ
referred to this as the “indiff erence of the indicator” diff erence to the brain damage; but in the absence of
(1927, p. 197). brain injury or lesion, to infer such from VIQ-PIQ
For individuals who have hearing impairments diff erences is a much riskier proposition.
or are deaf, have receptive or expressive language Such inferential problems from test scores or
defi cits, or are from ethnic minority groups with other “signs” in psychology have been pointed
limited English profi ciency; nonverbal assess- out at least as far back as 1955 (Meehl & Rosen,
ment of intellectual abilities is particularly useful 1955) and are also articulated by others (Lilienfeld,
(Bracken & McCallum, 1998b; Naglieri, 2003a). Wood, & Garb, 2000; McFall 2005; Swets, Dawes,
Th is is primarily because the verbal (English) meth- & Monahan, 2000; Watkins, 2009; Watkins,
ods would likely underestimate the intellectual Glutting, & Youngstrom, 2005) as it relates to the
abilities of such individuals. But there are other Reverend Th omas Bayes’ (1702–1761) theorem of
clinical groups for whom diff erences between verbal conditional probabilities and base rates. Leonard
and nonverbal (performance) estimates have been and Hsu (1999) and Nickerson (2004) provide
reported or hypothesized. Th ese include individu- excellent descriptions of Bayes’ theorem (Bayes,
als with traumatic brain injury, bilingualism, autis- 1763) and its implications and applications. One
tic disorder, Asperger’s disorder, and delinquents or feature of diagnostic tests often highlighted is s ensi-
psychopaths. Most intelligence test technical manu- tivity, which indicates the probability of obtaining a
als (i.e., WPPSI-III, WISC-IV, WAIS-IV, KAIT, positive test result, given that the person has the tar-
UNIT, KABC-II, DAS-II) typically provide clinical get disorder. However, in diagnostic use of a test, a
group and matched normal group comparisons in clinician is much more interested in p ositive predic-
test performance that sometimes illustrate similar tive power, or the probability of a person’s having the
verbal − nonverbal diff erences but frequently contain target disorder, given a positive test result. A simi-
small samples that are intended to be only prelimi- lar contrast is that of s pecifi city, which indicates the
nary investigations. probability of obtaining a negative test result, given
that the person d oes not have the target disorder;
verbal − nonverbal (viq-piq/vci-pri) and negative predictive power , the more important
comparisons indicator of the probability of a person’s not hav-
Kaufman and Lichtenberger (2006) devoted ing the target disorder, given a negative test result.
two chapters to VIQ and PIQ diff erences as they With respect to low base rates, it is diffi cult for tests
related to neuropsychology (brain functioning or to improve accuracy in individual cases (Lilienfeld
injury) and clinical research and use. Many of the et al., 2006; McFall, 2005; Meehl, 2001; Meehl &
studies were of adults and with WB or WAIS data, Rosen, 1955).
but some studies were with children and adolescents In the case of VIQ-PIQ diff erences and infer-
and WISC data. Th eir review of more than 50 stud- ences regarding brain injury and function, inverse
ies and approximately 2,700 patients who had uni- probabilities suggest that there may well be a much
lateral brain damage generally supported the PIQ greater proportion of individuals with brain injury
> VIQ for left-hemisphere lesions and VIQ > PIQ and lesions who show VIQ-PIQ diff erences (sensi-
for right-hemisphere lesions on the WB, WAIS, tivity) than individuals with VIQ-PIQ diff erences
and WAIS-R, but there was greater consistency who also have brain injury and lesions (positive
for the right-hemisphere lesion groups. Variables predictive power). Neuropsychologists are also
that appeared to have possible moderating eff ects probably more likely to see patients referred for
included age, sex, race or ethnicity, and educational evaluations who have brain damage and observed
attainment. Group diff erence studies are not suf- VIQ-PIQ (VCI-PRI) discrepancies; and while those
fi cient for individual diagnostic use of such signs, who have brain damage are more likely to show
and Kaufman and Lichtenberger noted calls by VIQ-PIQ (VCI-PRI) discrepancies, neuropsychol-
Matarazzo and colleagues (Bornstein & Matarazzo, ogists may overestimate the value of these VIQ-PIQ
1982, 1984; Matarazzo, 1972; Matarazzo & (VCI-PRI) diff erences because they are not likely to
90 psychometric versus actuarial interpretation of intelligence
see those with VIQ-PIQ (VCI-PRI) discrepancies mental retardation/intellectual disability, and emo-
who do not have brain damage. tional disability); additional VIQ-PIQ analyses
Review and summary of research provided by of this dataset were conducted for this chapter.
Kaufman and Lichtenberger (2006) regarding PIQ > Correlations between VIQ-PIQ discrepancies and
VIQ for delinquents or “psychopaths” indicated that all ASCA syndromes (core and supplementary)
some distinct group diff erence studies found such and global adjustment scales ranged from –.004
diff erences as suggested by Wechsler (1944, 1958), to .056 (p >.05) and ranged from –.003 to .099
but results overall were reportedly mixed. Kaufman ( p >.05) with measures of academic achievement.
and Lichtenberger noted that use of this PIQ > VIQ Furthermore, there were no signifi cant diff erences
“sign” as recommended by Wechsler should not be in VIQ-PIQ discrepancies between the four diag-
used for individual diagnosis due to a lack of empiri- nostic groups. Glutting, Youngstrom, Oakland, and
cal support. Inconsistency of PIQ > VIQ fi ndings Watkins (1996) also examined relations between
for individuals with autistic disorder was also noted, WISC-III scores and ASCA (and other measures)
in addition to small eff ect sizes, and thus determined and found low WISC-III IQ and index score corre-
to be of no diagnostic clinical use (Kaufman & lations ranging from –.27 to .18 (M = –.04) across
r
Lichtenberger). More recently, a large Swedish study all ASCA syndromes and global adjustment scales.
of individuals with Asperger’s disorder, autism, or While some research may indicate distinct group
pervasive developmental disorder–not otherwise diff erences with respect to VIQ > PIQ or PIQ >
specifi ed (PDD-NOS), based on DSM-IV criteria VIQ, this is necessary but not suffi cient for indi-
(American Psychiatric Association, 1994), found vidual diagnostic utility, and until such diagnostic
profi le level (overall IQ) diff erentiated Asperger’s utility is demonstrated in diff erentiating individuals
from autism and PDD-NOS, but scatter and shape within these groups, clinicians should not assume
of profi les were small (Zander & Dahlgren, 2010). that a VIQ/VCI and PIQ/POI/PRI diff erence is an
Within the autism group, a mean VCI-POI diff er- indicator, marker, or sign for that diagnostic group.
ence of 9 points (S D = 20.5) in favor of POI was Study of distinct clinical groups may well refl ect the
observed and represented a medium eff ect size, but problem of inverse probabilities where members of
Zander and Dahlgren also noted individual profi les a distinct group may likely demonstrate VIQ-PIQ
were too variable for individual diagnostic use of the “signs” (sensitivity), but those who demonstrate
Swedish version of the WISC-III (Wechsler, 1999) VIQ-PIQ “signs” may not necessarily be members
in diff erentiating among PDD diagnoses. of that distinct clinical group (positive predictive
Cronbach (1990), however, noted problems power). Kaufman and Lichtenberger (2006) noted
with interpreting diff erence scores due to their low that, “when evaluating V–P diff erences for individu-
reliability. VIQ-PIQ diff erence scores, for example, als instead of groups, extreme caution must be exer-
have been shown to have poor temporal stability cised” (p. 316). In the absence of diagnostic utility
(too low for individual clinical use) and thus likely research affi rming the diagnostic utility of scores
of questionable utility (Canivez & Watkins, 1998, for individual diagnostic purposes (especially their
1999, 2001; Cassidy, 1997). Th e inference from positive predictive power), interpretation of those
signifi cant VIQ-PIQ diff erences is that the indi- scores should probably be curtailed.
vidual has stronger cognitive skills in one area than
the other, as well as giving rise to speculation as to factor/broad-ability score comparisons
the implications of the diff erence. However, if the Beginning with the WISC-IV, revised Wechsler
diff erence score is not suffi ciently reliable, it cannot scales no longer provide VIQ and PIQ scores and
be valid or of diagnostic value. Also, what such an now only report factor index scores as Stratum II
analysis and inference ignores is the fact that VIQ abilities, as they are more factorially pure indexes
and PIQ scores are not independent and such infer- of latent abilities. Like Full Scale scores, interpreta-
ences from them are obscured by shared variance. tion of factor or broad-ability scores typically fi rst
While Canivez, Neitzel, and Martin (2005) did involves a presentation and description of the stan-
not examine VIQ-PIQ diff erences in their study dard score, percentile rank, and confi dence interval
on relationships between the WISC-III, KBIT, (obtained score or estimated true score) for each
Adjustment Scales for Children and Adolescents standard score to account for measurement error
(ASCA: McDermott, Marston, & Stott, 1993), and (AERA, APA, NCME, 1999). Classifi cation of
academic achievement; with a sample (N = 207) of factor or broad-ability scores within a descriptive
various students (non-disabled, learning disability, category (or range of descriptive categories around
canivez 91
the confi dence interval) refl ecting some deviation 2001) for example, it was found that across the
from average is also frequently made (i.e., “average,” total sample and across age, sex, race or ethnicity,
“below average,” “signifi cantly above average”). and disability groups that the VCI, POI, FDI, and
Like Full Scale scores, these normative descriptors PSI scores had lower stability coeffi cients than the
are intended to provide an illustration of how the FSIQ but more importantly, only the VCI and POI
individual performed relative to others their age and scores showed long-term temporal stability coef-
are a function of the interval level measurement the fi cients close to being high enough for individual
standardized scores represent. interpretation or decision-making (r ≥ .85; Hills,
Because there are multiple factor or broad-ability 1981; r ≥ .90; Aiken, 2000; Guilford & Fruchter,
scores, test authors and publishers provide in their 1978; Nunnally & Bernstein, 1994; Ponterotto
respective manuals procedures for comparing these & Ruckdeschel, 2007; Salvia & Ysseldyke, 1988,
scores to each other. Tables of critical values of dif- 2001). Th e FDI and PSI score stability coeffi cients
ference scores as well as base rates for diff erences were too low (too unstable) for individual clinical
are presented in test manuals and provide clini- use. Similar results were also obtained by Krohn and
cians a convenient way to determine which factor Lamp (1999) for the K-ABC and SB:FE.
or broad-ability scores diff er and how rare such a
diff erence was in the standardization sample. Like factorial/structural validity
VIQ-PIQ diff erences, the inference from signifi cant While some factor-based scores might possess
diff erences between factor or broad-ability scores is acceptable reliability coeffi cients (internal con-
that the individual has stronger cognitive skills in sistency, short-term stability, long-term stability)
one area than the other and there is speculation as to and reliability is a foundation for possible score
the implications of these strengths and weaknesses. validity, validity is ultimately more important.
Factor or broad diff erence scores that are not suffi - Also, “validity is always s pecifi c to some particular
ciently reliable cannot be valid or of value. Like the use or interpretation ” (Linn & Gronlund, 1995,
VIQ and PIQ scores, factor or broad area scores are p. 49). Investigations of the internal or structural
not independent and inferences from them are also validity of intelligence tests is often conducted
obscured by shared variance. via factor analyses (exploratory [EFA] a nd confi r-
matory [CFA]), but recently, some intelligence
factor/broad-ability psychometric test authors and publishers (Elliott, 2007b; Roid,
support 2003b, Wechsler, 2008b; McGrew & Woodcock,
Psychometrically, factor scores or broad-ability 2001) opted to report only results from CFA. Th is
scores typically have internal consistency estimates, is in contrast to previous practice (and some cur-
short-term stability estimates, and predictive valid- rent practice) wherein both EFA a nd CFA results
ity coeffi cients that are generally lower than the Full were both reported (Bracken & McCallum, 1998b;
Scale score but higher than individual subtest scores Elliott, 1990; Glutting et al., 2000b; Kaufman &
(Bracken & McCallum, 1998b; Elliott, 2007b; Kaufman, 1993; Naglieri & Das, 1997b; Wechsler,
Glutting, Adams, & Sheslow, 2000b; Kaufman 1991, 2002a, 2002b; Wechsler & Naglieri, 2006).
& Kaufman, 1983, 1993, 2004a, 2004b; Naglieri Gorsuch (1983) noted the complimentary nature of
& Das, 1997b; Th e Psychological Corporation, EFA and CFA, suggesting that greater confi dence in
1999; Reynolds & Kamphaus, 2003; Roid, 2003b; the internal structure of a test is obtained when EFA
Wechsler, 1997, 2002a, 2002b, 2003, 2008b; and CFA are in agreement. As noted by Frazier and
Wechsler & Naglieri, 2006; Woodcock et al., Youngstrom (2007), there is good cause for concern
2001). Th is is expected, as true score theory pre- regarding the disagreement between the number of
dicts that scores with more items and subtests will latent factors reported in contemporary intelligence
have less error variance and thus greater reliability tests based only on CFA procedures (or the most
and true score variance. Th is also means that factor liberal EFA factor-extraction criteria) and the num-
scores or broad-ability scores typically include more ber of factors suggested with EFA procedures using
error variance than the Full Scale score. However, the most psychometrically sound methods for deter-
some factor scores or broad-ability scores have bet- mining the correct number of factors to extract and
ter reliability estimates than others, partly related to retain. For example, DiStefano and Dombrowski
the number of subtests (and items) that comprise (2006) and Canivez (2008) provided markedly dif-
the factor-based score. In long-term stability studies ferent results for the SB-5 than the CFA results pre-
of the WISC-III (Canivez & Watkins, 1998, 1999, sented in its technical manual (Roid, 2003b).
92 psychometric versus actuarial interpretation of intelligence
Another EFA approach to investigate the inter- investigations such as predictive validity and incre-
nal structure of intelligence tests is the Schmid mental validity of lower-level scores beyond that
and Leiman (1957) procedure, which was recom- of higher-level scores (Haynes & Lench, 2003;
mended by Carroll (1993, 1995, 1997a, 2003); Hunsley, 2003; Hunsley & Meyer, 2003). In this
McClain (1996); Gustafsson and Snow (1997); way the relative importance of factor scores versus
Carretta and Ree (2001); Ree, Carretta, and Green the global Full Scale score may be assessed. However,
(2003); and Th ompson (2004). Because the narrow validity should not be confused with d iagnostic
abilities (subtests) and broad abilities (factors) are utility (Meehl, 1959; Mullins-Sweatt & Widiger,
themselves correlated, subtest performance on cog- 2009; Wiggins, 1988), as the latter is concerned
nitive abilities tests refl ect combinations or mixtures with the application of test score interpretation to
of both fi rst-order a nd second-order factors. Carroll the individual. It follows that construct validity
argued that variance from the second-order factor and criterion-related validity, Cronbach and Meehl
should be extracted fi rst to residualize the fi rst-order (1955) preferred construct validity, are a prerequi-
factors, leaving them orthogonal to each other and site without which utility is not possible.
the second-order factor. Th us, variability associated A major aspect of intelligence test use is its utility
with the higher-order factor is accounted for prior in assisting in the diagnosis or classifi cation of an
to interpreting variability in the lower-order factors. individual (e.g., MR, SLD, GT). Examination of
In this way, it is possible to see how the reliable test the diagnostic or predictive utility is also a prerequi-
variance is partitioned to higher- and lower-order site for the ethical use of test scores (Dawes, 2005).
dimensions. However, almost no test manuals pro- Ultimately, the greatest utility would be the ability
vide these analyses for practitioners to review. of a test or set of variables to accurately determine
When the Schmid and Leiman (1957) procedure the likelihood of treatment response under speci-
has been used with contemporary intelligence tests, fi ed conditions (i.e., treatment validity). However,
the higher-order factor (g ) accounted for the largest prediction is, in and of itself, important, regard-
portion of variance, and considerably smaller por- less of treatment utility (Glutting, Watkins, &
tions of variance remained at the lower-order (factors) Youngstrom, 2003) and is frequently investigated.
level (Bracken & McCallum, 1998b; Canivez, 2008, Th e importance of incremental validity inves-
2011; Canivez, Konold, Collins, & Wilson, 2009; tigations in general, and in the case of multilevel
Canivez & Watkins, 2010a, 2010b; Domrowski & intelligence test interpretation in particular, is based
Watkins, in press; Dombrowski, Watkins, & Brogan, on an important scientifi c principle articulated by
2009; Nelson & Canivez, 2012; Nelson, Canivez, William of Ockham (alt. “Occam”): the law of par-
Lindstrom, & Hatt, 2007; Watkins, 2006; Watkins, simony (Occam’s razor), which states “what can be
Wilson, Kotz, Carbone, & Babula, 2006). Th is is explained by fewer principles is needlessly explained
one reason the primary (if not exclusive) interpreta- by more” Jones, 1952, p. 620). Th us, science favors
tion should be at the Full Scale score level. Clinicians a less complex explanation over a more complex
should be provided such information about the por- explanation for phenomena. In the case of intel-
tions of test variance captured at the diff erent levels ligence test interpretation, the Full Scale score, an
of the test in test manuals to facilitate decisions about estimate of g , is a more parsimonious index than
the importance of the diff erent dimensions and what the lower-level factor or broad-ability scores (and
should be interpreted. Unfortunately, this informa- subtest scores) and satisfi es the law of parsimony.
tion is absent from most contemporary intelligence Intelligence test Full Scale scores demonstrate sub-
test technical manuals. However, decisions about the stantial criterion-related validity (Neisser et al.,
validity and interpretation of intelligence tests can- 1996; Carroll, 1993; Gottfredson, 1997, 2008,
not be suffi ciently answered or resolved using only 2009; Jensen, 1998; Lubinski, 2000; Lubinski &
structural validity or internal structure perspective Humphreys, 1997), so in order for the factor scores
(EFA or CFA) (Canivez et al., 2009; Carroll, 1997b; to be relevant, they must demonstrate m eaningful
Kline, 1994; Lubinski & Dawis, 1992). predictive validity b eyond that aff orded by the Full
Scale score.
factor/broad-ability incremental Besides describing performance on factor-based
validity scores or broad-ability scores, clinicians are often
When considering intelligence test validity and instructed to consider predictive utility and explana-
interpretation across multiple levels and scores from tion of performance in academic achievement areas
a test, it is critical to consider the e xternal validity refl ecting the higher- and lower-order factor scores.
canivez 93