VALIDITY AND DIAGNOSTIC ACCURACY OF SCORES FROM THE AUTISM DIAGNOSTIC ...

188 Pages·2012·0.45 MB·English

The Pennsylvania State University
The Graduate School
College of Education

VALIDITY AND DIAGNOSTIC ACCURACY OF SCORES FROM THE AUTISM
DIAGNOSTIC OBSERVATION SCHEDULE-GENERIC

A Dissertation in

School Psychology

by

Melissa A. Reid

© 2012 Melissa A. Reid
Submitted in Partial Fulfillment of the Requirements
for the Degree of
Doctor of Philosophy
August 2012
ii

The dissertation of Melissa A. Reid was reviewed and approved* by the following:

James C. DiPerna
Associate Professor of Education
Professor in Charge of the Program of School Psychology
Dissertation Adviser

Richard Hazler
Professor of Education

Robert Steven
Professor of Education

Beverly J. Vandiver
Associate Professor of Education

*Signatures are on file in the Graduate School
iii

Abstract
The purpose of this study was to examine the internal structure, relationships with other
variables, and diagnostic accuracy of scores on the Autism Diagnostic Observation Schedule –
Generic (ADOS-G; Lord et al., 1999) for the purpose of diagnostic decision-making. Participants
were 462 children enrolled in a public school district in the southern U.S. who were referred for
a school-based psychoeducational evaluation. Four hypotheses were tested with mixed results.
The first prediction was that items included in the Original Scoring Algorithm (OSA) would
reflect a uni-dimensional construct, and items included in the Revised Scoring Algorithm (RSA)
would reflect two constructs across modules. Exploratory factor analysis confirmed the one-
factor structure of the OSA across modules. However, a two-factor structure was not retained for
the Module 2 or Module 3 RSA. Second, it was predicted that total scores on the ADOS-G,
across modules and scoring algorithms, would demonstrate moderate to strong relations with
scores from other measures of autistic behavior, and weak relations with measures of emotional
functioning. Weak relationships were consistently measured between participants’ scores on the
ADOS-G across modules and algorithms and other measures of autistic and emotional
functioning. Third, it was predicted that scores obtained from application of the RSA would
result in greater diagnostic accuracy than those obtained from the OSA. Receiver Operating
Curve (ROC) analysis was conducted to determine the sensitivity and specificity of ADOS-G
scores. Consistent with hypotheses, the RSA typically resulted in greater diagnostic accuracy,
and a better balance between sensitivity and specificity than did the OSA. Finally, the fourth
hypothesis, which predicted that the diagnostic accuracy of the ADOS-G would be lower with an
independent criterion relative to an interdependent criterion, was not consistently supported. In
general, results of the current study confirm the structural validity and overall diagnostic
iv

accuracy of the ADOS-G, but also highlight some of the limitations of the instrument. Despite its
limitations, it was concluded that the strengths of the ADOS-G provide support for its continued
use in school-based psychoeducational evaluations for the diagnosis of students with Autism
Spectrum Disorders.
v

Table of Contents
List of Tables……………………………………………………………………………………viii
List of Appendices………………………………………………………………………………...x
Acknowledgements………………………………………………………………………………xii
Chapter 1. Introduction and Literature Review ...............................................................................1
Definition of Autism Spectrum Disorders……………………………………………………..2
Common Characteristics of Autism Spectrum Disorders……………………………………...3
Assessment and Diagnosis of Autism Spectrum Disorders…………………………………....5
Autism Diagnostic Observation Schedule……………………………………………………..8
Development and Evolution of the ADOS………………………………………………....9
Autism Diagnostic Observation Schedule-Generic…………………………………….....10
Rationale for Present Study…………………………………………………………………..36
Purpose and Hypotheses……………………………………………………………………...37
Chapter 2. Method……………………………………………………………………………….40
Participants…………………………………………………………………………………....40
Measures……………………………………………………………………………………...44
Autism Diagnostic Observation Schedule-Generic……………………………………….44
Gilliam Autism Rating Scale, Second Edition……………………………………………45
Behavior Assessment System for Children, Second Edition……………………………..47
Procedure……………………………………………………………………………………..50
Chapter 3. Results………………………………………………………………………………..53
Preliminary Analyses and Testing of Assumptions…………………………………………..53
ADOS-G Item Analysis…………………………………………………………………...53
Total, Scale, and Subscale Score Analysis………………………………………………..60
vi

Hypothesis 1: Factor Structure of the Original and Revised Scoring Algorithms……………60
Module 1-Original Scoring Algorithm……………………………………………………63
Module 1-Revised Scoring Algorithm……………………………………………………66
Module 2-Original Scoring Algorithm……………………………………………………70
Module 2-Revised Scoring Algorithm…………………………………………………….71
Module 3-Original Scoring Algorithm……………………………………………………77
Module 3-Revised Scoring Algorithm……………………………………………………79
Hypothesis 2: Relationships between Scores on the ADOS-G and Other Measures………...82
Module 1…………………………………………………………………………………..82
Module 2…………………………………………………………………………………..86
Module 3…………………………………………………………………………………..87
Hypothesis 3: Comparisons of Diagnostic Accuracy Indicators Across Scoring Algorithms..88
Original and Revised Scoring Algorithm Comparisons…………………………………..89
Updated Scoring Algorithms and Optimal Cut-Score Comparisons……………………...92
Hypothesis 4: Diagnostic Accuracy of Independent Clinical Diagnoses…………………….95
Chapter 4. Discussion……………………………………………………………………………99
Structural Validity Evidence………………………………………………………………...100
Module 1…………………………………………………………………………………100
Module 2…………………………………………………………………………………101
Module 3…………………………………………………………………………………102
Convergent and Discriminant Validity Evidence…………………………………………...103
Evidence of Diagnostic Accuracy…………………………………………………………...105
Module 1…………………………………………………………………………………105
vii

Module 2…………………………………………………………………………………107
Module 3…………………………………………………………………………………109
Independent Clinical Diagnoses…………………………………………………………….111
Summary of Evidence by Module and Scoring Algorithm…………………………………112
Module 1…………………………………………………………………………………112
Module 2…………………………………………………………………………………114
Module 3…………………………………………………………………………………116
Clinical Implications………………………………………………………………………...118
Limitations…………………………………………………………………………………..120
Future Research………………………………………………………………………………123
Conclusions……………………………………………………………………………………..124
References………………………………………………………………………………………126
Footnotes………………………………………………………………………………………..136
viii

List of Tables
Table 1. Sensitivity and Specificity of Original and Revised Scoring Algorithms by Research
Study…………………………………………………………………………………………….23
Table 2. Demographic Characteristics of Total Sample (N = 462) and Independent Clinical
Diagnosis Subsample (N = 100)…………………………………………………………………41
Table 3. Item Means, Standard Deviations, Skew and Kurtosis Values on Module 1 from the
ADOS-G (N = 82)………………………………………………………………………………..54
Table 4. Item Means, Standard Deviations, Skew and Kurtosis Values on Module 2 from the
ADOS-G (N =118)……………………………………………………………………………….56
Table 5. Item Means, Standard Deviations, Skew and Kurtosis Values on Module 3 from the
ADOS-G (N = 262)……………………………………………………………………................58
Table 6. Participant’ Means, Standard Deviations, Score Range, Skew, and Kurtosis Values on
the ADOS-G, GARS-2, and Selected Subscales from the BASC-2……………………………..61
Table 7. Structure Coefficients and Communalities for the ADOS-G Module 1 (Original Scoring
Algorithm) Items (N = 82)……………………………………………………………………….65
Table 8. Pattern Coefficients, Structure Coefficients, and Communalities for the ADOS-G
Module 1 (Revised Scoring Algorithm) Items (N = 66)…………………………………………67
Table 9. Structure Coefficients and Communalities for the ADOS-G Module 1 (Revised Scoring
Algorithm) Items (N = 66)……………………………………………………………………….69
Table 10. Structure Coefficients and Communalities for the ADOS-G Module 2 (Original
Scoring Algorithm) Items (N = 118)…………………………………………………………….72
Table 11. Pattern Coefficients, Structure Coefficients, and Communalities for the ADOS-G
Module 2 (Revised Scoring Algorithm) Items (N = 73)………………………………………...74
Table 12. Structure Coefficients and Communalities for the ADOS-G Module 2 (Revised
Scoring Algorithm) Items (N = 73)……………………………………………………………...76
Table 13. Structure Coefficients and Communalities for the ADOS-G Module 3 (Original
Scoring Algorithm) Items (N = 262)…………………………………………………………….78
Table 14. Structure Coefficients and Communalities for the ADOS-G Module 3 (Revised
Scoring Algorithm) Items (N = 261)…………………………………………………………….81
ix

Table 15. Pearson Correlations between Participants’ Total Scores on the ADOS-G Original and
Revised Scoring Algorithms for Module 3 and Parent and Teacher Ratings on the GARS-
2…………………………………………………………………………………………………..83
Table 16. Pearson Correlations between Participants; Total Scores on the ADOS-G Original,
Revised, and Updated Scoring Algorithms and Parent and Teacher Ratings on the BASC-2…..84
Table 17. AUC Values, Sensitivities, Specificities, Positive Predictive Values, Negative
Predictive Values, and Hit Rates of ADOS-G Scores on the Original and Revised Scoring
Algorithm………………………………………………………………………………………...90
Table 18. AUC Values and Optimal Cut-Scores for the ADOS-G Updated and Retained Scoring
Algorithms……………………………………………………………………………………….93
Table 19. Sensitivities, Specificities, Positive Predictive Values, Negative Predictive Values, and
Hit Rates of ADOS-G Scores on the Updated and Retained Scoring Algorithms………………94
Table 20. Sensitivities, Specificities, Positive Predictive Values, Negative Predictive Values, and
Hit Rates of ADOS-G Scores on the Original and Revised Scoring Algorithm Compared to
Clinical Diagnoses Made With and Without the ADOS-G (N = 100)…………………………..96
Table 21. Sensitivities, Specificities, Positive Predictive Values, Negative Predictive Values, and
Hit Rates of ADOS-G Scores from the Updated and Retained Original Scoring Algorithms
Compared to Clinical Diagnoses Made With and Without the ADOS-G (N = 100)……………98
x

List of Appendices
Appendix A: DSM-IV-TR Diagnostic Criteria for Autism Spectrum Disorders………………137
Appendix B: ……………………………………………………………………………………142
Table B1: Activities on the Autism Diagnostic Observation Schedule and their Purpose by
Module (Lord, Rutter, DiLavore, & Risi, 1999)…………………………………………….142
Table B2: Items Rated on the Autism Diagnostic Observation Schedule by Subdomain and
Module (Lord, Rutter, DiLavore, & Risi, 1999)…………………………………………….149
Table B3: Items Included in the Revised Scoring Algorithm on the Autism Diagnostic
Observation Schedule-Generic by Developmental Cell…………………………………….153
Appendix C…………………………………………………………………………………….156
Table C1: Correlation Matrix of Items Included in the ADOS-G Module 1, Original Scoring
Algorithm (N = 82)………………………………………………………………………….156
Table C2: Correlation Matrix of Items Included in the ADOS-G Module 1, Revised Scoring
Algorithm (N = 66)………………………………………………………………………….157
Table C3: Correlation Matrix of Items Included in the ADOS-G Module 2, Original Scoring
Algorithm (N = 118)………………………………………………………………………...158
Table C4: Correlation Matrix of Items Included in the ADOS-G Module 2, Revised Scoring
Algorithm (N = 73)………………………………………………………………………….159
Table C5: Correlation Matrix of Items Included in the ADOS-G Module 3, Original Scoring
Algorithm (N = 261)………………………………………………………………………...160
Table C6: Correlation Matrix of Items Included in the ADOS-G Module 3, Revised Scoring
Algorithm (N = 262)………………………………………………………………………...161
Appendix D…………………………………………………………………………………….162
Table D1: Corrected Item-Total Correlations and Cronbach’s Alpha if Item Deleted Values
for ADOG-G Original Scoring Algorithm One-Factor Solutions…………………………..162
Table D2: Corrected Item-Total Correlations and Cronbach’s Alpha if Item Deleted Values
for ADOG-G Revised Scoring Algorithm One-Factor Solutions…………………………...164
Table D3: Corrected Item-Total Correlations and Cronbach’s Alpha if Item Deleted Values
for ADOG-G Revised Scoring Algorithm Two-Factor Solutions…………………………..166
Appendix E……………………………………………………………………………………..168