Posterior Predictive Model Checking in Bayesian Networks by AAron Crawford A Dissertation ...

204 Pages·2014·2.02 MB·English

Posterior Predictive Model Checking in Bayesian Networks
by
AAron Crawford

A Dissertation Presented in Partial Fulfillment
of the Requirements for the Degree
Doctor of Philosophy

Approved April 2014 by the
Graduate Supervisory Committee:

Roy Levy, Chair
Samuel Green
Marilyn Thompson

ARIZONA STATE UNIVERSITY
May 2014
© 2014 AAron Vaughn Crawford
All Rights Reserved
ABSTRACT

This simulation study compared the utility of various discrepancy measures within a
posterior predictive model checking (PPMC) framework for detecting different types of
data-model misfit in multidimensional Bayesian network (BN) models. The investigated
conditions were motivated by an applied research program utilizing an operational
complex performance assessment within a digital-simulation educational context
grounded in theories of cognition and learning. BN models were manipulated along two
factors: latent variable dependency structure and number of latent classes. Distributions
of posterior predicted p-values (PPP-values) served as the primary outcome measure and
were summarized in graphical presentations, by median values across replications, and by
proportions of replications in which the PPP-values were extreme. An effect size measure
for PPMC was introduced as a supplemental numerical summary to the PPP-value.
Consistent with previous PPMC research, all investigated fit functions tended to perform
conservatively, but Standardized Generalized Dimensionality Discrepancy Measure
(SGDDM), Yen's Q , and Hierarchy Consistency Index (HCI) only mildly so. Adequate
3
power to detect at least some types of misfit was demonstrated by SGDDM, Q , HCI,
3
Item Consistency Index (ICI), and to a lesser extent Deviance, while proportion correct
(PC), a chi-square-type item-fit measure, Ranked Probability Score (RPS), and Good's
Logarithmic Scale (GLS) were powerless across all investigated factors. Bivariate
SGDDM and Q were found to provide powerful and detailed feedback for all
3
investigated types of misfit.
i
DEDICATION

To Linda, for steadfast friendship in trying times, for carrying an excessive burden
interminably, for sacrificing more of yourself and subsidizing more of this work than
anyone knows.
ii
ACKNOWLEDGMENTS

A heartfelt thanks to everyone who contributed to this work, particularly Dr. Samuel
Green and Dr. Marilyn Thompson for constructive criticism of the study design and
drafts of the paper, Dr. Dubravka Svetina for the suggestion of using multiple instances
of R on a single machine as a time-saving computational strategy, Dr. Hollis Lai for
sharing R code for ICI, Katie Kunze, Nedim Yel, and Derek Fay for useful feedback on
aspects of presentation and communication in figures and tables, and for responsive
suggestions during troubleshooting phases. A disproportionate debt of gratitude is owed
to Dr. Roy Levy for his generous guidance on all aspects of the project throughout the
entire process, and especially for mentoring with improbable patience and equanimity.
iii
TABLE OF CONTENTS
Page
LIST OF TABLES ……………………………………………………………………. viii
LIST OF FIGURES …...………………….…………………………………………...… x
INTRODUCTION ………………………...…………………………………………….. 1
Purpose of the Study …………………………………………………………….. 2
LITERATURE REVIEW …………..………………………………………………….... 5
Bayesian Networks …...……………………………………………………….… 5
Description of Bayesian Networks …………………………………….... 5
Bayesian Networks in Psychometrics …………………………………… 8
Assessing Data-Model Fit …………………………………………………….... 11
PPMC …………………………………………..……………………… 13
Description of PPMC ………………………………………… 13
Example Using Q …………..................……….....…………. 17
3
Applications of PPMC …..……...............………………….… 19
Fit Functions …..………………………………………………………. 19
Global Fit ………………...…………………………………… 20
Local Fit ………….…………………………………………… 24
Person Fit ……………...……………………………………… 29
Research on Discrepancy Measures ………………………....... 31
Label Switching …………………………………………………………….… 36
Summary ……………………………………………………………………… 36
METHOD ……………………………………………………………………………… 37
iv
Page
Simulation Study …………………...………………………………………… 38
Manipulated Factors …………..……………………………………… 38
Simple Structure: Models 1 and 2 ……………………………… 39
Context Effects: Models 3 and 4 ….………………………….… 46
Complex Structure: Models 5 and 6 .………………………...… 52
Latent Dependency Structures: Models 1, 3, and 5 ………….… 58
Latent Dependency Structures: Models 2, 4, and 6 ………….… 61
Conditions ……………………………………….….………………… 65
Replications…………………………………………………………… 66
Sample Size …………………………………………………………… 67
Estimation …………………………………………………………….. 67
Label Switching ……………………………………………….……… 68
Fit Functions …………………………………………………………………… 69
Deviance …...……………………………………………………......... 70
Proportion Correct …………………………….……………………… 70
Q …..……………………………………………………………….… 70
3
SGDDM ……….................................................................................… 71
-type Item Fit Index ..…………………………………………….… 71
(cid:2)
χ
Ranked Probability Score ………………………………………......… 71
Good’s Logarithmic Scale .………………………………………....… 71
Hierarchy Consistency Index ……………………………….............… 72
Item Consistency Index …………………………………………......... 72
v
Page
Outcome Variables ………………………………………………………….… 72
PPP-values …………......…………………………………………….… 73
Effect Size ……………………………………………………………… 73
Computing Time ……………………………………………………..… 74
RESULTS ……………………………………………………………………………… 75
MCMC ..……………………………………………………………………...… 75
Label Switching …………………………………………………………………75
Distributions of PPP-values …………………………...……………………….. 79
Global SGDDM ……………………………………………..…………………. 86
SGDDM Subscale θ …………………………………..………………………. 99
1
SGDDM Subscale θ ……………………………………..…………………… 100
2
SGDDM Subscale θ ……………………………………..………………...… 100
3
Bivariate SGDDM …………………………………………..……………...… 109
Q …………………………………………………………..……………….… 131
3
HCI ………………………………………………………..……………….…. 131
ICI ………………………………………………………………………….…. 140
DISCUSSION ……………………………………………….……………………… 157
Discrepancy Measures …………………………………………….………..… 157
Effect Size ………………………………………………………………….…. 161
Computing Time ……………………………………………………………… 164
Recommendations …………………………………………………………..… 165
Limitations ……………………………………………………...…………..… 167
vi
Page
REFERENCES ……………………………………………………………………..… 169
APPENDIX …………………………………………………………………………… 178
vii
LIST OF TABLES
Table Page
1. Q-matrix for Models 1 and 2 ................................................................................ 40
2. CPT Template 4 .................................................................................................... 42
3. CPT Template 8 ..................................................................................................... 44
4. CPT Template 9 .................................................................................................... 45
5. CPT Template 10 .................................................................................................. 48
6. Q-matrix for Models 3 and 4 ................................................................................ 49
7. CPT Template 11 .................................................................................................. 50
8. CPT Template 12 .................................................................................................. 51
9. CPT Template 13 .................................................................................................. 52
10. Q-matrix for Models 5 and 6 .............................................................................. 53
11. CPT Template 14 ................................................................................................. 55
12. CPT Template 15 ................................................................................................ 57
13. CPT Template 16 ................................................................................................. 57
14. CPT Template 1 .................................................................................................. 58
15. CPT Template 2 .................................................................................................. 59
16. CPT Template 3 ................................................................................................... 59
17. Marginal Latent Variable Proficiencies for Generating Models 1, 3, and 5 ..... 60
18. Marginal Latent Variable Proficiencies for Generating Model 3 ...................... 60
19. CPT Template 5 .................................................................................................. 61
20. CPT Template 6 ................................................................................................... 62
21. CPT Template 7 .................................................................................................. 62
viii