THE MEASUREMENT OF THE COMPLEXITY, ACCURACY, AND FLUENCY OF WRITTEN ...

302 Pages·2017·3.29 MB·English

by  Windows User

THE MEASUREMENT OF THE COMPLEXITY, ACCURACY, AND FLUENCY OF
WRITTEN ARABIC

A Dissertation
submitted to the Faculty of the
Graduate School of Arts and Sciences
of Georgetown University
in partial fulfillment of the requirements for the
degree of
Doctor of Philosophy
in Arabic and Islamic Studies

By

Michael Raish, M.A.

Washington, D.C.
April 19, 2017
Copyright 2017 by Michael Raish
All Rights Reserved

ii
THE MEASUREMENT OF THE COMPLEXITY, ACCURACY, AND FLUENCY OF
WRITTEN ARABIC

Michael Raish, M.A.

Thesis Advisor: Karin C. Ryding, Ph.D.

ABSTRACT

This study investigates the multicomponential nature of L2 Arabic writing by adapting a
number of direct measures of linguistic complexity, accuracy, and fluency (CAF) to the
measurement of written Arabic texts produced by L2 Arabic learners and Arabic Native
Speakers (NSs). Previous studies of L2 writing have established that such measures are reliable
indicators of learners’ developing interlanguage, however few efforts have thus far applied CAF
analyses to spoken or written Arabic. The results of the measures explored here are triangulated
with learner responses to several independent estimates of Arabic proficiency, including a novel
“short-cut” estimate in the form of an Arabic C-test. Instead of a longitudinal design, in which a
learner or group of learners are asked to provide data at different stages of L2 acquisition, the
current study entailed the collection Arabic production along a spectrum of writing ability,
spanning from learners at the end of their first year through Arabic NSs.
Significant findings of this study include the fact that learners appear to “move” toward
more natively-like written production as their overall facility with the Arabic language increases.
Furthermore, learners scores on the various selected CAF measures correlate highly with each
other, as well as with the selected independent measures. C-test consistently emerges via
regression as the most reliable predictor of variability in written CAF among the selected
explanatory variables, followed closely by self-estimated overall Arabic ability. Conversely,
however, this study establishes that sentence-based measures are shown to be unreliable for the
iii
modeling of learner-produced Arabic texts, as well as the fact that communicative Arabic writing
does not necessarily benefit from lemmatization prior to lexical analysis, in spite of features of
the Arabic script and morphology that may argue otherwise. Additionally, this analysis indicates
that the commonly used independent measure Years of Study is an unreliable predictor of learner
CAF among the participants in this study. Ultimately, the measures and analyses presented here
represent an initial effort to model L2 Arabic writing according to widely used CAF measures,
however the findings presented here nevertheless have important implications for the Teaching
Arabic as a Foreign Language (TAFL) field.

iv
This dissertation could not have been written without the continued and ongoing support
of many individuals, all of whom are deserving of my sincere thanks and gratitude.
My parents and siblings have always been a constant source of encouragement, and I am
deeply grateful for support. I am also grateful to Carolyn, who selflessly assisted in some way in
nearly every stage of this dissertation. Logic dictates that I couldn’t have finished it without her.
I sincerely thank the members of my committee, including my adviser Dr. Karin Ryding,
who untiringly provided feedback, support, and guidance. Dr. Ryding has long been an
appreciated mentor and advocate. Dr. John Norris was likewise a constant source of insight and
support—the instruments and study design used here grew out of projects that I began in his
classes, and his ongoing help was crucial. Dr. Cristina Sanz was likewise an unwaveringly
positive influence during all phases of this study, generously providing direction and
encouragement.
I likewise extend my sincere gratitude to all of the individuals who participated in the
many stages of this study, including the participants, as well as those good friends who helped
recruit participants. I thank them for their contributions and interest in Arabic.

My sincere thanks to all.

v
Table of Contents

Chapter 1: Introduction ................................................................................................................... 1
1.1 Background ................................................................................................................... 1
1.2 Study Overview ............................................................................................................ 3
Chapter 2: Review of Literature ..................................................................................................... 7
2.1 Approaches to Measurement and Assessment of L2 Arabic ........................................ 7
2.2 Alternative Testing Methods....................................................................................... 12
2.3 Challenges in the Assessment of Arabic Writing ....................................................... 14
2.4 The Complexity, Accuracy, Fluency Framework ....................................................... 19
2.5 Complexity .................................................................................................................. 21
2.6 Lexical Complexity ..................................................................................................... 34
2.7 Accuracy ..................................................................................................................... 37
2.8 Fluency ........................................................................................................................ 40
2.9 Genre Effect ................................................................................................................ 43
Chapter 3: The Development of an Arabic C-Test ....................................................................... 50
3.1 Introduction ................................................................................................................. 50
3.2 Background ................................................................................................................. 51
3.3 Developing Arabic C-tests .......................................................................................... 57
3.4 Developing a Pilot C-test ............................................................................................ 73
3.5 Learner Testing ........................................................................................................... 80
3.6 Results and Rasch Analysis ........................................................................................ 82
3.7 Discussion and Conclusions ....................................................................................... 86
Chapter 4: Methodology ............................................................................................................... 89
vi
4.1 Research Questions ..................................................................................................... 89
4.2 Participants and Recruitment ...................................................................................... 89
4.3 Instruments and Procedures ........................................................................................ 93
4.4 Scoring and Coding................................................................................................... 106
4.5 Syntactic Complexity ................................................................................................ 108
4.6 Lexical Complexity ................................................................................................... 118
4.7 Accuracy ................................................................................................................... 131
4.8 Fluency ...................................................................................................................... 145
Chapter 5: Results ....................................................................................................................... 146
5.1 Results Overview ...................................................................................................... 146
5.2 C-test and Related Independent Variables ................................................................ 148
5.3 C-test and Unrelated Independent Variables ............................................................ 157
5.4 C-test Correlation with Independent Variables ......................................................... 161
5.5 Arabic Writing at Three Levels ................................................................................ 162
5.5 Usefulness of Lemmatization ................................................................................... 168
5.6 Usefulness of Hapax Index ....................................................................................... 172
5.7 CAF measures ........................................................................................................... 173
5.8 Sentence-Based Measures ......................................................................................... 186
5.9 Regression Analysis .................................................................................................. 193
5.10 NS Writing and L1 Writing Ability ........................................................................ 206
5.11 Genre Effect in Arabic Writing ............................................................................... 207
Chapter 6: Discussion and Implications ..................................................................................... 220
6.1 Introduction ............................................................................................................... 220
vii
6.2 Research Question 1: Arabic CAF Measured at Three Levels ................................. 221
6.3 Research Question 2: Independent Variables that Explain Written Arabic CAF ..... 234
6.4 Research Question 3: Genre Effect on Written CAF ................................................ 237
6.5 Limitations ................................................................................................................ 241
6.6 Implications............................................................................................................... 243
Chapter 7: Conclusion................................................................................................................. 252
References ................................................................................................................................... 255
Appendix A: Arabic C-Test ........................................................................................................ 279
Appendix B: English Background Questionnaire ....................................................................... 283
Appendix C: Arabic Background Questionnaire ........................................................................ 288

viii
List of Figures

Figure 1: Arabic as an nth Language ............................................................................................ 81
Figure 2: Arabic Learner Educational Status ................................................................................ 92
Figure 3: Languages Spoken or Studied in Addition to Arabic .................................................... 93
Figure 4: Average Time for Completion of Writing Tasks (in minutes) ...................................... 94
Figure 5: C-test Score vs YoS ..................................................................................................... 150
Figure 6: Fluency ........................................................................................................................ 176
Figure 7: Syntactic Complexity .................................................................................................. 179
Figure 8: Lexical Complexity ..................................................................................................... 182
Figure 9: Accuracy (EfT/T) ........................................................................................................ 185
Figure 10: Words per Sentence (W/S) ........................................................................................ 188
Figure 11: Coordination (T/S) .................................................................................................... 191
Figure 12: Fluency Across Genre ............................................................................................... 209
Figure 13: Syntactic Complexity across Genre .......................................................................... 211
Figure 14: Accuracy across Genre .............................................................................................. 214
Figure 15: Lexical Complexity across Genre ............................................................................. 217

ix
List of Tables
Table 1: "dispersing the protesters" .............................................................................................. 64
Table 2: Modified Arabic Deletion Strategy ................................................................................ 66
Table 3: “complaint” Variants—All Newspapers ......................................................................... 70
Table 4: “complaint” variants—Moroccan at-Tajdid ................................................................... 71
Table 5: Use of يوهج jihawiyy by Region .................................................................................... 72
Table 6: Candidate Texts .............................................................................................................. 76
Table 7: 10-Text C-test ................................................................................................................. 83
Table 8: Key Item Quality Statistics ............................................................................................. 84
Table 9: Final 5-Text C-test Item Quality Statistics ..................................................................... 85
Table 10: Self-Assessment Correlation with C-test vs. Reported Semesters of Arabic Study ..... 86
Table 11: Summary of CAF Measures ....................................................................................... 107
Table 12: Arabic T-units in Narration ........................................................................................ 110
Table 13: Simple T-Units............................................................................................................ 113
Table 14: Simple T-Units............................................................................................................ 114
Table 15: Complex T-Units ........................................................................................................ 115
Table 16: Most Frequent Types .................................................................................................. 129
Table 17: NS Verbal Mood Error ............................................................................................... 135
Table 18: Colloquial Lexical Items............................................................................................. 138
Table 19: Colloquial Lexical Items............................................................................................. 138
Table 20: Summary of CAF Measures ....................................................................................... 147
Table 21: Learner C-Test Scores ................................................................................................ 149
Table 22: Learner C-Test Scores According to Group Membership .......................................... 152
x