Table Of ContentMokken Scale Analysis collectively refers to a set of methods to examine the fit
of data to two nonparametric Item Response Theory (IRT) models known as the
Monotone Homogeneity Model (MHM) and the Double Monotonicity Model (DMM).
As nonparametric IRT models, MHM and DMM are, compared to their parametric
Purya Baghaei
counterparts, easier to fit to the noisy data that social science researchers usually
work with. Furthermore, the logic behind these models is a lot easier to grasp by
researchers who do not have a strong background in algebra.
This book is an introductory treatment of the topic with examples from the field
Mokken
of language assessment and research. It describes the basics of MSA and includes
step-by-step tutorials to help the readers run the analyses with the R package
mokken. Furthermore, case studies are reported to illustrate the concepts intro-
duced throughout the book. The book is comprehensive and reader-friendly and
can be followed by most empirical researchers in the social sciences. It is suitable Scale Analysis
for all researchers and practitioners in the fields of behavioral and social sciences
who are engaged in test and scale development. It is an easy-to-use manual that
covers everything that you need to know to apply Mokken scaling confidently.
in Language
Assessment
ISBN 978-3-8309-4446-1
www.waxmann.com
Purya Baghaei
Mokken Scale Analysis
in Language Assessment
Waxmann 2021
Münster • New York
Bibliographic information published by die Deutsche Nationalbibliothek
Die Deutsche Nationalbibliothek lists this publication in the
Deutsche Nationalbibliografie; detailed bibliographic data
are available in the internet at http://dnb.dnb.de.
Print-ISBN 978-3-8309-4446-1
E-Book-ISBN 978-3-8309-9446-6
Waxmann Verlag GmbH, 2021
Steinfurter Straße 555, 48159 Münster, Germany
Waxmann Publishing Co.
P. O. Box 1318, New York, NY 10028, U. S. A.
www.waxmann.com
[email protected]
Cover Design: Anne Breitenbach, Münster
Print: CPI Books GmbH, Leck
Printed on age resistant paper acid-free as per ISO 9706
Printed in Germany
All rights reserved. No part of this publication may be reproduced, stored in
a retrieval system or transmitted in any form or by any means, electronic,
electrostatic, magnetic tape, mechanical, photocopying, recording or
otherwise without permission in writing from the copyright holder.
Contents
Preface ................................................................................................................. 7
Chapter 1: Mokken Scale Analysis: Core Issues ............................................. 9
1.1 Overview ........................................................................................................ 9
1.2 Guttman Scaling ........................................................................................... 11
1.3 Mokken Scale Analysis ................................................................................ 13
1.4 Comparison with the Rasch Model .............................................................. 20
1.5 Comparison with the Classical Test Theory ................................................. 22
1.6 Mokken Scale Analysis for Polytomous Items ............................................ 24
1.7 Reliability ..................................................................................................... 27
1.8 Summary ...................................................................................................... 28
Chapter 2: Mokken Scale Analysis: Advanced Issues .................................. 30
2.1 Overview ...................................................................................................... 30
2.2 Scalability Coefficients ................................................................................ 30
2.3 Automated Item Selection Procedure (AISP) .............................................. 36
2.4 Monotonicity ................................................................................................ 42
2.5 Invariant Item Ordering (IIO) ...................................................................... 45
2.6 Sample Size in MSA .................................................................................... 51
2.7 Contribution of MSA to Test Validation ...................................................... 53
2.8 Criticism of Mokken Scale Analysis ............................................................ 56
2.9 Summary ...................................................................................................... 58
Chapter 3: mokken Package Tutorial ............................................................. 60
3.1 Overview ...................................................................................................... 60
3.2 Automated Item Selection Procedure (AISP) .............................................. 60
3.3 Scalability Coefficients ................................................................................ 65
3.4 Monotonicity ................................................................................................ 69
3.5 Invariant Item Ordering (IIO) ...................................................................... 71
3.6 Nonintersection of ISRFs ............................................................................. 76
3.7 Reliability ..................................................................................................... 80
3.8 Identifying Outliers ...................................................................................... 80
3.9 Two-Level MSA .......................................................................................... 82
Chapter 4: Application of MSA to a Dichotomous Test ............................... 84
4.1 Overview ...................................................................................................... 84
4.2 Analysis ........................................................................................................ 84
4.3 Comparison with the Rasch Model .............................................................. 88
Chapter 5: Application of MSA to Polytomous Items .................................. 95
5.1 Overview ...................................................................................................... 95
5.2 Data Source and Material ............................................................................. 95
5.3 Analysis ........................................................................................................ 96
Chapter 6: Application of MSA to a Partial Credit Test ............................ 103
6.1 Overview .................................................................................................... 103
6.2 Introduction ................................................................................................ 103
6.3 Seven-Point Scale (Sample 1) .................................................................... 104
6.4 Twenty-One-Point Scale (Sample 1) .......................................................... 119
6.5 Seven-Point Scale (Sample 2) .................................................................... 132
6.6 Twenty-One-Point Scale (Sample 2) .......................................................... 134
6.7 Discussion .................................................................................................. 137
Chapter 7: Application of MSA to Rater-Mediated Performance
Assessment ...................................................................................................... 139
7.1 Overview .................................................................................................... 139
7.2 MSA for Performance Assessment ............................................................ 139
7.3 Analysis of Essay Writing .......................................................................... 143
Chapter 8: Application of MSA to Two-Level Data .................................... 150
8.1 Overview .................................................................................................... 150
8.2 Introduction ................................................................................................ 150
8.3 Two-Level Mokken Scale Analysis ........................................................... 155
8.4 Analyses and Results ................................................................................. 155
Bibliography ................................................................................................... 158
About the Author ........................................................................................... 168
6
Preface
The purpose of this monograph is to introduce two related nonparametric item
response theory (NIRT) models and a series of methods to test these models
known as Mokken Scale Analysis (MSA). Although MSA is almost as old as
some parametric IRT models, its application and spread in the social measure-
ment contexts has been modest. Only recently, there has been a surge in the
application of MSA in psychology and education. The availability of the user-
friendly mokken package in the open source statistical programming software R
has helped the spread of MSA over the past few years. This book is suitable for
all researchers who are involved in test development and questionnaire design.
MSA is a cogent and accessible technique for applied researchers who need to
develop and validate tests and questionnaires. The book can be used as supple-
mentary material in measurement courses in MA and PhD programmes in the
social sciences. In this exposition, I have done my best to explain the logic of
MSA in a simple language for social scientists in general and language testing
researchers in particular.
All the theoretical issues are covered in the first two chapters. Chapter 3 is a
tutorial on mokken package in R. I have provided the codes and applied them to
a polytomous dataset with outputs and interpretations. In chapters 4 to 8, I have
applied MSA to five different types of data. In Chapter 4, MSA is applied to a
dichotomous listening comprehension test. In Chapter 5, as an application to
Likert-type items, a foreign language reading anxiety questionnaire is analysed.
In Chapter 6, a C-Test battery, which is a language test for whose items partial
credit is given, is analysed. In Chapter 7, the very recent and innovative applica-
tion of the model to rater-mediated performance assessment is demonstrated.
And finally, in Chapter 8, two-level scores are analysed. I believe these five
types of data cover almost all the applications of MSA although the model could
in future be applied in more diverse measurement contexts. The purpose of the
chapters 4-8 is to demonstrate how MSA analyses are employed and interpreted
in applied settings. In some cases, the Rasch model and factor analysis are also
applied in parallel to MSA for comparison.
7
I would like to acknowledge the intellectual contribution of several col-
leagues who read the manuscript and provided useful comments. Andries van
der Ark from the University of Amsterdam, Rudy Ligtvoet from the University
of Köln, Daniela Crișan from the University of Groningen, Roger Watson from
Hull University, UK, Stefanie Wind from the University of Alabama, and Letty
Koopman from the University of Amsterdam reviewed different chapters of the
book and kindly offered their insights and recommendations. The nuances of the
technicalities of MSA would have been very sloppy without their comments and
corrections. A group of other colleagues and students proofread the manuscript
before publication. I am deeply indebted to Hamdollah Ravand from Vali-e-Asr
University of Rafsanjan, Iran, Mohammad Afsharrard, Mona Tabatabaee-Yazdi,
Roya Shoahosseini, and Farshad Effatpanah. Their comments and insights
helped tremendously to improve the arguments presented in the book. Any er-
rors that remain are, of course, mine.
Purya Baghaei
July, 2021
Mashhad, Iran
8
Chapter 1
Mokken Scale Analysis: Core Issues
1.1 Overview
Tests and questionnaires are prevalent in education and psychology. In fact, they
are an indispensable part of everyday practices of educators, teachers, clinicians,
and graduate students and instructors in these fields. Clinical decisions and
diagnoses as well as findings of research heavily depend on the quality of the
measures used. Furthermore, in high-stakes selection examinations, where can-
didates are given access to employment or education on the basis of test results,
the precision of the ordering of candidates is paramount. Therefore, validation
research to ascertain that tests accurately reflect levels of the relevant construct
is essential.
Mokken Scale Analysis (MSA), named after the Dutch mathematician and
political scientist Robert J. Mokken, is a series of methods to evaluate the fit of
data to Nonparametric Item Response Theory models (NIRT). The building
block of item response theory (IRT) models is the idea that educational and
psychological constructs are latent, i.e., are not directly observable and
measureable. These latent constructs can only become tangible by their manifes-
tations through test items. Test takers’ responses to items in an educational test
or a psychological questionnaire are manifestations of their locations on the
latent continuum and an index of the degree to which they possess the construct
of interest. However, the items or tasks in a test and examinees’ responses do
not necessarily coincide with the construct of interest. IRT models provide the
right apparatus to examine if there is a relationship between the items developed
by the test designer and the latent variable. IRT assumes that there is a latent
9
variable on which persons and items have a location and, with this assumption,
attempts to describe the structure in the observed variables (test items) (Sijtsma,
1998). The goal of IRT-based validation is to establish that the variation in the
item responses and the construct the researcher wants to measure co-refer to the
same entity (Baghaei & Tabatabaee-Yazdi, 2016; Borsboom, 2008).
The models under MSA are probabilistic latent trait models which are suita-
ble for instrument validation and placing respondents and items on an ordinal
scale. They can be applied to both dichotomous and polytomous items. Being
nonparametric, MSA models have less restrictive assumptions compared to
Parametric Item Response Theory (PIRT) models such as the Rasch model
(Rasch, 1960/1980). However, this comes at a price. MSA creates ordinal scales
for persons and items. In fact, unlike parametric IRT models, there is no proce-
dure for item easiness and person ability estimation on an equal-interval metric
scale. In MSA, person ability and item difficulty indices are similar to those
computed in classical test theory (CTT) methods. That is, for persons the total
raw scores are computed as ability indices and for items the proportions of cor-
rect answers are computed as indices of easiness. This limits the application of
the model in equating and adaptive testing (Sijtsma & van der Ark, 2017). MSA
models are suitable for validation purposes, scale construction, and checking the
fundamental measurement properties of tests (Wind, 2017a). The advantage of
NIRT models is that they produce better model-data fits compared to PIRT
models (Sijtsma & van der Ark, 2017).
Models under MSA are considered nonparametric IRT models because the
relationship between the latent trait and the probability of a correct response,
i.e., the item response function (IRF), need not follow a predetermined specific
shape. In the Rasch model, for example, IRFs should follow a logistic function
which is S-shaped. In MSA, as long as the IRFs are non-decreasing, they meet
the requirement of the model. Unlike PIRT models, in NIRT the relationship
between the latent trait and the item responses is represented with item response
functions, not item parameters.
The application of parametric IRT models implies “a relatively deep insight
into the structure of the variable to be measured and the properties of the items
by which it can be measured” (Mokken, 1971, p. 173). MSA is more appropriate
“in contexts in which the underlying response processes are not well under-
stood, such as affective variables” (Wind, 2017a, p. 50). MSA has mostly been
10
