the relationship between student evaluations and teacher quality in high school in saudi arabia

212 Pages·2015·3.24 MB·English

by  Abdulelah Alqarni

THE RELATIONSHIP BETWEEN STUDENT EVALUATIONS AND TEACHER
QUALITY IN HIGH SCHOOL IN SAUDI ARABIA: ITEM RESPONSE THEORY
ANALYSIS AND MULTILEVEL MODELING

A dissertation submitted to the
Kent State University College
of Education, Health, and Human Services
in partial fulfillment of the requirements
for the degree of Doctor of Philosophy

by

Abdulelah Alqarni
May 2015
© Copyright 2015, by Abdulelah Alqarni
All Rights Reserved

ii
A dissertation written by
Abdulelah Alqarni
B.A., King Khalid University, 2001
M.A., Umm Al-Qura University, 2007
Ph.D., Kent State University, 2015

Approved by

________________________________, Director, Doctoral Dissertation Committee
Tricia Niesz

________________________________, Member, Doctoral Dissertation Committee
Margarita Kokinova

________________________________, Member, Doctoral Dissertation Committee
Christopher Was

Accepted by

________________________________, Director, School of Foundations, Leadership,
Mark Kretovics and Administration

________________________________, Dean, College of Education, Health and
Daniel F. Mahony Human Services
iii
ALQARNI, ABDULELAH, Ph.D., May 2015 FOUNDATIONS, LEADERSHIP,
AND ADMINISTRATION

THE RELATIONSHIP BETWEEN STUDENT EVALUATIONS AND TEACHER
QUALITY IN HIGH SCHOOL IN SAUDI ARABIA: ITEM RESPONSE THEORY
(IRT) ANALYSIS AND MULTILEVEL MODELING (198 pp.)

Director of Dissertation: Tricia Niesz, Ph.D.

This study examines the multilevel factors (i.e., student’s achievement, Arabic
teacher quality measure [ATQM] scores, and rural and urban school) that influence
student ratings of teachers (SRT) in Saudi Arabia among third grade high school students
(i.e., equivalent to 12th graders in the United States). Rasch Analysis, Hierarchical Linear
Modeling, and Analysis of Covariance were used to investigate the following three main
objectives: (a) developing and adapting an ATQM, (b) developing a measure of SRT, and
(c) examining the relationship between student-, teacher-, and school-level factors on
student ratings of teacher on the SRT.
Specific to the first objective, a psychometrically sound measure of teacher quality
was developed. The results retained a 24-question measure focusing on four domains: (a)
Teacher Planning and Preparation, (b) Classroom Environment, (c) Teacher Instruction,
and (d) Teacher Professionalism. The second objective, a 25-question SRT measure was
examined. The third objective found that a significant relationship between teacher
quality scores and SRT scores, with higher teacher quality related to lower student ratings
of teacher scores. Additionally significant differences were found between urban schools
and rural schools on student ratings of teacher scores, with rural schools having higher
SRT scores.
Future research should continue to define the network of relationships between
pupil-, teacher-, and school-level factors and SRT, and maintain the measures revision
and validation process of both the ATQM and the SRT. The findings support
administrative demands for more efficient ways to monitor teacher quality, and in turn,
meet educational standards and improve school outcomes.
DEDICATION

This document is dedicated to my beloved parents and family. It is also dedicated to all
of my valued previous, current, and future educators, teachers, and advisors. It is also
dedicated to all of the children in the world who deserve high quality teachers.

iv
ACKNOWLEDGMENTS

First and foremost, praises and thanks to the God, the Almighty, for His showers
of blessings throughout my research work to complete the research successfully. There
are many people who have contributed to the work that is presented here.
Each of my dissertation committee members has inspired my work in unique and
highly significant ways. I would like to express my deep and sincere gratitude to my
research supervisor, Dr. Tricia Niesz, who served as chair of my dissertation committee
for providing invaluable guidance throughout this research. Dr. Niesz believed from the
start that my efforts had value and encouraged me without hesitation. Her commitment
to nurturing the diverse interests of her students was a constant inspiration for me. I
always appreciated her constructive comments, and her personal support of my academic
endeavors.
I am deeply grateful to Dr. Aryn Karpinski, my previous advisor and mentor
throughout my doctoral studies, who served as advisor and for supporting the idea of the
research. She has taught me Item Response Theory and Hierarchal Linear Model, which
helped me to carry out the research. Although she left Kent State University temporarily,
she agreed to mentor the quantitative part of my study.
I am grateful for the other members of my committee, especially Dr. Margarita
Kokinova and Dr. Christopher Was, who critically read my work, and provided
constructive comments, which were very beneficial in shaping my dissertation. Thank
you also to Dr. Kenneth Cushner for serving as the Graduate Faculty Representative for
my defense. These are lessons I will carry for my future work.
v
Many people in Saudi Arabia supported this study by sharing generously of their
time, often on short notice, during my fieldwork. Sr. Sulima Alqarni volunteered to be
my research assistant, and shared thoughtful insights on teacher quality that helped shape
my understanding of the topic in the context of Saudi Arabia. I salute all of the educators
who take their work seriously enough to sit down and write about it. Without them, I
would never have learned what it means to be a high quality teacher. I am also deeply
indebted to all of the teachers, students, and educators who have helped frame this
research. Without their input and participation, this study would not have been possible.
I am extremely grateful to my parents for their love, prayers, caring, and sacrifices
for educating and preparing me for my future. I am very much thankful to my wife and
my son and daughters for their love, understanding, prayers, and continuing support to
complete this research work. Also I express my thanks to my sisters and brothers-in-law
for their support and valuable prayers.
I thank King Abdul-Aziz University (KAU) where I work for their support
throughout the study and the research. I thank Saudi Arabia culture mission (SACM) for
their genuine support to complete this dissertation successfully. Finally, my thanks go to
all the people who have supported me to complete the research work directly or
indirectly.

vi
TABLE OF CONTENTS
Page

DEDICATION ................................................................................................................... iv
ACKNOWLEDGMENTS .................................................................................................. v
LIST OF FIGURES ............................................................................................................ x
LIST OF TABLES ............................................................................................................. xi
CHAPTER

I. INTRODUCTION ..........................................................................................................1
The Context of the Saudi Arabian Schooling System ............................................ 2
High School Education ........................................................................................... 3
Teacher Preparation ................................................................................................ 4
The Context of the Study ........................................................................................ 4
Teacher Evaluations ...................................................................................... 5
Statement of the Problem .............................................................................. 8
Research Questions ................................................................................................. 9
The Rationale for the Study .................................................................................. 10
Significance of the Study ...................................................................................... 11
Variables in the Study ........................................................................................... 13

II. LITERATURE REVIEW ............................................................................................17
Teacher Evaluation ............................................................................................... 17
Different Methods of Teacher Evaluation ............................................................ 19
Portfolios ..................................................................................................... 19
Value-Added Models .................................................................................. 19
Peer Evaluation ........................................................................................... 20
Self-Evaluation ............................................................................................ 21
Student Ratings of Teachers ........................................................................ 22
Initial Studies on Students Rating Teachers Globally and in Saudi
Arabia .......................................................................................................... 24
Student-Level Variable: Students’ Achievement.................................................. 25
Teacher-Level Variable: Teacher Quality ............................................................ 29
Studies Measuring Teacher Quality ...................................................................... 31
Teacher Quality and Student Ratings of Teacher ................................................. 35
School-Level Variable: School Type .................................................................... 44
Summary ............................................................................................................... 45

III. METHODOLOGY ......................................................................................................47
vii
Research Question 1: Development of the Arabic Teacher Quality Measure ...... 47
Adaptation of the ATQM ............................................................................ 47
The Pilot Phase of the ATQM ..................................................................... 49
Data Source: Data From Teachers .............................................................. 50
The Administration of the ATQM .............................................................. 51
Research Question 2: Development of the Student Ratings of Teachers (SRT)
Measure ................................................................................................................. 52
ATQM and SRT: Data Collection Procedures ............................................ 53
Data Analysis ........................................................................................................ 56
Research Question 1 .................................................................................... 56
Research Question 2 .................................................................................... 57
Research Question 3A ................................................................................. 57
Research Question 3B ................................................................................. 60

IV. RESULTS ...................................................................................................................63
Research Question 1 ............................................................................................. 63
Teacher Quality ........................................................................................... 64
Qualitative Information ............................................................................... 65
Item writing and content validity ......................................................... 65
Quantitative Data ......................................................................................... 67
Pilot study of ATQM ............................................................................ 67
Pilot Study Analysis .................................................................................... 70
Content Analysis of the 37-Item ATQM ..................................................... 74
Final ATQM Analysis ................................................................................. 77
Final ATQM descriptive statistics ........................................................ 77
Rasch summary statistics ..................................................................... 78
Rasch response scale ............................................................................ 81
Rasch probability curves ...................................................................... 83
Rasch item misfit diagnostics. .............................................................. 84
Rasch map of persons and items .......................................................... 86
Research Question 2 ............................................................................................. 88
Development and Examination of SRT ...................................................... 88
Qualitative Information ............................................................................... 89
Item writing and content validity ......................................................... 89
Quantitative Data: Previous Data Analysis of SRT .................................... 92
Final SRT Analysis ..................................................................................... 98
Rasch summary statistics ..................................................................... 99
Rasch response scale .......................................................................... 102
Rasch probability curves .................................................................... 104
Rasch item misfit diagnostics ............................................................. 105
Rasch map of persons and items ........................................................ 107
Research Question 3A......................................................................................... 109
Descriptive Statistics of the SRT .............................................................. 110
Student Ratings of Teachers’ Assumptions .............................................. 111
viii