Early Grade Reading Assessment (EGRA) in Public Schools of Afghanistan

146 Pages·2017·1.78 MB·English

Early Grade Reading Assessment (EGRA) in
Public Schools of Afghanistan

USAID.GOV EGR STAKEHOLDER REPORT – EGRA IN AFGHANISTAN I
EGRA IN PUBLIC SCHOOLS OF
AFGHANISTAN: 2016 EGR
General Education (GE) Results in
Afghanistan

RESOURCES, SKILLS, AND CAPACITIES IN
EARLY GRADE READING IN AFGHANISTAN
(EGR)

Contracted under AID-OAA-0-14-00055 TO AID-306-TO-15-00064

Access to Basic Education: All Children Reading (ABE ACR)

DISCLAIMER
This report is made possible by the support of the American people through the United States Agency for International
Development (USAID). The contents do not necessarily reflect the views of USAID or the United States Government.

USAID.GOV EGR AFGHANISTAN EGRA & SMES STAKEHOLDER REPORT I
ACKNOWLEDGEMENTS
Chemonics and its sub-contractor, School-to-School International, would like to thank
USAID/Afghanistan for their support. We would also like to recognize the fantastic staff from the
Ministry of Education’s Early Grade Reading (EGR) Working Group for being true partners and
leaders throughout the Early Grade Reading Assessment (EGRA) and School Management and
Effectiveness Study (SMES) design and implementation. In addition, the Provincial Education
Directorate (PED) played a crucial role in the data collection process, with 93 PED personnel
trained as assessors. Finally, this endeavor would not have been possible without the dedication of
the master trainers, quality assurance officers, data collection partners, subject experts, and
administrators who lent their continuous support throughout the assessment activity.

The following persons played a major role in the process:
Abdul Karim Afzali MOE EGR Working group
Khairuddin Andar MOE EGR Working group
Haqiqullah Atiq MOE EGR Working group
Mohammad Suhrab Didar MOE EGR Working group
S. Mohammad Ehsani MOE EGR Working group
Faizullah Faiez MOE EGR Working group
Mariam Munaza MOE EGR Working group
Mir Sayed Naqibulla MOE EGR Working group
Abdul Munir Negah MOE EGR Working group
Anahita Shams MOE EGR Working group

Nancy Parks Chemonics International
Julio Ramirez de Arellano Chemonics International
Akbar Hamidi Chemonics International
Farahnaz Farah Chemonics International
Khalilurahman Hanif Chemonics International
Muhammad Daud Muddakir Chemonics International
Patricia Bader Chemonics International
Russell Raymond Chemonics International
Bridget Burns Chemonics International

Mark Lynd School-to-School International
Michel Rousseau School-to-School International
Hetal Thukral School-to-School International
Fernanda Gandara School-to-School International
Casey McHugh School-to-School International
Drew Schmenner School-to-School International
Autumn Thomas School-to-School International
Randy Tarnowski School-to-School International
Louise Bahry School-to-School International
USAID.GOV EGR STAKEHOLDER REPORT – EGRA IN AFGHANISTAN II
TABLE OF CONTENTS
ACKNOWLEDGEMENTS II
ACRONYMS V
EXECUTIVE SUMMARY VI
1. INTRODUCTION 19
1.1 EGR Project............................................................................................................................................................................. 19
1.2 Purpose of EGRA & SMES ................................................................................................................................................... 19
1.3 EGRA & SMES Design ........................................................................................................................................................... 20
1.4 Role of the Ministry of Education (MOE) ........................................................................................................................ 21
2. METHODOLOGY 22
2.1 Research Questions .............................................................................................................................................................. 22
2.2 Sample ....................................................................................................................................................................................... 22
2.3 Tool Development ................................................................................................................................................................ 25
2.4 Training of Master Trainers, Supervisors and Assessors ............................................................................................ 27
2.5 Data Collection ...................................................................................................................................................................... 28
2.6 Data Analysis ......................................................................................................................................................................... 29
2.7 Quality of Assessment Tools .............................................................................................................................................. 30
2.8 Limitations ............................................................................................................................................................................... 30
3. DARI GE EGRA RESULTS 32
3.1 GE Analytical Sample, Dari-medium schools .................................................................................................................. 32
3.2 Results by EGRA task ........................................................................................................................................................... 33
4. PASHTO GE EGRA RESULTS 62
4.1 GE Analytical Sample, Pashto-medium schools .............................................................................................................. 62
4.2 Results by EGRA task ........................................................................................................................................................... 63
5. SMES THEMES 94
6. DARI SMES THEMES RESULTS 94
Theme: School Infrastructure, DARI ............................................................................................................................................... 95
Theme: Classroom Materials, DARI ................................................................................................................................................ 97
Theme: Teaching Methodology, DARI ............................................................................................................................................ 98
Theme: Safety and Security, DARI ................................................................................................................................................. 100
Theme: Parent and Student Satisfaction, DARI .......................................................................................................................... 102
Theme: Preschool and Mosque Education, DARI ...................................................................................................................... 104
Theme: Demographics, DARI ......................................................................................................................................................... 105
Theme: Teacher and Principal Expectations, DARI .................................................................................................................. 107
Theme: Home Environment, DARI ............................................................................................................................................... 109
Theme: Student Wellbeing, DARI ................................................................................................................................................. 110
7. PASHTO SMES THEMES RESULTS 111
Theme: School Infrastructure, PASHTO ..................................................................................................................................... 112
Theme: Classroom Materials, PASHTO ...................................................................................................................................... 113
Theme: Teaching Methodology, PASHTO .................................................................................................................................. 114
Theme: Safety and Security, PASHTO .......................................................................................................................................... 116
Theme: Parent and Student Satisfaction, PASTHO ................................................................................................................... 118
Theme: Preschool and Mosque Education, PASHTO............................................................................................................... 120
RSC-EGR TECHNICAL REPORT: AFGHANISTAN EGRA & SMES RESULTS III
Theme: Demographics, PASHTO .................................................................................................................................................. 121
Theme: Teacher and Principal Expectations, PASHTO ........................................................................................................... 124
Theme: Home Environment, PASHTO ........................................................................................................................................ 125
Theme: Student Wellbeing, PASHTO .......................................................................................................................................... 126
8. DISCUSSION 128
Overall Test Quality (for Dari and Pashto)................................................................................................................................. 128
EGRA Design and Administration ................................................................................................................................................. 128
DARI-medium GE schools: Summary of EGRA Results and Recommendations .............................................................. 129
PASHTO-medium GE schools: Summary of EGRA Results and Recommendations ....................................................... 130
Summary of GE SMES Themes Analysis, Dari and Pashto ...................................................................................................... 132
Classroom Practices .......................................................................................................................................................................... 132
REFERENCES 137
LIST OF TABLES 138
LIST OF FIGURES 144

RSC-EGR TECHNICAL REPORT: AFGHANISTAN EGRA & SMES RESULTS IV
ACRONYMS
CBE Community Based Education
CLSPM Correct Letter Sounds Per Minute
CNWPM Correct Nonwords Per Minute
CWPM Correct Words Per Minute
EGR Early Grade Reading
EGRA Early Grade Reading Assessment
EMIS Education Management Information System
GE General Education
GIRoA Government of the Islamic Republic of Afghanistan
LOI Language of Instruction
MOE Ministry of Education
ORF Oral Reading Fluency
PED Provincial Education Directorate
EGR Resources, Skills and Capacities in Early Grade Reading
RSI RSI Consulting
SSME Snapshot of School Management Effectiveness
SMES School Management Effectiveness and Safety
STS School-to-School International
USAID United States Agency for International Development

RSC-EGR TECHNICAL REPORT: AFGHANISTAN EGRA & SMES RESULTS V
EXECUTIVE SUMMARY

1. INTRODUCTION
The Government of the Islamic Republic of Afghanistan (GIRoA), through the Ministry of Education
(MOE), is working with USAID/Afghanistan to improve student performance in early grade reading.
As part of this effort, USAID has provided support to the GIRoA through the Resources, Skills, and
Capacities in the Early Grade Reading (EGR) Project to conduct research that can inform discussion
about challenges faced by teachers and students concerning early grade reading in Afghanistan and
avenues to address these challenges.

In the spring and summer of 2016, EGR conducted the first ever national Early Grade Reading
Assessment (EGRA) to measure the reading ability of second and fourth grade students, and at the
same time, they conducted a study of School Management Effectiveness and Safety to understand
factors affecting reading performance in Afghanistan.1 This report presents the findings of the EGRA
and contextual factors identified in the SMES associated with strong reading performance.

2. METHODOLOGY
This section describes the EGRA and SMES design, sampling process, development of the tools, data
collection, and analysis of the quality of the tools.

EGRA & SMES Design
The purpose of this EGRA and SMES exercise was to measure students’ early grade reading skills
and the factors that help or hinder their ability to learn to read throughout Afghanistan. To that end,
the EGR project, with the MOE, 2 designed two sets of tools: (1) the Early Grade Reading
Assessment (EGRA) and (2) the Snapshot of School Management Effectiveness (SSME).3 Importantly,
the EGRA and SMES tools were adapted into Dari and Pashto from models used in other countries,
not simply translated or used in their existing form. EGRAs are adapted because all tasks and
questions should reflect students’ achievement levels in each country and language, and thus, should
be culturally and linguistically appropriate as well as aligning with MOE and USAID goals and
priorities.

RESEARCH QUESTIONS
The primary research questions of the EGRA and SMES study addressed are as follows:
• What are Grade 2 and Grade 4 students’ reading skills?
• How do students’ reading skills vary by student characteristics, region, urbanicity, and other
contextual factors?
• Which contextual factors are most frequently associated with strong reading outcomes?4

1 The EGR project is being implemented by Chemonics International, with support from School-to-School International
(STS), and RSI Consulting (RSI).
2 The MOE was deeply engaged throughout this process and across all stages of EGRA and SMES design and
implementation, including tool development and adaptation, piloting, instrument revision and finalization, training of
assessors, administration of data collection in the field, as well as validation of preliminary results.
3 The SSME is a package of tools developed for USAID’s EdData II project and includes interviews, observations and
checklists designed to examine key school-level inputs and practices to improve education outcomes. The core SSME tools
can be accessed through the following link: https://eddataglobal.org/eddata/core-ssme-instruments.
4 Variation by school type (GE vs. CBE) is also included in the Summary report: GE & CBE EGRA Results (Unweighted) in
Afghanistan but could not be included here since it is not possible to compare weighted and unweighted results.
USAID.GOV EGR STAKEHOLDER REPORT – EGRA IN AFGHANISTAN VI
SAMPLING
The sample was derived5 using a two-stage cluster-based approach for each of two populations: one
for GE schools and one for CBE schools. First, stratified random sampling was used to select GE and
CBE schools in each of eight regions6 and 34 provinces plus one administrative area (Kabul City).7
Stratification criteria included school location (urban vs. rural) and school gender (mixed, girls, boys).
Second, within each school, students were randomly selected from each target grade. In mixed-
gender schools, students were stratified by gender, then selected randomly within each group.
Students were assessed in either Dari or Pashto based on the language of instruction (LOI) in their
school and classroom.

Note that EGRA data were collected in both GE and CBE schools. However, because of the high
rate of discrepancies found between official CBE school statistics and conditions on the ground, CBE
results could not be weighted or generalized and thus were excluded from this analysis. This report
presents results based on weighted EGRA data from Afghanistan’s General Education (GE) primary
schools, and are thus generalizable at the regional level to all GE schools in Afghanistan.8 The
unweighted EGRA results of both GE and CBE schools can be found in a separate report.

A total of 658 GE schools were included in this sample, and 11,771 students in Grades 2 and 4 in GE
schools were in all eight regions, in urban and rural environments, in warm and cold climates, and in
mixed-gender and in single-gender schools. 9 Note that in warmer regions, where the EGRA was
conducted near the end of the school year, students in Grades 2 and 4 were assessed. In colder
regions, where the EGRA was conducted near the beginning of the school year, students in Grades 3
and 5 were assessed as a proxy for the end of Grades 2 and 4. Throughout this report, results are
described for Grades 2 and 4 but include both groups.

Note also that assessments conducted in Dari and Pashto were analyzed separately, since results
from these two languages cannot be combined due to their differences in structure and difficulty.10
The following are the final sample sizes by schools’ language of instruction:

Dari Sample. 400 GE Dari-medium schools were reached across eight regions11 where 7,192
students were assessed: 3,638 in Grade 2 and 3,554 in Grade 4. The final Dari SMES sample included
383 principals, 692 teachers, and 373 parents or Shura members. 402 school inventories12 were
completed.

5 Sampling was conducted by Chemonics.
6 Additional information on the composition of provinces within each region can be found in the Technical Report.
7 Due to population density, Kabul City is treated as separate entity from Kabul province.
8 EGRA data were collected in CBE schools as well. However, it was not possible to weight or generalize these results due
to the high rate of discrepancies found between official statistics and conditions of schools on the ground. The unweighted
EGRA results of both GE and CBE schools can be found in a separate report.
9 Though the EGRA was designed to assess students’ reading achievement at the end of Grades 2 and 4, the EGRA was
administered to students in “cold schools” in Grades 3 and 5. This is because the timeframe for data collection fell at the
end of the school year for warm schools and at the beginning of the school year for cold schools. Hence, data were
collected in warm school in Grades 2 and 4 and in cold schools in Grades 3 and 5 as a proxy for the end of Grades 2 and
4. Throughout this report, results are reported for Grades 2 and 3 combined (to represent learning levels at the end of
Grade 2) and for Grades 4 and 5 combined (to represent learning levels at the end of Grade 4).
10 This is standard in EGRA analysis and reporting; in general, EGRA scores should not be compared across languages for
this reason.
11 For the Dari sample, the East region is not included in this analysis because the sample size was too small (14 students
total).
12 As compared with the EGRA Dari sample of 400 schools, these two additional schools reflect cases where SMES tools
were administered in Dari at a Pashto-medium school. While most SMES tools were administered in the language that
aligned with the associated schools’ language of instruction, in a small number of cases the tools were administered in the
respondent’s preferred language which did not align with the language of instruction.
RSC-EGR TECHNICAL REPORT: AFGHANISTAN EGRA & SMES RESULTS VII
Pashto Sample. 258 Pashto-medium schools were reached across seven regions,13 where 4,579
students were assessed: 2,400 in Grade 2 and 2,179 in Grade 4. The final Pashto SMES sample
includes 235 principals, 430 teachers, and 225 parents or Shura members. 246 school inventories
were completed.

3. TOOL DEVELOPMENT
EGRA
EGR and the MOE Working Group worked in collaboration to develop the EGRA and SMES tools.
The EGRA assessed six core reading skills or types of knowledge: phonological awareness, alphabetic
principle, phonics, passage reading, comprehension, and oral language. For each of these areas,
students were assessed with a corresponding EGRA task as presented in the table below. Mean
scores were calculated for each task. Moreover, some tasks were timed; for these tasks, fluency
rates were also calculated (see Table 1):

Table 1. EGRA TEST STRUCTURE14
Core Reading Timed/
Knowledge or Skill EGR EGRA Task Untimed Skill Demonstrated by Students’ Ability to:
Phonological Phonemic Awareness Untimed Respond correctly when the assessor reads
Awareness aloud a set of three words and asks the
student to identify the word that starts with
a different sound
Alphabetic Letter Sound Identification Timed Identify the sound of letters presented in a
Principle (1 minute) random order
Phonics Invented/Nonword Reading Timed Make letter-sound correspondences
(Decoding) (1 minute) (grapheme-phoneme correspondences)
through the reading of simple nonsense
words
Passage Reading Oral Reading Fluency (ORF) Timed Read a text with accuracy and at a sufficient
(1 minute) rate of speed
Comprehension Reading Comprehension Untimed Respond correctly to different types of
questions, including literal and inferential
questions about the text they have read
Oral Language Listening Comprehension Untimed Respond correctly to different types of
questions, including literal and inferential
questions about the text the assessors read
to

QUALITY OF ASSESSMENT TOOLS
The quality of the EGRA tools was measured by examining their internal consistency, item difficulty,
and item-total correlations. Based on the results of the pilot, tasks were revised to ensure high
internal consistency, appropriate levels of difficulty for Grade 2 and 4, and adequate item
discrimination. Analyses of test quality showed that the EGRA instruments were of high quality.

SMES

13 For the Pashto sample, no data were collected in Pashto-medium schools in the West Central, Northwest, and West
regions due to the small sample sizes.
14 Adapted from the EGRA Toolkit 2.0 (RTI International. 2015. Early Grade Reading Assessment (EGRA) Toolkit, Second
Edition. Washington, D.C. United States Agency for International Development).
RSC-EGR TECHNICAL REPORT: AFGHANISTAN EGRA & SMES RESULTS VIII
The SMES consisted of five tools: school inventory, student questionnaire, principal questionnaire,
teacher questionnaire, and parent/Shura questionnaire.

4. DATA COLLECTION
The EGRA and SMES were administered by the data collection firm RSI, and MOE Provisional
Education Directorate (PED) assessors between April 11 and August 21, 2016 across 34 provinces in
Afghanistan. Data were collected on tablets utilizing a customized Tangerine platform, an open-
source electronic data collection system specifically designed for administering the EGRA. In schools
that were determined to be in highly insecure areas, paper tools were utilized to administer the
EGRA and SMES questionnaires to reduce the profile of the assessment teams. The EGR project,
through RSI, implemented several strategies to track the progress of data collection and assure
quality. STS analyzed the data and drafted this report. The results are reported by language, grade
level, student gender, urbanicity, and region.

5. FINDINGS
Three kinds of results are presented in this section, each by language:
• EGRA mean scores (average percent correct) by task and grade level
• Oral reading fluency (ORF) by Grades 2 and 4
• Subgroup results by gender, urbanicity, and region

EGRA, GE Dari-medium Schools
OVERALL RESULTS
The EGRA used the same tasks to assess students in Grades 2 and 4. Table 2 shows that overall,
Grade 4 students performed better than Grade 2 students in Dari-medium GE schools. The greatest
difference could be found in ORF scores, where Grade 4 levels were more than double those of
Grade 2. Of course, it is expected to see Grade 4 students performing better than Grade 2 on the
same test as it is likely a sign that learning has probably continued since Grade 2.

Table 2. MEAN EGRA RESULTS BY GRADE, GE DARI MEDIUM SCHOOLS
EGRA task EGRA, Dari
Grade 2 Grade 4
Mean Mean
Phonemic Awareness 4.5 items correct 5.8 items correct
(number correct out of ten)
Letter Sound Identification 28.4 CLPM 38.9 CLPM
(correct letter sounds per minute)
Nonword Reading 14.9 CNWPM 25.2 CNWPM
(correct nonwords per minute)
Oral Reading Fluency 21.7 CWPM 44.3 CWPM
(correct words per minute)
Reading Comprehension 25.1% items 47.2% items
(percentage correct out of five) correct correct
Listening Comprehension 64.3% items 75.4% items
(percentage correct out of three) correct correct
For Grade 2 GE Dari, n = 3,638, for Grade 4 Dari, n = 3,554
These results show that students’ strongest performance in GE Dari-medium schools was in the
area skills that are foundational to learning to read with comprehension (“the mechanics of
RSC-EGR TECHNICAL REPORT: AFGHANISTAN EGRA & SMES RESULTS IX