ERIC ED449354: Technical Report and Data File User's Manual for the 1992 National Adult Literacy Survey.

626 Pages·6.8 MB·English

by  ERIC

DOCUMENT RESUME
CE 081 270
ED 449 354
AUTHOR
Kirsch, Irwin; Yamamoto, Kentaro; Norris, Norma; Rock,
Donald; Jungeblut, Ann; O'Reilly, Patricia; Berlin, Martha;
Mohadjer, Leyla; Waksberg, Joseph; Goksel, Huseyin; Burke,
John; Rieger, Susan; Green, James; Klein, Merle; Campbell,
Anne; Jenkins, Lynn; Kolstad, Andrew; Mosenthal, Peter;
Baldi, Stephane
Technical Report and Data File User's Manual for the 1992
TITLE
National Adult Literacy Survey.
National Center for Education Statistics (ED), Washington,
INSTITUTION
DC.
NCES-2001-457
REPORT NO
2001-01-00
PUB DATE
NOTE 648p.; Project Officer: Andrew Kolstad.
ED Pubs, P.O. Box 1398, Jessup, MD 20794-1398. Tel:
AVAILABLE FROM
877-433-7827 (Toll Free), Fax: [phone], TTY/TDD: 800
437-0833, E-mail: [email protected], Web site:
http://www.ed.gov/pubs/edpubs.html.
Research (143)
Reports
Non-Classroom (055)
PUB TYPE
Guides
Tests /Questionnaires (160)
MF03/PC26 Plus Postage.
EDRS PRICE
Adult Basic Education; *Adult Literacy; *Data Processing;
DESCRIPTORS
Educational Research; Illiteracy; Literacy Education;
*Material Development; *National Surveys; Prisoners;
Research Administration; Research Design; Research
Methodology; Research Problems; Sampling; Scaling; State
Surveys; Statistical Analysis; Validity; Weighted Scores
*National Adult Literacy Survey (NCES)
IDENTIFIERS
ABSTRACT
Chapter 1 of this report and user's manual describes design
and implementation of the 1992 National Adult Literacy Survey (NALS). Chapter
2 reviews stages of sampling for national and state survey components;
presents weighted and unweighted response rates for the household component;
and describes non-incentive and prison sample designs. Chapter 3 addresses
weighting procedures. Chapter 4 describes NALS's conceptual framework and
development of the background questionnaire and literacy tasks. Chapters 5
and 6 document field operations for the household and prison surveys.
Chapters 7-9 detail data processing; the missing data procedures; and models
and procedures used to scale NALS results, estimate respondents'
proficiencies, and conduct statistical analyses. Chapter 10 presents a
summary of literature on use of monetary incentives in survey research,
experimental features of the field test and national study, and research
results. Chapters 11-15 describe statistics, components of variance, and
statistical methods used to derive outcome estimates; estimate and test
discriminant validity of the three literacy scales from the perspective of
correlation or covariance; summarize establishment of NALS literacy levels;
explore importance of the response probability convention in reporting prose
literacy results; and discuss use of SPSS (Statistical Program for the Social
Sciences) and SAS (Statistical Analysis System) for analyzing NALS data.
Appendixes include 98 references, additional data and notes, and instruments.
(YLB)
Reproductions supplied by EDRS are the best that can be made
from the original document.
ATIONAL CENTER FOR EDUCATION STATISTICS
c7,
Technical Report
January 2001
Q
Technical Report and
Data File User's Manual
for the 1992 National
Adult Literacy Survey
U.S. DEPARTMENT OF EDUCATION
Office of Educational Research and Improvement
ED CATIONAL RESOURCES INFORMATION
CENTER (ERIC)
This document has been reproduced as
received from the person or organization
originating it.
0 Minor changes have been made to
improve reproduction quality.
Points of view or opinions stated in this
document do not necessarily represent
official OERI position or policy.
BEST COPY AVAILABLE
U.S. Department of Education
Office of Educational Research and Improvement
NCES 2001-457
2.
Customer Survey of OERI Publication Users
To help us improve future editions of this publication and give
5. For what purposes did you use this OERI
you better customer service, we would appreciate your
(Check all that apply.)
publication?
comments on this survey form. Please check the appropriate
Planning
box(es) below for each question. Responses will be kept
Policy or legislation
completely confidential. You may return the survey by mail or
FAX. It can be folded and taped closed to allow mailing to the
Administrative decisions
address listed on the reverse side of this form, or it can be
Teaching, class material
returned by FAX to 202-219-1321. Many thanks for
your customer feedbackit is very important to usl
1:1 Research/analysis
General information
la. Name of publication
Writing news articles, TV or radio material
Marketing, sales, or promotion
lb.
Publication number NCES 2001-457
Other (please describe)
lc. Author name
How did you receive a copy of this publication?
2.
6. Did the publication help you accomplish whatever you
Bought it
needed it for?
Borrowed it
Partially
No
C]l Yes
1:1 Mailing list membership
7. What is your occupation?
Telephone request
Administrator
Teacher
Parent
Internet request
Statistician
Researcher
Librarian
1:1 Other (please describe)
Student
Policy Analyst
Journalist/writer
Program Planner
Other (please specify)
Was this publication easy to get?
3.
Not at all
Somewhat
Very
How did you find out about this and other OERI
4.
8. How could this OERI publication (or other OERI
publications) better meet your needs?
(Check all that apply.)
publications?
(Check all that apply.)
Conferences
More important topics in education
Journal articles
More timely release of data
Teacher/educator
More text introductions to each section
Professional associations
More research statistics
Internet (WWW)
Shorter reports (less than 10 pages)
Publication announcement
Other (please describe)
Received in mail
OERI staff contact
Dissatisfied
Satisfied
Very Satisfied
Overall, how satisfied are you
9.
with this product?
Comprehensiveness of information
a.
Clarity of writing (readability, interpretability)
b.
Clarity of presentation (e.g., tables, charts)
c.
Timeliness of information
d.
Accuracy of information
e.
Clarity of technical notes
f.
E:11
Usefulness of resources and bibliography
g.
(=I
Organization
h.
Length
i.
CIE
j. Format
[:11
PAPERWORK BURDEN STATEMENT
Office of Education Research and improvement (OERI)
Publication Customer Survey
control number. The valid
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB
OMB control number for this information collection is 1880-0529.The time required to complete this information collection is estimated to average 10 minutes per response,
including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any
comments
concerning the accuracy of the time estimate(s), suggestions for improving this form, or concerns regarding the status of your individual submission of this form, write
directly
to: P. Quinn, Room 204, Media and Information Services, OERI, U.S. Department of Education, 555 New Jersey Avenue NW, Washlnaton. nn onon.A570.
BEAT COPY AVAILABLE
OERI Publication Customer Survey
NO POSTAGE
Media and Information Services
NECESSARY
U.S. Department of Education
IF MAILED
Washington, DC 20202
IN THE
UNITED STATES
Official Business
Penalty for Private Use, $300
BUSINESS REPLY MAIL
FIRST-CLASS MAIL
WASHINGTON DC
PERMIT NO. 012935
POSTAGE WILL BE PAID BY U.S. DEPARTMENT OF EDUCATION
U.S. Department of Education
Mail Code: 5570
400 Maryland Avenue, S.W.
Washington, D.C. 20277-2935
11111111111111111111111111111111111111111111111111111111
Fold on lineTAPE CLOSEDDO NOT STAPLE
10. Do you have any suggestions regarding the content or format of future editions of this publication or other comments?
.
4
MIS 1999-6532
NATIONAL CENTER FOR EDUCATION STATISTICS
Technical Report
January
2001
Technical Report and
Data File User's Manual
for the 1992 National
Adult Literacy Survey
Irwin Kirsch, Kentaro Yamamoto, Norma Norris, Donald Rock, Ann Jungeblut, and
Patricia O'ReillyEducational Testing Service
Martha Berlin, Ley la Mohadjer, Joseph Waksberg, Huseyin Goksel, John Burke,
Susan Rieger, James Green, and Merle Klein--- Westat, Inc.
Anne CampbellDine College
Lynn JenkinsWordsworth Writing and Editing
Andrew KolstadNational Center for Education Statistics
Peter MosenthalSyracuse University
Stephane BaldiAmerican Institutes for Research
Andrew Kolstad
Project Officer
National Center for Education Statistics
U.S. Department of Education
NCES 2001-457
Office of Educational Research and Improvement
5
U.S. Department of Education
Richard W. Riley
Secretary
Office of Educational Research and Improvement
C. Kent McGuire
Assistant Secretary
National Center for Education Statistics
Gary W. Phillips
Acting Commissioner
The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, and reporting
data related to education in the United States and other nations. It fulfills a congressional mandate to collect, collate,
analyze, and report full and complete statistics on the condition of education in the United States; conduct and publish
reports and specialized analyses of the meaning and significance of such statistics; assist state and local education
agencies in improving their statistical systems; and review and report on education activities in foreign countries.
NCES activities are designed to address high priority education data needs; provide consistent, reliable, complete, and
accurate indicators of education status and trends; and report timely, useful, and high quality data to the U.S. Department
of Education, the Congress, the states, other education policymakers, practitioners, data users, and the general public.
We strive to make our products available in a variety of formats and in language that is appropriate to a variety of audiences.
You, as our customer, are the best judge of our success in communicating information effectively. If you have any comments
or suggestions about this or any other NOES product or report, we would like to hear from you. Please direct your
comments to:
National Center for Education Statistics
Office of Educational Research and Improvement
U.S. Department of Education
1900 K Street, NW
Washington, DC 20006-5651
January 2001
The NOES World Wide Web Home Page is: http://nces.ed.gov
The NOES World Wide Web Electronic Catalog is: http://nces.ed.gov/pubsearchfindex.asp
Suggested Citation
U.S. Department of Education. National Center for Education Statistics. Technical Report and Data File User's Manual for
the 1992 National Adult Literacy Survey, NCES 2001-457, by Irwin Kirsch, Kentaro Yamamoto, Norma Norris, Donald
Rock, Ann Jungeblut, Patricia O'Reilly, Martha Berlin, Leyla Mohadjer, Joseph Waksberg, Huseyin Goksel, John Burke,
Susan Rieger, James Green, Merle Klein, Anne Campbell, Lynn Jenkins, Andrew Kolstad, Peter Mosenthal, and
Stephane Baldi. Andrew Kolstad, Project Officer. Washington, DC: 2000.
For ordering information on this report, write:
U.S. Department of Education
ED Pubs
P.O. Box 1398
Jessup, MD 20794-1398
or call toll free 1-877-4EDPubs.
Content Contact:
Andrew Kolstad
[phone]
TABLE OF CONTENTS
1
CHAPTER 1: THE NATIONAL ADULT LITERACY SURVEY: AN OVERVIEW
1.1 Introduction
1
3
1.2 Defining Literacy
3
1.3 The Sample
4
1.4 Weighting
5
1.5 The Survey Instrument: Measuring Literacy
7
1.6 Field Operations
8
1.7 Data Processing and Missing Data
9
1.8 Scaling and Proficiency Estimates
10
1.9 Establishing Literacy Levels
11
CHAPTER 2: SAMPLE DESIGN
11
2.1 Overview
13
2.2 Sampling for the National Component
14
2.2.1 First-Stage Sample
14
2.2.1.1 Westat's master sample of PSUs
14
2.2.1.2 Selecting the sample of PSUs for the national component
21
2.2.2 Second-Stage SampleSelecting Census Blocks (Segments)
22
2.2.2.1 Measures of size and sampling rates
23
2.2.2.2 Minimum segment size
24
2.2.2.3 Segment sample selection
24
2.2.2.4 TIGER maps
25
2.2.2.5 Listing sample segments
26
2.2.3 Third-Stage SampleSelecting Housing Units
26
2.2.3.1 Within-segment sampling rate
28
2.2.3.2 Overall probabilities of selection
28
2.2.3.3 Procedures for selecting missed structures and missed dwelling units
29
2.2.4 Fourth-Stage SampleSelecting Persons Age 16 or Older
30
2.3 The Non-Incentive Sample
..30
2.4 Sampling for the State Literacy Surveys
31
2.4.1 Sample of PSUs
31
2.4.2 Sample of Segments
32
2.4.3 Sample of Housing Units
32
2.4.4 Sample of Persons
32
2.5 Weighted and Unweighted Response Rates
34
2.6 Sampling for the Prison Survey
34
2.6.1 Sample of Correctional Facilities
34
2.6.1.1 Sampling frame and selection of correctional facilities
37.
2.6.2 Selection of Inmates Within Facilities
39
CHAPTER 3: WEIGHTING AND POPULATION ESTIMATES
39
3.1 Goals of Weighting
41
3.2 Calculating Sample Weights for the Household Population
41
3.2.1 Preliminary Steps in Weighting
43
3.2.2 Computing Base Weights
44
3.2.3 Nonresponse Adjustments and Poststratification
45
3.2.4 Compositing Data from the National and State Components
45
3.2.4.1 Composite estimation procedure
TABLE OF CONTENTS
CONTINUED
3.2.4.2 Deriving the PSU design effect
48
3.2.4.3 Estimating composite factors
49
3.2.5 Computing Final WeightPoststratification Through Raking
Ration Adjustments
54
3.3 Replicated Weights for Variance Estimation in the Household Population
55
3.3.1 Household Sample Replication for the National Component
57
3.3.2 Household Sample Replication for the State Component
57
3.3.3 Final Household Sample Replication for the National and State
Components
58
3.4 Calculating Sample Weights for the Prison Population
58
3.4.1 Computing Inmate Base Weights
58
3.4.2 Nonresponse Adjustments
62
3.4.2.1 Facility nonresponse adjustment
62
3.4.2.2 Inmate nonresponse adjustment
63
3.4.3 Poststratification Procedures
64
3.4.4 Final Inmate Weights
67
3.5 Replication Weights for Variance Estimation in the Prison Population
68
CHAPTER 4: DEVELOPMENT OF THE SURVEY INSTRUMENTS
70
4.1 Conceptual Framework
70
4.2 The Scope of the Background Questionnaire
72
4.2.1 General and Language Background
73
4.2.2 Educational Background and Experiences
73
4.2.3 Political and Social Participation
74
4.2.4 Labor Force Participation
74
4.2.5 Literacy Activities and Collaboration
75
4.2.6 Demographic Information
75
4.2.7 Prison Survey Background Questionnaire
76
4.2.8 Spanish Versions of the Questionnaires
76
4.3 Development of the Simulation Tasks
76
4.3.1 Organizing Framework for Task Development
77
4.3.2 Materials/Structures
77
4.3.3 Adult Contexts/Content
78
4.3.4 Processes/Strategies
79
4.3.5 Task Difficulty
87
4.3.6 Development of Scoring Guides
88
4.3.7 Assembling the Tasks for Administration
89
CHAPTER 5: THE HOUSEHOLD SURVEY
93
5.1 Overview
93
5.2 Listing
94
5.2.1 Staff Organization for Listing
94
5.2.2 Training Listers
95
5.2.3 Listing Materials
95
5.2.4 The Listing Operation
96
5.2.5 Quality Control Procedures
97
5.2.5.1 Quality control of listing sheets
97
5.2.5.2 Quality control of the listing operation
97
5.3 Data Collection Instruments and Interviewer Materials
99
5.3.1 The Soreener
99
5.3.2 Interview Guides for Exercise Booklets
101
5.3.3 Non-interview Report Forms
101
e ii
CONTINUED
TABLE OF CONTENTS
104
5.3.4 Interviewer Manuals
104
5.3.5 Field Aids
104
5.3.5.1 Aids used for locating and contacting respondents
105
5.3.5.2 Aids used for obtaining respondent cooperation
106
5.3.5.3 Aids used during the interview
106
5.4 Field Organization and Training
106
5.4.1 Field Organization
106
5.4.1.1 Lines of responsibility
107
5.4.1.2 Interviewer recruitment
109
5.4.2 Training
109
5.4.2.1 Supervisor training
110
5.4.2.2 Interviewer training
114
5.4.2.3 Editor training
114
5.5 Field Operations
115
5.5.1 General Approach to the Field Effort
116
5.5.2 Schedule and Production
117
5.5.3 Reporting Systems
117
5.5.3.1 Automated Survey Control System (ASCS)
118
5.5.3.2 Interviewer reports to the supervisor
118
5.5.3.3 Supervisor reports to the home office
119
5.5.3.4 Home office staff reports to ETS and to NCES
119
5.6 Quality Control of Data Collection
119
5.6.1 Introduction
119
5.6.2 Editing
121
5.6.3 Validation
122
5.6.4 Observation
123
5.6.5 Supervisor Observations
123
5.7 Response Rates
125
5.7.1 Reasons for Non-response
126
5.7.2 Characteristics of Non-respondents
128
5.7.3 Discussion
131
CHAPTER 6: THE PRISON SURVEY
131
6.1 Sample Design
132
6.2 Gaining Cooperation
133
6.3 Interviewer Selection and Training
134
6.4 Data Collection
134
6.5 Quality Control
136
CHAPTER 7: PROCESSING THE DATA
136
7.1 Receipt Control
136
7.1.1 Screener
137
7.1.2 Background Questionnaire
137
7.1.3 Exercise Envelope
138
7.2 Coding and Scoring
138
7.2.1 Coding Background Questionnaires
139
7.2.2 Scoring Simulation Tasks
140
7.3 Data Entry
141
7.4 Editing and Quality Control
iii
9
TABLE OF CONTENTS
CONTINUED
CHAPTER 8: ESTIMATING LITERACY PROFICIENCIES WITH
AND WITHOUT COGNITIVE DATA
142
8.1 The Normal Treatment of Missing Cognitive Data
143
8.1.1 Omitted Answers and Questions Not Reached
143
8.1.2 Statistical Imputation Through Scaling
144
8.2 Reasons Cognitive Data Were Missing/Not Reached
146
8.2.1 Non-interview Reports and Low Literacy Skills
151
8.2.2 Internal Evidence for the Validity of Reasons
153
8.3 Using 'Reasons' to Improve Treatment of Missing Cognitive
Data
155
8.3.1 Five Logical Imputation Methods Considered
156
8.3.2 Five Methods Applied to 1991 Field Test Data
158
8.3.3 The Method Selected
163
8.4 Final Evaluation
164
CHAPTER 9: SCALING AND PROFICIENCY ESTIMATES
165
9.1 Scaling
165
9.2 Scaling Methodology
168
9.2.1 The Scaling Model
168
9.2.2 Design for Linking the 1992 Scales to the 1985 Scales
169
9.2.3 Item Parameter Estimation
170
9.3 Proficiency Estimation Using Plausible Values
177
9.3.1 Generating Proficiency Scores
177
9.3.2 Linking the 1992 Scale to the 1985 Scale
182
9.3.3 Evaluation of Differential Group Performance
183
9.4 Statistical Tests
187
9.4.1 Analysis of Plausible Values
187
9.4.2 Partitioning the Estimation Error Variance: A Numerical
Example
188
9.4.3 Minimum Sample Sizes for Reporting Subgroup Results
190
9.4.4 Estimates of Standard Errors with Large Mean Squared Errors
190
CHAPTER 10: THE ROLE OF INCENTIVES IN LITERACY SURVEY
RESEARCH
191
10.1 Literature Review
191
10.2 The 1991 Field Test
193
10.2.1 Field Test Design
193
10.2.2 Summary of Field Test Results
194
10.2.3 Field Test Response Rates
196
10.2.4 Representation of the Target Population in the Field Test
198
10.2.5 Relationship Between Incentive Level, Self-Selection, and
Performance
in the Field Test
200
10.2.6 Survey Costs for the Field Test
208
10.2.7 Conclusions from the 1991 Field Test
209
10.3 The 1992 Incentive Experiment
210
10.3.1 Sample Design for the 1992 National Adult Literacy Survey
210
10.3.2 1992 Incentive Experiment Design
211
10.3.3 Analysis of Response Rates
212
10.3.3.1 Screener
213
10.3.3.2 Background questionnaire and exercise booklet
214
10.4 Summary and Conclusion
217
10
iv
fly