Table Of ContentDOCUMENT RESUME
CE 081 904
ED 454 399
Mundie, Karen; Thompson, Debbie; Joyce, Michelle
AUTHOR
Piloting the AIM Project: Measuring Progress for Program
TITLE
Evaluation and Accountability.
Greater Pittsburgh Literacy Council, PA.
INSTITUTION
Pennsylvania State Dept. of Education, Harrisburg. Bureau of
SPONS AGENCY
Adult Basic and Literacy Education.; Department of
Education, Washington, DC.
PUB DATE 2000-00-00
NOTE
32p.
CONTRACT
98-00-0004
AdvancE State Literacy Resource Center, Pennsylvania
AVAILABLE FROM
Department of Education, 333 Market St., 11th Floor,
Harrisburg, PA 17126-0333. Tel: 800-992-2283 (Toll Free);
TTY: 717-783-8445; Fax: 717-783-5420; e-mail:
[email protected].
Reports
Evaluative (142)
PUB TYPE
MF01/PCO2 Plus Postage.
EDRS PRICE
Adoption (Ideas); Adult Basic Education; Adult Educators;
DESCRIPTORS
Adult Literacy; Check Lists; Comparative Analysis; English
(Second Language); *Evaluation Methods; Feasibility Studies;
Guidelines; *Literacy Education; Mentors; *Performance Based
Assessment; Pilot Projects; Portfolio Assessment; Portfolios
(Background Materials); Pretests Posttests; *Statewide
Planning; *Student Evaluation; Teacher Attitudes; Tutoring;
Tutors; Volunteers
*Pennsylvania; 223 Project
IDENTIFIERS
ABSTRACT
The AIM (Assessment, Instruction, Mastery) system is a
performance-based assessment that was developed in Oregon. The AIM system was
piloted by 83 volunteers and staff from 18 volunteer-based programs to assess
the system's usefulness as a means of collecting and aggregating data on
student progress in adult literacy and similar programs in Pennsylvania. Six
programs that had received AIM training in prior years also worked with the
assessment on a more intensive level. They reported that, of the 192 students
to whom AIM was assigned, 44 completed some skills on the AIM checklist and
43 showed posttest gains, and 10 students completed the checklist they were
assigned and moved on to the next level. Based on the sample portfolios
received, it was concluded that tutors gave more attention to correct
documentation and collection of evidence during the current project year than
in past years. Close mentoring of tutors using the AIM checklists was found
to ensure better student progress on the AIM checklists. The AIM training was
revised again after the second year. A role-playing activity for tutors was
added. The project was slated for continuation. AIM tutor tips and an AIM
student progress report form are appended.
(MN)
Reproductions supplied by EDRS are the best that can be made
from the original document.
Piloting the AIM Project:
Accountability
Program Evaluation and
Measuring Progress for
Karen Mundie: Program Director
Debbie Thompson: Program Manager
Michelle Joyce: Reading Specialist
1999-2000
Greater Pittsburgh Literacy Council
100 Sheridan Square
PERMISSION TO REPRODUCE AND
DISSEMINATE THIS MATERIAL HAS
Pittsburgh, PA 15206
BEEN GRANTED BY
412-661-7323
OF EDUCATION
U.S. DEPARTMENT
Research and Improvement
of Educational
Of
INFORMATION
RESOURCES
EDU ATIONAL
TO THE EDUCATIONAL RESOURCES
CENTER (ERIC)
been reproduced as
INFORMATION CENTER (ERIC)
This document has
organization
person or
received from the
originating it.
been made to
Minor changes have
quality.
improve reproduction
Amount of Grant: $30, 000
this
opinions stated in
Points of view or
necessarily represent
document do not
or policy.
official OERI position
Project Number: 98-00-0004
in part by the U.S.
The activity which is the subject of this report was supported
Department of Education. However, the opinions expressed herein do not necessarily
on or the
Department of Educati
reflect the position or policy of the U.S.
Pennsylvania Department of Education, and no official endorsement by these
agencies should be inferred.
2
BEST COPY AVAILABLE
Project No.: 98-00-0004
ABSTRACT PAGE
Greater Pittsburgh Literacy Council
Grant Recipient:
100 Sheridan Sq., 4th Floor
Pittsburgh, PA 15206
412-661-7323
Measuring Progress for Program Evaluation and
Program Name: Piloting the AIM Project:
Accountability
Grant Allocation: $30,000
Project Period: July 1, 1999-June 30, 2000
Mundie, Michelle Joyce
Project Director: Debbie Thompson, Karen
pilot the AIM (Assessment, Instruction, Mastery), a
Project Purpose: The project proposed to
and
Oregon, in volunteer-based programs (5 new programs
performance-based assessment developed in
recommendations to the PDE
from previous year) in Pennsylvania and to make
any of the 26 programs
data on student progress for
means of collecting and aggregating
on the usefulness of this system as a
these and similar programs.
and staff from eighteen literacy programs attended
Project Outcomes: Eighty-three volunteers
Six programs that had received AIM training in prior years
training on the use of the AIM system.
level. They reported that AIM was assigned to 192
worked with this assessment on a more intensive
skills on the AIM checklist and forty-three
students. Forty-four of these students completed some
the checklist they were assigned and five moved on to
showed post-test gains. Ten students completed
the next level.
information collected this year. Based upon the
Impact: Two important findings came out of the
correct documentation and collection of
sample portfolios received, tutors gave more attention to
pilot project conducted by the Literacy Council of
evidence than in past years. Based upon the
AIM checklists ensured better student progress
Lancaster-Lebanon, close mentoring of tutors using the
on the AIM checklists.
again revised after the second year to reflect
Product or Training Developed: The AIM training was
offered more intensive work on lesson planning
what was learned in years one and two. This training
role-playing activity was added to increase the tutors
and documentation of mastery of skills. A
the learning process.
understanding of the idea of the adult learner as a facilitator in
The project will continue in a fourth year. All
Project Continuation and/or Future Implications:
be offered the opportunity for on-site training and additional
programs previously involved will
available to new programs in their regions.
materials. The PDC's and TLC will also have this training
the AIM guides and disseminating them to
GPLC will continue to work closely with Oregon in revising
programs presently using the AIM system.
data can be
Conclusions/Recommendations: GPLC is still unable to establish that student progress
Until this can occur, we are unable to
collected and aggregated in sufficient numbers across programs.
auxiliary assessment to Pennsylvania
recommend that the PDE offer the AIM as an alternative or
volunteer-based literacy programs.
3
final report are intended for use by administrators,
Additional Comments: The AIM and this
volunteer-based literacy programs.
program staff, and tutors in
Table of Contents
Section 1: Project Team
Section 2: Project History
Section 3: Project Plan
Section 4: Project Activities
Section 5: Project Results
Section 6: Project Recommendations
Section 7: Appendix
5
Section 1
Project Team
Evaluation
Measuring Progress for Program
Piloting the AIM Project:
and Accountability
Manager
Debbie Thompson, Program
Council
Greater Pittsburgh Literacy
Council,
Manager for Greater Pittsburgh Literacy
Debbie Thompson, MA, Program
offices in Allegheny County. In addition to her
supervises ten area literacy
the agency[]s Program Improvement Team and is a part
supervisory role, Debbie leads
She is also a member of the state-wide
of the regional implementation team.
of evaluating Pennsylvania's professional
evaluation team currently in its second year
for Tutors of Literacy in the Commonwealth,
development system, is the Chairman
data sharing strand and lead trainer for the
LLA/LVA
and is a trainer for the EQUAL
certified Laubach supervising trainer and has
Accreditation strand. Debbie is a
worked in education for eighteen years.
Karen Mundie, Program Director
Council
Greater Pittsburgh Literacy
for Greater Pittsburgh Literacy Council, has
Karen Mundie, MA, Program Director
twenty-five years and program director for eight.
been a literacy professional for
exemplary
of fifteen and has directed a number of
Karen supervises a support staff
brought tutor training expertise to other literacy
353 projects including one which
She is the First Vice-President of the Pennsylvania
programs in Pennsylvania.
Education and is the lead trainer for the
Association for Adult and Continuing
also a part of the regional implementation
EQUAL data sharing strand. Karen is
literacy
national project designed to document success in
team and has worked on a
Works Literacy Partners.
programs, the What
Michelle Joyce, Reading Specialist
Greater Pittsburgh Literacy Council
Greater Pittsburgh Literacy Council, has
Michelle Joyce, MS, Reading Specialist for
Michelle has directed previous 353
led the agency's reading program for nine years.
place of literacy council's in workplace
projects including a research project on the
of 1) assessment and 2) adults with
education. She is a statewide trainer in the areas
working group on measuring gains
learning differences and is part of a national
What Works Literacy Partners. Michelle
through authentic assessment sponsored by
and the regional implementation
is a member of both the EQUAL training team
team.
7
Section 2
Project History
Progress for Program Evaluation
Piloting the AIM Project: Measuring
and Accountability
8
Literacy Education awarded Greater Pittsburgh
The Bureau of Adult Basic and
pilot a performance based assessment developed in
Literacy (GPLC) a 353 grant to
Mastery), which showed promise as an
Oregon, AIM (Assessment, Instruction,
in volunteer literacy programs. Karen
instrument to measure student progress
identified two staff members who would lead
Mundie, Program Director for GPLC,
Program Manager, and Michelle Joyce, Reading
this project: Debbie Thompson,
Specialist.
students (basic and ESL) in adult
The AIM was designed for use with literacy
student and tutor form a partnership to make
education programs. In this system the
using an accomplishment checklist
decisions on both instruction and assessment
Performance Levels. They target certain
whose items are based on federal Student
mastered, work together to choose what
skills for instruction and as each skill is
Examples of mastery must be concrete
would best exemplify mastery of that skill.
placed in a portfolio. When a
and are attached to a progress summary and
the next level.
designated number of skills are mastered, they move on to
results on standardized assessments
For many beginning and intermediate learners
work they and their tutors have invested in
do not truly reflect the amount of hard
and tutors become frustrated because they
the learning process. Often these students
relationship between their work and the assessment instrument.
are unable to see the
that do not show subtle or
Although necessary, standardized tests are gross measures
should not be the only method for
small incremental changes in skills, and therefore
help literacy programs better
measuring student performance. In order to
funders, and the tutoring pair, AIM was
demonstrate learner progress for themselves,
using a technique for
developed. It is designed to match curriculum with assessment
-the accomplishment checklist. In
collecting information and demonstrating progress-
of the learning process.
this way, assessment can be an on-going, natural part
GPLC staff developed an initial
To implement the AIM project in Pennsylvania,
materials. Among the
training, revised the training twice, and prepared support
lead volunteer tutors
materials developed was a "tutor tip" packet designed to
of student skills. This program
through the process of evaluation and documentation
and asked to assist in organizing regional trainings for
year the PDC's were contacted
Six programs
volunteer programs interested in piloting the AIM in their programs.
offered on-site
where selected to working more intensively with AIM and were
staff was given an orientation
training. Data was collected throughout the year. TLC
asked to include AIM, when appropriate, in
to the AIM materials and training and
GPLC has met twice to share their findings and
tutor training. Staff from Oregon and
The most
develop a plan for further collaboration and standardization of the process.
who are using AIM in their
recent meeting included a visit to sites in Oregon
9
Conference, institutes, and
presentations at PAACE Midwinter
programs. Through
AIM on a
has been able to share information about
other regional activities, GPLC
of a number of literacy providers.
statewide basis and gain the interest
has kept
being revised by the Oregon staff. GPLC staff
The AIM checklists are now
possible.
staff and offered assistance and input whenever
in contact with the Oregon
specific skills to be learned and more concrete
The revision will include more
and students can select to demonstrate
examples of evidence from which tutors
ESL checklists will be replaced by communication skills
mastery. In addition, the
and ESL.
checklist, appropriate for both basic
10
