Table Of ContentSystemwide Assess Prog.qxd 21/8/03 9:11 AM Page a
A POLICY
MAKER'S
GUIDE TO
Systemwide
Assessment
Programs
by
Margaret Forster
Systemwide Assess Prog.qxd 21/8/03 9:11 AM Page b
This publication is the result of research that forms
part of a program supported by a grant to the
Australian Council for Educational Research by
State, Territory and Commonwealth governments.
The support provided by these governments is
gratefully acknowledged.
The views expressed in this publication are those of
the author and not necessarily those of the State,
Territory and Commonwealth governments.
First published 2001 by the Australian Council for Educational Research Ltd.
19 Prospect Hill Road, Camberwell, Melbourne, Victoria 3124, Australia.
Copyright © 2001 Australian Council for Educational Research
All rights reserved. Except as provided for by Australian copyright law, no part
of this book may be reproduced without written permission from the publisher.
ISBN 0-86431-359-4
Printed by RossCo Print
Systemwide Assess Prog.qxd 21/8/03 9:11 AM Page 1
Table of Contents Page
Introduction 2
What are the purposes of systemwide
assessment programs? 3
Why the interest in systemwide
assessment programs? 4
What are some examples of systemwide
assessment programs? 6
How are data collected? 7
How are data reported? 8
How are trends monitored? 20
How are data used to improve learning? 27
What concerns have been raised? 37
A case study 44
A checklist of considerations 46
Useful websites 48
End Notes 49
Page
1
Systemwide Assess Prog.qxd 21/8/03 9:11 AM Page 2
This guide provides policy makers with research-based information about
systemwide assessment programs.
Good decision making at all levels of an education system is facilitated by easily
accessible, relevant, and reliable information.
Many indicators provide useful input to educational decision making; but the
most important indicators are those which address the central concern of
education: the promotion of student learning.
Education systems monitor student learning—with the fundamental intention of
promoting learning—by collecting, analysing and reporting student achievement
data. Given that state, national and international achievement studies are both
time consuming and expensive, it seems prudent to reflect on this effort:
What is the purpose of these programs?
How are data reported and used?
How can we ensure that data will provide evidence for informed decision
making?
INTRODUCTION
The assessment programs considered in • limited background information
this handbook focus on the collection (characteristics of students).
and analysis of systemwide information.
Achievement data usually are collected
Two kinds of information usually are through standardised tests administered
collected: to either representative samples or entire
cohorts of students. Background
• data on student achievement in
information is collected by means of
particular subject areas at particular
questionnaires completed by students or
grade levels; and
teachers.
Page
2
Systemwide Assess Prog.qxd 21/8/03 9:11 AM Page 3
WHAT ARE THE PURPOSES OF SYSTEMWIDE
ASSESSMENT PROGRAMS?
Systemwide assessment programs Full-cohort programs provide:
provide systematic and regular measures
• parents with information about their
of student learning. They are designed
child’s progress to assist them to make
to investigate and monitor the ‘health’of
decisions about the best ways to
an education system and to improve
support their child; and
student learning by providing
information to stakeholders at different • students with information about their
levels of the system. They provide progress to assist them to take an
active role in monitoring their own
• policy makers with information to
learning.
monitor standards over time, to
monitor the impact of particular Programs that are not full cohort can
programs, and to make decisions about provide this information for a limited
resource allocation; and number of students.
• schools (principals, councils) and
teachers with information about whole
school, class and individual pupil
performance that they can use to make
decisions about resource allocation
and to support learning in the
classroom.
The purpose of a systemwide monitoring program
The purpose of British Columbia’s Foundation Skills Assessment (FSA) program is
stated explicitly.1 The program is intended to
• provide information to districts about the performance of their students in
relation to provincial expectations and standards in order to assist districts to plan
for improvement;
• provide information to the public about the performance of students provincially
in relation to expectations and trends over time;
• measure the achievement of students in reading comprehension, first-draft
writing, and selected components of numeracy;
• determine if there are any trends in student performance at the district and
provincial levels; and
• determine if there are groups of students who under perform with respect to
provincial standards. Page
3
Systemwide Assess Prog.qxd 21/8/03 9:11 AM Page 4
WHY THE INTEREST IN SYSTEMWIDE ASSESSMENT
PROGRAMS?
The management of an education system [In the United States] the testing
is a complex and expensive operation. If enterprise in K-12 education has
decisions are to be informed, then mushroomed in the last quarter-century;
dependable information on educational Americans want numbers when they look
outputs is required. Systemwide at students, schools, state education
programs provide this information for systems, and how America’s students
system level monitoring and resource compare to those of other countries.
allocation. Among political leaders, testing is turning
into a means of reform, rather than just a
Of increasing interest, however, is the
way of finding out whether reforms have
role large-scale assessment programs can
been effective.4
play as agents of reform and
accountability—to provide both In some countries, student achievement
direction and motivation to schools, data collected through systemwide
teachers, parents and students. In the assessment programs are used as a
State of South Australia, for example, measure of schools’contributions to
assessment is seen as the missing link in student learning. In some, schools are
earlier curriculum planning and ‘rewarded’or ‘punished’on the basis of
programming which was not informed, students’results (see pages 34-35). In
as a matter of course, by student others, public comparisons of schools’
achievement information.2 In British achievements in the form of ‘league
Columbia, State assessments are seen as tables’are made.
part of the ongoing process of
educational reform as Figure 1 below
illustrates.3
Relevant
District assessment results
information
collected
by district Annual provincial
assessment
Assessment follow through
Interpretation of
district results
Revisit goals
Make recommendations Develop Monitor action
and set goals action plan plan progress
Page
4 Figure 1 Assessment and educational reform – a continuous process
Systemwide Assess Prog.qxd 21/8/03 9:11 AM Page 5
United States government publications Significant commentary also focuses on
emphasise the need for system driven the unintended negative consequences
accountability measures to improve of assessment driven reform (see pages
student learning: 38-44).
No school improvement can succeed The recent role of achievement data in
without real accountability for results. assessment driven reform is illustrated in
Turning around low-performing schools the lower feedback loop in Figure 2.
requires that state and district leaders Once assessments of student learning
take active steps to set high expectations (data collection) have been reported and
for schools and students, establish the evaluated (figure centre), information is
means to measure performance against then disaggregated to provide
those expectations, and create policies to evaluations at school level. These
identify and provide assistance to those evaluations are publicised. Rewards and
schools and students that fail to meet sanctions are applied to schools to
high standards for performance.5 encourage improvements in student
learning.
Commentary on this reform and
accountability agenda focuses on its The upper feedback loop in Figure 2
understandable political appeal: illustrates the traditional approach to
system use of student achievement data.
Compared with reforms such as targeting
Once assessments of student learning
instructional time, professional
(data collection) have been reported and
development for teachers, and reducing
evaluated, decisions are then made at
class sizes, state assessment programs are
system level about the best ways to
relatively inexpensive. The assessments
improve student learning through the
also can be mandated (unlike changes in
allocation of resources to disadvantaged
classroom practice), can be rapidly
schools, to programs, and to teacher
implemented, and have a public
professional development.
visibility.6
System level improvement (and accountability)
Resource
Resource Resource
allocation to
allocation to allocation to
professional
programs schools
development
Evaluate
Collect data Report data
findings
Apply rewards Publicise Disaggregate
and sanctions school level findings at
to schools findings school level
School level accountability (and improvement)
Page
Figure 2 Using achievement data for improvement and accountability 5
Systemwide Assess Prog.qxd 21/8/03 9:11 AM Page 6
WHAT ARE SOME EXAMPLES OF SYSTEMWIDE
ASSESSMENT PROGRAMS?
The table below lists systemwide
assessment programs referred to in this
handbook. Education systems collect a
range of achievement data at key year
levels of schooling. Most monitoring
systems collect data on literacy and
numeracy achievement, some, on a range
of other learning outcomes.
Some examples of systemwide assessment programs
Program Year level/s Learning outcomes Website information
assessed assessed
Australia
Australian 3, 5, 7, 9 Literacy, numeracy www.decs.act.gov.au/
Capital schools/assessment
Territory
New South 3, 5, 7 Literacy, numeracy www.det.nsw.edu.au
Wales
Western 3, 5, 7, 10 Literacy, numeracy, www.eddept.wa.edu.
Australia science, technology, au/centoff/annrep99/
the arts, health & AnnualReport.pdf
physical education,
studies of society
& environment
South Australia 3, 5 literacy, numeracy
Victoria 3, 5, 7 literacy, numeracy, www.bos.vic.edu.au
science, studies of
society and environment
in alternate years
Northern 3, 5, 7 www.ntde.gov.au
Territory
Queensland 3, 5 literacy, numeracy www.qscc.quld.edu.au
Tasmania 3, 5 literacy, numeracy www.tased.edu.au
Canada
Brit. Columbia 4, 7, 10 Reading, numeracy, www.bced.gov.bc.ca/
first draft writing assessments
United States
Colorado 4, 8 Reading, Lang Arts www.ced.state.co.us/
Kentucky 4, 5, 7, 8 maths www.kde.state.ky.us/
Nth Carolina 4, 8 www.dpi.state.nc.us
Texas 4, 8 www.tea.state.tx.us
Page Tennessee 4, 8 www.state.tn.us/
6
education/
Systemwide Assess Prog.qxd 21/8/03 9:11 AM Page 7
HOW ARE DATA COLLECTED?
Full-cohort testing performances of carefully drawn
representative samples of students.
Some education systems collect student
Samples may be drawn to ensure
achievement data through full-cohort
adequate representation of particular
assessment programs. The Year 3 and
categories of students so that the
Year 5 literacy and numeracy assessments
average performances of students in
in Australian States and Territories, in
those categories can be compared and
many states of the United States, and in
monitored.
parts of Canada are examples, although
in some instances ‘full-cohort’means the Although sample surveys cannot provide
cohort of government schools only. all parents with information on the
progress of individual students, or local
Typically, data in these programs are
school communities with information on
collected through the use of machine-
school results, sample surveys have a
scored paper and pen tests, although for
number of important advantages over
some learning outcomes, students’
full-cohort testing.
extended responses or on-the-spot
performances are assessed by classroom Sample surveys are capable of providing
teachers or by centrally trained assessors evidence about a rich and varied set of
(for example, writing outcomes, and learning goals. Full-cohort testing
physical education outcomes). Data are programs inevitably address only those
aggregated to provide summaries of outcomes that can be assessed for many
group performances, either at the level thousands of students at a time. This
of a system or at the level of a school. constraint limits the range of learning
outcomes that the program is able to
An advantage of these programs is that
address.
they can provide reliable information to
all parents on individual student progress Because sample surveys usually do not
in a few crucial areas of school learning. report on individual students, it is not
necessary for all students to attempt the
Sample surveys
same set of assessment tasks. Different
Some systems collect achievement data students can attempt different but
through sample surveys which are overlapping sets of tasks (known as a
designed to provide summary ‘multiple-matrix’design) to allow system
information at the system level only. The reporting on a wide range of valued
US National Assessment of Educational curriculum goals.
Progress (NAEP) is an example of a survey
Sample surveys are also less expensive
of this kind conducted at a national level.
overall (though more expensive per
But some systems conduct similar
student) than full-cohort testing, and
surveys at a State or district level; for
tend to be less intrusive into classroom
example, the Western Australian
time (though they may require extensive
Monitoring Standards in Education
commitment from a very small number
Program.
of teachers and students).
Page
Programs of this kind are based on the 7
