On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations

288 Pages·2004·4.95 MB·English

by  National Research Council, Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials

On
Evaluating
Curricular
Effectiveness
JUDGING THE QUALITY OF
K--12 MATHEMATICS EVALUATIONS
Committee for a Review of the Evaluation Data on the
Effectiveness of NSF-Supported and Commercially Generated
Mathematics Curriculum Materials
Jere Confrey and Vicki Stohl, Editors
Mathematical Sciences Education Board
Center for Education
Division of Behavioral and Social Sciences and Education
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Govern-
ing Board of the National Research Council, whose members are drawn from the
councils of the National Academy of Sciences, the National Academy of Engineer-
ing, and the Institute of Medicine. The members of the committee responsible for
the report were chosen for their special competences and with regard for appropri-
ate balance.
This study/publication was supported by Contract/Grant No. ESI-0102582 between
the National Academy of Sciences and the National Science Foundation. Additional
funding was provided by an award from the Presidents’ Committee of the National
Academies. Any opinions, findings, conclusions, or recommendations expressed in
this publication are those of the author(s) and do not necessarily reflect the views of
the organizations or agencies that provided support for the project.
Library of Congress Cataloging-in-Publication Data
On evaluating curricular effectiveness : judging the quality of K-12 mathematics
evaluations / Committee for a Review of the Evaluation Data on the Effectiveness
of NSF-Supported and Commercially Generated Mathematics Curriculum
Materials, Mathematical Sciences Education Board, Center for Education,
Division of Behavioral and Social Sciences and Education.
p. cm.
Includes bibliographical references.
ISBN 0-309-09242-6 (pbk.) — ISBN 0-309-53287-6 (pdf)
1. Mathematics—Study and teaching—Evaluation. I. National Research
Council (U.S.). Committee for a Review of the Evaluation Data on the
Effectiveness of NSF-Supported and Commercially Generated Mathematics
Curriculum Materials.
QA11.2.O5 2004
510'.71—dc22
2004015000
Additional copies of this report are available from the National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; [phone] or
[phone] (in the Washington metropolitan area); Internet, http://
www.nap.edu.
Printed in the United States of America.
Copyright 2004 by the National Academy of Sciences. All rights reserved.
National Research Council. (2004). On Evaluating Curricular Effectiveness: Judg-
ing the Quality of K-12 Mathematics Evaluations. Committee for a Review of the
Evaluation Data on the Effectiveness of NSF-Supported and Commercially Gener-
ated Mathematics Curriculum Materials. Jere Confrey and Vicki Stohl, Editors.
Mathematical Sciences Education Board, Center for Education, Division of Behav-
ioral and Social Sciences and Education. Washington, DC: The National Academies
Press.
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
http://www.nap.edu/catalog/11025.html
The National Academy of Sciences is a private, nonprofit, self-perpetuating society
of distinguished scholars engaged in scientific and engineering research, dedicated
to the furtherance of science and technology and to their use for the general welfare.
Upon the authority of the charter granted to it by the Congress in 1863, the Acad-
emy has a mandate that requires it to advise the federal government on scientific and
technical matters. Dr. Bruce M. Alberts is president of the National Academy of
Sciences.
The National Academy of Engineering was established in 1964, under the charter of
the National Academy of Sciences, as a parallel organization of outstanding engi-
neers. It is autonomous in its administration and in the selection of its members,
sharing with the National Academy of Sciences the responsibility for advising the
federal government. The National Academy of Engineering also sponsors engineer-
ing programs aimed at meeting national needs, encourages education and research,
and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is presi-
dent of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of
Sciences to secure the services of eminent members of appropriate professions in the
examination of policy matters pertaining to the health of the public. The Institute
acts under the responsibility given to the National Academy of Sciences by its
congressional charter to be an adviser to the federal government and, upon its own
initiative, to identify issues of medical care, research, and education. Dr. Harvey V.
Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences
in 1916 to associate the broad community of science and technology with the
Academy’s purposes of furthering knowledge and advising the federal government.
Functioning in accordance with general policies determined by the Academy, the
Council has become the principal operating agency of both the National Academy
of Sciences and the National Academy of Engineering in providing services to the
government, the public, and the scientific and engineering communities. The Coun-
cil is administered jointly by both Academies and the Institute of Medicine. Dr.
Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the
National Research Council.
www.national-academies.org
Copyright © National Academy of Sciences. All rights reserved.
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
http://www.nap.edu/catalog/11025.html
COMMITTEE FOR A REVIEW OF THE EVALUATION DATA ON
THE EFFECTIVENESS OF NSF-SUPPORTED AND COMMERCIALLY
GENERATED MATHEMATICS CURRICULUM MATERIALS
JERE CONFREY (Chair), Department of Education, Washington
University in St. Louis, MO
CARLOS CASTILLO-CHAVEZ, Department of Mathematics and
Statistics, Arizona State University
DOUGLAS A. GROUWS, Department of Learning, Teaching, and
Curriculum, University of Missouri
CAROLYN MAHONEY, Division of Academic Affairs, Elizabeth City
State University, NC
DONALD SAARI, Institute for Mathematical Behavioral Sciences,
University of California, Irvine
WILLIAM SCHMIDT, College of Education, Michigan State University
PATRICK W. THOMPSON, Department of Teaching and Learning,
Vanderbilt University, TN
WILLIAM VELEZ, Department of Mathematics, University of Arizona
CAROLE B. LACAMPAGNE, Study Director
VICKI STOHL, Research Associate
DIONNA J. WILLIAMS, Senior Program Assistant
v
Copyright © National Academy of Sciences. All rights reserved.
MATHEMATICAL SCIENCES EDUCATION BOARD
2004
JOAN LEITZEL (Chair), President Emerita, University of New
Hampshire
JERE CONFREY (Vice Chair), Department of Education, Washington
University in St. Louis, MO
THOMAS BANCHOFF, Department of Mathematics, Brown
University, RI
JAN DE LANGE, Freudenthal Institute, Utrecht University, The
Netherlands
LOUIS GOMEZ, School of Education and Social Policy, Northwestern
University, IL
DOUGLAS A. GROUWS, Department of Learning, Teaching, and
Curriculum, University of Missouri
ARTHUR JAFFE, Department of Mathematics, Harvard University, MA
ERIC JOLLY, Science Museum of Minnesota
JIM LEWIS, Department of Mathematics and Statistics, University of
Nebraska-Lincoln
GEORGE MCSHAN, National School Boards Association, TX
KAREN MICHALOWICZ, Mathematics Department, The Langley
School, VA
JUDITH MUMME, WestEd, CA
CASILDA PARDO, Teaching and Learning Systems, Valle Vista
Elementary School, NM
SUE PARSONS, Instructor, Cerritos College, CA
MARGE PETIT, The National Center for the Improvement of
Educational Assessment, VT
DONALD SAARI, Institute for Mathematical Behavioral Sciences,
Universityof California, Irvine
RICHARD SCHEAFFER, Professor Emeritus, University of Florida
FRANCIS SULLIVAN, Center for Computing Sciences, MD
HUNG HSI WU, Department of Mathematics, University of California,
Berkeley
vi
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
http://www.nap.edu/catalog/11025.html
Foreword
The Mathematical Sciences Education Board (MSEB) of the National
Research Council was established in 1985 to provide national leadership
and guidance for policies, programs, and practices supporting the improve-
ment of mathematics education at all levels. Curriculum materials for grades
K-12 play a central role in what mathematics topics are taught in our
schools, how the topics are sequenced and presented to students, what
levels of understanding are expected, what skills students will develop and
when. Schools, practitioners, policy makers, and the public depend on
evaluations of materials undertaken during their development and imple-
mentation in making decisions about the appropriate uses of the materials.
The MSEB recognized that the nature and quality of the evidence used to
judge claims of success and failure are critical elements in enabling the
community to make sound judgments. This report presents a synthesis of
the evidence used in the evaluations of several sets of recently developed
curriculum materials, provides a framework for the design of evaluation
studies of curriculum materials, and gives conclusions and recommenda-
tions to guide future efforts in evaluating curriculum materials.
The report was prepared by a committee of experts who devoted their
time, skills, and scholarship to this project over the past two years. On
behalf of the MSEB, I want to thank each of them for their commitment to
the important and difficult set of issues this study comprised. I especially
want to commend Jere Confrey and extend deepest appreciation to her for
her extraordinary leadership and commitment as chair of this project. In
addition to her leadership of the committee, Jere played an extensive role
vii
Copyright © National Academy of Sciences. All rights reserved.
viii FOREWORD
in drafting and redrafting the report through the final stages of committee
consultation and the intensive review process. Her dedication to maintain-
ing the highest standards of scholarship and the full engagement of the
committee, despite her many other professional obligations, was exem-
plary. The report bears the imprint of her commitment to intellectual
and empirical rigor; the field of mathematics education is the fortunate
beneficiary.
Joan R. Leitzel
Chair, MSEB
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
http://www.nap.edu/catalog/11025.html
Acknowledgments
This report reflects the efforts of many people, and the committee is
most grateful for their contributions. First, we wish to acknowledge and
thank our sponsor, the National Science Foundation’s Directorate for Edu-
cation and Human Resources, for their support of this study. The National
Academies’ Presidents’ Fund also provided some additional support for the
report.
The committee was aided greatly by the individuals who participated in
our evidence-gathering workshops. They helped us understand the nuances
of curriculum development and implication and ways to evaluate the effi-
cacy of such curricula once out in the field. The first evidence gathering
workshop was held in September 2002 and focused on how to define or
evaluate “effectiveness” of school mathematics curricula. The committee
gained valuable insights from the experts who presented at that workshop:
Johanna Birckmayer, Pacific Institute of Research and Evaluation; David
Francis, University of Houston; Pendred Noyce, The Noyce Foundation;
Andrew Porter, Wisconsin Center for Education Research; Frank Wang,
Saxon Publishers; Jan Mokros, TERC; Thomas Romberg, University of
Wisconsin–Madison; Sheila Sconiers, ARC Implementation Center; Andrew
Isaacs, University of Chicago; Judith Zawojewski, Illinois Institute of Tech-
nology; Solomon Garfunkel, COMAP, Inc; Norman Webb, Wisconsin Cen-
ter for Education Research; Richard Askey, University of Wisconsin–Madi-
son; Roger Howe, Yale University; William McCallum, University of
Arizona; M. Kathleen Heid, Pennsylvania State University; Richard Lehrer,
Vanderbilt University; Richard Lesh, Purdue University; Mark Jenness,
ix
Copyright © National Academy of Sciences. All rights reserved.