Innovations in software engineering for defense systems

88 Pages·2003·3.81 MB·English

by  All authors, Siddhartha R Dalal, J H Poore, Michael L Cohen, Oversight Committee for the Workshop on Statistical Methods in Software Engineering for Defense Systems., National Research Council (U.S.). Committee on National Statistics.

Innovations in
Software
Engineering for
Defense
Systems
Oversight Committee for the Workshop on Statistical Methods
in Software Engineering for Defense Systems
Siddhartha R. Dalal, Jesse H. Poore, and
Michael L. Cohen, editors
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
and
Committee on Applied and Theoretical Statistics
Division on Engineering and Physical Sciences
THE NATIONAL ACADEMIES PRESS • 500 Fifth Street, NW • Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing
Board of the National Research Council, whose members are drawn from the councils
of the National Academy of Sciences, the National Academy of Engineering, and the
Institute of Medicine. The members of the committee responsible for the report were
chosen for their special competences and with regard for appropriate balance.
This material is based upon work supported by the National Science Foundation under
Grant No. SBR-9709489. Any opinions, findings, conclusions, or recommendations
expressed in this publication are those of the author(s) and do not necessarily reflect the
views of the National Science Foundation.
International Standard Book Number 0-309-08983-2 (Book)
International Standard Book Number 0-309-52631-0 (PDF)
Library of Congress Control Number 2003110750
Additional copies of this report are available from National Academies Press, 500 Fifth
Street, N.W., Lockbox 285, Washington, DC 20055; [phone] or (202) 334-
3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Printed in the United States of America.
Copyright 2003 by the National Academy of Sciences. All rights reserved.
Suggested citation: National Research Council. (2003). Innovations in Software Engi-
neering for Defense Systems. Oversight Committee for the Workshop on Statistical Meth-
ods in Software Engineering for Defense Systems. S.R. Dalal, J.H. Poore, and M.L.
Cohen, eds. Committee on National Statistics, Division of Behavioral and Social Sci-
ences and Education. Washington, DC: The National Academies Press.
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of
distinguished scholars engaged in scientific and engineering research, dedicated to the
furtherance of science and technology and to their use for the general welfare. Upon
the authority of the charter granted to it by the Congress in 1863, the Academy has a
mandate that requires it to advise the federal government on scientific and technical
matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of
the National Academy of Sciences, as a parallel organization of outstanding engineers.
It is autonomous in its administration and in the selection of its members, sharing with
the National Academy of Sciences the responsibility for advising the federal govern-
ment. The National Academy of Engineering also sponsors engineering programs
aimed at meeting national needs, encourages education and research, and recognizes
the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National
Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of
Sciences to secure the services of eminent members of appropriate professions in the
examination of policy matters pertaining to the health of the public. The Institute acts
under the responsibility given to the National Academy of Sciences by its congressional
charter to be an adviser to the federal government and, upon its own initiative, to
identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is
president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences
in 1916 to associate the broad community of science and technology with the
Academy’s purposes of furthering knowledge and advising the federal government.
Functioning in accordance with general policies determined by the Academy, the
Council has become the principal operating agency of both the National Academy of
Sciences and the National Academy of Engineering in providing services to the govern-
ment, the public, and the scientific and engineering communities. The Council is
administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M.
Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National
Research Council.
www.national-academies.org
OVERSIGHT COMMITTEE FOR THE WORKSHOP ON
STATISTICAL METHODS IN SOFTWARE ENGINEERING FOR
DEFENSE SYSTEMS
DARYL PREGIBON (Chair), AT&T Laboratories, Florham Park,
New Jersey
BARRY BOEHM, Computer Science Department, University of
Southern California
SIDDHARTHA R. DALAL, Xerox Corporation, Webster, New York
WILLIAM F. EDDY, Department of Statistics, Carnegie Mellon
University
JESSE H. POORE, Department of Computer Science, University of
Tennessee, Knoxville
JOHN E. ROLPH, Marshall School of Business, University of Southern
California
FRANCISCO J. SAMANIEGO, Division of Statistics, University of
California, Davis
ELAINE WEYUKER, AT&T Laboratories, Florham Park, New Jersey
MICHAEL L. COHEN, Study Director
MICHAEL SIRI, Project Assistant
iv
COMMITTEE ON NATIONAL STATISTICS
2002-2003
JOHN E. ROLPH (Chair), Marshall School of Business, University of
Southern California
JOSEPH G. ALTONJI, Department of Economics, Yale University
ROBERT M. BELL, AT&T Laboratories, Florham Park, New Jersey
LAWRENCE D. BROWN, Department of Statistics, University of
Pennsylvania
ROBERT M. GROVES, Survey Research Center, University of Michigan
JOEL L. HOROWITZ, Department of Economics, Northwestern
University
WILLIAM D. KALSBEEK, Department of Biostatistics, University of
North Carolina
ARLEEN LEIBOWITZ, School of Public Policy Research, University of
California, Los Angeles
THOMAS A. LOUIS, Department of Biostatistics, Johns Hopkins
University
VIJAYAN NAIR, Department of Statistics, University of Michigan
DARYL PREGIBON, AT&T Laboratories, Florham Park, New Jersey
KENNETH PREWITT, School of International and Public Affairs,
Columbia University
NORA CATE SCHAEFFER, Department of Sociology, University of
Wisconsin, Madison
MATTHEW D. SHAPIRO, Department of Economics, University of
Michigan
ANDREW A. WHITE, Director
v
COMMITTEE ON APPLIED AND THEORETICAL STATISTICS
2002-2003
SALLIE KELLER-McNULTY (Chair), Los Alamos National Laboratory
SIDDHARTHA DALAL, Xerox Corporation, Webster, New York
CATHRYN DIPPO, Consultant, Washington, DC
THOMAS KEPLER, Duke University Medical Center, Durham,
North Carolina
DANIEL L. McFADDEN, University of California, Berkeley
RICHARD OLSHEN, Stanford University
DAVID SCOTT, Rice University, Houston, Texas
DANIEL L. SOLOMON, North Carolina State University
EDWARD C. WAYMIRE, Oregon State University, Corvallis, Oregon
LELAND WILKINSON, SPSS, Incorporated, Chicago
SCOTT T. WEIDMAN, Director
vi
Acknowledgments
he Workshop on Statistical Methods in Software Engineering for
Defense Systems, jointly sponsored by the Committee on National
TStatistics and the Committee on Applied and Theoretical Statistics
of the National Research Council (NRC), grew out of a need to more fully
examine issues raised in the NRC reports Statistics, Testing, and Defense
Acquisition: New Approaches and Methodological Improvements (NRC, 1998)
and Reliability Issues for Defense Systems: Report of a Workshop (NRC, 2002).
The former stressed, among other things, the importance of software re-
quirements specification, and the latter demonstrated the benefits gained
from model-based testing. Concurrent with these activities, there was an
increasing recognition in the Office of the Secretary of Defense that soft-
ware problems were a major source of defects, delays, and cost overruns in
defense systems in development, and that approaches to addressing these
problems were very welcome.
The Workshop on Statistical Methods in Software Engineering for
Defense Systems, which took place July 19-20, 2001, was funded jointly by
the offices of the Director of Operational Test and Evaluation (DOT&E)
and the Undersecretary of Defense for Acquisition, Technology, and Logis-
tics (OUSD(AT&L)). DOT&E, headed by Tom Christie, was ably repre-
sented at various planning meetings by Ernest Seglie, the science advisor
for DOT&E, and OUSD (AT&L) was represented by Nancy Spruill. Both
Ernest Seglie and Nancy Spruill were enormously helpful in suggesting
topics for study, individuals to serve as expert defense discussants, and in-
vii
viii ACKNOWLEDGMENTS
vited guests to supplement floor discussion. Their contribution was essen-
tial to the success of this workshop.
We would also like to thank the following Office of the Secretary of
Defense employees for their assistance: Walter Benesch, Michael Falat,
Jack Ferguson, Ken Hong Fong, Liz Rodriguez Johnson, Steven King, Bob
Nemetz, Rich Turner, and Bob Williams. We are very grateful to Norton
Compton for his help on a number of administrative issues and his can-do
spirit. Scott Weidman, staff director of the NRC’s Committee on Applied
and Theoretical Statistics, provided very helpful advice in structuring the
workshop. We would also like to thank Art Fries of the Institute for De-
fense Analysis, Dave Zubrow of the Software Engineering Institute, and
David Nicholls of IIT Research Institute for their assistance.
The Oversight Committee for the Workshop on Statistical Methods in
Software Engineering for Defense Systems met and interacted only infor-
mally, often via telephone and e-mail, and in person in small groups of two
and three. It relied to a great extent on the content of the presentations
given at the workshop. Therefore, the Committee would like to express its
gratitude to the speakers and the discussants: Frank Apicella, Army Evalua-
tion Center, Vic Basili, University of Maryland, Patrick Carnes, Air Force
Operational Test and Evaluation Center, Ram Chillarege, consultant, Tom
Christian, WR-ALC, Delores Etter, Office of the Secretary of Defense,
William Farr, Naval Surface Warfare Center, Jack Ferguson, Office of the
Undersecretary of Defense for Science and Technology, Janet Gill, PAX
River Naval Air Systems, Software Safety, Constance Heitmeyer, Naval Re-
search Lab, Brian Hicks, Joint Advanced Strike Technology, Mike
Houghtaling, IBM, Jerry Huller, Raytheon, Ashish Jain, Telcordia Tech-
nologies, Stuart Knoll, Joint Staff, J-8, Scott Lucero, Army SW Metrics,
Army Evaluation Center, Luqi, Naval Postgraduate School, Ron Manning,
Raytheon, David McClung, University of Texas, Margaret Myers, Office of
the Assistant Secretary of Defense for Command, Control, Communica-
tions and Intelligence, Jim O’Bryon, Office of the Director of Operational
Test and Evaluation, Lloyd Pickering, Army Evaluation Center, Stacy J.
Prowell, University of Tennessee, Manish Rathi, Telcordia Technologies,
Harry Robinson, Microsoft, Mike Saboe, Tank Automotive and Armaments
Command, Ernest Seglie, Office of the Director of Operational Test and
Evaluation, Linda Sheckler, Penn State University, Applied Research Lab,
Kishor Trivedi, Duke University, Amjad Umar, Telcordia Technologies, and
Steven Whitehead, Operational Test and Evaluation Force Technical Advi-
ACKNOWLEDGMENTS ix
sor. Further, we would like to give special thanks to Constance Heitmeyer
and Amjad Umar for their help in writing sections of the report.
We would also like to thank Michael Cohen for help in organizing the
workshop and in drafting sections of the report, and Michael Siri for his
excellent attention to all administrative details. We would also like to thank
Cameron Fletcher for meticulous technical editing of the draft report,
catching numerous errors.
This report has been reviewed in draft form by individuals chosen for
their diverse perspectives and technical expertise, in accordance with proce-
dures approved by the Report Review Committee of the NRC. The pur-
pose of this independent review is to provide candid and critical comments
that will assist the institution in making the published report as sound as
possible and to ensure that the report meets institutional standards for ob-
jectivity, evidence, and responsiveness to the study charge. The review
comments and draft manuscript remain confidential to protect the integ-
rity of the deliberative process.
We thank the following individuals for their participation in the re-
view of this report: Robert M. Bell, AT&T Research Laboratories, Florham
Park, NJ, Philip E. Coyle, III, Consultant, Los Angeles, CA, Michael A.
Houghtaling, IBM/ Systems Group/ Storage Systems Division, Tucson, AZ,
and Lawrence Markosian, QSS Group, Inc., NASA Ames Research Center,
Moffett Field, CA. Although the reviewers listed above have provided many
constructive comments and suggestions, they were not asked to endorse the
conclusions or recommendations nor did they see the final draft of the
report before its release. The review of this report was overseen by William
G. Howard, Jr., Independent Consultant, Scottsdale, AZ. Appointed by
the NRC, he was responsible for making certain that an independent ex-
amination of this report was carried out in accordance with institutional
procedures and that all review comments were carefully considered. Re-
sponsibility for the final content of this report rests entirely with the
authoring committee and the institution.
Daryl Pregibon, Chair
Workshop on Statistical Methods in Software
Engineering for Defense Systems