Eliciting and Analyzing Expert Judgment: A Practical Guide (ASA-SIAM Series on Statistics and Applied Probability)

492 Pages·2001·26.94 MB·English

by  Mary A. Meyer

Eliciting and
Analyzing
Expert Judgment
ASA-SIAM Series on
Statistics and Applied Probability
The ASA-SIAM Series on Statistics and Applied Probability is published
jointly by the American Statistical Association and the Society for Industrial and
Applied Mathematics. The series consists of a broad spectrum of books on topics
in statistics and applied probability. The purpose of the series is to provide
inexpensive, quality publications of interest to the intersecting membership of the
two societies.
Editorial Board
Richard F. Gunst Sallie Keller-McNulty
Southern Methodist University, Editor-in-Chief Los Alamos National Laboratory
Robert L. Mason Max D. Morris
Southwest Research Institute, Editor-in-Chief Iowa State University
Janet P. Buckingham Jane F. Pendergast
Southwest Research Institute University of Iowa
James A. Calvin Robert N. Rodriguez
Texas A&M University SAS Institute, Inc.
Gary C. McDonald
General Motors R&D Center
Smith, P. L., A Primer for Sampling Solids, Liquids, and Cases: Based on the Seven
Sampling Errors of Pierre Gy
Meyer, M. A. and Booker, J. M., Eliciting and Analyzing Expert Judgment:
A Practical Guide
Latouche, G. and Ramaswami, V., Introduction to Matrix Analytic Methods in
Stochastic Modeling
Peck, R., Haugh, L., and Goodman, A., Statistical Case Studies: A Collaboration
Between Academe and Industry, Student Edition
Peck, R., Haugh, L., and Goodman, A., Statistical Case Studies: A Collaboration
Between Academe and Industry
Barlow, R., Engineering Reliability
Czitrom, V. and Spagon, P. D., Statistical Case Studies for Industrial Process
Improvement
Eliciting and
Analyzing
Expert Judgment
A Practical Guide
Mary A. Meyer
Jane M. Booker
Statistical Sciences Group
Los Alamos National Laboratory
Los Alamos, New Mexico
ASA
Society for Industrial and Applied Mathematics American Statistical Association
Philadelphia, Pennsylvania Alexandria, Virginia
Copyright ©2001 by the American Statistical Association and the Society for
Industrial and Applied Mathematics.
This SIAM edition is an unabridged republication of the work first published by
Academic Press Limited, London, 1991.
1 0 9 8 7 6 5 4 3 21
All rights reserved. Printed in the United States of America. No part of this book
may be reproduced, stored, or transmitted in any manner without the written
permission of the publisher. For information, write to the Society for Industrial
and Applied Mathematics, 3600 University City Science Center, Philadelphia,
PA 19104-2688.
Library of Congress Card Number: 00-105923
ISBN 0-89871-474-5
m
ZlJLCJLJIL.is a registered trademark.
For Dr. Thomas R. Bement,
versatile statistician,
co-developer of PREDICT,
dedicated colleague,
and wonderful friend
This page intentionally left blank
Contents
Preface to ASA-SIAM Edition xxi
Preface xxiii
List of Figures xxv
List of Tables xxvii
List of Examples xxix
PART I: INTRODUCTION TO EXPERT JUDGMENT
1 Introduction 3
What Is Expert Judgment? 3
When Expert Judgment Is Used 4
General Attributes of Expert Judgment 6
Expert Judgment Covered in This Book 7
How Expert Judgment Is Elicited 9
Philosophy Guiding the Elicitation 11
Philosophy Guiding the Analysis 12
How To Use This Book 13
2 Common Questions and Pitfalls
Concerning Expert Judgment 17
Summary of Common Questions and Pitfalls 17
List of Common Questions 17
List of Pitfalls 18
Questions 19
What Does It Mean When the Experts Disagree? 19
Is Expert Judgment Valid Data? 20
Is the Gathering and Analyzing of Expert
Judgment Scientific? 21
Are Experts Bayesian? 23
Do Experts Provide Better Data? 24
Can Experts Be Fully Calibrated? 26
vii
viii CONTENTS
Pitfalls 27
Interviewers, Knowledge Engineers, and Analysts
Can Introduce Bias 27
Experts Are Limited in the Number of Things That
They Can Mentally Juggle 29
The Level of Detail in the Data (Granularity) Affects
the Analyses 30
The Conditioning Effect Poses Difficulties in
Gathering and Analyzing Expert Data 32
3 Background on Human Problem Solving
and Bias 35
Why Is It Necessary to Have an Understanding
of Human Problem Solving? 35
What Is Involved in Solving Problems and
in Responding? 35
The Four Cognitive Tasks 35
Two Models of Human Memory and Information
Processing 37
Bias 38
Two Views of Bias 40
Potential Impact of Bias 41
Sources of Bias 42
Motivational Bias 42
Cognitive Bias 44
Program for Handling Bias 44
Determining Which Steps to Apply 47
The Reason for Focusing on Bias 47
The Selection of the Definition of Bias,
Motivational or Cognitive 50
Interest in Particular Types of Bias 51
PART II: ELICITATION PROCEDURES
4 Selecting the Question Areas and Questions 57
Steps Involved in Selecting the Questions 57
CONTENTS ix
Illustrations of the Variation in Project Goals,
Question Areas, and Questions 58
Sources of Variation 59
Executing the Steps with the Assistance of Sponsors,
Project Personnel, and Experts 61
Determining in Which Steps the Advisory or External
Experts Will Assist 63
The External Experts Do Not Participate in Question
Area Selection, Question Identification, or
Refinement, Except Minimally Just Before Having
Their Judgments Elicited 63
The External Experts Receive the Question Areas
and Develop the Questions and Refine Them, or
They Receive the Questions and Refine Them 64
The External Experts Help Select the Question Areas
and Assist in the Identification and Refinement of
the Questions 64
Checklist for Selected Questions 65
Common Difficulties—Their Signs and Solutions .. 67
Difficulty: Sponsor Cannot Provide Clear
Information on the Project's Goal, the Data to
Be Gathered, or the Question Areas 67
Difficulty: The Question Developed From the
Question Area Remains Too Broad 67
Difficulty: Too Many Questions Have Been
Selected for the Amount of Time Available .. 68
5 Refining the Questions 69
Reasons for Structuring the Questions 69
Techniques for Structuring the Questions 71
Presentation of the Question Information 71
Types of Question Information Needed 71
Background 71
Assumptions 71
Definitions 72
Ordering of Information 72
Roles of Project Personnel and Experts 73
Decomposition of the Question 74
Considerations in Question Decomposition 74