Design, Evaluation, and Analysis of Questionnaires for Survey Research (Wiley Series in Survey Methodology)

391 Pages·2007·16.85 MB·English

by  Willem E. Saris

DE S IG N , EVALU,ATI 0 N ,
AND ANALYSIS (OF
QUESTIONNAIRES FOR
SURVEY RESEARCH
WILLEM E. SARIS
ESADE
Universitat Ramon Llull
Barcelona, Spain
IRMTRAUD N. GALLHOFER
ESADE
Universitat Ramon Llull
Barcelona, Spain
BICENTENNIAL
WILEY-IWTERSCIENCE
A John Wiley & Sons, Inc., Publication
This Page Intentionally Left Blank
DESIGN, EVALUATION,
AND ANALYSIS
OF
QUESTIONN AlRES FOR
SURVEY RESEARCH
THE w1 LEY BlCENTENN IAL-KNOWLEDGE FOR GEN€ RATIONS
6
ach generation has its unique needs and aspirations. When Charles Wiley first
opened his small printing shop in lower Manhattan in 1807, it was a generation
of boundless potential searching for an identity. And we were there, helping to
define a new American literary tradition. Over half a century later, in the midst
of the Second Industrial Revolution, it was a generation focused on building the
future. Once again, we were there, supplying the critical scientific, technical, and
engineering knowledge that helped frame the world. Throughout the 20th
Century, and into the new millennium, nations began to reach out beyond their
own borders and a new international community was born. Wiley was there,
expanding its operations around the world to enable a global exchange of ideas,
opinions, and know-how,
For 200 years, Wiley has been an integral part of each generation’s journey,
enabling the flow of information and understanding necessary to meet their needs
and fulfill their aspirations. Today, bold new technologies are changing the way
we live and learn. Wiley will be there, providing you the must-have knowledge
you need to imagine new worlds, new possibilities, and new opportunities.
Generations come and go, but you can always count on Wiley to provide you the
knowledge you need when and where you need it!
c-\ 4
WILLIAM J. PESCE PETER BOOTH WlLEV
PRESIDENT AND CHIEF EXECUTIVE OmCER CHAIRMAN OF THE BOARD
DE S IG N , EVALU,ATI 0 N ,
AND ANALYSIS (OF
QUESTIONNAIRES FOR
SURVEY RESEARCH
WILLEM E. SARIS
ESADE
Universitat Ramon Llull
Barcelona, Spain
IRMTRAUD N. GALLHOFER
ESADE
Universitat Ramon Llull
Barcelona, Spain
BICENTENNIAL
WILEY-IWTERSCIENCE
A John Wiley & Sons, Inc., Publication
Copyright C 2007 by John Wiley & Sons, Inc. All rights reserved.
Published by John Wiley & Sons, Inc., Hoboken, New Jersey
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form
or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as
permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior
written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to
the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax
[phone], or on the web at www.copyright.com. Requests to the Publisher for permission should
be addressed to the Permissions Department, John Wiley & Sons, Inc., 1 I1 River Street, Hoboken, NJ
07030, (201) 748-601 I, fax [phone], or online at
http::iwww.wiley.conL’go/permission.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in
preparing this book, they make no representations or warranties with respect to the accuracy or
completeness of the contents of this book and specifically disclaim any implied warranties of
merchantability or fitness for a particular purpose. No warranty may be created or extended by sales
representatives or written sales materials. The advice and strategies contained herein may not be
suitable for your situation. You should consult with a professional where appropriate. Neither the
publisher nor author shall be liable for any loss of profit or any other commercial damages, including
but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our
Customer Care Department within the United States at [phone], outside the United States at
[phone] or fax [phone].
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may
not be available in electronic format. For information about Wiley products, visit our web site at
www.wiley.com,
Wiley Bicentennial Logo: Richard J. Pacific0
Library of Congress Cataloging-in-Publication Data:
Saris. Willem E.
Design, evaulation, and analysis of questionnaires for survey research /
Willem E. Saris, Irmtraud N. Gallhofer.
p. cm. - (Wiley series in survey methodology)
Includes bibliographical references.
ISBN 978-0-470-1 1495-7 (cloth)
1. Social surveys. 2. Social surveys-Methodology. 3. Questionnaires. 4.
Interviewing. I. Gallhofer, Irmtraud N. 11. Title.
HN29.S29 2007
300.72’3-dc22 2007001697
Printed in the United States of America
1 0 9 8 7 6 5 4 3 2 1
Preface
Designing a survey involves many more dlecisions than most researchers
realize. Survey specialists, therefore, speak of the art of designing survey ques-
tions (Payne However, this book introduces methods and procedures that
1951).
can make questionnaire design a scientific activity. This requires knowledge of
the consequences of the many decisions thai: researchers take in survey design
and how these decisions affect the quality of the questions.
It is desirable to be able to evaluate the quality of the candidate questions
of the questionnaire before collecting the data. However, it is very tedious to
manually evaluate each question separately on all characteristics, mentioned
in the scientific literature, that predict the quality of the questions. It may even
be said that it is impossible to evaluate thle effect of the combination of all
these characteristics. This requires special tools that did not exist to date. A
computer program that can evaluate all the questions of a questionnaire on a
number of characteristics and provide an estimate of the quality of the ques-
tions based on the coded question characteristics would be very helpful. This
program could be a tool for the designer of the survey who can determine, on
the basis of the computer output, which questions in the survey require further
study in order to improve the quality of the data collected.
Furthermore, after a survey is completed it is useful to have information
about the data quality collected in order to correct for the errors in the data.
Therefore, there is a need for a computer program that can evaluate all ques-
tions of a questionnaire on a number of characteristics and provide an esti-
mate of the quality of the questions. Such information can be used to improve
the quality of the data analysis.
In order to further such an approach, we have
Developed a system for coding characteristics of survey questions and
I.
the more general survey procedure
2. Assembled a large set of studies that used multitrait-multimethod
(MTMM) experiments to estimate the reliability and validity of questions
1
VI DESIGN, EVALUATION, AND ANALYSIS OF QUESTIONNAIRES FOR SURVEY RESEARCH
3. Carried out a meta-analysis that relates these question characteristics to
the reliability and validity estimates of the questions
4. Developed a semiautomatic program that predicts the validity and
reliability of new questions based on the information available from the
meta-analysis of MTMM experiments
We think that these four steps are necessary to change the development of
questionnaires from an “art”i nto a scientific activity.
While this approach helps to optimize the formulation of a single ques-
tion, it does not necessarily improve the quality of survey measures. Often
researchers use complex concepts in research that cannot be measured by a
single question. So, several indicators are used. Moving from complex concepts
to a set of questions that together may provide a good measure for the concept
is called operationalization. In order to develop a scientific approach for ques-
tionnaire design we have also provided suggestions for operationalization of
complex concepts.
The purpose of this book is, first, to specify a three-step procedure which will
generate questions to measure the complex concept defined by the researcher.
The approach of operationalization is discussed in Part I of this book.
The second purpose of the book is to introduce to survey researchers the
different choices researchers can make and are makingwhile designing survey
questionnaires. This topic is covered in Part I1 of this book.
Part 111 of this book discusses quality criteria for survey questions, the way
these quality criteria have been evaluated in experimental research and the
results of a meta-analysis over many of such experiments that allow researchers
to determine the size of the effects of the different decisions on the quality of
the questions.
Part IV indicates how all this information can be used efficiently in the
design and analysis of surveys. Therefore, the first chapter introduces the
program “survey quality predictor” (SQP),w hich can be used for the predic-
tion of the quality of survey items on the basis of cumulative information
concerning the effect of different characteristics of the different components of
survey items on the data quality. The discussion of the program will be specific
enough so that the reader can use it to improve hidher own questionnaires.
The information about data quality can and should also be used after a
survey has been completed. Measurement error is unavoidable, and this infor-
mation is useful for how to correct it. The exact mechanics of it are illustrated
in several chapters of Part IV. We start out by demonstrating how this informa-
tion can be applied to estimate the quality of measures of complex concepts,
followed by a discussion on how to correct for measurement error in survey
research and how to cope with measurement error in cross cultural research.
In the last chapter we discuss how one can cope with measurement error in
cross-cultural research.
THE QUALITYO F MEASURESF OR CONCEPTS-BY-POSTULATION I VII
In general, we hope to contribute to the scientific approach of questionnaire
design and the overall improvement of survey research with this book.
Before closing this preface we have to mention the important contribution of
Frank Andrews, who suggested this approach in the early 1990s and inspired us
to continue his important work. The data sets of his studies are included in the
database on which the SQP program is based. Avery important contribution to
this book has also been made by Annette Scherpenzeel and Richard Koltringer,
who designed and performed many of the experiments on the basis of which
we can now make more general statements. We would also like to thank Albert
Satorra for his helpful discussions about statistical problems. An important
contribution in the development of this approach has been the cooperation
over the years with the members of the International Research group of Meth-
odology and Comparative Surveys (IRMCS) that came together on a yearly
basis in Slovenia, among other places to discuss the latest developments. We
especially have to mention Anuska Ferligoj. who organized most of the meet-
ings. We are also very grateful to William van der Veld and Daniel Oberslti,
who made the first versions of the SQP program. Important advice concerning
sentence grammar in Chapter 2 of the book has been obtained from Dr. P.J.
van der Voort. We are very grateful for his help. In the last phase of producing
the book we got a lot of helpful comment from Christine Punzo (Wiley) and
without the help of our type setter Kjeld de Ruyter of Puntspatie we would not
have been able to produce this book.
In addition we like also to thank the Netherlands Organization for Scientific
Research (NWO) for its financial support not only for the data collection but
also for the development of the programs. Avery important role was also played
by Sophia Kusyk, who was able to transform some of our awkward English
phrases into proper ones.
Last but not least, we would like to thank the many students who have
commented on the different versions of this book and the program. Without
their stimulating support and criticism, this book would not have been made.
Willem E. Saris
Irmtraud Gallhofer