Table Of ContentContents Contents
Mixed-Mode Official Surveys
Statistics in the Social and Behavioral Sciences Series
Series Editors
Jeff Gill, Steven Heeringa, Wim J. van der Linden, Tom Snijders
Recently Published Titles
Bayesian Demographic Estimation and Forecasting
John Bryant and Junni L. Zhang
Multivariate Analysis in the Behavioral Sciences, Second Edition
Kimmo Vehkalahti and Brian S. Everitt
Analysis of Integrated Data
Li-Chun Zhang and Raymond L. Chambers
Multilevel Modeling Using R, Second Edition
W. Holmes Finch, Joselyn E. Bolin, and Ken Kelley
Modelling Spatial and Spatial-Temporal Data: A Bayesian Approach
Robert Haining and Guangquan Li
Handbook of Automated Scoring: Theory into Practice
Duanli Yan, André A. Rupp, and Peter W. Foltz
Interviewer Effects from a Total Survey Error Perspective
Kristen Olson, Jolene D. Smyth, Jennifer Dykema, Allyson Holbrook, Frauke Kreuter,
and Brady T. West
Measurement Models for Psychological Attributes
Klaas Sijtsma and Andries van der Ark
Big Data and Social Science: Data Science Methods and Tools for Research
and Practice, Second Edition
Ian Foster, Rayid Ghani, Ron S. Jarmin, Frauke Kreuter and Julia Lane
Understanding Elections through Statistics: Polling, Prediction, and Testing
Ole J. Forsberg
Analyzing Spatial Models of Choice and Judgment, Second Edition
David A. Armstrong II, Ryan Bakker, Royce Carroll, Christopher Hare, Keith T. Poole
and Howard Rosenthal
Introduction to R for Social Scientists: A Tidy Programming Approach
Ryan Kennedy and Philip Waggoner
Linear Regression Models: Applications in R
John P. Hoffman
Mixed-Mode Official Surveys: Design and Analysis
Barry Schouten, Jan van den Brakel, Bart Buelens, Deirdre Giesen, Annemieke Luiten,
and Vivian Meertens
For more information about this series, please visit: https :/ /ww w .rou tledg e .com /Chap
man-- H allC R C- St a tist i cs- i n - the - Soci a l- an d - Beh a vior a l- Sc i ence s / boo k - ser i es/ C H STSO
BESCI
Mixed-Mode Official Surveys
Design and Analysis
Barry Schouten, Jan van den Brakel,
Bart Buelens, Deirdre Giesen, Annemieke Luiten,
and Vivian Meertens
First edition published 2022
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
© 2022 CRC Press
CRC Press is an imprint of Taylor & Francis Group, LLC
Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have
attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright hold-
ers if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged
please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or
utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including pho-
tocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission
from the publishers.
For permission to photocopy or use material electronically from this work, access www .copyright .com or contact the
Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. For works that are not
available on CCC please contact mpkbookspermissions @tandf .co .uk
Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used only for iden-
tification and explanation without intent to infringe.
ISBN: 9781138618459 (hbk)
ISBN: 9781032102962 (pbk)
ISBN: 9780429461156 (ebk)
DOI: 10.1201/9780429461156
Typeset in Palatino
by Deanta Global Publishing Services, Chennai, India
Contents
Preface ..............................................................................................................................................xi
Acknowledgements ....................................................................................................................xiii
Authors ...........................................................................................................................................xv
Part I Introduction
1. Foreword to Mixed-Mode Official Surveys: Design and Analysis ..............................3
1.1 Why a Book on Mixed-Mode Survey Design and Analysis? ..................................3
1.2 Modes and Devices .......................................................................................................4
1.3 Outline of the Book .......................................................................................................6
2. Designing Mixed-Mode Surveys ........................................................................................9
2.1 Introduction ...................................................................................................................9
2.2 Why Mixed-Mode Survey Designs? ...........................................................................9
2.3 How Can Modes Be Combined? ...............................................................................13
2.4 How Do Modes Affect Quality? ...............................................................................16
2.5 How to Reduce Mode Impact on Quality? ..............................................................22
2.6 Case Studies .................................................................................................................26
2.6.1 Dutch Crime Victimization Survey .............................................................27
2.6.2 Dutch Labour Force Survey .........................................................................28
2.6.3 European Health Interview Survey ............................................................29
2.7 Summary ......................................................................................................................30
Part II Mode Effects
3. Mode-Specific Measurement Effects ................................................................................33
3.1 Introduction .................................................................................................................33
3.2 Measurement Features of Modes ..............................................................................35
3.3 Mode-Specific Answering Behaviors and Response Styles ..................................42
3.4 Detection of Mode-Specific Measurement Effects .................................................49
3.5 Summary ......................................................................................................................55
4. Mode-Specific Selection Effects ........................................................................................57
4.1 Introduction .................................................................................................................57
4.2 Coverage Issues in CAPI, CATI, Mail, and Web Surveys ......................................58
4.2.1 CAPI Coverage ...............................................................................................58
4.2.2 CATI Coverage ...............................................................................................59
4.2.3 Mail Coverage.................................................................................................63
4.2.4 Web Coverage .................................................................................................63
v
vi Contents
4.3 Response and Subgroup Response Rates in Mixed-Mode Designs ....................68
4.3.1 Mixed-Mode Designs at Statistics Netherlands ........................................70
4.4 Summary ......................................................................................................................79
Notes ........................................................................................................................................79
Part III Design
5. Mixed-Mode Data Collection Design ...............................................................................83
5.1 Introduction .................................................................................................................83
5.2 Mode and Mode Combinations ................................................................................85
5.3 Concurrent and Sequential Designs ........................................................................90
5.3.1 Concurrent or Sequential Order? ................................................................90
5.3.2 Sequential, but Which Sequence? ................................................................93
5.4 Costs ..............................................................................................................................94
5.5 Sample Composition and Data Quality ...................................................................95
5.6 Communication Strategies to Increase Web Response .........................................97
5.6.1 Contacting the Sample Person or Household ............................................99
5.6.2 Reminders .....................................................................................................100
5.6.3 Invitation Letters, Flyers, and Envelopes: Statistics Netherlands’
Experiments ..................................................................................................101
5.6.3.1 Experimental Manipulations ......................................................102
5.6.3.2 Results ............................................................................................106
5.6.4 Incentives in Web and Mixed-Mode Surveys ..........................................110
5.6.4.1 Effect on Sample Composition and Data Quality ...................112
5.6.4.2 Incentives in European NSIs ......................................................113
5.6.4.3 Incentive Experiments by Statistics Netherlands ....................114
5.6.4.4 An Incentive Experiment in the Longitudinal Labor
Force Survey2 ................................................................................118
5.7 Summary ....................................................................................................................120
Notes ......................................................................................................................................121
6. Mixed-Mode Questionnaire Design ...............................................................................123
6.1 Introduction ...............................................................................................................123
6.2 General Goals and Challenges in Questionnaire Design ...................................123
6.2.1 Why Questionnaire Matters .......................................................................123
6.2.2 Striking a Balance between Conflicting Goals and Stakeholders ........124
Clients (Partly Overlap with Data Users) .................................................125
Respondents .................................................................................................125
Interviewers ..................................................................................................125
Programmers ................................................................................................125
Data Users (Partly Overlap with Clients) .................................................125
Project Managers ..........................................................................................126
Questionnaire Designers .............................................................................126
6.2.3 Questionnaire Design as an Iterative Process .........................................126
6.3 Questionnaire Design and Mode-Specific Measurement Errors .......................127
6.3.1 The Cognitive Response Process Model...................................................127
6.3.2 Main Sources of Mode-Specific Measurement Error ..............................128
Contents vii
6.3.3 Even Small Design Differences May Matter ............................................129
6.3.4 Device-Specific Measurement Errors ........................................................130
6.4 Questionnaire Design for Mixed-Mode Surveys .................................................133
6.4.1 Minimizing Differences or Minimizing Error? ......................................133
6.4.2 Mixed-Mode Requirements in the Questionnaire Design Process ........134
6.4.3 Mixed-Device Questionnaire Design .......................................................136
6.5 Testing and Evaluating Mixed-Mode Questionnaires ........................................138
6.5.1 Testing and Evaluation as Part of the Questionnaire Design Process .138
6.5.2 Mixed-Mode Questionnaire Testing and Evaluation .............................141
Step 1 Make an Explicit Mode Risk Assessment .....................................141
Step 2 Decide if the Test Should Compare Modes ..................................141
Step 3 Decide Which Modes to Test, and When in the
Development Process...................................................................142
Step 4 Test Relevant Modes as Realistically as Practically Possible .........142
Step 5 In Analyses and Reporting, Distinguish between Usability
and Content Findings, and Reflect on Mode and Device
Specificity Regarding Findings ..................................................143
6.6 Guidelines for Mixed-Mode Questionnaire Design ............................................143
6.6.1 Keep the Stimulus and Response Task as Similar as Possible in All
Modes ............................................................................................................143
6.6.2 Organize the Questionnaire Design Process to Prevent and Detect
Mode-Specific Measurement Errors ..........................................................144
6.6.3 Prevent Social Desirability Bias .................................................................145
6.6.4 Prevent Satisficing........................................................................................145
6.7 Summary ....................................................................................................................146
Part IV Analysis
7. Field Tests and Implementation of Mixed-Mode Surveys .........................................149
7.1 Introduction ...............................................................................................................149
7.2 Design of Field Experiments ...................................................................................152
7.3 Design-Based Inference for Field Experiments ....................................................154
7.3.1 Measurement Error Model and Hypotheses Testing .............................154
7.3.2 Parameter and Variance Estimation ..........................................................156
7.3.3 Wald Test and Related Tests .......................................................................158
7.3.4 Extensions .....................................................................................................159
7.3.5 Software ........................................................................................................160
7.4 Time Series Methods ................................................................................................161
7.5 Small Area Estimation Methods .............................................................................164
7.6 The Introduction of a Hybrid Mixed-Mode Design in the Dutch Crime
Victimization Survey ................................................................................................166
7.6.1 Design ............................................................................................................166
7.6.2 Results ...........................................................................................................168
7.7 Discussion ..................................................................................................................172
7.8 Summary ....................................................................................................................174
viii Contents
8. Re-interview Designs to Disentangle and Adjust for Mode Effects .......................175
8.1 Introduction ...............................................................................................................175
8.2 Decomposition of Relative Mode Effects ...............................................................177
8.3 Estimating Components of the Relative Mode Effects ........................................179
8.3.1 Experimental Design ...................................................................................179
8.3.2 Estimation Strategies ...................................................................................181
8.3.3 Assumptions .................................................................................................183
8.4 Application .................................................................................................................184
8.5 Adjusting Measurement Bias Using Re-interview Designs ...............................190
8.6 Simulation Study .......................................................................................................195
8.7 Extension of Re-interview Design to Multiple Modes .........................................197
8.8 Conclusions ................................................................................................................198
8.9 Summary ....................................................................................................................199
9. Mixed-Mode Data Analysis..............................................................................................201
9.1 Introduction ...............................................................................................................201
9.2 Analyzing Mixed-Mode Survey Response Data ..................................................202
9.2.1 Literature .......................................................................................................202
9.2.2 Problem Statement, Definitions, and Notation ........................................204
9.3 Correcting Differential Measurement Bias ...........................................................205
9.3.1 Counterfactuals ............................................................................................205
9.3.2 Mode-Specific Estimators ...........................................................................207
9.4 Balancing Differential Measurement Bias .............................................................208
9.4.1 Mode Calibration .........................................................................................208
9.4.2 Choosing Calibration Levels ......................................................................211
9.5 Handling Measurement Bias in Practice ...............................................................212
9.5.1 Testing Assumptions ...................................................................................212
9.5.2 Calibration Levels and Mixing Coefficients ............................................212
9.5.3 Comparison with Single-Mode Designs ..................................................214
9.6 Applications ...............................................................................................................214
9.6.1 Health Survey ...............................................................................................214
9.6.2 Crime Victimization Survey.......................................................................215
9.6.3 Labor Force Survey ......................................................................................218
9.7 Summary ....................................................................................................................219
Part V The Future of Mixed-Mode Surveys
10. Multi-Device Surveys ........................................................................................................223
10.1 Introduction ...............................................................................................................223
10.2 Smart Surveys ............................................................................................................225
10.2.1 A Taxonomy of Smart Surveys ..................................................................225
10.2.2 Sensor Data ...................................................................................................228
10.2.3 Other Types of External Data .....................................................................231
10.3 Total Survey Error .....................................................................................................233
10.3.1 Representation and Measurement of Smart Surveys .............................233
10.3.2 Criteria to Include New Types of Data .....................................................235
Contents ix
10.4 Methodology for Hybrid Data Collection .............................................................237
10.4.1 Active Versus Passive Data Collection ......................................................237
10.4.2 Data Collection Strategies ...........................................................................238
10.4.3 Measurement Strategies ..............................................................................240
10.4.4 Estimation Strategies ...................................................................................241
10.4.5 Logistics and Operations ............................................................................241
10.5 Case Studies ...............................................................................................................243
10.5.1 Smart Survey Criteria ..................................................................................243
10.5.2 A Case Study Elaborated: Physical Activity ............................................246
10.6 Summary ....................................................................................................................248
Note ........................................................................................................................................249
11. Adaptive Mixed-Mode Survey Designs ........................................................................251
11.1 Introduction ...............................................................................................................251
11.2 Elements of Adaptive Multi-Mode Survey Designs ............................................253
11.2.1 Survey Mode as a Design Feature .............................................................254
11.2.2 Population Strata ..........................................................................................256
11.2.3 Quality and Cost Objectives in Multi-Mode Surveys ............................261
11.2.4 Optimization Strategies ..............................................................................265
11.3 Case Studies ...............................................................................................................266
11.3.1 The Dutch LFS ..............................................................................................267
11.3.2 The Dutch HS ...............................................................................................269
11.4 Summary ....................................................................................................................272
12. The Future of Mixed-Mode Surveys ..............................................................................273
12.1 Introduction ...............................................................................................................273
12.2 What Are the Current Developments? ...................................................................273
12.3 What Are the Open Areas in the Methodology of Mixed-Mode Surveys ........275
12.4 Open Areas in Logistics and Implementation of Mixed-Mode Surveys ..........277
12.5 Will There Be Mixed-Mode Surveys in the Future?.............................................278
References ...................................................................................................................................281
Index .............................................................................................................................................301
