Experimental Methods in Survey Research

525 Pages·2019·5.743 MB·English

by  Mick P. Couper, Graham Kalton, Lars Lyberg, J. N. K. Rao, Norbert Schwarz, Christopher Skinner

ExperimentalMethodsinSurveyResearch
WILEYSERIESINSURVEYMETHODOLOGY
EstablishedinPartbyWalterA.ShewhartandSamuelS.Wilks
Editors:MickP.Couper,GrahamKalton,LarsLyberg,J.N.K.Rao,NorbertSchwarz,
ChristopherSkinner
EditorEmeritus:RobertM.Groves
Acompletelistofthetitlesinthisseriesappearsattheendofthisvolume.
Experimental Methods in Survey Research
TechniquesthatCombineRandomSamplingwithRandomAssignment
Editedby
PaulJ.Lavrakas
NORC,IL,US
MichaelW.Traugott
UniversityofMichigan,MI,US
CourtneyKennedy
PewFoundation,DC,US
AllysonL.Holbrook
UniversityofIllinois-Chicago,IL,US
EdithD.deLeeuw
UniversityofUtrecht,NL,TC
BradyT.West
UniversityofMichigan,MI,US
Thiseditionfirstpublished2019
©2019JohnWiley&Sons,Inc.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,inany
formorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptaspermittedbylaw.
Adviceonhowtoobtainpermissiontoreusematerialfromthistitleisavailableathttp://www.wiley.com/go/
permissions.
TherightofPaulJ.Lavrakas,MichaelW.Traugott,CourtneyKennedy,AllysonL.Holbrook,EdithD.deLeeuw,and
BradyT.Westtobeidentifiedastheauthorsoftheeditorialmaterialinthisworkhasbeenassertedinaccordance
withlaw.
RegisteredOffice
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
EditorialOffice
111RiverStreet,Hoboken,NJ07030,USA
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWileyproductsvisitusat
www.wiley.com.
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Somecontentthatappears
instandardprintversionsofthisbookmaynotbeavailableinotherformats.
LimitofLiability/DisclaimerofWarranty
Whilethepublisherandauthorshaveusedtheirbesteffortsinpreparingthiswork,theymakenorepresentationsor
warrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthisworkandspecificallydisclaimall
warranties,includingwithoutlimitationanyimpliedwarrantiesofmerchantabilityorfitnessforaparticularpurpose.
Nowarrantymaybecreatedorextendedbysalesrepresentatives,writtensalesmaterialsorpromotionalstatements
forthiswork.Thefactthatanorganization,website,orproductisreferredtointhisworkasacitationand/or
potentialsourceoffurtherinformationdoesnotmeanthatthepublisherandauthorsendorsetheinformationor
servicestheorganization,website,orproductmayprovideorrecommendationsitmaymake.Thisworkissoldwith
theunderstandingthatthepublisherisnotengagedinrenderingprofessionalservices.Theadviceandstrategies
containedhereinmaynotbesuitableforyoursituation.Youshouldconsultwithaspecialistwhereappropriate.
Further,readersshouldbeawarethatwebsiteslistedinthisworkmayhavechangedordisappearedbetweenwhen
thisworkwaswrittenandwhenitisread.Neitherthepublishernorauthorsshallbeliableforanylossofprofitorany
othercommercialdamages,includingbutnotlimitedtospecial,incidental,consequential,orotherdamages.
LibraryofCongressCataloging-in-PublicationData
Names:Lavrakas,PaulJ.,editor.
Title:Experimentalmethodsinsurveyresearch:techniquesthatcombine
randomsamplingwithrandomassignment/editedbyPaulJ.Lavrakas,[and
fiveothers].
Description:NewJersey:Wiley,2020.|Series:Wileyseriesinsurvey
methodology|Includesbibliographicalreferencesandindex.|
Identifiers:LCCN2019015585(print)|LCCN2019021945(ebook)|ISBN
9781119083757(AdobePDF)|ISBN9781119083764(ePub)|ISBN9781119083740
(hardback)
Subjects:LCSH:Socialsurveys.|Socialsciences–Research–Methodology.|
Surveys–Evaluation.|BISAC:SOCIALSCIENCE/Statistics.
Classification:LCCHM538(ebook)|LCCHM538.E972020(print)|DDC
300.72/3–dc23
LCrecordavailableathttps://lccn.loc.gov/2019015585
Coverdesign:Wiley
Coverimage:©AngelinaBabii/Shutterstock
Setin10/12ptWarnockProbySPiGlobal,Chennai,India
PrintedintheUnitedStatesofAmerica
10 9 8 7 6 5 4 3 2 1
Wededicatethisbookwithgratitudetothecontributionsmadetotheuseofexperimentaldesign
byDonaldT.Campbell,ThomasD.Cook,RobertBoruch,MarilynB.Brewer,JeanneConverse,
StephenFienberg,StanleyPresser,HowardSchuman,andEleanorSinger,andtotheirinfluence
onourownusageofsurvey-basedexperiments.
vi
Contents
ListofContributors xix
PrefacebyDr.JudithTanur xxv
AbouttheCompanionWebsite xxix
1 ProbabilitySurvey-BasedExperimentationandtheBalancingofInternal
andExternalValidityConcerns 1
PaulJ.Lavrakas,CourtneyKennedy,EdithD.deLeeuw,BradyT.West,AllysonL.Holbrook,
andMichaelW.Traugott
1.1 ValidityConcernsinSurveyResearch 3
1.2 SurveyValidityandSurveyError 5
1.3 InternalValidity 6
1.4 ThreatstoInternalValidity 8
1.4.1 Selection 8
1.4.2 History 9
1.4.3 Instrumentation 10
1.4.4 Mortality 10
1.5 ExternalValidity 11
1.6 PairingExperimentalDesignswithProbabilitySampling 12
1.7 SomeThoughtsonConductingExperimentswithOnlineConvenience
Samples 12
1.8 TheContentsofthisBook 15
References 15
PartI IntroductiontoSectiononWithin-UnitCoverage 19
PaulJ.LavrakasandEdithD.deLeeuw
2 Within-HouseholdSelectionMethods:ACriticalReviewandExperimental
Examination 23
JoleneD.Smyth,KristenOlson,andMathewStange
2.1 Introduction 23
2.2 Within-HouseholdSelectionandTotalSurveyError 24
2.3 TypesofWithin-HouseholdSelectionTechniques 24
2.4 Within-HouseholdSelectioninTelephoneSurveys 25
2.5 Within-HouseholdSelectioninSelf-AdministeredSurveys 26
Contents vii
2.6 MethodologicalRequirementsofExperimentallyStudyingWithin-Household
SelectionMethods 27
2.7 EmpiricalExample 30
2.8 DataandMethods 31
2.9 AnalysisPlan 34
2.10 Results 35
2.10.1 ResponseRates 35
2.10.2 SampleComposition 35
2.10.3 Accuracy 37
2.11 DiscussionandConclusions 40
References 42
3 MeasuringWithin-HouseholdContamination:TheChallengeof
InterviewingMoreThanOneMemberofaHousehold 47
ColmO’Muircheartaigh,StephenSmith,andJaclynS.Wong
3.1 LiteratureReview 47
3.2 DataandMethods 50
3.2.1 ThePartnerStudyExperimentalDesignandEvaluation 52
Investigators 53
Field/ProjectDirectors 53
3.2.1.1 EffectsonResponseRatesandCoverage 54
3.3 TheSequenceofAnalyses 55
3.4 Results 55
3.4.1 ResultsofExperimentalGroupAssignment 55
3.5 EffectonStandardErrorsoftheEstimates 57
3.6 EffectonResponseRates 58
3.6.1 ResponseRatesAmongPrimes 58
3.6.2 ResponseRatesAmongPartners 60
3.7 EffectonResponses 61
3.7.1 CasesConsideredintheAnalyses 61
3.7.2 SequenceofAnalyses 61
3.7.3 DependentVariablesofInterest 61
3.7.4 EffectofExperimentalTreatmentonResponsesofPrimes 61
3.7.4.1 Missingness 61
3.7.4.2 SubstantiveAnswers 62
3.7.5 ChangesinResponsePatternsOverTimeAmongPrimes 62
3.7.5.1 Missingness 62
3.7.5.2 SubstantiveAnswers 62
3.7.5.3 SocialDesirability 62
3.7.5.4 NegativeQuestions 62
3.7.5.5 PositiveQuestions 63
3.7.5.6 EffectonResponsesofPartners 63
3.7.6 Conclusion 63
3.7.6.1 Design 63
3.7.6.2 StatisticalConsiderations 64
3.8 SubstantiveResults 64
References 64
viii Contents
PartII SurveyExperimentswithTechniquestoReduce
Nonresponse 67
EdithD.deLeeuwandPaulJ.Lavrakas
4 SurveyExperimentsonInteractionsandNonresponse:ACaseStudyof
IncentivesandModes 69
A.BianchiandS.Biffignandi
4.1 Introduction 69
4.2 LiteratureOverview 70
4.3 CaseStudy:ExaminingtheInteractionBetweenIncentivesandMode 73
4.3.1 TheSurvey 73
4.3.2 TheExperiment 73
4.3.3 DataandMethods 75
4.3.4 ResultsoftheExperiment:IncentivesandModeEffects 77
4.3.4.1 EffectsonParticipation 77
4.3.4.2 EffectsonSubgroups 78
4.3.4.3 EffectsonDataQuality 80
4.3.4.4 EffectsonCosts 81
4.4 ConcludingRemarks 83
Acknowledgments 85
References 86
5 ExperimentsontheEffectsofAdvanceLettersinSurveys 89
SusanneVogl,JenniferA.Parsons,LindaK.Owens,andPaulJ.Lavrakas
5.1 Introduction 89
5.1.1 WhyAdvanceLetters? 89
5.1.2 WhatWeKnowAboutEffectsofAdvanceLetters? 90
5.1.2.1 OutcomeRates 90
5.1.2.2 NonresponseBias 91
5.1.2.3 CostEffectiveness 91
5.1.2.4 PropertiesofAdvanceLetters 93
5.2 StateoftheArtonExperimentationontheEffectofAdvanceLetters 93
5.3 CaseStudies:ExperimentalResearchontheEffectofAdvanceLetters 95
5.4 CaseStudyI:ViolenceAgainstMeninIntimateRelationships 96
5.4.1 SampleDesign 96
5.4.2 Fieldwork 97
5.4.3 SamplingBias 97
5.4.4 AnalyticalStrategies 97
5.4.5 Results 98
5.4.5.1 EffectofAdvanceLettersonOutcomeRatesandReasonsforRefusals 98
5.4.5.2 RecruitmentEffort 98
5.4.5.3 EffectofAdvanceLettersonReportingonSensitiveTopics 98
5.4.6 Discussion 99
5.5 CaseStudyII:TheNeighborhoodCrimeandJusticeStudy 100
5.5.1 StudyDesign 100
5.5.2 ExperimentalDesign 100
5.5.3 Fieldwork 101
5.5.4 AnalyticalStrategies 102
5.5.5 Results 103
Contents ix
5.5.6 Discussion 103
5.6 Discussion 106
5.7 ResearchAgendafortheFuture 107
References 108
PartIII OverviewoftheSectionontheQuestionnaire 111
AllysonHolbrookandMichaelW.Traugott
6 ExperimentsontheDesignandEvaluationofComplexSurvey
Questions 113
PaulBeatty,CarolCosenza,andFloydJ.FowlerJr.
6.1 QuestionConstruction:DanglingQualifiers 115
6.2 OverallMeaningsofQuestionCanBeObscuredbyDetailedWords 117
6.3 AreTwoQuestionsBetterthanOne? 119
6.4 TheUseofMultipleQuestionstoSimplifyResponseJudgments 121
6.5 TheEffectofContextorFramingonAnswers 122
6.5.1 Alcohol 123
6.5.2 Guns 123
6.6 DoQuestionnaireEffectsVaryAcrossSub-groupsofRespondents? 124
6.7 Discussion 126
References 128
7 ImpactofResponseScaleFeaturesonSurveyResponsestoBehavioral
Questions 131
FlorianKeuschandTingYan
7.1 Introduction 131
7.2 PreviousWorkonScaleDesignFeatures 132
7.2.1 ScaleDirection 132
7.2.2 ScaleAlignment 133
7.2.3 VerbalScaleLabels 133
7.3 Methods 134
7.4 Results 136
7.5 Discussion 141
Acknowledgment 143
7.A QuestionWording 143
7.A.1 ExperimentalQuestions(OneQuestionPerScreen) 143
7.A.2 ValidationQuestions(OnePerScreen) 144
7.A.3 GfKProfileQuestions(NotPartoftheQuestionnaire) 145
7.B TestofInteractionEffects 145
References 146
8 ModeEffectsVersusQuestionFormatEffects:AnExperimental
InvestigationofMeasurementErrorImplementedinaProbability-Based
OnlinePanel 151
EdithD.deLeeuw,JoopHox,andAnnetteScherpenzeel
8.1 Introduction 151
8.1.1 OnlineSurveysAdvantagesandChallenges 151
8.1.2 Probability-BasedOnlinePanels 152
8.1.3 ModeEffectsandMeasurement 152