Qualitative evaluation of the design and layout of the 1996 census questionnaires.

98 Pages·1994·2.9 MB·English

by  Rosalie Bambrick, Brenda Belanger, Jean-Pierre Morin, Sandrine Prasil.

92N0020E Recensement Census
no.26
c.l
1^1 SCtaantiasdtaique SCtaantiasdtiacs Canada

"^;;^?;i^i^^riTAT'STiQue
CANADA CANADA
ju\i. 7 1995
LIBRARY
BlBLlOTHeoUJ..
-r -nu
'(.-*' '• —
_-—- -NeT-1?:qport No.26
Qualitative Evaluation oftheDesignand
Layoutofthe 1996CensusQuestionnaire
Rosalie Bambrick
Brenda Belanger
Jean-Pierre Morin
Sandrine Prasil
Statistics Canada
February, 1994

COGNITIVERESEARCH:
DESIGNANDLAYOUTOFTHE
1996CENSUSQUESTIONNAIRES
ObservationReportofthe
TorontoandOttawaInterviews
ConductedinEnglish.
February1toFebruary21,1994
CensusOperationsDivision
CommunicationsDivision
QuestionnaireDesignResourceCentre
STATISTICSCANADA

Background
Over the past year. Statistics Canada has undertaken an extensive series of public
consultationsandresearchprojectsaimedatdevelopingaquestionnaireforthe1996Census.
Mostofthisworkhasfocusedoncontentissuesofthequestionnaire. Basedontheresults
oftheseconsultationsandresearch,apreliminaryversionofthe1996Censusquestionnaire
wastestedduringtheNationalCensusTost(NCT)inearlyNovember 1993. Apreliminary
versionofthe1996Census2Aquestionnairewasalsodeveloped.
ThefollowingreportpresentstheresultsofaseparatestudywhichbeganinJanuary,1994.
The study investigated the impact that the design and layout of the NCT and 2A
questionnaireshadonrespondentbehaviouranddataquality. Cognitiveaspects,suchas
respondentunderstandingandeaseofcompletingthequestionnaires,wereexplored.
ObjectivesoftheResearchProject
Themainpurposeoftheresearchprojectwastoinvestigaterespondentreactionstothe
NationalCensusTest(NCT)andthe2Aquestionnaires. Thestudyfocusedonnon-content
issuestoexaminetheimpactthatthedesignandlayou.thadonrespondentbehaviourand
dataquality.
Cognitiveresearchtechniqueswereusedtoexamine:
• HowpeoplewentaboutcompletingtheNCTand2Aquestionnaires{respondentbehaviour);
• Problemsorconfusionencounteredbyrespondentswhilecompletingtheforms;
• Theextenttowhichrespondentsreadandunderstoodinstructionsandquestions.
The research addressed all aspects of the design and layout of the NCT and 2A
questionnaires. The studyalsoassessedthefollowing issuesfrom thepoint ofview of
respondents:
• Thegraphicdesignofthequestionnaires,including:
-layoutoftheforms
-colourandshading
-sizeoftheprintfont
-bold-faceprint
-sizeofrespondententryfields
-useofofficecodingmarks
-placementofinstructionsandexamples
• Responseburdenandthelengthoftheforms.
• Respondents'reactionstothecoverpagesandLogooptions.
• Inreadingthe"MessagefromtheChiefStatisticianofCanada,"howdidrespondentsreact
tothestatementaboutmandatoryresponseandtheassuranceofconfidentiality?
• DidrespondentsreadandunderstandtheGuidethataccompaniestheforms?
• Didrespondentsreadtheinstructionsontheforms? Didtheyunderstandthem?
• Didthestep-by-stepapproachmakeiteasiertocompletetheform?
• FortheNCTquestionnaire: Didrespondentsnoticetheheadingsformajorsections(e.p.,
Socio-cglturalInformationandMobility)?
• Didrespondentsreadandunderstandtheinstructions,definitions,notesandexamplesin
(orbefore)variousquestions? Thesearementionedinthefollowing reportonlywhen
problemswithspecificquestionsoccurred.
• Didaninstructionassistrespondentsorcomplicatetheform?
• FortheNCTquestionnaire: Didrespondentsfollowtheskipinstructionscorrectly?
• Werethereanywordsorconceptsthatrespondentsdidnotunderstandormisinterpret?
• Respondents'suggestionsforimprovementstothedesignandlayoutofthequestionnaires.
Methodology
The studyinvolved in-depth interviews with both English-speaking and French-speaking
Canadians to get feedback about their opinions and reactions to the NCT and 2A
questionnaires. Consultations with respondents were carried out by the Contemporary
ResearchCentreand StatisticsCanada. Twofocusgroups, 20one-on-oneinterviewsin
observationfacilities,and60in-fiomeinterviewswereconducted.
RespondentswererecruitedbytheContemporaryResearchCentre. Therecruitmentcriteria
werebasedonthefollowingcharacteristics:
•sex(50%men,50%women)
•age(30%20to34years,30%35to49yearsand30%50to65years)
•education(lowerlevelsandthosewithoutuniversitydegrees)
•employmentstatus(70%employed,30%unemployed).
•ethnicity(somelandedimmigrantsandnewCanadians)
•diversityofrelationships(somehouseholdswithroom-matesorboarders)
Thirtyofthe60in-homeinterviewswereconductedinEnglishbyStatisticsCanada. Halfof
the30interviewswereconductedintheTorontoareaandtheremaining15tookplaceinthe
region of Ottawa. Fifteen National Census Test questionnaires and 15 2A Census
questionnaireswerecompleted byrespondents duringthesein-depth consultations. The
resultsofthe30in-homeinterviewsconductedinFrenchbyStatisticsCanadaarepresented
in a separate report. The 20 in-depth interviews and the 2 focusgroups conducted in
observation facilities are addressed separately in the Contemporary Research Centre's
forthcomingreport.
CognitiveResearchReportofEnglishInterviews
ThefollowingreportprovidesGeneralCommentstodescribehowrespondentscompletedthe
questionnairesandtofocusoncertainbehavioursobservedduringtheinterviews. Adetailed
report follows inchart form, including Questionnaire Items, Respondents' Problems and
Comments,andRespondents'Suggestions. Accordingtorespondents,suggestedchanges
would improve comprehension levels of census questionnaire items and facilitate the
collectionofaccuratecensusdata.
GeneralComments
1. The2AandtheNCTquestionnaireswereassessedbymostrespondentsasbeingsimple
andstraightforward. Respondentsgenerallyhadnoproblemswiththelegibilityoftheblack
type. Whenaskedtoelaborate,respondentsthoughtthatfontsize,bolding,whitecirclesand
responseboxesmadethequestionnaireeasytocorfiplete.
Theareasthatcausedthemostproblemsonboththe2AandtheNCTformsinvolvedsteps
orquestionswithcomplicatedwording,extensiveinstructions,orthoserequiringrespondents
tomakesubjectivedecisionsorassessments.
2. Respondentsdidnotalwaysdowhatonewouldexpectthemtodowhencompletingthe
censusquestionnaire.
Mostrespondentscompletedtheformasquicklyaspossible. Theybarelyscannedthecover,
searchingforwheretobegin,andthenimmediatelyfocussedonStep 1. Mostsaidthey
continuedtobrowsethroughthequestionnaire,readingaslittleinformationaspossible,and
stoppedreadingaquestionassoonastheyfelttheyhadenoughinformationtoprovidean
answer.
Respondents initially read the firstfew words that directly followed a step or question
number. Forlongersections,theyreadwordsinboldletters,andthenattemptedtoanswer
thequestion. Participantsonlylookedfor,orreadfurtherfor,moreinformationiftheyfelt
theycouldnotimmediatelyanswerthequestion.
3. Theskipinstructionswerefollowedbyabouthalfoftherespondents. Someoftheskips
werefollowedbysomeoftheparticipants,butnotbyothers. Behaviourofrespondentswas
contradictorywhentheydemonstratedorstatedthattheywantedtocompletetheformas
quicklyaspossible,butthentheyreadmorequestionsthanwasrequiredbynotfollowingthe
skip instructions, even when they admitted having read that they could skip. Some
respondentsstatedthattheyreadquestionstheydidnotneedtoreadbecausetheydidnot
wantto"missanything",orbecausetheyweresimplycurious. Althoughrespondentsoften
read questions they did not need to, they also did not read important instructions or
Informationthatwasneededtoaccuratelyansweringcertainquestions.
4. TTiequalityofthedatasufferedwhenrespondentsassumedtheyknewwhatthequestion
wasaskingwithouthavingreadtheinstructions.Thisresultedinproblemsofunderreporting,
suchashotincludingpaidvacationorsickleavewhenrecordingthenumberofweeksworked
inthepreviousyear.
.
5. Thematrixdesign ofthe questionnaire was followed bymost of the respondents.
However,manyrespondentswith3ormorepersonsintheirhouseholdansweredacrossthe
pageratherthandownthematrix. Somerespondentsswitchederraticallyfromusingthe
matrixdownwardformattoansweringacrossthepage,dependingonthecomplexityofthe
questionsbeinganswered. Afewrespondentsfirstquicklyscannedthequestionsoneach
page,answeredthequestionsthatwerethequickestandeasiesttoanswer,andthenwould
gobacktotheremainingquestionsonthepagethatappearedmoredifficulttocomplete.This
explainedafewofthereasonswhyrespondentsfollowedsomeoftheskipinstructionswhile
completelymissingothers.
6. The Guide andthe Census Help Line (CHL) would be used only as a last resort.
RespondentsstatedthatiftheyhaddifficultiestheymightchecktheGuide,buttheywould
ratheraskotherhouseholdmembers,friends,oraneighbourbeforetryingto"getthroughto
someone"bytelephone. Manyrespondentsthoughtitwouldbefrustratingtryingtoreach
StatisticsCanadausingatollfreenumbersincethelinewouldalwaysbebusy.
Whenprompted during theinterviewtochecktheGuide foranswerstotheirquestions,
respondents were frustrated when the Guide was of no help. Some commented that
informationtheyhadalreadybrowsedthroughinthequestionnairewasunnecessarilyrepeated
intheGuide.
MostrespondentsdidnotseetheGuideandCHLnoteatthebottomofpage2,nordidthey
seethenoteinthewhiteareaatthetopsectionofpage4oftheNCTquestionnaire. Afew
peopledescribed their impression when reading the note asmaking them "feel stupid"
suggestingthat"dummieswhoneedhelp"should"usetheGuideorcallustollfree...'. They
suggestedthatthemessagewouldbelessthreateningifitwasre-wordedperhapsas"when
youneedhelp..." oras'Ifyouneedmoreinformation,pleaseusetheGuideorcallustoll
free...*.Thiswouldbelessintimidatingsuggestingthateveryoneratherthanjust"dummies"
needhelpcompletingtheform.
7. Confidentialityandobligation to complete the Census questionnaire was generally
assumed,believedandunderstoodbyrespondents.Afewrespondentsthoughtthat"nothing
isconfidentialanymore"duetonewcomputertechnology.
Noneoftheparticipantswereobservedreadingthesecrecyprovisionsonthebackofthe
censuspackageenvelope. However,whenaskedtoreadthisstatement,respondentswere
reassured, and pleasedtoknowthatdatacollectedand stored byStatistf(::s Canadaare
protectedunderthesecrecyprovisionsoftheStatisticsAct.
8 7770areasofgreatestdifficuhyforrespondentsincluded:
Forboththe2AandtheNCTquestionnaires:
• 2AandNCT,Step2.decidingwhichordertolisthouseholdmembersandwhomtolist
first.
• 2AandNCT,Question 2, "RelationshiptoPerson 1". Mostrespondentshad problems
tryingtodecidehowtorespondtothisquestionforPerson1. Whenaskedtodescribethe