Using a "Study of Studies" to help statistics students assess research findings

0.04 MB·

by  Azka Javaid

Using a “Study of Studies” to help statistics students assess
research findings
∗
AzkaJavaid Xiaofei (Susan)Wang NicholasJ.Horton
7
1
January 31,2017
0
2

n
a
J

9
2

]
T
O
.
t
a
t
s
[

1
v
8
3
4
8
0
.
1
0
7
1
:
v
i
X
r
a
∗Addressforcorrespondence: DepartmentofMathematicsandStatistics,AmherstCollege,AC#2239,POBox5000,Amherst,
MA01002-5000. Phone:[phone],email:[email protected]
1
Usinga “StudyofStudies”tohelp statisticsstudentsassess research findings
Introduction
TheAmericanStatisticalAssociation’sUndergraduate GuidelinesWorkgroupintheCurriculumGuidelines
forUndergraduate ProgramsinStatisticalScienceemphasizetheimportanceofstudydesignasanessential
skill needed for undergraduate programs in statistics (ASA 2014). Communication skills are also stressed
alongwithteamworkandcollaboration asessential elements forstatistical practice. Inaddition, theGuide-
lines for Assessment and Instruction in Statistics Education (GAISE)College Report notes the importance
ofconcepts like bias and causal inference instudy design (ASA2016). Toachieve these goals, theGAISE
CollegeReportincludesseveralrecommendations,oneofwhichreiteratestheimportanceoffosteringactive
learning throughdiscussions (p. 18-20).
Thisarticle describes anactivity thatisappropriate forstudents inintroductory andintermediate statis-
ticscourses topractice interpreting research resultsandscrutinizing thedesign andanalysis ofstudies. The
activityincludes acomponent ofgroupworktoimprovestudents’ communication andcollaboration skills.
Traditional textbooks used in the introductory statistics curriculum stress the importance of survey de-
sign. Asanexample,thefourtheditionofIntroStats(IS)byRichardDeVeaux,PaulF.VellemanandDavid
E.Bock,reinforces conceptsrelevanttoexperimental designincluding bias,randomization andsamplesize
inChapter12(“SampleSurveys”)(DeVeaux,Velleman&Bock2013). Additionalsampledesignconcepts
like observational studies, control groups, statistical significance and confounding variables are presented
in Chapter 13 (“Experiments and Observational Studies”) of Intro Stats. Other textbooks (e.g., OpenIntro
Statistics, (Diez,Barr&C¸etinkayaRundel2015))followasimilarapproach.
Howcantextbookreadingsbereinforcedinaclass? Wedescribeanactivitytohelpstudentsexploreas-
pectsofdesign,assessresearchfindingsinpublishedpapers,andcritiquerepresentations andinterpretations
oforiginal research.
The Activity
Study ofStudies
“Study ofStudies” isacolumn regularly published bytheTheAtlantic magazine. Each“Study ofStudies”
analyzes a different topic using published research articles, with full citations provided at the end. Table 1
liststhename,author, anddatefortheallthepastpublished “Studies.”
Forthisactivity,weutilizedoneofthe“StudyofStudies”articlestitled“DinerBeware: Howrestaurants
trickyouintoeatinglessandspendingmore”(http://www.theatlantic.com/magazine/archive/2015/04/d
(Lam 2015). Author Bourree Lam analyzes how restaurants manipulate seating arrangement, server pos-
ture, plate color and size, and music to attract more customers and revenue. Twelve research articles are
2
Title Author Publication Date
YouCanBeTooBeautiful JamesHamblin March2013
TheQueenBee’sGuidetoParenting LindseyAbrams April2013
Various Ways You Might Accidentally Get JamesHamblin May2013
Drunk
The Unexpected Ways a Fetus Is Shaped by a LindseyAbrams June2013
Mother’sEnvironment
TheWorstTimetoHaveSurgery JamesHamblin July/August 2013
IsThereReallySuchaThingasa‘Workaholic’? JordanWeissmann September2013
ViolenceIsContagious RebeccaJ.Rosen October2013
WhyYouLookLikeYourDog SarahYager November2013
HowWomenChangeMen SarahYager December2013
WhoCheats-and Why JulieBeck January/February 2014
WhyYouCan’tKeepaSecret SarahYager March2014
TheOptimalOffice JulieBeck April2014
OurGullibleBrains SarahYager May2014
FunnyorDie JulieBeck June2014
WhatisArt? MatthewHutson July/August 2014
HowtoLookSmart JulieBeck September2014
StatusAnxiety MatthewHutson October2014
KeepingtheFaith EmmaGreen November2014
FakingIt JulieBeck December2014
YouAreJustLikeMe! MatthewHutson January/February 2015
TheSecretofSuperstition MatthewHutson March2015
DinerBeware BourreeLam April2015
WhenEmotionalIntelligence GoesWrong AndrewGiambrone May2015
TheHypocrisyofProfessional Ethicists EmmaGreen June2015
PalmReadingIsReal? EleanorSmith July/August 2015
AScientificLookatBadScience BourreeLam September2015
WhyWeCompete MatthewHutson October2015
TheStrangeOriginsofUrbanLegends MatthewHutson November2015
WhyYouBoughtThatUglySweater EleanorSmith December2015
AStrategicGuidetoSwearing StephanieHayes January/February 2016
PeopleArePrettyBadAtReadingFaces NaomiSharp March2016
CEOsBehavingBadly AlyzaSebenius April2016
HowtoBoastontheSly MatthewHutson May2016
LifeIsn’tFair MatthewHutson June2016
TheScienceofBeerGoggles StephanieHayes July/August 2016
TheCharismaEffect MatthewHutson September2016
DoPeopleNeedSmallTalktoBeHappy? StephanieHayes October2016
HowVotersRespondtoElectoralDefeat BenRowen November2016
WhyKidsNeedRecess AliaWong December2016
Table1: Listofpast“StudyofStudies”published byTheAtlantic
3
summarized in short excerpts with full citations provided as footnotes. Table 2 displays the list of papers
included inthe“DinerBeware”column.
TitleandAuthor Publication
OdorsandConsumerBehaviorinaRestaurant[Gue´guenandPetr InternationalJournalofHospital-
2006] ityManagement
PlateSizeandColorSuggestibility [Ittersum andWansink2012] JournalofConsumerResearch
AssessingtheInfluenceoftheColorofthePlateonthePerception Flavour
of a Complex Food in a Restaurant Setting [Fiszman, Giboreau
andSpence2013]
DiningintheDark[Scheibehenne, ToddandWansink2010] Appetite
The Effect of Musical Style on Restaurant Customers’ Spending EnvironmentandBehavior
[North,ShilcockandHargreaves2003]
The Influence of Background Music on the Behavior of Restau- JournalofConsumerResearch
rantPatrons[Milliman1986]
TheImpactofRestaurantTableCharacteristicsonMealDuration CornellHotelandRestaurantAd-
andSpending[KimesandRobson2004] ministration Quarterly
How a Crowded Restaurant Affects Consumers’ Attribution Be- InternationalJournalofHospital-
havior[Tse,SinandYim2002] ityManagement
LowerBuffetPricesLeadtoLessTasteSatisfaction [Just,Sigirci JournalofSensory Studies
andWansink2014]
Determinants and Consequences of Female Attractiveness and ArchivesofSexualBehavior
Sexiness[Lynn2009]
EffectofServerPostureonRestaurantTipping[LynnandMynier Journal of Applied Social Psy-
1993] chology
EffectonRestaurant TippingofMaleandFemaleServersDraw- Journal of Applied Social Psy-
ing a Happy, Smiling Face on the Backs of Customers’ Checks chology
[RindandBordia1996]
Table 2: 12 papers included in Bourree Lam’s “Study of Studies” on Restaurants and Dining (April, 2015
TheAtlanticMagazine)
Implementation
Thestudents wereprovided withcopies oftheone-page “DinerBeware”column, whichwasreadaloud by
the class. Next, they were split into groups of two to four students and each was provided a copy of one
ofthe twelve research articles cited in “Diner Beware.”Theresearch articles ranged inlength from 4 to28
pages,withanaverage of10pages.
The students were asked to skim the research article and as a group, summarize the original research
study design (i.e., describe the study’s sample design, determine if the study was randomized or observa-
tional). Theywereaskedtoassessthevalidityoftheclaimspresentedinthe“DinerBeware”regardingtheir
research article.
Students a brief set of slides summarizing their original article using RMarkdown (Baumer, C¸etinkaya
4
Rundel, Bray, Loi & Horton 2014). The RMarkdown slides were then shared with the class via RPubs, a
platform for web publishing from RStudio (the slides could be submitted in other ways e.g. by emailing
the instructor). Lastly, students were given 5-10 minutes to present their findings. The student presen-
tations were intended to improve communication skills as well as allow students to gain experience with
technological innovations likeRPubs.
An example of this process can be presented with the research article titled “Odors and consumer be-
havior inarestaurant” (Gue´guen &Petr2006). Gue´guen and Petr’s workanalyzed the effect oflemon and
lavender scents on the duration of timeand the amount of money spent by customers in arestaurant. They
carried their study from 8 pm to 11 pm on three Saturdays in May with 88 patrons and hypothesized that
lavender is considered arelaxing odor whilelemon isastimulating odor. Another example ispresented by
theresearcharticletitled“TheImpactofRestaurantTableCharacteristics onMealDurationandSpending”
(Kimes&Robson2004). Inthispaper,SherriE.KimesandStephaniK.A.Robsonassessedhowtabletype
andtablelocation canaffectaverage spending perminute(SPM)ofacustomer.
Lam summarizes Gue´guen and Petr’s research article with only the following: “particular scents also
have an effect: diners who got a whiff of lavender stayed longer and spent more than those who smelled
lemon, or no scent” (Lam 2015). Similarly Lam provides a terse summary for Kimesand Robson’s article
stating that “Diners at banquettes stayed the longest...Diners at bad tables-next to the kitchen door, say-
spent nearly as much as others but soon fled.” The students were asked to reconcile these statements with
theconclusions presented intheoriginal researcharticles.
Results
The activity was conducted with Introductory and Intermediate statistics students at Amherst College in
Fall 2015 and Spring 2016 academic semesters. The Amherst College Institutional Review Board (IRB)
approved this study. On average, 20-25 students in each class engaged in the activity. Approximately 80
minuteswereallottedtotheactivity.
In summary, students correctly identified basic conceptual elements in the original studies’ designs.
These elements include sample size, the research question, conclusion, and the classification of the study
asobservational orrandomized. Manystudents wereskeptical ofthebriefclaimsabout theoriginal studies
giveninthe“StudyofStudies.”
Forexample, student workcorrectly identified Gue´guen and Petr’ssample size ofpatrons from asmall
pizzeriainBrittany,France. Thestudentsalsodescribehow“lavender,butnotlemon,increasedthelengthof
stay ofcustomers and theamount ofpurchasing,” which indicates that the students’ picked up onGue´guen
andPetr’shypothesis andresearch conclusions.
The students criticized the way the conclusions were portrayed in the “Diner Beware” article; the stu-
dents identify how the “Diner Beware” summary does not account for the possibility of “cultural bias/
geographicalbias.”Geographicalbiasstemsfromthefactthatthestudywasonlyconductedinasmalltown
5
in France and so the conclusions regarding scent and customer spending behavior may not generalize well
topeople ofnon-French heritageorindividuals fromurbanareas.
Intheirarticle,Gue´guenandPetracknowledgethatasmallsamplesizeandtheuseofonlyonerestaurant
are limitations of their study; students picked up on these caveats. Student also recognized how “limiting
the study to three Saturdays in May between 8pm-11pm further creates sampling bias (targets a specific
population).” Daytime and weekday visitors are evidently not represented. Moreover since there was no
replication, itishighlypossible thatanotherfactormayhaveconfounded theresults.
Student analysis of Kimes and Robson’s article also revealed comprehension of the research’s design.
In their analysis, the students correctly identify the sample size of 1,413 and the single-blinded nature of
the study, since in students’ words, “the participants did not know the true nature of the experiment.” The
students expressed skepticism regarding the causal statements made in Lam’s article regarding Kimes and
Robson’s study, considering the observational nature of thestudy and the fact that Kimesand Robson “ex-
cluded some information, like the bar and patio seating” and that they “only took data from busy times.”
KimesandRobson’slimitationsstemfromthefactthattheyonlyusedonerestauranttodrawconclusions, a
shortcomingthatrelatestothelimitationstudentspickedupinregardstothelimitedfocusofthestudy(i.e.,
inattention paid to less busy hours). Another student group summarized the limitation of Lam’s synthesis
as the inability to generalize the original research’s findings since the “conclusion for this specific restau-
rant may not apply to all restaurants.” Students’ propensity for critique allows for challenging conventions
which produces askepticism and curiosity driven outlook. Thisoutlook though may need to challenged or
reconsidered sincestudentskepticism maybeexcessive.
Discussion
We described an activity that linked summaries of research studies with published scientific papers. In
general, students accurately reported the original research’s study design, in particular, the study’s sample
size, whether it was observational or experimental, and the general hypothesis as well as the overarching
conclusions. Students were often critical of the extremely terse representations of the original research by
Lam’s“DinerBeware”articleinTheAtlantic’s“StudyofStudies”column. Thisisnotsurprising giventhat
the goal of the “Study of Studies” is to introduce provocative or idiosyncratic research findings and not to
comprehensively revieworassessthem.
Time-permitting, the instructor might spend some time debunking misplaced criticism, ensuring that
the students have a thorough understanding of the original research, can acknowledge credible published
findings,andnotdevelop“knee-jerk” skepticism.
Overall, this activity was successfully implemented. It raised awareness about study design as well
as secondary representations of original research. The activity can be undertaken with introductory and
intermediate statistics students in a single class period and may help improve communication skills by
fostering discussion about experimental design. We recommend that the study be undertaken after one or
6
more lectures in study design. Conducting the study after few lectures would provide an informal student
assessment andintheprocess, helpreinforce previously-learned studydesignconcepts.
We believe numerous other articles published in the “Study of Studies” column could be utilized in
the same way as the “Diner Beware” article. Depending on student interest, other suitable articles include
“GullibleBrains: Howoursensesinfluenceourthoughts,”“CEOsBehavingBadly: Whatachiefexecutive’s
golfgameandhandwritingsayabouthiscompensationand hisleadership,”“StatusAnxiety: Whatthelogos
you’re wearing really say,” and“TheScience ofBeerGoggles: Alcohol makespeople impulsive, vain, and
uncharitableand itjustmighthelpthemmaintaincommittedrelationships” (seeTable1foracomprehensive
listofcandidate articles).
Acknowledgements
Wewouldlike toacknowledge support from theGregory CallFundfrom theAmherst CollegeDeanofthe
Faculty. WethankTasheenaNarraidoo forherassistance.
References
ASA (2014), ‘Curriculum guidelines for undergraduate programs in statistical science’,
https://www.amstat.org/asa/education/Curriculum-Guidelines-for-Undergraduate-Programs-in-
Statistical-Science.aspx.
ASA (2016), ‘Revised guidelines for assessment and instruction in statistics education (GAISE) college
report’, http://www.amstat.org/education/gaise.
Baumer, B., C¸etinkaya Rundel, M., Bray, A., Loi, L. & Horton, N. J. (2014), ‘R markdown: Integrating a
reproducible analysis toolintointroductory statistics’, Technology Innovations inStatistics Education
8,1–29.
DeVeaux,D.,Velleman,P.F.&Bock,D.E.(2013),IntroStats(4thedition), Pearson.
Diez,D.M.,Barr,C.D.&C¸etinkaya Rundel,M.(2015), OpenIntroStatistics (3rdedition), OpenIntro.
Gue´guen, N. & Petr, C. (2006), ‘Odors and consumer behavior in a restaurant’, International Journal of
Hospitality Management25,335–339.
Kimes, S. E. & Robson, S. K. A. (2004), ‘The impact of restaurant table characteristics on meal duration
andspending’, CornellHotelandRestaurant Administration Quarterly45,333–346.
Lam,B.(2015),‘Dinerbeware: Howrestaurantstrickyouintoeatinglessandspendingmore’,TheAtlantic
April,29–29.
7