Harnessing the Web for Population-Scale Physiological Sensing: A Case Study of Sleep and Performance

1.9 MB·

by  Tim Althoff

Harnessing the Web for Population-Scale Physiological
Sensing: A Case Study of Sleep and Performance
Tim Althoff∗ Eric Horvitz Ryen W. White Jamie Zeitzer
StanfordCenterforSleep
StanfordUniversity MicrosoftResearch
SciencesandMedicine
[email protected] {horvitz, ryenw}@microsoft.com [email protected]
7 ABSTRACT andapplications.Furthermore,cognitiveperformanceisdecreased
1 Humancognitiveperformanceiscriticaltoproductivity,learning, significantlyafterlossofsleep[20]. Understandingthereal-world
0 andaccidentavoidance. Cognitiveperformancevariesthroughout impact of sleep deficiency is critical. It has been estimated that
2 thecostoffatiguetoU.S.businessesexceeds$150billionayear
each day and is in part driven by intrinsic, near 24-hour circa-
in absenteeism, presenteeism, workplace accidents, poor and de-
b dianrhythms. Priorresearchontheimpactofsleepandcircadian
layeddecision-makingandotherlostproductivityontopofthein-
e rhythmsoncognitiveperformancehastypicallybeenrestrictedto
creasedhealthcarecostsandriskofdisease[24]. Despitetheim-
F small-scale laboratory-based studies that do not capture the vari-
portantinfluences, temporalvariationsofreal-worldperformance
abilityofreal-worldconditions,suchasenvironmentalfactors,mo-
5 are not well understood and have never been characterized on a
tivation,andsleeppatternsinreal-worldsettings.Giventheselimi-
2 largescale[39].
tations,leadingsleepresearchershavecalledforlargerinsitumon-
itoringofsleepandperformance[39].Wepresentthelargeststudy Models of daily patterns in human cognitive performance rely
] typically on representations of three biological processes: circa-
C todateontheimpactofobjectivelymeasuredreal-worldsleepon
dianrhythms(time-dependent,behavior-independent,near24-hour
H performanceenabledthroughareframingofeverydayinteractions
oscillations) [43], homeostatic sleep pressure (the longer awake,
with a web search engine as a series of performance tasks. Our
. the more tired you become) [13], and sleep inertia (performance
s analysisincludes3millionnightsofsleepand75millioninterac-
c tiontasks.Wemeasurecognitiveperformancethroughthespeedof impairmentexperiencedimmediatelyafterwakingup)[4,19].
[ Whilemodelsofthesebiologicalprocessescapturewellthepat-
keystrokeandclickinteractionsonawebsearchengineandcorre-

2 latethemtowearabledevice-definedsleepmeasuresovertime.We terns of cognitive performance in the laboratory [4, 13], they are
basedonexperimentalstudiesinwhichparticipantsaredeprivedof
v demonstratethatreal-worldperformancevariesthroughouttheday
sleepandundertakeregular,artificialtaskstomeasureperformance
3 and is influenced by both circadian rhythms, chronotype (morn-
insteadofnon-intrusivelycapturingperformancethrougheveryday
8 ing/eveningpreference), andpriorsleepdurationandtiming. We
tasks in real-world environments. In addition, these studies typi-
0 developastatisticalmodelthatoperationalizesalargebodyofwork
callyincludeparticipantsthatfitaspecificphysicalandpsycholog-
7 onsleepandperformanceanddemonstratesthatourestimatesof
ical profile (e.g., those with depressed mood are often excluded).
0 circadianrhythms,homeostaticsleepdrive,andsleepinertiaalign
Further, participants in an artificial setting can be influenced by
. withexpectationsfromlaboratory-basedsleepstudies.Further,we
1 their understanding of the study and subconsciously change their
quantifytheimpactofinsufficientsleeponreal-worldperformance
0 behavior to fit the interpretation of its motivation and goals [35].
and show that two consecutive nights with less than six hours of
7 While laboratory studies have been critical in developing under-
sleepareassociatedwithdecreasesinperformancewhichlastfora
1
standingsofthebasicbiologicalprocessesthatunderliecognitive
: periodofsixdays. Thisworkdemonstratesthefeasibilityofusing
v onlineinteractionsforlarge-scalephysiologicalsensing. performance,theyfailtoaccountformyriadinfluencesinthereal-
i world, including motivation, mood, illness, environmental condi-
X
tions,behavioralcompensationincludingcaffeineintake,andsleep
1. INTRODUCTION
r patterns in the wild that are far more complicated than those en-
a Maintainingoptimalcognitiveperformancehasbeenfoundtobe forcedinresearchstudies. Howtheseandotherfactorsalterreal-
importantinlearning[26], productivity[16], andavoidingindus- world cognitive performance is not well understood. Therefore,
trial and motor vehicle accidents [16, 20]. Studies have demon- sleepscientistshavecalledforlarge-scalereal-worldmeasurements
stratedthatcognitiveperformancevariesthroughouttheday[43], ofperformanceandsleepasanecessarystepto“totransformour
likely influencing the quality of our efforts and engagements–in- understandingofsleep”and“toestablishhowtomanagesleepto
cludinghowweuseandinteractwithvehicles,devices,resources, improveproductivity,healthandqualityoflife”[39].
∗ResearchdoneduringaninternshipatMicrosoftResearch. ThisWork.Werespondtotheappealfromthesleepresearchcom-
munitywithalarge-scalestudyofsleepandperformanceenabled
(cid:13)c2017InternationalWorldWideWebConferenceCommittee throughreframingeverydayinteractionswithawebsearchengine
(IW3C2),publishedunderCreativeCommonsCCBY4.0License. asaseriesofperformancetasks. Inparticular, weuseindividual
WWW2017,April3–7,2017,Perth,Australia. keystrokeswhentypingasearchqueryandtheclicksonsearchre-
ACM978-1-4503-4913-0/17/04.
sultsasasourceofpreciselytimedinteractions. Wedemonstrate
http://dx.doi.org/10.1145/3038912.3052637
thatthetimingoftheseinteractionsvariesbasedonbiologicalpro-
cessesandcanbeusedtostudytheinfluenceofdifferentquantities
ofsleeponperformance. Searchengineinteractionsofferinsight
.
aboutreal-worldcognitiveperformanceastheyareanintegralpart tiredwebecome)[13]. Thecircadianrhythmactsinoppositionto
ofmanypeople’slivesandworkeveryday. Morethan90%ofUS thehomeostaticdriveforsleepthataccumulatesacrosstheday,en-
onlineadultsusewebsearchengines,whichnowhandlebillionsof ablingasingle,consolidatedperiodofwakefulnessthroughoutthe
searcheseachday[38]. day. Athirdprocesshasbeenproposedcalledsleepinertia[43],
Our dataset comprises over 3 million nights of sleep tracked whichcorrespondstotheperformanceimpairmentexperiencedim-
by wearable sensors from 31 thousand users over a period of 18 mediatelyafterwakingup[4,19]. Inadditiontotheinfluenceof
months and 75 million subsequent real-world performance mea- dailyrhythmsonthestructureofsleepandperformance,thereare
surementsbasedonkeystrokesandclickswithinawebsearchen- alsoshorter, 90-minuteoscillations, ultradianrhythms, thatorga-
gine(Section3). Thisconstitutesthelargestprospectivestudyof nizetheoccurrenceofNREMandREMstagesduringsleep. Ul-
real-world human performance and sleep to date (more than 400 tradianrhythms,circadianrhythms,andhomeostaticsleeppressure
times larger than the second largest comparable study which had canallimpactthestructure,andlikelyfunction,ofsleep[17].
only76participants[29]). Humanpreferencesandnaturaltendencyintherelativetimingof
Wefirstdemonstratethatreal-worldhumancognitiveperformance sleepandwakearecalledchronotypesandareatleastpartlybased
capturedthroughsearchengineinteractionsvariesthroughoutthe on genetics [40]. Cognitive performance depends on chronotype
dayinadailyrhythm(Section4).Wefindthatperformanceislow- andtimeofday[31]; thatis,early/morningtypes(“lark”)tendto
est during habitual sleep times when it is reduced by up to 31%. be higher performing earlier in the day while late/evening types
Boththeshapeandmagnitudeofthistemporalvariationarecon- (“owl”) are higher performing later. Sleep deprivation has been
sistentwithcontrolledlaboratory-basedstudies,providingvalida- linked to significant decreases in cognitive performance that lead
tionofourlarge-scaleperformancemeasures. Wealsoshowthat toincreasedriskforaccidentsandinjury[20].
performancevariesbasedonchronotype(morning/eveningprefer- A recent study correlated performance on cognitive exercises
ence)withearlyrisersperformingslowestat04:00h(4am)andlate withasleepmeasurebasedonretrospectiveself-reportsof“typical
risersperformingslowestat07:00h. sleep”in160thousandusers[42]. However,thismeasuresuffers
We then develop a statistical model based on chronobiological frompotentialbiases[28]anddoesnotenablethestudyofperfor-
research and demonstrate that it successfully disentangles circa- mancevariationovertimebasedontimeofdayandsleeptiming.
dianrhythms,homeostaticsleepdrive,sleepinertia,andpriorsleep Anotherstudyshowedthatinsomniawithshortsleepisassociated
duration—keyfactorsconsideredinthesleepliterature(Section5). withcognitivedeficitsin678subjects[22]butonlymeasuredasin-
Wequantifythatperformancevariesby23%basedontimeofday, glenightofsleeptocharacterizetypicalsleeppatternsaftertaking
by19%basedontimesincewakeup, andby5%basedonsleep performancemeasurements,leadingtosimilarlimitations.Accord-
duration (Section 5.3). We validate our methodology by demon- ingtoarecentmeta-analysis[29],thelargeststudythatmeasured
stratingcloseagreementbetweenourmodelestimatesbasedona bothsleepandperformanceconcurrentlyhad76participants.
largeamountofperformancemeasurementsinthewildandsmaller
TechnologyUseandInteractionPatterns.Interactionpatternsof
controlledsleepstudiesinartificiallaboratorysettings.
differentdevicesandapplicationshavebeenstudiedonsmallscale
Aftervalidatingourapproach,weextendpriorlaboratory-based
tobetterunderstandmobiledeviceusage[12],todetectstress[44],
sleepresearchthroughestimatesofhowsleepimpactsperformance
usedasbiometricsignalsforauthentication[32],andlinkedtobi-
inreal-worldsettings. Inparticular,wequantifytheimpactofone
ological processes [33, 34] including alertness [1]. For example,
ormultiplenightsofinsufficientsleeponreal-worldperformance
lesssleepwaslinkedtoshorterdurationoffocusofattentionina
(Section 6). We demonstrate that very short and very long sleep
studywith40participants[30]. Large-scaleinteractiondatahave
durations,andirregulartimingofsleepareassociatedwith3%,4%
beenusedtogaininsightsintohumanbehaviorintheareasofmood
and 7% lower performance, respectively. We also show that two
rhythms[23],diet[46],conversationstrategies[5],socialnetworks
consecutivenightswithfewerthansixhoursofsleepareassociated
and mobile games encouraging health behaviors [7, 8, 41], and
withsignificantlydecreasedperformanceforaperiodofsixdays.
healthanddisease-relatedsearchbehaviors[36,47].
Our study is also the first to demonstrate that ambient streams
ofdata, suchaspatternsofinteractionswithdevices, canbehar- ThisWork. Existingresearchonsleepandperformanceiseither
nessed as large-scale physiological sensors to study and continu- small-scaleandlaboratory-based[29]orreliesonsubjectivemea-
ouslyandnon-intrusivelymonitorhumanperformanceatpopula- suressuchassurveyscapturing“typical”sleep[42]whichdonot
tionscale. Theinsightsandmethodologydevelopedinthiswork allowfortemporalcoordinationofsleepandperformancemeasure-
arerelevanttosleepscientistsinpursuitoflarger-scalereal-world ments. Asacomplementandextensionofresearchtodateonper-
measurements of performance, to computer scientists who build formanceinartificiallaboratorysettings,westudyreal-worldcog-
toolsandapplicationsthatmaybeaffectedbyvariationsinhuman nitiveperformancewhichwemeasurethroughinteractionswitha
performance, and to the growing community of researchers who websearchengine. Weuseobjectivemeasurementsofsleep(time
havebeenexploringusesofdatafromonlineactivitiestoaddress in bed) from wearable devices which are preferred to subjective
questionsandchallengesintherealmofpublichealth. self-reportsthatcanbesignificantlybiased[28]andthatenableus
tostudyperformancevariationovertimeinreferencetosleeptim-
ing.Thisworkrepresentsthelargeststudyofobjectivelymeasured
2. RELATEDWORK
sleepandreal-worldperformancetodate,employingasubjectpool
thatisordersofmagnitudelargerthanthelargestcomparableprior
CircadianProcessesinSleepandPerformance. Empiricalstud-
study [29]. Our study demonstrates on a large scale that interac-
ieshavefounddailyrhythmsinhumanperformanceincludingalert-
tionswithdevicesareinfluencedbybiologicalprocessesandsleep.
ness,attention,reactiontime,memory,andhigherexecutivefunc-
tions such as planning [11]. The daily variations in performance
3. DATASET
havebeenfoundtobemodulatedprimarilybytwoprocesses[18]:a
circadianrhythm(time-dependent,behavior-independent,near24- Ourdatasetcontainsover75millionsearchengineinteractions
hour oscillations) [43] and a homeostatic sleep drive (the longer and sleep measurements for 31,793 US users of Microsoft prod-
awake,themoretiredwebecomeandthemorewesleep,theless ucts who agreed to link their Bing searches and Microsoft Band
DatasetStatistics cludea3-axisaccelerometer,gyrometer,andopticalheartratesen-
sor. TheMicrosoftBandemploysinternallyvalidatedproprietary
Observationperiod 18months
algorithmsforestimationofsleepandwefocusondurationoftime
#users 31,793
inbed(hereinreferredtoas“sleepduration”). Timeinbedisde-
#nightsofsleeptracked 3,102,209
lineated either by manual input of the user (i.e., explicit taps on
#queries 24,590,345
thedevicebeforegoingtosleepandimmediatelyafterwakingup)
#filteredquerieswithclicks 6,906,791
orautomaticallybasedonmovementiftheuserdoesnotprovide
#keystrokesextracted 68,779,113
manualinput. Theuseofaneventmarkertodenotebedtimingis
#totalinteractions 75,685,904 widelyusedinsleepresearchinlieuoforinconcertwithsleepdi-
Averagekeystroketime 225ms aries[9]. Followingstandardpractice[45], weexcludeanysleep
Averageclicktime 9.28s durationmeasurementsbelow4andabove12hoursoftimeinbed.
Medianage 38 Asevidencethatoursleepmeasurementshavefacevalidity,we
%female 6.1% showthattheymatchpublishedsleepestimates.Figure1illustrates
%underweight(BMI<18.5) 1.4% averagetimeinbedacrossageandgender. Timeinbeddecreases
%normalweight(18.5≤BMI<25) 32.4% withageandishigherinfemalesthanmalesconsistentwithpub-
%overweight(25≤BMI<30) 39.2% lishedestimates[10,14,45]. Walchetal.[45]reportverysimilar
%obese(30≤BMI) 27.0% timesandadifferenceof17minutesbetweenfemalesandmales.
Mediantimeinbed 7.26h Withtheexceptionof60to70yearoldsubjects,wefinddifferences
between12and17minutes. Thereisnodifferenceforoldersub-
Table1:Datasetstatistics.BMIreferstobodymassindex. jects,whichmatchessurvey-basedestimatesbyBasneretal.[10].
Wetakethesealignmentswithpublishedresearchasevidencefor
the validity of using wearable device-based sleep data for large-
8.0 Male
scalepopulationstudiesofsleepandperformance.
7.8 Female
s)
ur7.6
o
d (h7.4 4. PERFORMANCEMEASURESBASEDON
e7.2
e in b7.0 INTERACTIONSDURINGSEARCH
m6.8 Next, we describe two human performance measures derived
Ti
6.6 from search engine interactions that we use to study daily varia-
6.4 tion in performance. We show how these measures exhibit vari-
(10, 20] (20, 30] (30, 40] (40, 50] (50, 60] (60, 70]
Age ationsinperformanceovertimeandbasedonchronotype(morn-
ing/evening preference) consistent with findings from laboratory-
Figure1: Averagesleepdurationacrossageandgender. Our based sleep studies. This demonstrates that performance signals
measurements are consistent with previous estimates [10, 14, generatedfromeverydaysearchengineinteractionsvarybasedon
45] (Section 3). Error bars in all figures correspond to 95% biologicalprocesses.Wemodeltheseprocessesandinfluencesex-
confidenceintervalsofthecorrespondingmeanestimates. plicitlyinSection5.
4.1 PerformanceMeasures
dataforuseingeneratingadditionalinsightsorrecommendations
about their sleep or activity. Basic dataset statistics and demo- We study two real-world performance measures in this work
graphicinformationontheusersaresummarizedinTable1.Demo- since it is possible that different measures would respond differ-
graphicvariables(age,gender,bodymassindex)areself-reported entlytosleepdeprivationassleepstudieshaveshowndifferential
through the Microsoft Health app. While the user age and over- effectsofsleepdeprivationondifferentmeasuresofcognition.
weight/obesitystatuscloselytrackofficialestimatesintheUnited
KeystrokeTime. Thefirstmeasureisbasedonkeystroketiming.
States,wenotethatoursampleispredominantlymale.
The search engine’s search box registers every single keystroke
Performance.Wemeasureperformancethroughthetimingoftwo and sends a request for query completions to the search engine’s
typesofinteractionswithasearchengine(MicrosoftBing):(1)in- servers. Weusethetimingbetweentwosuchrequestsasthetime
dividual keystrokes within the search box that are tracked by the of a single keystroke if the two queries are different by exactly
search engine so it can automatically suggest query completions, one character (not every request is received on the server side)
and(2)clicksontheresultpageafterasearchquery. Section4.1 andwithintwoseconds(largertimesindicatelongerthoughtpro-
provides more details on each of these measures and we discuss cesses or separate sessions). This threshold is sensible as an av-
howtoaccountforpotentialconfoundssuchasthetypeofquery eragekeystrokebyanaveragetypisttakesabout240milliseconds
in Section 5.1. We exclude search engine interactions originat- (50wordsperminuteat5charactersperword[15]).
ingfrommobiledevicessincesuchinteractionpatternsandtiming
ClickTime. Thesecondmeasureisbasedonthetimetoclickon
arefundamentallydifferentfromthoseondesktopdevices. While
asearchresultafterasearchresultpageisdisplayed. Wemeasure
userscouldpotentiallyaccessthesearchenginefrommultiplema-
thetimebetweenthesearchqueryandthefirstclickonanyresult
chines, wenotethatformostusersthisisunlikelytobethecase
onthefirstpage. Clicktimesovertwominutesareexcludedsince
andthatusingdifferentkeyboardsandmicethroughoutthedayis
they might stem from interrupted sessions. We account for click
unlikelytoexplainthetimingdifferencesobservedinthiswork.
positionandquerytypeasdescribedinSection5.1.
Sleep. Sleepdatafromwearabledevicesprovidesobjectivemea- Webelievethatinvestigatingmeasuresthatcaptureperformance
surementswhichhavebeenpreferredtosubjectiveself-reportsthat ontwodifferenttasksprovidesrobustnessandbreadthtoouranal-
may be significantly biased [28]. To estimate sleep, we consider yses. The two tasks rely on different mixes of sensing, reflec-
signalsfromwrist-wornactivitytrackers(MicrosoftBand)thatin- tion,planning,andformulating,executing,andmonitoringofmo-
(a) (b)
(cid:21)(cid:27)(cid:19)
Worse performance
(cid:21)(cid:26)(cid:19)
(cid:80)(cid:86)(cid:12)(cid:21)(cid:25)(cid:19) (cid:81)(cid:71)(cid:86)(cid:12)(cid:20)(cid:19)(cid:17)(cid:19)
H(cid:3)(cid:11)(cid:21)(cid:24)(cid:19) (cid:70)(cid:82)
(cid:80) H
H(cid:3)(cid:87)(cid:76)(cid:21)(cid:23)(cid:19) H(cid:3)(cid:11)(cid:86) (cid:28)(cid:17)(cid:24)
(cid:78) (cid:80)
(cid:86)(cid:87)(cid:85)(cid:82)(cid:21)(cid:22)(cid:19) (cid:78)(cid:3)(cid:87)(cid:76)
(cid:46)H(cid:92)(cid:21)(cid:21)(cid:19) (cid:38)(cid:79)(cid:76)(cid:70) (cid:28)(cid:17)(cid:19)
(cid:21)(cid:20)(cid:19)
Better performance
(cid:21)(cid:19)(cid:19)
(cid:19) (cid:25) (cid:20)(cid:21) (cid:20)(cid:27) (cid:21)(cid:23) (cid:19) (cid:25) (cid:20)(cid:21) (cid:20)(cid:27) (cid:21)(cid:23)
(cid:43)(cid:82)(cid:88)(cid:85)(cid:86)(cid:3)(cid:86)(cid:76)(cid:81)(cid:70)H(cid:3)(cid:80)(cid:76)(cid:71)(cid:81)(cid:76)(cid:74)K(cid:87)(cid:3)(cid:79)(cid:82)(cid:70)(cid:68)(cid:79)(cid:3)(cid:87)(cid:76)(cid:80)H (cid:43)(cid:82)(cid:88)(cid:85)(cid:86)(cid:3)(cid:86)(cid:76)(cid:81)(cid:70)H(cid:3)(cid:80)(cid:76)(cid:71)(cid:81)(cid:76)(cid:74)(cid:75)(cid:87)(cid:3)(cid:79)(cid:82)(cid:70)(cid:68)(cid:79)(cid:3)(cid:87)(cid:76)(cid:80)H
Figure2:Timeofday-dependentvariationinkeystroke(a)andclicktiming(b).Highervaluesindicateworseperformance.Boththe
shapeoftemporalvariationwithfastestperformanceafewhoursafterwakeandslowestperformanceduringhabitualsleeptimes
aswellasthemagnitudeofvariationareconsistentwithcontrolledlaboratory-basedstudies[3,18,20,48](Section4.2).
time points instead of actual within user variation throughout the
280
Average sleep midpoint day. However,weverifiedthatthetemporalvariationweobserve
270
(0, 5] is due to within user variation throughout the day by confirming
s)260 (5, 7] thatthepatternsoftemporalvariationareeffectivelyidenticalfor
e (m250 (7, 12] rawmeasurementsandwithin-usernormalizedvariants(Z-scores;
m240 onlineappendix[6]). Wealsoverifiedthatperformancevariation
e ti during the weekend is similar to variation during the week (on-
k230
eystro220 ltiwneeeanpppeenrfdoirxm[a6n])ceanddurwinegthweereefkodraeydsoannodtwfueretkheenrddsiffinerethnitsiapteapbeer-.
K210 Finally,weconsideredalternativeperformancemeasuresbasedon
200 backspaceusageinkeystrokesandspellingerrorsinsearchqueries.
190 Sincewefoundresultstobesimilartokeystrokeandclicktiming
0 6 12 18 24 butmorenoisyduetolessfrequentmeasurements,wereportresults
Hours since midnight local time
onkeystrokeandclicktiminginthispaper.
Figure3:Variationinkeystroketimethroughoutthedayvaries 4.2 TemporalVariationofKeystrokeand
withchronotype(morning/eveningpreference)whichisdefined
ClickTimes
based on the average point of mid sleep (Section 4.3). Users
thattypicallysleepearly(lightcolor)performslowestatabout Next,wevalidateourmethodologybyconsideringthefindings
04:00h,whilemediumorlatesleepers(darkercolors)perform obtained from small-scale controlled sleep studies. It is well es-
slowestat05:00hand06:00-07:00h,respectively. Thisclosely tablished that human performance varies over time and follows a
matches their habitual sleep time and is consistent with con- circadian rhythm [3, 48]. Keystroke and click timing also vary
trolledlaboratory-basedstudies[31]. throughout the day in a daily rhythm as illustrated in Figure 2.
Keystroketimes(Figure2a)areontheorderof240milliseconds
which closely matches the expected typing speed of an average
torplans[37]. Studiesofthepotentialsubprocessesforeachtask typist(240milliseconds; 50wordsperminuteat5charactersper
andhowtheymightbedifferentiallyinfluencedbysleepisbeyond word, see [15]). Click times (Figure 2b) are on the order of 10
the scope of this paper. However, our search engine interactions seconds.Notethatbothmeasuresfollowasimilarpatternthrough-
capture performance in everyday tasks that are highly relevant to out the day. Users are fastest to type and click a few hours after
manyoccupations, ascapturedbytypingandsearchingforinfor- typical wake times and the timing increases again in the evening
mation [38], and allow us to non-intrusively measure changes in hours(inparticularforclicktimes).Performanceisslowestduring
real-worldperformancethroughouttheday. habitualsleeptimes(e.g.,04:00h)closelymatchingaccidentrisk
Notethatalltimingmeasurementsaretakenontheserverside rates[20]andtheanticipatedcircadiannadir(i.e.,thetimeofgreat-
and not the client side. Therefore, it is important to consider the estcircadiansleepdrive)[18]. Furthermore,controlledlaboratory
potentialinfluenceofnetworklatencyfactors. Wefoundthatthe experimentshaveshownthatperformancetypicallyvariesby15to
networklatencychangesonlyverylittlebetweentwoconsecutive 30percentoverthecourseofadayacrossavarietyofsimplemotor
requests(lessthan1millisecond)andthusanylatencyeffectscan- andcognitivetasks[3,48]. Forkeystrokeswemeasureavariation
celoutwhenwetakethetimedifferencebetweentworequests(de- of31%andforclicktimesavariationof12%.
tails in online appendix [6]). This demonstrates that variation in The consistent agreement in shape and magnitude of variation
networklatencydoesnotaffectouranalyses. Furthermore,varia- with controlled lab experiments on human performance and for
tionsinsiterenderingtime(i.e.,measuringtimefromfirstscripttill two different tasks suggest that these large-scale measures based
pageloadcompletedincludingdynamiccontents)aremuchsmaller onsearchengineinteractionscanbeusedtostudysleepandper-
(orderofmilliseconds)comparedtovariationinclicktimes. formance.Theproposedmeasurescanbecollectednon-intrusively
Thetemporalvariationsensedinperformancecouldpotentially at unprecedented scale and shine light on how real-world perfor-
be an artifact of different users contributing timings at different mancevariesthroughoutthedayandwithchangesinsleep.
4.3 PerformanceVariationbyChronotype PotentialConfoundingFactors. Wecontrolforseveralfactorsin
A person’s chronotype encompasses the propensity for the in- ourmodeltoavoidconfounding.Forkeystrokes,wecontrolforthe
dividualtosleepataparticulartimeduringa24-hourperiodand exactcharactertypedorremovedsincedifferentcharactersmight
isatleastpartlybasedongenetics[40]. Studieshaveshownthat takeavaryingamountoftime(e.g.,typingan“a”,oracapital“A”,
performancedependsonthealignmentofchronotypeandtimeof orhittingbackspace). Forclicktimes,itisexpectedthatclicking
day [31]; early types tend to be higher performing earlier in the onresultsfurtherdownthelistofresultswilltakemoretime,which
day while late types are higher performing later. The individual holdstrueinourdata(onlineappendix[6]). Wethereforecontrol
chronotype of each user can be defined based on the mid-sleep fortheclickpositioninourmodel.
pointonfreedays(MSF)whichisthehalfwaypointbetweengo- Clicking on a result link is preceded by a cognitive process–
ingtosleepandwakingup[25,40]. Manypeoplecompensatefor interpretingthewordsdisplayedonlinksanddecidingwhichlinkto
sleptdebtaccumulatedduringworkdaysbysleepinglongeronfree click–whichcanbequickinthecaseofnavigationalqueries(e.g.,
days;thatis,thesleepmidpointweobserveislaterthantheinternal “facebook”) or much slower in the case of informational queries
biologicalclockwoulddictateonthefreedays. Therefore, sleep (e.g.,“Whatisthehomeostaticsleepdrive?”). Formally,thisdis-
scientistsuseamidsleeppointthatiscorrectedforoversleep(indi- tinctioncanbecapturedthroughtheconceptofclickentropy,which
catedbySC)[25]:MSF =MSF−0.5(SD −(5∗SD +2∗ measureshow“surprising”thedistributionoverclickedURLsfor
SC F W
SD )/7),whereSD andSD aresleepdurationandfreedays agivenqueryis[21].Wefindthatinformationalqueriestakeabout
F F W
andworkdays,respectively,andSD −(5∗SD +2∗SD )/7 two seconds longer than navigational queries on average (online
F W F
correspondstothedifferenceinsleepdurationonfreedaysandthe appendix [6]). Therefore, we control for the click entropy of the
averageday.Wecomputethiscorrectedmidpointforeveryuserin queryprecedingtheclickinourmodel.
thedatasetusingweekdaysasworkdaysandweekenddaysasfree An extreme way of controlling for varying queries is to com-
days(MedianMSF =4.70). pareclicktimesforexactlyidenticalqueries(e.g.,popularqueries
SC
Weshowthatkeystroketimesthroughoutthedayvarywithchro- suchas“facebook”). Weverifiedthatthisyieldsverysimilarre-
notype(Figure3),matchingresultsfromprevioussleepstudies[31] sults,albeitwithlargerconfidenceintervalssincethesamplesize
andthusprovidingfurthervalidationofourmethods. Wefindthat isreduceddramaticallycomparedtoincludingallqueriesandcon-
earlysleepersareslowestatabout04:00h,whilemediumorlate trollingforclickentropy,demonstratingthattheobservedpatterns
sleepers are slowest at 05:00 h and 06:00-07:00 h, respectively. arenotduetoaparticularmixofquerytypes.
Thiscloselymatcheseachgroup’shabitualsleeptimeanddemon- In addition, we tested for learning effects as issuing the same
stratesthevalidityandpowerofthislargedataset;foreachchrono- querymultipletimesmightleadtoimprovedperformance. How-
typegroup,wehavemillionsofmeasurementsevenduringtypical ever,mostqueries,73.1%,areuniqueinthedatasetandonly4.1%
sleeptimesthatallowustoestimatetheseperformancecurves.We of queries occur more than three times. Further, we did not find
findsimilarresultsforclicktimes. anyevidenceforimprovingperformanceovertimeforfrequently
occurring queries. This is likely because most users were fairly
5. MODELINGPERFORMANCE proficientattypingbeforethestartofourobservationperiod.
Havingdemonstratedthatperformanceofsearchengineinterac-
5.2 MathematicalFormulation
tionsvaryovertimeandbasedonbiologicalprocesses(Section4),
wenowoperationalizeandextendaconceptualmodelofsleepand Wenowdescribetheformulationofthemodelforkeystroketim-
performance from chronobiology [4, 13] to explain the variation ing.Themodelforclicktimesisparallel,wherewecontrolforthe
observedinperformancemeasurements. Classicsleepmodelsare clickpositionandclickentropyinsteadofthekeystroketype. We
based on circadian rhythms and homeostatic sleep drive [13]. In areinterestedinestimatinghow(1)timeofday,(2)timeafterwake
addition,weconsidersleepinertiaandsleepduration[4,43].Back- up,and(3)sleepdurationinfluenceperformance. Weassumethat
groundonrelevantbiologicalprocessesiscoveredinSection2. all these effects are additive as supported by evidence presented
in[2].Mathematically,weformulateafixed-effectsmodel
5.1 ConceptualModel
y =α+fk(xk)+ft(xt)+fw(xw)+fd(xd)+(cid:15) ,
Wemodelthekeystrokeandclicktimingbasedon(1)timeofday i i i i i i
where y is the keystroke time for observation i, α is a constant
inlocaltime,(2)timeinhoursafterwakeup,and(3)sleepduration i
intercept,andfk,ft,fw,fdaretheunknownfunctionsofinterest
the previous night. We know (1) from the time of the keystroke
forkeystroketype,timeofday,timesincewakeup,andsleepdura-
orclicktimemeasurement,and(2)and(3)fromwearabledevice-
tion,respectively,withcorrespondinginputfeaturesxk,xt,xw,xd,
definedsleepmeasurements(Section3). i i i i
and(cid:15) isthei-thresidual.
Sincemanypeoplewakeupduringthesamemorninghoursev- i
Instead of estimating arbitrary functions, we use fine-grained
eryday,timeofdayandtimesincewakeuparenaturallycorrelated
piecewiseconstantapproximations.Wediscretizeeachinputspace
and challenging to disentangle. In laboratory-based sleep stud-
(e.g.,betweenmidnightand01:00h,orbetween01:00hand02:00h,
ies, the goal of exploring the distinct influences of the factors is
or between 0 and 15 minutes after waking up, etc.). We denote
achievedby“forceddesynchrony”protocols[43], wheresubjects
thefunctionsmappinginputfeaturesxt,xw,xdtotheirrespective
aredeprivedofsleepforextendedperiodsoftime. Insteadofsim- i i i
binsasbt,bw,bd (notethatkeystroketypexk isalreadydiscrete).
ilarinterventions,weemploymathematicalmodelingwithalarge- i
Further,weusethefunctionsck,ct,cw,cd tomapthediscretized
scale dataset of real-world sleep and performance measurements
featurestoaconstantvalue.Thesimplifiedmodelthenbecomes
and use the variation observed across millions of observations to
disentangletherelativecontributionsofcircadianandhomeostatic yi =α+ck(xki)+ct(bt(xti))+cw(bw(xwi ))+cd(bd(xdi))+(cid:15)i.
factors. The large-scale dataset contains numerous performance The outcome of interest in this modeling task are the functions
measurements during usual (day) and unusual (late night) times ct,cw,cdwhichexpresstheindependentimpactof(1)timeofday,
(e.g., Figure 3) that we can use to understand the relative contri- (2)timesincewakeup,and(3)sleepdurationonperformancetim-
butionsofthesefactorstoperformanceintheopenworld(seefor- ings the next day. We estimate all parameters (α,ck,ct,cw,cd)
mulationofadditivemodelinSection5.2). including95%confidenceintervalsthroughleastsquaresoptimiza-
(a) (b)
(cid:86)(cid:12) (cid:24)(cid:19) s) 2.5
(cid:80) d
(cid:80)H(cid:3)(cid:11) (cid:23)(cid:19) Hcon 2.0
(cid:82)(cid:78)H(cid:3)(cid:87)(cid:76) (cid:22)(cid:19) mH (s 1.5
(cid:78)H(cid:92)(cid:86)(cid:87)(cid:85) (cid:21)(cid:19) click ti 1.0
(cid:82)(cid:3) o 
(cid:81)(cid:3)(cid:87) (cid:20)(cid:19) n t 0.5
(cid:82) o
(cid:88)(cid:87)(cid:76) uti
(cid:69) b
(cid:81)(cid:87)(cid:85)(cid:76) (cid:19) ntri 0.0
(cid:82) o
(cid:38) C
(cid:237)(cid:20)(cid:19) −0.5
(cid:19) (cid:25) (cid:20)(cid:21) (cid:20)(cid:27) (cid:21)(cid:23) 0 6 12 18 24
(cid:43)(cid:82)(cid:88)(cid:85)(cid:86)(cid:3)(cid:86)(cid:76)(cid:81)(cid:70)H(cid:3)(cid:80)(cid:76)(cid:71)(cid:81)(cid:76)(cid:74)(cid:75)(cid:87)(cid:3)(cid:79)(cid:82)(cid:70)(cid:68)(cid:79)(cid:3)(cid:87)(cid:76)(cid:80)H Hours sincH midnight local timH
(c) (d)
(cid:86)(cid:12) (cid:20)(cid:19) s) 1.5
(cid:80) d
(cid:80)H(cid:3)(cid:11) (cid:19)(cid:24) Hcon 1.0
(cid:82)(cid:78)H(cid:3)(cid:87)(cid:76) (cid:237)(cid:24) mH (s
(cid:82)(cid:3)(cid:78)H(cid:92)(cid:86)(cid:87)(cid:85)(cid:237)(cid:237)(cid:20)(cid:20)(cid:24)(cid:19) o click ti 00..05
(cid:81)(cid:3)(cid:87)(cid:237)(cid:21)(cid:19) n t
(cid:82) o
(cid:69)(cid:88)(cid:87)(cid:76)(cid:237)(cid:21)(cid:24) buti−0.5
(cid:82)(cid:81)(cid:87)(cid:85)(cid:76)(cid:237)(cid:22)(cid:19) ontri
(cid:38) C
(cid:237)(cid:22)(cid:24) −1.0
(cid:19) (cid:23) (cid:27) (cid:20)(cid:21) (cid:20)(cid:25) 0 4 8 12 16
(cid:43)(cid:82)(cid:88)(cid:85)(cid:86)(cid:3)(cid:68)(cid:73)(cid:87)H(cid:85)(cid:3)(cid:90)(cid:68)(cid:78)H(cid:88)(cid:83) Hours aftHr wakHup
(e) (f)
(cid:86)(cid:12) (cid:27) s) 0.3
(cid:80) d
(cid:80)(cid:72)(cid:3)(cid:11) (cid:25) econ 0.2
(cid:82)(cid:78)(cid:72)(cid:3)(cid:87)(cid:76) (cid:21)(cid:23) me (s 0.1
(cid:78)(cid:72)(cid:92)(cid:86)(cid:87)(cid:85) (cid:19) click ti 0.0
(cid:82)(cid:3) o −0.1
(cid:81)(cid:3)(cid:87)(cid:237)(cid:21) n t
(cid:88)(cid:87)(cid:76)(cid:82)(cid:237)(cid:23) utio−0.2
(cid:69) b
(cid:81)(cid:87)(cid:85)(cid:76)(cid:237)(cid:25) ntri−0.3
(cid:82) o
(cid:38) C
(cid:237)(cid:27) −0.4
(cid:23) (cid:24) (cid:25) (cid:26) (cid:27) (cid:28) (cid:20)(cid:19) (cid:20)(cid:20) 4 5 6 7 8 9 10 11
7(cid:76)(cid:80)(cid:72)(cid:3)(cid:76)(cid:81)(cid:3)(cid:69)(cid:72)(cid:71)(cid:3)(cid:11)(cid:75)(cid:82)(cid:88)(cid:85)(cid:86)(cid:12) 7ime in bed (hours)
Figure4:Contributionstokeystroke(a,c,e;blue)andclicktime(b,d,f;red)performanceofdifferentfactorsincludedinourmodel.
Resultsaresimilarforbothperformancemeasuresandmatchestimatesfromcontrolledsleepstudiesinthelaboratory(Section5).
Forexample,variationoverthetimeofdayct(a,b)showsthatperformanceisslowestduringhabitualsleeptimesnearthepresump-
tivecircadiannadir(04:00h;seemaintext). Variationacrosstimeafterwakeupcw (c,d)showseffectsofsleepinertiaduringthe
firsttwohoursafterwake.Thereisrelativestabilityforaroundeighthoursinkeystroketimebutasteadydeclineinclicktimeafter
thatpoint. Sleepdurationscd (e,f)of7.0-7.5hoursareassociatedwithoptimalperformanceaccordingtoourmeasures. However,
notethattheimpactonoverallvariationissmallercomparedtotimeofday(a,b)andtimesincewakeup(c,d).
tion. Wealsoexperimentedwithmixedeffectsmodelscontrolling areillustratedinFigure4. Impactonkeystroketimingsareshown
for variation across users and across queries through random ef- in blue (Figure 4a,c,e) and impact on click times are shown in
fects.Whilestandardmixedmodellibrariesdonotscalewelltothe red (Figure 4b,d,f). Note that the shapes of these functions for
sizeofourdataset,wefoundthatthesemodelsleadtoverysimi- keystrokesandclicktimesareverysimilarandsmooth,eventhough
larestimatescomparedtothefixedeffectsmodeldescribedabove therearenoconstraintsthatwouldforcethistooccur.Furthermore,
whenusingsubsetsofthedata. wenotethatthetemporalvariationincognitiveperformanceisnot
explainedbyvariationindifferentusersthatcontributetimingsat
5.3 Results differentpointsthroughouttheday(i.e.,populationdifferences)but
Thefunctionsct,cw,cdmodelingtheinfluenceoncognitiveper- areduetowithinuservariation(onlineappendix[6]).
formance of time of day, time since wake up, and sleep duration
TimeofDay. Cognitiveperformanceonbothkeystrokeandclick thatusersperformedsignificantlyslowerwheninbedfewerthan
tasks varies with time of day (Figure 4a,b) and is slowest during 6 or more than 9 hours, consistent with the results described in
habitual sleep time around 04:00-06:00 h. Performance quickly Section5.3. Inthoseconditions,theaveragekeystroketimeswere
improvesaftertypicalwaketimesandbecomesslightlyslowerin aboutfourandsevenmillisecondslongercomparedtosleepingbe-
theeveningforbothkeystrokeandclicktimes(19:00h). Thetwo tween 7 and 9 hours (increases of 2.7% and 4.0%, respectively;
curvesconsistentlymatchestimatesofcircadianrhythmprocesses bothp(cid:28)10−10;Mann–WhitneyU-test,whichisusedforallhy-
in sleep obtained through controlled laboratory experiments [18, pothesistestsinthissection).
49]. Notethatthemagnitudeofvariationissubstantialataround Timingofsleepisalsoasignificantfactorforperformancethe
40millisecondsforkeystrokesandover2.1secondsforclicktimes, nextday(Figure5b).Whilesleepingearlierthanusualmakesonly
whicharechangesof18%and23%,respectively,relativetoaver- a difference of about 1 milliseconds or 0.5% (p (cid:28) 10−10), go-
agetimingforeach(Table1). ingtobedanhourormorelaterthanusualisassociatedwithsig-
nificantlyworseaverageperformanceofabout14millisecondsor
Time after Awakening. Cognitive performance also varies sub-
7.3% longer keystrokes (p (cid:28) 10−10). Note that we limited the
stantially with the time after wake up (Figure 4c,d). The magni-
sleepdurationtobebetween7and8hourslongforthisanalysis
tude of the variation is relatively large at about 42 milliseconds
sothattheseresultsdemonstratetheimpactoftimingindependent
or19%forkeystrokesaboutslightlyover1.6secondsor17%for
ofdifferencesinduration(i.e.,thosegoingtosleeplaterhadanor-
click times. Within the first two hours, performance rapidly im-
mallengthoftimeinbeddespitegoingtosleeplate). Wefurther
proves(i.e.,lowertimings).Thisdemonstratesawell-knowneffect
verifiedthattheseresultsarenotduetopeoplesleepinglaterand
in sleep studies called sleep inertia (Section 2). After this point,
longeronweekendswhentheymightbetypingslowerduetoless
performance is best and slowly worsens until a point of poorest
workpressureaswefindsimilarpatternsandeffectsizesusingjust
performanceisreachedataround16hoursofwaketime,consistent
weekdaydata. Thus,theseresultscouldpointtoaninteractionbe-
withthehomeostaticsleepdrive[13]. Thiscorrespondsexactlyto
tween the circadian clock and the ultradian rhythm of sleep (i.e.,
thepointwhenmostpeoplewouldgotosleepagain(i.e.,atypical
the cycling of sleep stages): sleeping at different phases can re-
sleep duration of 8 hours). We excluded data beyond the typical
sultindifferentsleeporganization[17]. Ourfindingssuggestthat
wake period of 16 hours because the data becomes more sparse
sleeping later in one’s circadian cycle does not satisfy the neural
and to avoid potential selection effects with regard to the people
recovery needed for proper daytime performance, while sleeping
whochoosetostayawakeforexceptionallylongperiodsoftime.
earlierdoesnothavethesamenegativeeffects.
However we found similar patterns between both keystrokes and
clicktimesevenbeyondthispoint. Wenotethatkeystroketimeis 6.2 MultipleNightsofInsufficientSleep
relativelystableforaboutsixhourswhileclicktimescontinuously
Above,wereportedontheeffectofasinglenightofsleepwith
increase,likelyduetothedifferencesincognitiveandmotorcom-
particular duration and timing on the next day. Here, we exam-
petencies for the tasks, and due to differences in the sensitivities
inewhethermultipleinsufficientnightsofsleepmeasurablyaffect
ofthosecompetenciestostatusofsleepandcircadianrhythm. In
performanceandhowlongthiseffectappearstopersist. Forpur-
summary,theestimatesderivedfromourmodelcloselycapturethe
poses of this analysis, we define an “insufficient” night of sleep
initialsleepinertiaandtheincreasinghomeostaticsleepdrivefirst
(“I”) to have a time in bed of under six hours (as in [22]), and a
discoveredthroughlaboratory-basedstudies[4,43,49].
“sufficient”nightofsleep(“S”)tohaveatimeinbedofatleastsix
TimeinBed. Keystrokesandclicktimevarywiththeamountof hours. Weconsiderthreedifferentscenarios: twonightsofsleep
timeinbedduringthepreviousnight(Figure4e,f). However,we with more than six hours each (SS), one night over and the next
note that this variation, 12 milliseconds for keystrokes (5%) and nightundersixhours(SI),andtwonightsundersixhoursofsleep
0.25 seconds for click times (3%), is much smaller than the pre- (II).Wemeasuretheperformanceafterthosetwonightsofsleepfor
vious two factors. For both measures, we find a clear U-shaped aperiodofsevendays,reducingtheperformanceoneachofthese
curve with its center, indicating optimal performance, at 7.0-7.5 sevendaystoasinglevalue—theaverageperformanceduringthe
hoursofsleep.Bothsleepingtoolittle(under7hours)ortoomuch first16hoursafterwakeup(i.e.,typicalwakeperiod). Wedonot
(morethan8-9hours)areassociatedwithdecreasedperformance. considerlongersleeppatternshereduetothelargenumberofpos-
U-shapedrelationshipswithrespecttosleepdurationhavebeenre- sible combinations and data reduction associated with individual
portedforseveraloutcomes(e.g.,mortality[27]).Wefurtherinves- sleeppatterns(e.g.,apersonmightnottracktheirsleepeverysin-
tigatetheimpactofinsufficientsleeponperformanceinSection6. gle night). Intentionally not controlling for sleep both preceding
and following the two nights of interest, we are addressing how
insufficientsleepimpactsreal-worldperformancegivenreal-world
6. INFLUENCEOFINSUFFICIENTSLEEP
choices.Wearenot,however,examiningtheunderlyingbiological
ONPERFORMANCE processesofrecoveryfromsleeploss. Wenotethatthestartofthe
Followingourstudiestovalidatethemethodology(Section4and sleeppatternswasdistributedallthroughouttheweek; forexam-
Section 5), we now extend prior laboratory-based sleep research ple, two nights of sufficient sleep (SS) did occur both during the
withestimatesofhowsleepinfluencesperformanceinreal-world weekaswellasovertheweekend. Wedefinerecoverytimeasthe
settings.Inparticular,westudytheimpactofoneormultiplenights numberofdaysittakestoreachperformancelevelscomparableto
ofinsufficientsleeponperformanceoverthefollowingdays. thoseafterasufficientsleepschedule(SS).
Results.Multipleinsufficientnightsofsleephaveasignificantim-
6.1 SingleNightsofInsufficientSleep
pactonaveragekeystroketiming(Figure6). Performanceisbest
We first consider single nights of sleep and analyze how very aftertwosufficientnightsofsleep,slightlybutmeasurablyworse
shortorverylongsleepdurations, aswellasdifferencesinsleep after one insufficient night of sleep, and significantly worse after
timingfromtheusualpatternswithinauser,impactperformance. twoinsufficientnightsinarow.Overthefirst24hours,havingone
Weonlyshowresultsforkeystroketiminghere;theresultsaresim- insufficient night of sleep is associated with 1.2% slower perfor-
ilarforclicktimes(e.g.,Figure2andFigure4). Figure5ashows mance(p (cid:28) 10−10)andtwoinsufficientnightsofsleepare4.8%
(a) (b)
(cid:21)(cid:25)(cid:19) (cid:21)(cid:25)(cid:19)
6(cid:79)HHS(cid:3)(cid:80)(cid:76)(cid:71)S(cid:82)(cid:76)(cid:81)(cid:87)(cid:3)(cid:71)(cid:76)(cid:73)(cid:73)H(cid:85)H(cid:81)(cid:70)H
(cid:21)(cid:24)(cid:19) (cid:21)(cid:24)(cid:19) (cid:73)(cid:85)(cid:82)(cid:80)(cid:3)(cid:68)(cid:89)H(cid:85)(cid:68)(cid:74)H(cid:3)(cid:11)K(cid:12)
(cid:11)(cid:16)(cid:23)(cid:15)(cid:3)(cid:16)(cid:20)(cid:64)
(cid:80)(cid:86)(cid:12)(cid:21)(cid:23)(cid:19) (cid:80)(cid:86)(cid:12)(cid:21)(cid:23)(cid:19) (cid:11)(cid:16)(cid:20)(cid:15)(cid:3)(cid:20)(cid:64)
H(cid:3)(cid:11) H(cid:3)(cid:11) (cid:11)(cid:20)(cid:15)(cid:3)(cid:23)(cid:64)
(cid:80)(cid:21)(cid:22)(cid:19) (cid:80)(cid:21)(cid:22)(cid:19)
H(cid:3)(cid:87)(cid:76) H(cid:3)(cid:87)(cid:76)
(cid:82)(cid:78)(cid:21)(cid:21)(cid:19) (cid:82)(cid:78)(cid:21)(cid:21)(cid:19)
(cid:86)(cid:87)(cid:85) 7(cid:76)(cid:80)H(cid:3)(cid:76)(cid:81)(cid:3)(cid:69)H(cid:71)(cid:3)(cid:11)K(cid:12) (cid:86)(cid:87)(cid:85)
(cid:92) (cid:11)(cid:23)(cid:15)(cid:3)(cid:25)(cid:64) (cid:92)
H(cid:21)(cid:20)(cid:19) H(cid:21)(cid:20)(cid:19)
(cid:46) (cid:11)(cid:25)(cid:15)(cid:3)(cid:26)(cid:64) (cid:46)
(cid:21)(cid:19)(cid:19) (cid:11)(cid:26)(cid:15)(cid:3)(cid:28)(cid:64) (cid:21)(cid:19)(cid:19)
(cid:11)(cid:28)(cid:15)(cid:3)(cid:20)(cid:21)(cid:64)
(cid:20)(cid:28)(cid:19) (cid:20)(cid:28)(cid:19)
(cid:19) (cid:23) (cid:27) (cid:20)(cid:21) (cid:20)(cid:25) (cid:19) (cid:23) (cid:27) (cid:20)(cid:21) (cid:20)(cid:25)
(cid:43)(cid:82)(cid:88)(cid:85)(cid:86)(cid:3)(cid:68)(cid:73)(cid:87)H(cid:85)(cid:3)(cid:90)(cid:68)(cid:78)H(cid:88)(cid:83) (cid:43)(cid:82)(cid:88)(cid:85)(cid:86)(cid:3)(cid:68)(cid:73)(cid:87)H(cid:85)(cid:3)(cid:90)(cid:68)(cid:78)H(cid:88)S
Figure5:Theimpactofsleepduration(a)andtiming(b)onperformancethenextday.Sleeptimingismeasuredthroughdifference
fromthetypicalsleepmidpointandwecontrolforsleepduration.Wefindthatsleepinglessthan7ormorethan9hoursisassociated
withslowerperformance(a). Sleepingearlierthanusualdoesnotmakealargedifferencebutgoingtobedanhourormorelater
thanusualisassociatedwithsignificantlyworseperformancethenextday(b).
ofFigure6thatvisualizesaverageperformancethroughouteachof
226
Sleep pattern thesevendaysisincludedintheonlineappendix[6].
224 SS Notethattheseresultsarenotsimplyduetohavingfundamen-
e (ms)222 SIII tWalhlyiledifsfoemreentuusesersrsacroenmtriobruetelitkoeleyacthoogfetthfeewtheerctuhravnessi(xSSh,oSuIr,sIIo)f.
m220 sleepthanothers,wedofindsimilareffectsbyrestrictingeachof
e ti thethreecurvestobeestimatedfromtheexactsamesetofusers.
strok218 Wenotethat,sinceweenforcenoconstraintsontimeinbeddur-
ey216 ing the seven days following the sleep pattern, additional nights
K
ofinsufficientsleepcouldoccurduringthefollow-upperiod,con-
214 tributingtothedurationoftherecoveryperiod. Thus,weneedto
212 explorewhetherthereisahigherlikelihoodofsleepdeficiencieson
1 2 3 4 5 6 7 days following the initial observed two-day period of insufficient
Days after sleep pattern
sleep. We find that, on average, SS is followed by 0.4 nights of
insufficientsleepduringthefollowingsevendays,whereasSIand
Figure6: Comparingtheimpactonperformanceofzero(SS),
IIarefollowedby1.2and2.5suchnights. Thus,additionaldays
one(SI),ortwo(II)consecutiveinsufficientnightsofsleep(less
ofinsufficientsleepfortheSIandIIcasesmayhaveaninfluence
thansixhoursoftimeinbed). Onenightofinsufficientsleep
on the overall time to returning to baseline performance. Never-
isassociatedwithsignificantlyslowerkeystroketimesandtwo
theless, ourfindingsshowreal-worldtimingofreturntobaseline
insufficientnightsinarowexhibitasignificantlylargereffect.
performance. Weleavetofutureworkthestudyofmorecomplex
Judgingbywhenaveragekeystroketimedropsbelowthehori-
real-worldpatternsofsleepandsleepdeficitandtheinfluencesof
zontaldashedlinerepresentingtheslowestperformanceforthe
sleepdeficitsonperformance.
groupwithtwonightsofsufficientsleep(SS),weobservethatit
takessixnightsofsleeptoreturntobaselineperformancelevels
aftertwonightsofinsufficientsleep(day7)andthreenightsto 7. CONCLUSION
returntobaselineperformancelevelsafteronenightofinsuffi-
Understanding human performance and its relation to sleep is
cientsleep(day4)givenreal-worldsleepschedules.
criticaltoproductivity[16],learning[26],andavoidingaccidents
[16, 20]. Human performance is not constant but exhibits daily
variations [43]. Existing research on sleep and performance has
slower(p (cid:28) 10−10)comparedtotwonightswithlongerthansix
typicallybeenrestrictedtosmall-scalelaboratory-basedstudiesin-
hoursofsleepeach(2.7%and7.3%increasesforclicktimes,re-
volving artificial performance tasks in an artificial environment.
spectively;bothp(cid:28)10−10). Notethattheseeffectestimatestake
Therefore,novelmethodsoflarge-scalereal-worldmonitoring,like
into account any real-world behavioral compensation such as in-
wehavepresented,arenecessarytoadvanceourunderstandingof
creased caffeine intake that will help improve performance after
sleepandperformance[39].
sleeploss. ThehorizontaldashedlineinFigure6correspondsto
theslowestkeystroketimeaftertwonightsofsufficientsleep(SS), Summary of Results. We presented the largest study to date on
which we use as aconservative point of reference to judge when sleepandperformanceinthewild. Usinganewapproachtonon-
performance after insufficient sleep (SI and II) has returned to a intrusive measurement for both cognitive performance and sleep
performance below this point. We find that, on average, it takes we were able to study more than 400 times the number of users
threenightstomakeuponeinsufficientnightofsleep(SIcrosses compared to the second largest study. We correlated human per-
dashedlineonday4)andsixnightstwomakeuptwoinsufficient formancebasedoninteractionswithawebsearchenginetosleep
nightsofsleepinarow(IIcrossesdashedlineonday7). Wefind measures detected by a wearable device. We demonstrated that
verysimilarresultsfortheimpactonthevariance(i.e.,insteadof real-world performance varies throughout the day and based on
mean) of keystroke timing as well as for click times. A version chronotype and prior sleep, in close agreement with small-scale
laboratory-basedstudies.Wedevelopedastatisticalmodelthatop- [7] T.Althoff,P.Jindal,andJ.Leskovec.Onlineactionswith
erationalizesrecentchronobiologicalresearchandshowedthatour offlineimpact:Howonlinesocialnetworksinfluenceonline
estimatesofcircadianrhythms,homeostaticsleepdrive,andsleep andofflineuserbehavior.InWSDM,2017.
inertiacloselymatchpublishedresultsofcontrolledsleepstudies. [8] T.Althoff,R.W.White,andE.Horvitz.Influenceof
Further, we contribute to existing sleep research through quanti- PokémonGoonphysicalactivity:Studyandimplications.J
fying extended periods of lower real-world performance that are MedInternetRes,18(12):e315,2016.
associatedwithsingleandmultiplenightsofinsufficientsleep. [9] S.Ancoli-Israel,R.Cole,C.Alessi,M.Chambers,
Implications.Wehavedemonstratedthathumanperformancecan W.Moorcroft,andC.Pollak.Theroleofactigraphyinthe
be measured in a real-world setting without any additional hard- studyofsleepandcircadianrhythms.Sleep,26(3):342–392,
ware or explicit testing by exploiting existing search engine in- 2003.
teractionsthatoccurbillionsoftimesperday. Wehavevalidated [10] M.Basner,K.M.Fomberstein,F.M.Razavi,S.Banks,J.H.
ourmethodologyandshownthathumanperformance,asmeasured William,R.R.Rosa,andD.F.Dinges.Americantimeuse
throughthesesignals,variesthroughoutthedayandbasedonchrono- survey:sleeptimeanditsrelationshiptowakingactivities.
typeandsleep,incloseagreementwithcontrolledlaboratory-based Sleep,30(9):1085–1095,2007.
studies. Beyondtherelevanceoftheresultstoextendinginsights [11] K.BlatterandC.Cajochen.Circadianrhythmsincognitive
aboutsleepandperformance,ourfindingsmoregenerallyhighlight performance:Methodologicalconstraints,protocols,
thepotentialpowerofharnessingonlineactivitiestostudyhuman theoreticalunderpinnings.Physiology&Behavior,
cognition,motorskills,andpublichealth. Large-scalephysiologi- 90(2):196–208,2007.
calsensingfromonlinedataenables [12] M.Böhmer,B.Hecht,J.Schöning,A.Krüger,andG.Bauer.
• studiesofsleepandperformanceoutsideofsmalllaboratory FallingasleepwithAngryBirds,FacebookandKindle:A
settings,andwithoutactivelyinducingsleepdeprivation, largescalestudyonmobileapplicationusage.InMobileHCI,
• non-intrusive measurement of cognitive performance with- 2011.
outforcingindividualstointerrupttheirworktoperformsep- [13] A.A.Borbély.Atwoprocessmodelofsleepregulation.
arateartificialtasks[39], Humanneurobiology,1982.
• the identification of realistic measures of real-world cogni- [14] BureauofLaborStatistics,AmericanTimeUseSurvey.
tiveperformancebasedonfrequenttasksandinteractions, Averagesleeptimesperday,byageandsex.Archivedat:
• andcontinuousmonitoringofsuchmeasures. http://www.webcitation.org/6lPcEntyS,2015.
Suitableexamplesforsuchdataincludecontinuoususagepatterns
[15] S.K.Card,T.P.Moran,andA.Newell.Thekeystroke-level
from computing applications such as email, programming envi-
modelforuserperformancetimewithinteractivesystems.
ronments, bug report systems, office suites, and others. Any in- CACM,23(7):396–410,1980.
sightsonperformanceandproductivitygainedthroughmonitoring
[16] H.R.ColtenandB.M.Altevogt.Sleepdisordersandsleep
theseapplicationscouldbeusedtoimprovetheuser’sawarenessof
deprivation:Anunmetpublichealthproblem.2006.
suchpatternsandtoadapttheuserexperienceappropriately(e.g.,
[17] D.-J.DijkandC.A.Czeisler.Contributionofthecircadian
schedulingtasksintelligentlyinordertopreventorminimizehu-
pacemakerandthesleephomeostattosleeppropensity,sleep
manerror;schedulingmeetingsbasedonparticipantsperformance
structure,electroencephalographicslowwaves,andsleep
and chronotype profiles). There are great opportunities ahead to
spindleactivityinhumans.J.Neurosci.,15(5):3526–3538,
investigatehowsuchinsightscouldbeusedtopersonalizeapplica-
1995.
tionsbasedonrelevantbiologicalprocessesandchronotypes.
[18] D.-J.Dijk,J.F.Duffy,andC.A.Czeisler.Circadianand
Acknowledgments.WethankJureLeskovec,EmmaPierson,Marinka sleep/wakedependentaspectsofsubjectivealertnessand
Zitnik,DavidHallac,DavidJurgensandtheanonymousreviewers cognitiveperformance.JSleepRes,1(2):112–117,1992.
fortheirvaluablefeedbackonthemanuscript. [19] D.F.Dinges.Areyouawake?Cognitiveperformanceand
reverieduringthehypnopompicstate.InSleepand
8. REFERENCES Cognition,pages159–75.1990.
[1] S.Abdullah,E.L.Murnane,M.Matthews,M.Kay,J.A. [20] D.F.Dinges.Anoverviewofsleepinessandaccidents.J
Kientz,G.Gay,andT.Choudhury.Cognitiverhythms: SleepRes,4(s2):4–14,1995.
Unobtrusiveandcontinuoussensingofalertnessusinga [21] Z.Dou,R.Song,andJ.-R.Wen.Alarge-scaleevaluationand
mobilephone.InUbiComp,2016. analysisofpersonalizedsearchstrategies.InWWW,2007.
[2] P.AchermannandA.A.Borbély.Simulationofdaytime [22] J.Fernandez-Mendoza,S.Calhoun,E.O.Bixler,S.Pejovic,
vigilancebytheadditiveinteractionofahomeostaticanda M.Karataraki,D.Liao,A.Vela-Bueno,M.J.Ramos-Platon,
circadianprocess.BiolCybern,71(2):115–121,1994. K.A.Sauder,andA.N.Vgontzas.Insomniawithobjective
[3] J.Ackerman.SexSleepEatDrinkDream:Adayinthelifeof shortsleepdurationisassociatedwithdeficitsin
yourbody.HoughtonMifflinHarcourt,2008. neuropsychologicalperformance:Ageneralpopulation
[4] T.ÅkerstedtandS.Folkard.Thethree-processmodelof study.Sleep,33(4):459–465,2010.
alertnessanditsextensiontoperformance,sleeplatency,and [23] S.A.GolderandM.W.Macy.Diurnalandseasonalmood
sleeplength.ChronobiolInt,14(2):115–123,1997. varywithwork,sleep,anddaylengthacrossdiversecultures.
[5] T.Althoff,K.Clark,andJ.Leskovec.Large-scaleanalysisof Science,333(6051):1878–1881,2011.
counselingconversations:Anapplicationofnaturallanguage [24] P.Hemp.Presenteeism:Atwork-butoutofit.Harvard
processingtomentalhealth.TACL,2016. BusinessReview,82(10):49–58,2004.
[6] T.Althoff,E.Horvitz,R.W.White,andJ.Zeitzer.Online [25] M.Juda,C.Vetter,andT.Roenneberg.Chronotype
appendixofthispaper.2017. modulatessleepduration,sleepquality,andsocialjetlagin
http://stanford.io/2ejFPhD. shift-workers.JBiolRhythms,28(2):141–151,2013.
[26] P.Kelley,S.W.Lockley,R.G.Foster,andJ.Kelley. [37] J.J.PilcherandA.J.Huffcutt.Effectsofsleepdeprivation
Synchronizingeducationtoadolescentbiology:‘letteens onperformance:Ameta-analysis.Sleep,1996.
sleep,startschoollater’.LearnMediaTechnol, [38] K.Purcell.Searchandemailstilltopthelistofmostpopular
40(2):210–226,2015. onlineactivities.PewResearchCenter.Archivedat:
[27] D.F.Kripke,R.N.Simons,L.Garfinkel,andE.C. http://www.webcitation.org/5I2STSU61,2011.
Hammond.Shortandlongsleepandsleepingpills:Is [39] T.Roenneberg.Chronobiology:Thehumansleepproject.
increasedmortalityassociated?ArchGenPsychiatry, Nature,498(7455):427–428,2013.
36(1):103–116,1979. [40] T.Roenneberg,A.Wirz-Justice,andM.Merrow.Life
[28] D.S.Lauderdale,K.L.Knutson,L.L.Yan,K.Liu,andP.J. betweenclocks:Dailytemporalpatternsofhuman
Rathouza.Self-reportedandmeasuredsleepduration. chronotypes.JBiolRhythms,18(1):80–90,2003.
Epidemiology,19(6):838–845,2008. [41] A.Shameli,T.Althoff,A.Saberi,andJ.Leskovec.How
[29] J.LimandD.F.Dinges.Ameta-analysisoftheimpactof gamificationaffectsphysicalactivity:Large-scaleanalysisof
short-termsleepdeprivationoncognitivevariables.Psychol walkingchallengesinamobileapplication.InWWW,2017.
Bull,136(3):375,2010. [42] D.A.Sternberg,K.Ballard,J.L.Hardy,B.Katz,P.M.
[30] G.Mark,S.T.Iqbal,M.Czerwinski,P.Johns,andA.Sano. Doraiswamy,andM.Scanlon.Thelargesthumancognitive
Neuroticscan’tfocus:Aninsitustudyofonlinemultitasking performancedatasetrevealsinsightsintotheeffectsof
intheworkplace.InCHI,2016. lifestylefactorsandaging.FrontHumNeurosci,7:292,2013.
[31] R.L.MatchockandJ.T.Mordkoff.Chronotypeand [43] H.P.VanDongenandD.F.Dinges.Circadianrhythmsin
time-of-dayinfluencesonthealerting,orienting,and fatigue,alertness,andperformance.Principlesandpractice
executivecomponentsofattention.ExpBrainRes, ofsleepmedicine,20:391–9,2000.
192(2):189–198,2009. [44] L.M.Vizer,L.Zhou,andA.Sears.Automatedstress
[32] F.MonroseandA.Rubin.Authenticationviakeystroke detectionusingkeystrokeandlinguisticfeatures:An
dynamics.InCCS,pages48–56,1997. exploratorystudy.IntJHumComputStud,67(10):870–886,
[33] E.L.Murnane,S.Abdullah,M.Matthews,T.Choudhury, 2009.
andG.Gay.Social(media)jetlag:Howusageofsocial [45] O.J.Walch,A.Cochran,andD.B.Forger.Aglobal
technologycanmodulateandreflectcircadianrhythms.In quantificationof“normal”sleepschedulesusingsmartphone
UbiComp,2015. data.SciAdv,2(5),2016.
[34] E.L.Murnane,S.Abdullah,M.Matthews,M.Kay,J.A. [46] R.West,R.W.White,andE.Horvitz.Fromcookiesto
Kientz,T.Choudhury,G.Gay,andD.Cosley.Mobile cooks:Insightsondietarypatternsviaanalysisofwebusage
manifestationsofalertness:Connectingbiologicalrhythms logs.InWWW,2013.
withpatternsofsmartphoneappuse.InMobileHCI,2016. [47] R.W.White,S.Wang,A.Pant,R.Harpaz,P.Shukla,
[35] M.T.Orne.Onthesocialpsychologyofthepsychological W.Sun,W.DuMouchel,andE.Horvitz.Earlyidentification
experiment:Withparticularreferencetodemand ofadversedrugreactionsfromsearchlogdata.JBiomed
characteristicsandtheirimplications.AmPsychol, Inform,59:42–48,2016.
17(11):776,1962. [48] J.A.Wise,V.D.Hopkin,andD.J.Garland.Handbookof
[36] J.Paparrizos,R.W.White,andE.Horvitz.Screeningfor aviationhumanfactors.CRCPress,2009.
pancreaticadenocarcinomausingsignalsfromwebsearch [49] K.P.WrightJr,C.A.Lowry,andM.K.LeBourgeois.
logs:Feasibilitystudyandresults.JOncolPract,pages Circadianandwakefulness-sleepmodulationofcognitionin
737–44,2016. humans.FrontHumNeurosci,5:50,2012.