Flexible Online Repeated Measures Experiment

0.32 MB·English

by  Yu Guo

Flexible Online Repeated Measures Experiment
Yu Guo Alex Deng
Microsoft Microsoft
OneMicrosoftWay OneMicrosoftWay
Redmond,WA98052 Redmond,WA98052
[email protected] [email protected]
ABSTRACT 1. INTRODUCTION
5 Online controlled experiments, now commonly known as Manyrecentpublicationsattesttothepowerofusingonline
1 A/Btesting,arecrucialtocausalinferenceanddatadriven A/B testing as the golden rule for making causal inference
0 decisionmakinginmanyinternetbasedbusinesses. Whilea inwebfacingcompanieslargeandsmall. Byrandomassign-
2 simple comparison between a treatment (the feature under ment of feature to otherwise balanced groups of users and
test) and a control (often the current standard), provides a measuring subsequent changes in user behavior, A/B test-
n
startingpointtoidentifythecauseofchangeinKeyPerfor- ingisolateseffectoffeaturechange,i.e. thetreatmenteffect
a
J manceIndicator(KPI),itisofteninsufficient,asthechange from extraneous sources of variance.
wewishtodetectmaybesmall,andinherentvariationcon-
2
tainedindatamayobscuremovementsinKPI.Tohavesuffi- Toperformstatisticalinferenceinbothpointestimationand

cientpowertodetectstatisticallysignificantchangesinKPI, hypothesistestingforthetreatmenteffect,whilecontrolling
]
P an experiment needs to engage a sufficiently large propor- typeIerroratpre-specifiedlevel,wewoulddesirelowertype
A tionoftraffictothesite,andalsolastforasufficientlylong II error, or equivalently, higher powered experiments. That
duration. This limits the number of candidate variations is,wewishtobeabletodetecttheeffectwhenthereisany.
.
t to be evaluated, and the speed new feature iterations. We Runningunderpoweredexperimentshavemanyperils. Not
a
t introduce more sophisticated experimental designs, specifi- onlywouldwemisspotentiallybeneficialeffects,wemayalso
s cally the repeated measures design, including the crossover getfalseconfidenceaboutlackofnegativeeffects. Statistical
[
designandrelatedvariants,toincreaseKPIsensitivitywith powerincreaseswithlargereffectsize,andsmallervariances.

1 thesametrafficsizeanddurationofexperiment. Inthispa- Let us look at these aspects in turn.
v perwepresentFORME(FlexibleOnlineRepeatedMeasures
0 Experiment),aflexibleandscalableframeworkforthesede- Whiletheactualeffectsizefromapotentialnewfeaturemay
5 signs. Weevaluatethetheoreticbasis,designconsiderations, notbeknown,wegenerallyselectasizethatmakesbusiness
4 practical guidelines and big data implementation. We com- sense, i.e. one that justifies the cost of feature development
0 pareFORMEtoanexistingmethodologycalledmixedeffect and ongoing maintenance of the code base. Dramatic fea-
0 model and demonstrate why FORME is more flexible and tures that drastically alter user behavior and get reflected
1. scalable. We present empirical results based on both sim- in KPI as large effect sizes are few and far in between. Of-
0 ulation and real data. Our method is widely applicable to tenthecandidatefeaturehasbutasmalleffectontheKPI.
5 online experimentation to improve sensitivity in detecting Nonetheless, by accumulating a portfolio of small changes,
1 movements in KPI, and increase experimentation capabil- a business can achieve big business success. Quote from
: ity. Rule #2 of Kohavi et al. (2014), winning is done inch by
v
inch. This is especially true for mature web facing busi-
i
X CategoriesandSubjectDescriptors nesses where most low hanging fruits were picked already.
r G.3 [ Probability and Statistics]: Experiment Design
a In general one expects variance to decrease with increased
samplesize. Butthisisnotalwaystrue. Foronlinebusiness,
GeneralTerms
at first glance it may seem that the number of visitors may
Measurement, Experimentation Design, Web search, A/B
belarge,andwithacasuallookpeoplemaythinkthepower
Testing
to detect any change is large. In reality, however, intrinsic
variation between users is large and may obscure the small
movement in KPI. Variation in measured treatment effect
comes from various sources. Exogenous to the treatment
itself includes user to user variation, e.g. some users from
slower internet connection would always have slower page
load time regardless of what experiments are run. Variance
for some metrics does not decrease over time, instead they
plateauaftersomeperiodoftime(say,twoweeks),andrun-
ning longer experiments no longer results in corresponding
benefits (see Kohavi et al. 2012, Section 3.4). This poses
a limitation to any online experimentation platform, where within-subject variation. We also discuss practical consid-
fast iterations and testing many ideas can reap the most erations to repeated measures design, with variants to the
rewards. crossoverdesigntostudythecarryovereffect,includingthe
“re-randomized”design (row 5 in table 1).
1.1 Motivation
1.2 MainContributions
Toimprovesensitivityofmeasurement,apartfromaccurate
implementation and increase sample size and duration, we Inthispaper,weproposeaframeworkcalledFORME(Flex-
can employ statistical methods to reduce variance. Using ibleOnlineRepeatedMeasuresExperiment). Wemadecon-
the user’s pre-experiment behavior as a baseline for his/her tributions in both novel application and new methodology.
post-experiment behavior, we can reduce the variance in
measured treatment effect. The experiment setup in a two- • Novel applications. We propose different experiment
weekexperimentisshowninTable1. ThetypicalA/Btest designs with repeated measurement. We demonstrate
is illustrated in the first row. In the past we have used through real examples the value of these new designs
regression to reduce variance (CUPED: Controlled Experi- comparingtotraditionalA/Btest. Methodsformodel
ments Using Pre-Experiment Data, see Deng et al. (2013)) assumption checking is also presented. We also com-
and have achieved good results, e.g. reducing variance in pare different designs for practical use and propose a
numberofqueriesperuniqueuserinagiventimeperiodby general workflow for practitioners.
40-50%. CUPEDhasthebenefitofhavingreadilyavailable • NewMethodology. Wereviewstandardrepeatedmea-
baselinedata“forfree”. Thisimprovementisperformedwith sures models in the framework of mixed effect mod-
existingdesign,usingthe“free”dataascovariatesonlyinthe els. We present a new method to fit the model that
analysis stage. CUPED is in fact a form of repeated mea- is scalable to big data. Our method is flexible in
suresdesign,wheremultiplemeasuresonthesamesubjects the sense that it makes far less assumptions than tra-
are taken over time. In particular, in the pre-experiment ditional method based on mixed effect model (Bates
stage, all users received the default feature C (control) and et al. 2012a). It naturally handles missing data with-
none received the new feature T (treatment). outmissingatrandomassumption(commoninonline
experimentation) and still provides unbiased average
Pre- treatment effect estimation when mixed effect model
Experiment
Experiment fails. FORME canfit different types of repeatedmea-
Groups Week0 Week1 Week2 sures models under the same framework. It also can
1 -T- beappliedtometricsbeyondthosedefinedasasimple
A/B Test
2 -C- average,suchasmetricsdefinedasafunctionofother
CUPED 1 C -T- metrics.
2 C -C-
Parallel 1 T T 2. ILLUSTRATIONOFFORME
2 C C
In this sections we will take a close look at several designs,
1 T C
Crossover with a treatment and a control, and with experiments car-
2 C T
riedoutover several periods. Many commononline metrics
1 C C
2 C T display different patterns between weekdays and weekends.
Re-Randomized
3 T C Therefore experiments at Bing and many large IT compa-
4 T T nies, in general are run for at least a full week to account
for the difference between weekdays and weekends. In the
Table 1: Repeated Measures Designs following section we assume the minimum experimentation
“period”to be one full week, and may extend to up to two
In this paper we extend the idea further by employing the weeks. To facilitate our illustration, in all the derivation
repeated measures design in different stages of treatment in this section we assume all users appear in all periods,
assignment. The traditional A/B test can be analyzed us- i.e. no missing measurement. We also restrict ourselves
ing the repeated measures analysis, reporting a“per week” to metrics that are defined as simple average and assume
treatment effect, as show in row 3“parallel”design in ta- treatment and control have the same sample size. We fur-
ble 1. The two week experiment can be considered to be ther assume treatment effects for each subjects are fixed.
conducted in two periods, even though users received the We emphasis this is just for illustration purpose and our
same treatment assignment during both periods. In one methoddoesnotrelyontheseassumptionsandwedescribe
of the new designs, the“crossover”design, in contrast, we how we handle missing data and more complicated metrics
swaptreatmentassignmenthalfwaythroughtheexperiment in Section 4. Impatient reader who are familiar with re-
(row 4 in table 1). Each user will be exposed to both ver- peated measures analysis might jump over to Section 5 to
sions of the treatments, instead of only one of the two in seedetailsofFORME’smodelassumptionsandcomparison
the usual A/B testing scenario. In sequence, a user will re- to linear mixed effect model.
ceive either T followed by C, or C followed by T, with the
flight re-assignment happening at the same moment for all Denote the metric value mean in the treatment group as
users. Instead of randomizing treatments to users, we ran- µ , and that in control as µ . We are interested in the
T C
dom treatment sequences (TC or CT) to users. This way average treatment effect (ATE) δ = µ −µ which is a
T C
eachuserservesashis/herowncontrolinthemeasurement. fixed effects in the model in this section. This way, various
In fact, the crossover design is a type of repeated measures designs considered can be examined in the same framework
design commonly used in biomedical research to control for and easily compared.
We will proceed to show, with theoretical derivations, that 2.1 TwoSampleT-test
given the same total traffic Let X denote the observed average metric value in control
group and Y denote that in the treatment group. Since
• Variance using CUPED ≤ T-Test usersarerandomlyassignedintoeithertreatmentorcontrol
• With CUPED: Variance in parallel design ≤ Cumula- group, X and Y are thus independent. For simplicity of
tive Design notation,weassumedvarianceinthetwogrouptobeequal.
• Variance in Crossover design ≤ Parallel Design Given large enough sample size, under CLT, and plug in
observed sample variances, we have:
Denoteobservedsamplevaluesinthetreatmentgroupsand
ttiomreofpmereiotrdicsavsalXu(cid:126)e,saXndftohreidriffmeeraennstβt(cid:126)i.mNeopteeritohdastiXn(cid:126)diesxaedvebcy- (cid:104)−CT(cid:105):(cid:104)XY (cid:105)∼N(cid:16)(cid:2)δ+µµ(cid:3),(cid:104)s02X s02Y (cid:105)(cid:17)
i
i,andthetreatmenteffectδcanbeformulatedasafunction where µ is mean metric value in the control group, δ is the
of β(cid:126) depending on model specification. Under the central treatmenteffectcomparedtothecontrolgroup,ands2X and
limit theorem (CLT), with sufficiently large samples X(cid:126) is s2Y are variances of X and Y respectively. Here (cid:126)λ = (µ,δ),
asymptotically normal andthe−2logLikelihood,denotedbyl,ofparametervector
β(cid:126)((cid:126)λ)=(µ,δ)T given observed data is then
X(cid:126) ∼N(β(cid:126),Σ).
l=(cid:104) X−µ (cid:105)T (cid:104)s2X 0 (cid:105)−1(cid:104) X−µ (cid:105)+const
The likelihood of β(cid:126) given observed data is then Y−δ−µ 0 s2Y Y−δ−µ
L= √ 1 exp(cid:0)1(X(cid:126) −β(cid:126))TΣ−1(X(cid:126) −β(cid:126))(cid:1) Solving for MLE of δ and obtain its variance as
2π|Σ|12 2 Var(δˆ)|TTest =s2X +s2Y (1)
To get maximum likelihood estimates (MLE) of β(cid:126), denoted It is simply the sum of variances from the treatment and
by βˆ, we seek to minimize -2log(Likelihood) control groups, which is the asymptotic variance of X−Y.
1
l=− (X(cid:126) −β(cid:126))TΣ−1(X(cid:126) −β(cid:126))+const 2.2 UsePre-experimentDataforVarianceRe-
2
duction
Solving ∂l =0givesMLEofβ(cid:126). Anditsvariance-covariance
∂β(cid:126) At the analysis level, different models seek to explain the
matrix is amountofvariationinobserveddata,whichmaycomefrom
(cid:20) ∂2l (cid:21)−1 intrinsic, within-user difference, as well as variation intro-
Var(βˆ)= =1/I(β(cid:126)) duced by differential treatment. For example, users that
∂β(cid:126)∂β(cid:126)T connect through broadband tend to have faster page load
time than people using dial-up connection. This difference
where Fisher Information
existsregardlessofwhichtreatmentconditionstheusersare
I(β(cid:126))=−E(cid:104)(cid:0) ∂ logf(X|β(cid:126))(cid:1)Tlogf(X|β(cid:126))(cid:12)(cid:12)β(cid:126)(cid:105). exposedto,andisthusirrelevantwhenmeasuringdifference
∂β(cid:126) (cid:12) introduced by different treatments. As a result, the mea-
surements on the same users over time tend be positively
In the following sections we will explicitly model the mean
correlated.
β(cid:126) asafunctionofotherparametersβ(cid:126)((cid:126)λ),oneofthecompo-
nents is treatment effect δ, and study expected variance of
CUPED and previous work has established that by includ-
the MLEs of(cid:126)λ. In fact this is simply:
ing covariates that are unrelated to the treatment, we can
Var(λˆ)=I((cid:126)λ)−1 improve sensitivity and reduce variance of estimated treat-
ment effect. Specifically, the users’ pre-experiment behav-
=(cid:2)(cid:0)∂β(cid:1)TΣ−1E(cid:2)(X−β)(X−β)T(cid:3)Σ−1∂β(cid:3)−1 iorsservers as agood baseline for their behavior duringthe
∂λ ∂λ experiment. Byincludingpre-experimentdataasacovariate
=(cid:2)(cid:0)∂β(cid:1)TΣ−1∂β(cid:3)−1 in the regression model for treatment effect, we can reduce
∂λ ∂λ the variance of the estimated treatment effect.
Coefficientofvariation(CV)definedasthemeanoverstan-
Denote the pre-experiment average metric value to be X
darddeviationofametric,determinesthesensitivityorthe 0
and Y for the later control and treatment groups respec-
power of the experiment, given the same sample size. To 0
tively. By CLT
study sensitivity or power of various experimental designs,
osotnnacbvelaewriaeactrhiooasnvseodfeisffetesatrbiemlnistahtmeeddeateshffuaerticntegffsipezceetrisosoidlzesel,yrw.eemSpcaaeinncisfithrceaelnlalytf,iovctehulyes (cid:34)−CCC(cid:35):XXY001∼N(cid:16)(cid:20)δ+µ+µµµ+θθ(cid:21),[Σ0 Σ0](cid:17),Σ=(cid:104)ρss020s1 ρss021s1(cid:105)
diagonal cell in Var(λˆ) corresponding to treatment effect δˆ T Y1
gives its variance, and is our main focus in the following of where θ is the difference between the pre-experiment and
this section. experimentperiods,i.e. thelongitudinaleffect,andρisthe
correlation between the two periods. Here(cid:126)λ=(µ,δ,θ). We
Analysis from randomized two-group experiments employs assume correlation ρ in both treatment and control groups
thetwosamplet-testundertheusualA/Btestingscenario. to be the same for simplicity. Results do not dependent
Asagentleintroductionwewillfirstlookatthet-testusing onthisassumption. Eventhoughthetwotreatmentgroups
this notation. are still independent, metric value measured on the same
group of users across different time periods are in general In practice, also there is a lot of non-recurring users. We
correlated. As we will later see, this correlation effectively opted to show empirical results instead in results section.
reduces variances on δˆ. Similarly we can solve for MLEs
fromsolvingpartialderivativeofl =0andderivevariances Carefulreadersmayhavenoticed,thismethodmakesakey
for these estimates. assumptionthattreatmenteffectδ remainsthesameinthe
Var(δˆ)| =2s2(cid:0)1−ρ2(cid:1) (2) twoweeks. Tocheckthisassumption,wecanexplicitlytest
CUPED 1 for δ =δ by fitting the model this way:
1 2
It’s easy to see (2) has smaller variance of δˆ than (1) by  
aaenmcrtootusisnmtteimopfee,2rρiio.2eds.21s,.wtiAhtihssunasomenro-szu’enrbtoeihcsaovprirooesrliatitisvioeun.suρTahallemyoacnmognosudinsifftteenort-f EXXYY1212=(cid:20)δ2δµ+1+µ+µθ+µθ(cid:21)
variance reduced is ρ2 that of the original variance. and test for the equivalence of MLEs H : δˆ = δˆ. The
0 1 2
parallel design is appropriate if we fail to reject H .
0
2.3 Cumulativevs. ParallelDesign
Notethatinthepreviousdesignwemakenoassumptionon 2.4 CrossoverDesign
thedurationofthepre-experimentandexperimentperiods. Now with the preliminary background information setup,
Empirical studies in Deng et al. (2013) have shown that we then look at variation reduction achieved through the
usingone-weekpre-experimentdataprovidessimilaramount crossover design.
of variance reduction as using even longer durations. For
simplicity, in practice we recommend using one-week such ThecrossoverdesignemploysasimilarideatoCUPED.In-
data. steadofusingpre-experimentdataasthebaseline,incross-
over experiments, each user is exposed to both treatments
And we have mentioned that to capture the difference be- sequentially, while the order of treatment groups is deter-
tweenweekdayandweekends,werecommendrunningexper- minedrandomly. Eachuser’sbehaviorwhileheorsheison
iments for whole weeks, typically 14 days. Assuming treat- thecontrolconditionservesasabaselineforhisorherbehav-
ment effect is the same across time, this gives us two ways ior on the treatment condition. By accounting for within-
of reporting treatment effects, i.e. reporting cumulative ef- user variation, analysis based on the crossover design also
fectsforthewhole14days,andreportingweeklytreatment reduces variance for the estimated treatment effect.
effect as a weighted average between observed values in the
twoweeks. Forthelatter,usingthesamenotationasabove, Incausalinference,weoftenseektoeliminateanyconfound-
we have ingfactorsandisolatetherootcauseofobserveddifference.
(cid:34)−CCTT(cid:35):XXYY1212∼N(cid:16)(cid:20)δ+µδ++µµ+µθθ(cid:21),[Σ0 Σ0](cid:17),Σ=(cid:104)ρss121s2 ρss122s2(cid:105) DcWoumineesthofirpnaom2t0eo0wb7os)re,krrv(aRinnogdsoetmhnbeiazcauotmuionntaenridsfaucRtsuuebdailntion1tm9h8ae3k;peMottehonregtiacaonlnoaturnotd-l
group as the surrogate for counterfactual. This surrogate
only works on average. In reality often some imbalance in
We can solve for MLE and their variances.
some observed or unobserved factors will remain. Cross-
s2s2(cid:0)1−ρ2(cid:1) overdesignuseseachtestsubjectashisorherowncontrol,
Var(δˆ)| =2 1 2 (3)
Parallel s2+s2−2ρs s thus reducing the influence of confounding covariates, and
1 2 1 2
achieve better sensitivity in estimating treatment effect.
Fortheformer,ifthemetricvalueisstrictlyadditiveacross
time,anexamplebeingrevenue,underourtoymodelwhere
Distribution of observed sample averages is:
all users appear in both periods, the cumulative treatment
effect wo(cid:104)ul−CTd(cid:105)b:e(cid:104)δ˜XY=11++2XYδ22,(cid:105)si∼ncNe (cid:16)(cid:104)2δ2+µ2+µθ+θ(cid:105),[Σ0 Σ0](cid:17), (cid:34)−CCTT(cid:35):XXYY1212∼N(cid:16)(cid:20)δ+µδ++µµ+µθθ(cid:21),[Σ0 Σ0](cid:17),Σ=(cid:104)ρss121s2 ρss122s2(cid:105)
Σ=Var(X1+X2). Similarly, treatment effect estimate has variance
Using (1), variance for the MLE is s2s2(cid:0)1−ρ2(cid:1)
Var(δˆ)| =2 1 2 (5)
Var(δˆ˜)| =2Var(X +X )=2(s2+s2+2ρs s ) Crossover s21+s22+2ρs1s2
Cumulative 1 2 1 2 1 2
(4) Comparing (3) to (5), it is obvious that in the crossover
Comparing coefficient of variation (CV) in (3) to (4), design, treatment effect has smaller variance as long as the
correlation ρ is positive. Similar to CUPED, the amount of
Var(δˆ˜) − Var(δˆ) = (s1+s2)2(s1−s2)2 ≥0 sensitivityimprovementisdeterminedbythesizeofρ. The
4δ2 δ2 2δ2(s21+s22−2ρs1s2) larger the correlation between time periods, the more im-
provementthecrossoverdesignhasovertheparalleldesign.
Equalityholdswhenthetwoperiodshaveidenticalvariance,
Theequivalenceoftreatmenteffectcanbesimilarlychecked
i.e. s = s . In other words, for additive metrics which
1 2
as in section 2.3.
variation over time is large, reporting weekly metrics alone
will improve metric sensitivity.
2.5 AbsoluteorRelativeChange?
For non-additive metrics, such as ratio metrics like Click Sofarinthispaperweconsideredtheabsolutetreatmentdif-
ThroughRate(CTR),thederivationbecomesmoreinvolved. ference δ = µ −µ . In practice we measure thousands of
T C
metrics simultaneously. These metrics may have vastly dif- no carry over effect exists. Usually a“wash-out”period can
ferent magnitude in their treatment effects. Even the same be injected in between treatment periods.
metric measured over different duration, or over different
sample sizes may have different absolute δ’s. This renders 3.1 Wash-outPeriod
comparison of effect size across different experiments diffi-
This approach calls for a“wash-out”period after the end
cult. To overcome this difficulty, we often seek to measure
of the first period, where all users will receive the control.
percent delta, %δ = δ ·100%. The relative change is less
µC Datafromthewash-outperiodcanbeanalyzedinsimilarly
influencedbythebase differenceandisamorerobustmea-
in linear mixed model to estimate the carry over effect and
sure of treatment effect. In online experimentation we usu-
subsequentlyinformthedesignoflaterstage. Weleavethis
ally deal with hundreds of thousands of samples, therefore
as an exercise to the reader.
CLT still holds and relative change would still have asymp-
toticnormality. Theadditivemodeldescribedabovecanbe
3.2 EstimateCarryoverEffect
readilyadaptedtomodelrelativedifferenceinsteadofabso-
lutedifference,byformulatingtheexpectedgroupmeansin Inthecrossoverdesignwhereonlytwogroupsareallocated,
the mixed variance-covariance structure model. For exam- it is not hard to see that potential carry over effect is con-
ple, the crossover model with relative treatment effect can founded with the week to week difference of treatment ef-
be written as: fect. Using only the crossover model, we can only measure
one of these two effects. As an alternative, at the cost of
(cid:34)−CCTT(cid:35):(cid:34)XXYY¯¯¯¯1212(cid:35)∼N(cid:16)(cid:34)µ(µ1(µ+1+µ+δθ)δ+)θ(cid:35),[Σ0 Σ0](cid:17),Σ=(cid:104)ρss121s2 ρss122s2(cid:105) lwfeoesllsomewffiainycgiee4nst-cgiymrogauatpiendtohevseeigrcnat.hrreytroavdeirtioeffneacltnoenxp-clricoistsloyveursidnegsitghne,
Theoreticderivationtoshowvariancereductioncanbecom-
plex, but MLE estimates and their variances can be easily In a 4-group re-randomized design, we conduct the experi-
solved using numeric methods. ment over two periods, and split the users into four equally
sizedgroups,onereceivingcontrolsinbothperiods,onere-
2.6 TheUnifiedTheme ceivingtreatmentsinboth,andonereceivingtreatmentfol-
We illustrated different model designs of FORME. Careful lowed by control, and the last receiving control followed by
readersmightalreadynoticedthattheunifiedthemehereis treatment, we can then tease apart carry over effect and
to study the joint distribution of X(cid:126) and Y(cid:126), which by cen- treatment effect. We call this the re-randomized design, as
trallimittheoremisknowntobemultivariatenormal. Each it is equivalent to having another round of user randomiza-
model specification maps to the mean vector β(cid:126) of this mul- tionbetweenthefirstandthesecondperiod. Usingnotation
tivariatenormal. Thereforeforanymeanvectorbasedona from the linear mixed model, the model considering a po-
model specification, we can solve the MLE and estimate its tential carry over effect α is then
vliaersiainncheowusitnogeFstiismheart’estIhnefocromvaatriioann.ceTmheatdriiffixcinulgtye,nheroawlecvaesre, −CT XX12 δ+µµ+θ
wainrietphnaprotrtiecdsueelnfiacrne,edwofeamsailsssisominnpgeleeddaatasaawavnaeydraitgnoeg.deenFceoidrraeclrwfoohsrsemothveeetrrridwcseestichgaannt, −−CCCT:YYZZ1212 ∼N(cid:16)α+µδδ+++µµµµ+θθ,(cid:20)Σ000 Σ000 Σ000 Σ000(cid:21)(cid:17),
safely assume the treatment effect in both periods are the T W1 δ+µ+θ
same without any carry over effect. We will address these T W2
indetailsinSection3andSection4. Section5explainswhy Σ=(cid:104) s21 ρs1s2(cid:105)
FORME is more flexible and scalable than existing method ρs1s2 s22
of fitting linear mixed effect model. Thecarryovereffectisinthegroupthatreceivedfirsttreat-
mentandthenrevertedbacktocontrolinthesecondstage.
3. CARRYOVEREFFECT Using observed data we can estimate carry over effect α as
an additional term in β(cid:126). When α is not statistically signifi-
Thecrossoverdesignisnotwithoutconcerns. Animportant
cantunderpre-specifiedtypeIerrorcutoff(usually0.05),it
assumption in crossover model is that the treatment effect
remains the same in the experimental periods. Since test issafetodropthetermαfromβ(cid:126),andre-fitthemodel. This
subjectsrandomlyreceiveallcombinationsoftreatmentsin way, we gain one more degree of freedom and thus reduces
sequence, different users will receive difference sequence or variances in the MLEs.
“order”thetreatments. Itispossiblethattheorderinwhich
users are exposed to treatments may change the effect. For Thisapproachenablesdirectestimationofcarryovereffect.
an extreme example, suppose our treatment introduced a It can also be considered a hybrid approach between the
bug that results in severely negative user experience, and crossover and parallel design, as half of the users received
these group of users fail to revisit the website in the later crossed treatments, and half received the same treatments
crossoverperiod,thetreatmenteffectisthendifferentinthe in the two periods. It is not hard to see this design will
two periods. We call this the carry over effect, as the users achieve sensitivity improvement between the crossover and
exposed to treatment first, and then the control later may parallel designs.
behave differently from the other group.
4. MISSING VALUES AND METRICS BE-
In some experiments where the treatment condition is less
YONDAVERAGE
noticeable to the users, the expected treatment effect is
small, and based on historical insight, it is safe to assume 4.1 Lossoffollow-upandintenttotreat
Loss of follow-up is a common term in clinical studies. It ing it as simple average over page level measurements this
refers to patients who were active participants during some needsextracare. Abetterapproachistoseethisasaratio
period of the trial, but stopped participating at some point oftwouserlevelmetrics: clicksperuserandpage-viewsper
of follow-up. This can lead to incomplete study results and user.
potential bias when the attrition is not random. Intention- (cid:80) (cid:80) (cid:80)
Clicks Clicks / I
to-treatanalysisiscommonlyemployed,whenthesubjects’ (cid:80)useri i = (cid:80)useri i (cid:80)useri i,
Pages Pages / I
initial assigned treatment is used regardless of actual re- useri i useri i useri i
ceived treatment. In online A/B testing the idea is similar; where (cid:80) I is the the count of appeared users. The
usersareassignedtotreatmentgroupsatsomepointintime same deltuas-emrietihod mentioned in Section 4.1 naturally ex-
before the experiment starts, often by user id, but may or tends here, with slightly more complicated formula. Since
may not appear in the actual duration of the experiment. delta-methodappliesingeneraltoanycontinuousfunction,
Thismissingpatternisfarfromrandom,thereforemethods we can handle any metric that is defined as a continuous
that rely on strong MCAR assumption (missing completely function of other metrics.
at random) are not appropriate and even MAR (missing at
random) assumption is questionable as it requires missing
5. FLEXIBLEANDSCALABLEREPEATED
pattern to be random conditioned on observed covariates.
MEASURESANALYSISVIAFORME
One way tosee measurements are notmissingat randomis
to realize infrequent users are more likely to have missing 5.1 ReviewofExistingMethods
values and the absence in a specific time window can still Itiscommontoanalyzedatafromrepeatedmeasuresdesign
provideinformationontheuserbehaviorandinrealitythere with the repeated measures ANOVA model and the F-test,
might be other factors causing user to be missing that are undercertainassumptions,suchasnormality,sphericity(ho-
not even observed. Instead of throwing away data points mogeneity of variances in differences between each pair of
where user appeared in only one period and is exposed to within-subject values), equal time points between subjects,
only one of the two treatments, in practice, we included an and no missing data. Such assumptions in general do not
additionalindicatorforwhetherornottheuserappearedin hold for large-scale online experiments, where the assign-
the study in the period. ment of users into different treatment group may not be
completely balanced.
Specifically, we use an additional indicator for the pres-
ence/absence status of a user in each experimentation pe- A generally more applicable method is to analyze the data
riod. For user j in period i, let Iij =1 if user j appears in usingthelinearmixedeffectmodel,forwhichcompletebal-
period i, and 0 otherwise. For each user j in period i, in- ance is not necessary (Bates et al. 2012a). In particular, a
steadofonescalarmetricvalue(Xij),weaugmenteditinto linear mixed effect model treat each measurement of a sub-
a vector (Iij,Xij). When Iij = 0, i.e. user is missing, we ject as a data point, and model the measurement as
defineX =0. Underthissimpleaugmentation,themetric
ij
valueX forperiodi,takingaverageoverthosenon-missing Y =θ+αX+βZ+(cid:15)
i
measurements,isthesameas (cid:80)(cid:80)kkXIiikk. Inthisconnection,to Here θ is the global mean and α stands for the vector of all
obtain MLE and its variance, we only need to estimate the deterministicfixedeffectswhileβisthevectorofallrandom
covariance matrices for each group across time periods, i.e. effects and (cid:15) is noise. X and Z are covariates in the model.
Cov(Xi,Xi(cid:48))=Cov(cid:18)(cid:80)(cid:80)kkXIiikk,(cid:80)(cid:80)kk(cid:48)(cid:48)XIii(cid:48)(cid:48)kk(cid:48)(cid:48)(cid:19) IpAnesroiaoundrsecoxafastmehspeltmeh,eeyaosnuaerreepmoisnesnditbic,laeutsomerrosiddoe,flatnfroderaartnmeypeeonatthteeadrsscmiogvenaamrsiuearnteets.,
=Cov(cid:18)Xi,Xi(cid:48)(cid:19) using lme4’s formula syntax (Bates et al. 2012a;b) is
Ii Ii(cid:48) Y ∼1+IsTreatment+Period+(1|UserID),
wherethelastequalityisbydividingbothnumeratorandde-
where the only difference of this model to the usual lin-
nominatorbythesametotalnumberofuserswhohaveever
ear model behind two sample test is the extra random ef-
appeared in the experiments. Thanks to the central limit
fect(clustered by UserID) to model user “baseline”. More
theorem, the vector (Ii,Xi,Ii(cid:48),Xi(cid:48)) is also asymptotically complicated models exist to further model interaction and
normal. Plugging in observed sample means and covari-
joint random effects.
ance matrix, Cov(Xi,Xi(cid:48)) can be trivially computed with
the delta-method; see Deng et al. (2013, Appendix B) for
Random effect makes modeling within-subject variability
a similar example; also see (Van der Vaart 2000) for a text
possible. In repeated measures data, users might appear in
book treatment of the delta-method.
multipleperiods,representedasmultiplerowsinthedataset.
As a result, rows of the dataset are not independent but
4.2 MetricsBeyondAverage withdependenciesclusteredbyuser. Toseethis,eachuser’s
Treatment groups are assigned to users, but not all metrics “baseline”measurementiscapturedasarandomeffect. The
are simple averages across users. We can define a metric as same user in different period will share the same“baseline”
a function of other metrics. One important family of met- random effect, therefore resulting in dependency. Mixed ef-
rics is page level metrics such as click through rate. Page fect model effectively takes advantage of this and is able to
level metrics use number of page-views as their denomina- estimatethevarianceoftherandomeffectwhilereducingthe
tor. At first glance it might look like just a simple aver- variance of average treatment effect. In the case of cross-
age. Sincetreatmentsareassignedtousers(theindependent over design, the model can take advantage of the positive
unit), page-views are therefore not independent. Consider- correlationbetweenthetwoperiodsofthesameuser,which
improvesaccuracyintheestimationoftreatmenteffect,sim- linear mixed effect model. In our experience FORME is 1
ilartotheillustrationwederivedinSection2.4. Treatment to 2 magnitudes faster than lme4 with much less memory
effect can be modeled as either a fixed effect or random ef- footprint even without map-reduce type parallelism.
fect1. If our interest is the average treatment effect, we can
model it as a fixed effect. Note that modeling treatment In the remaining of this section, we explain why FORME
effect as fixed effect does not mean we need to assume it is is both more scalable and flexible than linear mixed effect
fixed,whichingeneralisnotsincedifferentsubjectsreactto model.
the treatment differently, but rather because the focus here
is the mean of the random treatment effect, not the vari- 5.2 FORMEisScalable
ance of the random treatment effect. One can still fit the
Insteadofmodelingatthelevelofeachindividualmeasure-
model with random treatment effect and the results gener-
ments, FORME sees the problem from a higher level and
ally agree, though fixed effect is believed to be more robust
takeadvantageofbigdata. Basedoncentrallimittheorem,
against model assumptions; see Wooldridge (2012).
metrics of interest in each period for treatment and control
followsnormaldistribution. UsingthesamenotationinSec-
We point out two issues of using traditional mixed effect
tion2,thismultivariatenormalrandomvectorisdenotedby
model, and claim that FORME is a better alternative on
X ,Y ,i=0,...,2, with mean β(cid:126)((cid:126)λ) and certain covariance
axes of flexibility and scalability. i i
matrix. These metric values are correlated with each other
via common user level random effects modeled explicitly in
First, linearmixedeffectmodel(andalsogeneralizedlinear
linear mixed effect model but not in FORME. This is be-
mixedeffectmodel)isafamilyofparametricmodels,andre-
cause when our interest is only in the average treatment
lies on full knowledge of the likelihood function to perform
effect, the estimates of those random effects are irrelevant.
parameter fitting. This means the model need to rely on
Instead,FORMEseestheaveragetreatmenteffectδ asjust
distributionalassumptionssuchasnormality. Inparticular,
all random effects are typically modeled as normally dis- oneparameterinthemeanvectorofthemetricvaluesβ(cid:126)((cid:126)λ).
tributed or jointly normally distributed. And noise (cid:15) need That is, when modeling metric values directly using mul-
to be either i.i.d normal or the modeler needs to provide a tivariate normal distribution with parameters in the mean
knowncovariancematrix. Theseassumptionsareindispens- vector, all the complexities involving the structures of the
able in the theory and pivotal in the fitting of the model. randomeffectsareburiedinthecovariancematrixofmulti-
For our application in online A/B Testing, many of these variate normal and we are left with a simple task, which is
assumptionsareinappropriate. Tonameafew,forametric to estimate the parameters(cid:126)λ of this multivariate normal.
like revenue per user, it is inappropriate to model the user
“baseline”revenue per week as normally distributed due to FORMEestimates(cid:126)λbyfittingMLE.Theuseofasymptotic
itslargeskewness. Alsothenoiseterm(cid:15)ishardtojustifyto statistics also guarantees that the estimates are normally
betrulyindependentofotherrandomeffect. Aheavieruser distributedwithcovariancematrixderivedfromFisher’sIn-
mighthave bigger“baseline”revenue, andalsobigger noise, formation(VanderVaart2000). Notethatthescaleofthis
and bigger (or smaller in some cases) treatment effect. It stepismuchsmallerthantheMLEfittingofatypicallinear
alsoassumesdataaremissingatrandom. Modelersoflinear mixedeffectproblem. FORMEonlyneedtofitamultivari-
mixedeffectmodelwillneedtomodifythemodelbymaking ate normal with small dimension, typically smaller than 12
randomeffectsjointlyrandom,orincludingmoreinteraction (6 in a crossover design: treatment and control for each of
terms. However the more complicated the model, the more the pre-experiment, period 1 and period 2.)
questionsonmodelassumptionswillarise. WeshowinSec-
tion 6 through simulation study, linear mixed effect model Themaincomputationburdenisthereforeintheestimation
fitted in R package lme4(Bates et al. 2012a) could result in of covariance matrix. Fortunately, this step only involves
biasedestimationoftheaveragetreatmenteffectwhenthere estimation of pair-wise covariance between metric values,
iscorrelationbetweendatamissingpatternanduserrandom andtheyallcanbemap-reducedwithonepassofthedata.
effect. To handle missing data and general form of metrics (as a
continuous function of other metrics), delta-method can be
Second,fittingmixedeffectmodelcouldbeexpensive. Avail- employed(Section4). Theapplicationofdelta-methodonly
able packages in SAS or R are based on fitting MLE or involves slightly more complicated covariance matrix so we
REML(restrictedmaximumlikelihood). Ineithercase,much need to estimate more covariance pairs in one map-reduce
effort is taken to estimate the variance of random effect(s) pass of the data, inducing negligible increase in complexity.
or covariance matrix if they are jointly random. Fitting al-
gorithm takes the full dataset with each row representing a 5.3 FORMEisFlexible
measurement. InonlineA/Btesting,wheretensofmillions
FORMEisnotonlyscalablebutalsomoreflexible. Because
of users are involved, this dataset could be large. In model
FORME doesn’t explicitly model random effects as linear
fitting, each iteration requires some operations on this full
mixed effect models do, FORME makes no distributional
dataset. Making the efficiency of model fitting a concern
assumptions on random effects and noises (cid:15). FORME also
in big data scenario. To the authors’ best knowledge, there
make zero assumption on missing data pattern. FORME
is no literature on the topic of big data implementation of
needs only one critical assumption , i.e. that central limit
theoremisapplicable,whichisrarelyviolatedinonlineA/B
1When there are only two measurements for a subject like
crossover design, modeling treatment effect and user“base- testing, since traffic size is large enough even for the most
line”bothasrandomeffectisunidentifiable. Butthemodel highlyskewedmetricssuchasRevenue(Kohavietal.2014).
can be fit if there are more measurements per subject. Specifically, FORME can be applied to all these cases:
1. Data can have arbitrary missing pattern. In other and all conditions have roughly 50% user missing in each
words not assumptions on missing at random. period.
2. Treatment Effect is random.
3. TreatmentEffectanduserrandomeffect(baseline)are First of all in condition 3 when there is a random treat-
not independent. ment effect, we found lme4 consistently gave biased esti-
4. Noises (cid:15) are not i.i.d. mation(when ground truth effect is 6.6, FORME estimates
5. Noise and random effects are not independent. are very close to ground truth while lme4 always gave bi-
6. Interactions. (e.g. treatment and control have differ- ased estimation around 7.2). This is because lme4 relies on
ent user random effect distribution, etc.) the assumption of missing at random, and it is violated as
randomeffectisnegativelycorrelatedtothechanceofmiss-
To close this section, we make the final remark that the ing. We believe this is a fundamental problem issue with
flexibility of FORME really comes from its simplicity, com- mixed effect model as missing data pattern is often corre-
paring to linear mixed effect model. We believe FORME lated with some underlying user characteristics that is cor-
is also easier for practitioners to understand. The cost of relatedwithuser’sresponsetotreatment. Onemightargue
FORME to put less assumptions than mixed effect model thatmixedeffectmodelcanbefixedbythrowingmoreinter-
is the expectation that when mixed effect model assump- actionterms. Howeverinpracticemorecomplexmodelsare
tions hold, FORME estimate could possess larger variance often not identifiable (parameters more than data points)
than mixed effect model estimate. Next we’ll explore these andtheyonlymakesmoreassumptions. Wealsonotedthat
through simulation study. exceptforcondition3,lme4providedunbiasedestimatesfor
condition 2, 4 and 5 where some assumptions in mixed ef-
6. RESULTS fect model are violated. We believe central limit theorem
also helped in this case for lme4 to stay unbiased. But bias
6.1 SimulationfromKnownDistributions
in condition 3 seems to be more fundamental. We leave a
WecomparevariancesreportedfromourFORMEproduces more thorough study of the bias of lme4 with violation of
to the traditional linear mixed model under various simu- different assumptions in future work.
lation assumptions. As illustration we used the crossover
design. We simulate a total of 2N users, where N =10000 WealsocomparedvarianceinLMEandFORMEunderthe
and randomly split them into two treatment groups. crossovermodelbelowinFigure1. BothFORMEandlme4
X =µ+δ +u +(cid:15) ,(cid:15) ∼N(0,σ2) provided very good estimation of variance. And also as ex-
ij ij i ij ij
pected FORME pays a price for its flexibility and almost
whereiisindexforuserandjfortimeperiod. (cid:15)ij represents “model free”as variance from lme4 estimations are gener-
random noises and ui represents random user“baseline”ef- allysmaller. Thevariancegapisbiggerwhenmissingrateis
fect. δij isthetreatmenteffectforuseriinperiodj (0ifnot higher and between-periods correlation is higher. Although
intreatment). Inthismodel,thebetweenperiodcorrelation not shown, in the conditions when either there is no miss-
is then σ2 . If user i is in treatment for time period j, ingdata,orthecorrelationis0,FORMEandlme4estimates
σu2+σ2 havethesamevariance. Althoughlme4estimatehassmaller
δ ∼N(δ,σ2)×p ,whereδ×E(p )isthegroundtruthaver-
ij δ i i variance,itspotentialbiasisashow-stoppersincefortreat-
agetreatmenteffectsize,p isacontinuousvaluebetween0
i ment effect estimation a low variance estimate is not useful
and1,anditrepresentstheuser’sactivitylevel. Wedesigned
if biased.
p tobecorrelatedtou . Thiswayweallowtreatmenteffect
i i
tovarybyhowfrequentauservisitsthesite. Finallyweal-
lowX tobemissingwithprobabilityofmax(90%,1−p ).
ij i
This is intuitive since a less active user would be missing
more often. Note that in this simulation study we know
exactly what the true average treatment effect is. We sim-
ulated this process K = 10000 times so that we can have
a good estimate of the ground truth variance of treatment
effectestimatedbyFORMEandmixedeffectmodel(lme4).
We want to learn the following for both FORME and lme4
from this simulation study: 1) is estimate unbiased, 2) is
variance estimation correct. If both methods are unbiased,
thenwewanttoknowwhichonehassmallervariance. With-
out loss of generality, we used µ = 0, σ = 4, σ = 2 or 4,
√ u
δ=10,andσ =0.1 12. Wechose5simulationconditions
δ
as the following:
Figure 1: Effect Variance in LME and FORME
1. Normalnoise,notreatmenteffect,normaluserrandom
effect u ∼N(0,σ2) 6.2 SimulationfromEmpiricalData
i u
2. Normal noise, no treatment effect, Poisson user ran- Next we randomly sampled from our in-house data a small
dom effect u ∼Poisson(σ2) subset of N = 1250 users, randomly split the users into
i u
3. Normal noise, with random treatment effect that is equal sized subsets, and applied various designs. We then
correlated with user random effect: N(δ,σ2)×p simulate K = 10000 bootstrap samples (with replacement)
δ i
4. NoiseiscorrelatedwithUseractivitylevel: σ=2×p from this dataset, fit FORME and report estimated MLEs.
i
5. NoiseiscorrelatedwithUseractivitylevel: σ=4×p The variance based on these MLEs are then compare to
i
the variance estimated from Fisher Information using the
full dataset. Figure 2 shows the two agrees well. Note the
cumulativeeffecthaddifferenteffectsizefromtherestofthe
designs. Forthisparticularmetric,usingCUPEDresultsin
roughly 50% reduction in variance. Crossover design shows
reduction of around 50% compare to the parallel design.
Figure 3: Percent samples needed to achieve the
same sensitivity for four metrics. Baseline is the
crossover design.
significanceinKPI,wemaychoosetoterminatetheexperi-
mentalready. Howeverinpractice,wegenerallyrecommend
Figure 2: Effect Variance from Fisher’s Information runningtheexperimentslongenoughtogainenoughpower
and Bootstrap method for not only the KPI, but other metrics designed to moni-
tor data quality and serve as guardrail against unexpected
changes.
6.3 RealExperiments
Otherwise, in running the second stage, we can use domain
Finally, we report results from three typical metrics in one
knowledge to inform about carry over effect. If historical
of our real experiments. Here we used percent change as
experiments in similar feature iterations indicate potential
the effect size. This way, weekly effect size is comparable
carryovereffect,werecommendrunningacomplete4-group
to cumulative effect size across two weeks. The variance of
crossoverexperiment,sowecandirectlyestimatecarryover
effect size therefore indicates sample size needed to achieve
effect. Otherwise, we recommend using the 2-group cross-
the same sensitivity. Figure 3 displays the percent samples
over design to achieve the maximum power for KPI. If we
needed to achieve the same sensitivity for three metrics us-
are not sure, it is still possible to leave a few days’“wash-
ing various models, with the crossover design as baseline.
out”period after completing the first stage, and see if any
Therefore crossover design had value of 100. All models
carry over effect can be observed.
included CUPED since pre-experiment data always exists
andisfree. Thecrossoverdesignconsistentlyhadthefewest
• No swapping: When it is critically important to en-
samplesneeded. Nextthere-randomizedesignhadvaluebe-
sure consistent users experience, such as changing the
tween crossover and the parallel design. Cumulative design
entirelayoutofasite,itmaynotbedesirabletoshow
follows. When the re-randomize model includes a leftover
usersthenewsiteforaweek,andthenswapthemback
effect, the samples needed can be larger than cumulative
to the old site. The experience may be too jarring to
design for metric 2. Note that compared to the previous
users and hurt the brand. In such cases, we do not
benchmark,thecumulativedesign,thecrossoverdesigncan
recommend re-assigning treatment variants half way
save up to 2/3 the traffic for metric 3, while for others, the
through.
traffic savings is in the 30-40% range. This is due to inher-
• Crossover: Relatively small changes that are less di-
ent difference in week to week correlation in different met-
rectly noticeable are better candidates for treatment
rics. Notethedrasticreductioninvarianceforsuchmetrics
swapping. If similar experiments from the past, or
means the same feature can be tested with only 1/3 of the
earlierexplorationdatadonotindicatethepresenceof
original traffic!
carryovereffect,thecrossoverdesigncanbeemployed.
• Re-randomized: If we suspect the presence of car-
7. PRACTICALCONSIDERATIONS
ryover effect, the re-randomized design enables us to
At the design stage, we face a few choices under the same measure it directly and should be used here.
frameworkofrepeatedmeasuresdesign. Experimentersshould • Wash-out and decide: If we have little informa-
use domain knowledge and past experiments to inform the tion to judge carry over effect, we can run the first
design. This is rather an art than pure science. Here we week of the experiment, and then leave a few days as
give guidelines according to our own experience. a “washout” period. The next stage is data driven.
Using such data we can estimate the carry over effect
7.1 RecommendedWorkFlow explicitly.
Due to the flexibility in a two-stage setup in repeated mea- – Ifthereisnosignificantcarryovereffect,proceed
sures design, we can use the information gathered in the as the crossover design.
first stage to inform procedures in the next stage. We rec- – Otherwise, proceed as the re-randomized design.
ommend using crossover design as validation stage experi-
ments, for which we already have gathered exploratory di- Having collected experiment data, they can then analyzed
rectional data. If the first stage already result in statistical in the following work flow to achieve the most power.
No swapping: 8.1 Extendingtomorefrequentswaps
– Test equivalence of treatment effect across time The crossover design achieves sensitivity by exposing users
– Iftheyareequivalence,reporttreatmenteffectin to both treatment variants in sequence, by swapping the
the“pertimeunit”metricvaluesbyanalyzingus- treatment assignment once during the experiment. Using
ing the parallel model, including pre-experiment each subject as his or her own control and this design to
data. account for within-subject variance. A natural extension
– Otherwise, analyze only cumulative effects, and of the idea is to swap treatment groups more than once.
includingpre-experimentdata. NotethisisCUPED. Essentially, this changes to a more granular randomization
Crossover design: unit,fromuserstopageviews. Exploratoryworkshowsthis
– Test equivalence of treatment effect across time indeed achieves further variance reduction.
– Iftheyareequivalence,reporttreatmenteffectin
the“pertimeunit”metricvaluesbyanalyzingus- However,thisalsoraisestheconcernforinconsistentuserex-
ingthecrossovermodel,includingpre-experiment perience,diminishedtreatmenteffectsize,strongerlearning
data. effect,andlackofalongertermmeasure. Despitethesecon-
– If, however, unexpected significant difference is cerns,itremainsavaluableoptioninearlystageexperiments
found, you have several choices to quickly select promising features for further iteration.
∗ Report the two treatment effects separately
∗ To understand the difference properly, an-
8.2 Limitationsandconcerns
otherphraseoftheexperimentcanbeadded,
using re-randomized design. With a total of Duetouserbehaviordifferencesbetweenweekdayandweek-
three weeks’ data, we can see whether the ends,weusuallyrecommendrunningeachphaseofthecross-
treatmenteffectdifferenceisduetotrueweek- over design for at least a full week. A crossover experiment
to-week to difference, and study its trend, or thenrequirestwocompleteweekstogatherdata,whichhin-
due to carry over effect. dersagility. Anotherlimitationisthatforveryhighlyvisible
Re-randomized design: featureslikechangingprominentUIfeatures,suchswapping
– Test equivalence of treatment effect across time maynotbedesirablesinceitmayconfusetheusers. Finally,
and presence of carryover effect not all features can be tested this way, as there might be
– Reducethemodelifanyoftheeffectsarenotsta- a“learning”effect, where we can’t have the users exposed
tistically significant, and report treatment effect. to treatment “unlearn” the feature, while having controls
naivetothetreatment. Forexample,ifthewebsiteprovides
This carries the subtle difference of reporting a treatment new features and personalized content to signed in users to
effect in the entire duration of the experiment, versus that encourage higher rate of signing in and staying signed in.
pertimeunit(aweekhere). Wearguethataslongasweekly Theseuserscannotthenbeforcedtologoutintothecontrol
treatment effects are stable over time, reporting weekly ef- group. Ma et al. (2011) shows one interesting case where
fect is intuitive, easy to understand, and easy to compare crossover design can be extended to tackle this issue.
across different experiments. In real life, various things can
happen during an experiment, and we may end up with an
8.3 FurtherImprovementsofFORME
experimentthatranonlyinpartialweeks. Inthesecases,re-
We’veshowninSection6.1thatmixedeffectmodelvialme4
portingtreatmenteffectintheentiredurationwillbebetter
provides a competing estimate of the average treatment ef-
than throwing away data or ignore weekdays difference.
fect that could be biased when missing data pattern corre-
latewithuserrandomeffect,butoftenwithsmallervariance
7.2 SampleSizeConsiderations
thanFORME.WenotedthatFORMEhastopaysomeprice
Whiledirectestimationinsamplesizeisdifficultinthelinear
to be more flexible and robust, similar to nonparametric
mixed model, in practice there is an easy work around. In
model usually is less efficient than their parametric coun-
the traditional design, using CLT, with a simple two-sided
terparts. However we suspect that efficiency of FORME
test for H :δ=0, sample sizes can be easily calculated.
0 can be further improved to match the efficiency of mixed
δ2 effectmodelevenunderperfectmixedeffectmodelassump-
n=(z +z )2/
1−α/2 1−β Var(δ) tion. Such improvement would be very desirable. But even
without such improvement we believe the bias when there
where α is the allowed false positive rate, usually 0.05, and
is missing data that is not missing at random is a big issue
1−β is the desired power, usually at 80% to 90%.
for mixed effect model to be adopted in online controlled
experiment. And FORME should be used instead.
From historical data we can record the amount of variance
reduced for each metric. The magnitude is determined by
inherent variance in the metric, and correlation across time References
periods,bothofwhichareobservedtobefairlystableacross Bakshy,E.andFrachtenberg,E.(n.d.),‘StatisticsandOpti-
many experiments. Suppose the variance for metric X in malDesignforBenchmarkingExperimentsInvolvingUser
crossover experiment is k% that in the conventional t-test. Traffic’.
IfN subjectsarerequiredtodetectachangeofδ%int-test Bates, D., Maechler, M. and Bolker, B. (2012a), ‘Fitting
with,say,80%power,thenk%N isthereducedsamplesize linear mixed-effects models using lme4’, J. Stat. Softw.
to achieve the same power. p. 51.
Bates,D.,Maechler,M.andBolker,B.(2012b),‘lme4: Lin-
8. DISCUSSIONSANDFUTUREWORK ear mixed-effects models using S4 classes’.