Speculative Approximations for Terascale Analytics

0.66 MB·English

by  Chengjie Qin

Speculative Approximations for Terascale Analytics
ChengjieQin FlorinRusu
UniversityofCalifornia,Merced
5200NLakeRoad
Merced,CA95343
{cqin3,frusu}@ucmerced.edu
5 December2014
1
0
2
Abstract
n
a Modelcalibrationisamajorchallengefacedbytheplethoraofstatisticalanalyticspackagesthatareincreasingly
J
usedinBigDataapplications. Identifyingtheoptimalmodelparametersisatime-consumingprocessthathastobe

1 executedfromscratchforeverydataset/modelcombinationevenbyexperienceddatascientists. Wearguethatthe

incapacitytoevaluatemultipleparameterconfigurationssimultaneouslyandthelackofsupporttoquicklyidentify
]
B sub-optimalconfigurationsaretheprincipalcauses.
Inthispaper,wedeveloptwodatabase-inspiredtechniquesforefficientmodelcalibration.Speculativeparameter
D
testingappliesadvancedparallelmulti-queryprocessingmethodstoevaluateseveralconfigurationsconcurrently.The
.
s numberofconfigurationsisdeterminedadaptivelyatruntime,whiletheconfigurationsthemselvesareextractedfrom
c adistributionthatiscontinuouslylearnedfollowingaBayesianprocess. Onlineaggregationisappliedtoidentify
[
sub-optimalconfigurationsearlyintheprocessingbyincrementallysamplingthetrainingdatasetandestimatingthe

1 objective function corresponding to each configuration. We design concurrent online aggregation estimators and
v definehaltingconditionstoaccuratelyandtimelystoptheexecution.
5 We apply the proposed techniques to distributed gradient descent optimization – batch and incremental – for
5 supportvectormachinesandlogisticregressionmodels. WeimplementtheresultingsolutionsinGLADEPF-OLA
2 –astate-of-the-artBigDataanalyticssystem–andevaluatetheirperformanceoverterascale-sizesyntheticandreal
0 datasets. Theresultsconfirmthatasmanyas32configurationscanbeevaluatedconcurrentlyalmostasfastasone,
0 whilesub-optimalconfigurationsaredetectedaccuratelyinaslittleasa1/20thfractionofthetime.
.
1
0
5 1 Introduction
1
:
v Big Data analytics is a major topic in contemporary data management and machine learning research and prac-
i tice. Many platforms, e.g., OptiML [5], GraphLab [50, 51, 32], SystemML [4], SimSQL [53], Google Brain [28],
X
GLADE[42,14]andlibraries,e.g.,MADlib[29],Bismarck[19],MLlib[17],VowpalWabbit[2],havebeenproposed
r
a toprovidesupportfordistributed/parallelstatisticalanalytics.
ModelcalibrationisafundamentalproblemthathastobehandledbyanyBigDataanalyticssystem. Identifying
theoptimalmodelparametersisaninteractive,human-in-the-loopprocessthatrequiresmanyhours–ifnotdaysand
months–evenforexperienceddatascientists. Fromdiscussionswithskilleddatascientistsandourownexperience,
weidentifiedseveralreasonsthatmakemodelcalibrationadifficultproblem. Thefirstreasonisthattheentireprocess
has to be executed from scratch for every dataset/model combination. There is little to nothing that can be reused
from past experience when a new model has to be trained on an existing dataset or even when the same model is
appliedtoanewdataset. Thesecondreasonisthemassivesizeoftheparameterspace—bothintermsofcardinality
anddimensionality. Moreover,theoptimalparameterconfigurationisdependentonthepositioninthemodelspace.
Andthird,parameterconfigurationsareevaluatediteratively—oneatatime. Thisisproblematicbecausethecomplete
evaluationofasingleconfiguration–evensub-optimalones–cantakeprohibitivelylong.
1
Motivating example. Gradient descent optimization [7] is a fundamental method for model calibration due to its
generalityandsimplicity. Itcanbeappliedvirtuallytoanyanalyticsmodel[19]–includingsupportvectormachines
(SVM),logisticregression,low-rankmatrixfactorization,conditionalrandomfields,anddeepneuralnetworks–for
which the gradient or sub-gradient can be computed or estimated. All the statistical analytics platforms mentioned
previouslyimplementoneversionoranotherofgradientdescentoptimization. Althoughthereisessentiallyasingle
parameter, i.e., the step size, that has to be set in a gradient descent method, its impact on model calibration is
tremendous. Findingagood-enoughstepsizecanbeatime-consumingtask. Moreso,inthecontextofthemassive
datasetsandhighly-dimensionalmodelsencounteredinBigDataapplications.
The standard practice of applying gradient descent to model calibration, e.g., MADlib, Vowpal Wabbit, MLlib,
Bismarck,illustratestheidentifiedproblemsperfectly. Foranewdataset/modelcombination,anarbitrarystepsizeis
chosen. Modeltrainingisexecutedforafixednumberofiterations. Sincetheobjectivefunctioniscomputedonlyfor
the result model – due to the additional pass over the data it incurs – it is impossible to identify bad step sizes in a
smallernumberofiterations. Theprocessisrepeatedwithdifferentstepvalues,chosenbasedonpreviousiterations,
until a good-enough step size is found. Certain systems, e.g., Google Brain, support the evaluation of multiple step
sizesconcurrently. Thisisdonebyexecutingindependentjobsonamassivecluster,withoutanysortofsharing.
Problemstatement. Weconsidertheabstractproblemofdistributedmodelcalibrationwithiterativeoptimization
methods, e.g., gradient descent. We argue that the incapacity to evaluate multiple parameter configurations simul-
taneously and the lack of support to quickly identify sub-optimal configurations are the principal causes that make
model calibration difficult. It is important to emphasize that these problems are not specific to a particular model,
butrathertheyareinherenttotheoptimizationmethodusedintraining. Thetargetofourmethodsistofindoptimal
configurations for the tunable hyper-parameters of the optimization method, e.g., step size, which, in turn, facili-
tatethediscoveryofoptimalvaluesforthemodelparameters. Therefore, weinvestigatespeculativeprocessingand
intra-iterationapproximationsfortheoptimizationofdistributedmodelcalibration. Speculativeiterationprocessing
allows concurrent evaluation of multiple parameter configurations in a single pass over the training data—without
theproportionalincreaseinrunningtime. Intra-iterationapproximationallowsforfasterconvergencedetection—and
corresponding reduction in iteration time. When put together, these techniques have the potential to reduce model
calibrationtimesignificantly.
Contributions. In this paper, we develop two database-inspired techniques for efficient model calibration. Specu-
lativeparametertestingappliesadvancedparallelmulti-queryprocessingmethodstoevaluateseveralconfigurations
concurrently. Online aggregation is applied to identify sub-optimal configurations early in the processing by incre-
mentallysamplingthetrainingdatasetandestimatingtheobjectivefunctioncorrespondingtoeachconfiguration.
Our major contribution is the conceptual integration of parallel multi-query processing and approximation for
efficient and effective large-scale model calibration with gradient descent methods. This requires novel technical
solutions as well as engineering artifacts in order to bring significant improvements to the state-of-the-art. Specific
contributionsinclude:
• We design speculative parameter testing algorithms that evaluate multiple configurations simultaneously. The
number of configurations is determined adaptively and dynamically at runtime. The configurations are drawn
fromaparametricdistributionthatiscontinuouslyupdatedusingaBayesianstatistics[23]approach.
• Wedesignconcurrentonlineaggregationestimatorsanddefinehaltingconditionstoaccuratelyandtimelystop
theexecution. Weprovideefficientparallelsolutionsfortheevaluationofthespeculativeestimatorsandoftheir
correspondingconfidenceboundsthatguaranteefastconvergence.
• WeapplythesetechniquestodistributedgradientdescentoptimizationforSVMandlogisticregressionmodels.
Asaprerequisite,weformalizegradientdescentoptimizationasadatabaseaggregationproblem.
• Weprovideanextensivecomparisonbetweenbatchandincrementalgradientdescentthatrevealsthat–contrary
tothegenerallyacceptedopinion–batchgradientdescentmethodsarebettersuitedfordistributedprocessing
overmassivedatasetsduetotheirlinearityproperties. Withtheproposedtechniquesintegrated,batchgradient
descentalwaysoutperformstheincrementalmethodbothinconvergencespeedandexecutiontime.
• WeimplementtheproposedsolutionsinGLADEPF-OLA[42,14,40,37]–astate-of-the-artBigDataanalytics
system–andevaluatetheperformanceoverterascale-sizesyntheticandrealdatasets.Theresultsconfirmthatas
2
manyas32configurationscanbeevaluatedconcurrentlyalmostasfastasone,whilesub-optimalconfigurations
can be detected in a 1/20th fraction of the time. These translate in more than 100X faster convergence over
MLlibonSpark[17]and2XoverVowpalWabbit[2]onaterascale-sizerealdataset.
Outline. The model calibration problem is presented in Section 2. Gradient descent solutions are introduced in
Section 3, while distributed gradient descent is discussed in Section 4. Speculative parameter testing is presented
in Section 5. Intra-iteration approximation with online aggregation is detailed in Section 6. Experimental results
that evaluate thoroughly the efficiency and efficacy of the proposed methods and compare the GLADE PF-OLA
implementation against the state-of-the-art are presented in Section 7. Section 8 discusses relevant related work,
whileSection9concludesthepaper.
2 Problem Formulation
Considerthefollowingmodelcalibrationproblemwithalinearlyseparableobjectivefunction:
N
(cid:88)
Λ(w(cid:126))=minw∈Rd f(w(cid:126),x(cid:126)i;yi)+µR(w(cid:126)) (1)
i=1
in which a d-dimensional vector w(cid:126), d ≥ 1, known as the model, has to be found such that the objective function is
minimized. The constants x(cid:126) and y , 1 ≤ i ≤ N, correspond to the feature vector of the ith data example and its
i i
scalarlabel,respectively. Functionf isknownasthelosswhileRisaregularizationtermtopreventoverfitting. µis
aconstant. Forexample,theobjectivefunctioninSVMclassificationwith-1/+1labelsandL -normregularizationis
1
givenby(cid:80) (cid:0)1−y w(cid:126)T ·x(cid:126)(cid:1) +µ||w(cid:126)|| .
i i i + 1
Gradientdescentrepresents,byfar,themostpopularmethodtosolvetheclassofoptimizationproblemsgivenin
Eq.(1). Gradientdescentisaniterativeoptimizationalgorithmthatstartsfromanarbitrarypointw(cid:126)(0) andcomputes
new points w(cid:126)(k+1) such that the loss decreases at every step, i.e., f(w(k+1)) < f(w(k)). The new points w(cid:126)(k+1)
aredeterminedbymovingalongtheoppositeΛgradientdirection. Formally,theΛgradientisavectorconsistingof
(cid:104) (cid:105)
entriesgivenbythepartialderivativewithrespecttoeachdimension,i.e.,∇Λ(w(cid:126))= ∂Λ(w(cid:126)),...,∂Λ(w(cid:126)) . Computing
∂w1 ∂wd
the gradient for the formulation in Eq. (1) reduces to the gradient computation for the loss f and the regularizer R,
respectively. Thelengthofthemoveatagiveniterationisknownasthestepsize, denotedbyα(k). Withthese, we
canwritetherecursiveequationcharacterizinganygradientdescentmethod:
(cid:16) (cid:17)
w(cid:126)(k+1) =w(cid:126)(k)−α(k)∇Λ w(cid:126)(k) (2)
Inordertocheckforconvergence,theobjectivefunctionΛhastobeevaluatedatw(cid:126)(k+1)aftereachiteration. Conver-
gencetotheglobalminimumisguaranteedonlywhenΛisconvex.Thisimpliesthatboththelossf andtheregularizer
Rareconvex.
ThespecificproblemweconsiderinthispaperishowtosolvetheoptimizationformulationgiveninEq.(1)using
genericgradientdescentmethodswhenthetrainingdatasetconsistingofN(vector,label)pairs{((cid:126)x ,y ),...,((cid:126)x ,y )}
1 1 N N
ispartitionedintoM subsets. Eachsubsetisassignedtoadifferentprocessingnodeforexecution. Wefocusonthe
case when N is extremely large and each of the M subsets are disk-resident. Gradient and loss computation can
takeaprohibitiveamountoftimeinthiscase. Moreover,gradientdescentmethodsarehighlysensitivetoaseriesof
parametersthatrequirecarefultuning,i.e.,repeatedexecutionwithdifferentconfigurations,foreverydataset. Thus,
noveltechniquesarerequiredinordertoscalegradientdescenttothelargestBigDatamodels.
3 Gradient Descent Methods
In this section, we introduce the basic gradient descent optimization algorithms. We start with the standard batch
gradientdescentalgorithmwhichisadirectimplementationofthetheory. Then,wepresentincrementalorstochastic
3
gradientdescent,apopularalternativetailoredfordatasetscontainingalargenumberofexamples.Inadditiontothese
primitivemethods,wealsodiscusstwoderivedalgorithms—coordinatedescentandL-BFGS.Weconcludethesection
withacomparisonofthetwostandardgradientdescentapproaches.
3.1 BatchGradientDescent
Thepseudo-codeforgradientdescentisgiveninAlgorithm1. Thisisalsoknownasbatchgradientdescent(BGD).
Thealgorithmtakesasinputthedataexamplesandtheirlabels,thelossfunctionf andthegradientoftheobjective
∇Λ, and initial values for the model and step size. The optimal model is returned. The main stages are gradient
computationandmodelandstepsizeupdate. Theyareexecuteduntilconvergenceisachieved. Convergencecanbe
specified as a fixed number of iterations or based on the loss, e.g., the loss difference across consecutive iterations
decreasesbelowagiventhreshold. Inthelatercase,thelosshastobecomputedaftereveryiteration,whichincursan
additionalpassoverthedata.
Model and step size update. The standard approach to compute the updated model w(cid:126)(k+1), once the direction
of the gradient is determined, is to use line search methods [8]. w(cid:126)(k+1) is found by iteratively trying different step
sizesalongtheoppositegradientdirectionuntilthedecreaseinlossisaboveauser-definedthreshold,i.e.,theWolfe
conditions[8]. Linesearchachievesatradeoffbetweenoptimalityandruntimebychoosingonlyanapproximationto
theoptimalstepsize,butinshortertime. Nonetheless,linesearchisstilliterativeinnatureandrequiresobjectiveand
gradientlossevaluation. Theseinvolvemultiplepassesovertheentiredata. Awidelyusedalternativeistofixthestep
sizetosomearbitraryvalueandthendecreaseitasmoreiterationsareexecuted,i.e.,α(k) →0ask →∞. Theinitial
stepsizeα(0) andthedecayarehighlysensitiveparameters,specifictoeachdataset,thatrequireintensivetuning. By
fixingthestepsize,theburdenisessentiallymovedfromruntimeevaluationtoofflinetuning.
Algorithm1BatchGradientDescent(BGD)
Input: {((cid:126)x ,y )} ,f,∇Λ,w(cid:126)(0),α(0)
j j 1≤j≤N
Output: w(cid:126)(k−1)
1. Initializew(cid:126)(0)andα(0)withrandomvaluesifnotprovided
2. Letk =1
3. while(true)do
4. ifmodel convergence({f(w(cid:126)(l))}0≤l<k)thenbreak
5. Computegradient: ∇Λ(w(cid:126)(k−1)){((cid:126)xj,yj)}1≤j≤N
6. Updatemodel: w(cid:126)(k) =w(cid:126)(k−1)−α(k−1)∇Λ(w(cid:126)(k−1))
7. Updatestepsizeα(k)
8. Letk =k+1
9. endwhile
10. return w(cid:126)(k−1)
SQLrepresentation. WeformulatethebatchgradientsolutiontotheobjectivefunctioninEq.(1)asSQLaggregate
queries. This allows us to identify database-specific optimizations. There are two dataset-wide operations in Algo-
rithm 1—loss evaluation and gradient computation. Both of them can be expressed as SQL queries over a relation
T ((cid:126)x,y)inwhicheachtuplerepresentsatrainingexample. Thelossatagivenpointw(cid:126) isevaluatedbythequery:
SELECT SUM(f(w(cid:126), (cid:126)x; y)) FROM T
inwhichthevectoroperationsinvolvingtheexamplefeaturevector(cid:126)xandthemulti-dimensionalpointw(cid:126) caneither
beexpressedasarrayfunctionsorbemappedexplicitlyintoarithmeticoperators. Gradientcomputationatapointw(cid:126)
requiresoneaggregateforeverydimension,asshowninthefollowingSQLquery:
SELECT SUM( ∂f (w(cid:126), (cid:126)x; y)), ..., SUM( ∂f (w(cid:126), (cid:126)x; y))
FROM T ∂w1 ∂wd
4
It is important to emphasize that both the point w(cid:126) and the function f and its gradient ∇f are constants at a given
iterationinthetwoqueriesgivenabove. Tomakethispointclear,weshowtheactualqueriescorrespondingtoSVM
classificationwith-1/+1labels:
SELECT SUM((cid:80)d 1−yx w ) FROM T
i=1 i i
(cid:16) (cid:17)
WHERE (cid:80)d 1−yx w > 0
i=1 i i
SELECT SUM(−yx ), ..., SUM(−yx ) FROM T
1 d
(cid:16) (cid:17)
WHERE (cid:80)d 1−yx w > 0
i=1 i i
Forothermodeltypes,onlytheformulaofthelossfunctionanditsgradientchange. Thereststaysthesame. Most
importantly,thequeryshapeisunchanged.
Basedontheseexamples,weformalizetheBGDcomputationswiththefollowingabstractSQLquery:
SELECT SUM(f (t)), ..., SUM(f (t)) FROM T (3)
1 p
inwhichpdifferentaggregatesarecomputedoverthetrainingtuplesinrelationT. Atleasttwoinstancesofthisquery
havetobeexecutedperiteration—oneforthegradientandonefortheloss. Theexactnumberistypicallymuchlarger
andisdeterminedbythenumberoftimesthelossiscomputedinlinesearch.
3.2 IncrementalGradientDescent
TheproblemwithBGDisthatonepass–ormanymoreiflinesearchisexecutedtofindtheoptimalstepsize–over
theentiredataisrequiredinordertomoveasinglestep. Thisisabigissueinourtargetscenario–massivenumber
of examples N – since the time per iteration is linear in N and many iterations are typically required in order to
achieveconvergence. Incrementalorstochasticgradientdescent(IGD)[7]addressesthisissuebytakingN stepsper
iteration. Thedirectionofeachstepisgivenbythegradientcorrespondingtoasingledataexample(cid:126)x . Essentially,
i
theentiregradientisapproximatedwithasingleterminthesummation,i.e.,∇Λ(w(cid:126)) ≈ ∇f(w(cid:126),x(cid:126);y ). Ifweignore
i i
theregularizerRthesteprecurrencebecomes:
(cid:16) (cid:17)
w(cid:126)(k+1) =w(cid:126)(k)−α(k)∇f w(cid:126)(k),(cid:126)x ;y (4)
η(k) η(k)
where η(k) returns the kth example in a random permutation of the data. The permutation is necessary to guarantee
that progress towards convergence is made inside an iteration. Moreover, a different permutation should be used at
each iteration to avoid stalling at a non-minimum point and increase the convergence rate. Given the large number
ofstepstakeninsideasingleiteration, thestepsizeα(k) hastobecarefullytunedtominimizeextremeoscillations.
Executinglinesearchaftereverystepisoutofthequestion. Algorithm2summarizestheIGDdifferencescompared
toBGD.Lines5and6inAlgorithm1arereplacedwiththeforloopgiveninAlgorithm2, wherew(cid:126)(k) = w(cid:126)(k−1)
(0)
andw(cid:126)(k) =w(cid:126)(k),respectively.
(N)
Algorithm2IncrementalGradientDescent(IGD)
1. fori=1toN do
(cid:16) (cid:17)
2. Approximategradient: ∇f w(cid:126)((ik−)1),(cid:126)xη(i);yη(i)
(cid:16) (cid:17)
3. w(cid:126)((ik)) =w(cid:126)((ik−)1)−α(k)∇f w(cid:126)((ik−)1),(cid:126)xη(i);yη(i)
4. endfor
Mini-batch gradient descent. BGD takes a single step per iteration. IGD takes one step for every data example.
An intermediate solution is to estimate the gradient using more than a single term in the summation Eq. (1) and
take one step for each group of terms. If there are n, 1 < n < N, such groups then n steps are taken in a single
iteration. Thestepsizeαhastobeconfiguredaccordingly. Thisalternativeisknownasmini-batchgradientdescent.
5
While it has been shown that there are situations when the mini-batch solution outperforms both the batch and the
incrementalmethods,theadditionofanotherparameter–thesizeofabatchorthenumberofbatches–onlyincreases
thecomplexityoftuning. ThiscannotbeignoredwhenthenumberofdataexamplesN islarge.
SQLrepresentation. ThemaindifferencebetweenBGDandIGDwithrespecttoSQLrepresentationisthatw(cid:126) is
not constant for an entire iteration. In pure IGD, w(cid:126) is updated for every example in the input, while in mini-batch
gradient descent, w(cid:126) is updated after a fixed number of examples. This can be expressed in SQL as the following
UPDATEstatement:
UPDATE W
SET w =w −α(k) · ∂f (w(cid:126), (cid:126)x; y), ...,
1 1 ∂w1
w =w −α(k) · ∂f (w(cid:126), (cid:126)x; y)
d d ∂wd
where W(w ,...,w ) is a table with a single tuple containing the model. The pair ((cid:126)x;y) is a tuple from table T
1 d
corresponding to an input example. The update is executed for all the tuples in T, extracted by a cursor in random
order. Asaconcreteexample,forSVMclassificationwehavethefollowingSQLstatement:
UPDATE W
SET w =w −α(k)· yx , ..., w =w −α(k)· yx
1 1 1 d d d
Thelossiscomputedforthelastw(cid:126) generatedattheendofaniteration,usingthesameSQLqueryasinBGD.
3.3 BeyondFirst-OrderGradientDescent
Apartfromthefirst-ordergradientdescentmethodspresentedinthissection,coordinatedescent[50]andL-BFGS[2,
28] are two alternatives that build upon BGD and IGD. Coordinate Descent (CD) optimizes a multi-dimensional
functionbyminimizingitalongonedimensionatatime. Insteadofmovingalongtheoverallgradientdirection,CD
takes steps along each coordinate direction sequentially. The search along each coordinate is done by line search,
which requires a step size. L-BFGS is a quasi-Newton method that searches the model space by approximating the
inverseHessianmatrixoftheobjectivefunction. Differentfromgradientdescent,L-BFGSmaintainsahistoryofthe
lastmupdatestomodelw andgradient∇Λ. Thedirectiond(cid:126)atiterationk isgivenbyd(cid:126)(k) = −H(k)∇Λ(k), where
H(k)istheapproximationtotheHessian. WithoutgoingintodetailsonhowH(k)iscomputedfromhistoricalmodels
andgradients,wepointoutthat,aftercomputingthesearchdirectiond(cid:126)(k),alinesearchisperformed.
Thetechniquesweproposeinthispapercanbeappliedtoanyofthegradient-basedoptimizationmethods–first-
andsecond-order–usedforlargescalemodelcalibration.Weemphasizethatthenumberoftunablehyper-parameters,
i.e.,parametersoftheoptimizationmethod,issmallinallthesemethods. Inthecaseofcoordinatedescent,onlythe
step size has to be tuned. Or, perhaps, the number of coordinates to update at the same time, if parallel coordinate
descentisconducted.InL-BFGS,onlythesizeofthehistoryandthestepsizeusedinlinesearchhavetobecalibrated.
Therelativelysmallspaceoftunablehyper-parametersallowsformoreopportunitiestoapplythetechniquesproposed
inthispaper.
3.4 Discussion
Considering the alternative gradient descent methods discussed in this section, a comparison from the perspective
of our specific problem, i.e., Eq. (1) with the assumption that N is extremely large, is required to clarify the merits
of each approach. According to the thorough survey by Bertsekas [7], two cases have to be considered. Far from
theminimum,IGDcanhaveaconvergencerateasmuchasN timesfasterforidenticallossfunctionsf andlargeN.
Closetotheminimum,IGDrequiresdiminishingstepsizesinordertoconverge.Thisresultsinsub-linearconvergence
rate—slower than the linear convergence of the batch method. Based on these observations, a hybrid approach [2],
thatfirstexecutesincrementalgradientfollowedbybatchgradientwhennoprogressismadeanymore,islikelytobe
theoptimalsolutioninpractice. ItisalsoimportanttoremarkthatIGDrequiresdatarandomizationateachiteration
inordertoachievetheN factorinconvergencespeedup. Withthesamecyclicalorderusedacrossalliterations,IGD
doesnotprovideanytheoreticalconvergenceimprovementbeyondBGD[7].
6
4 Distributed Gradient Descent
Inthissection,wepresentparallelalgorithmsfordistributedgradientdescentoptimization. Withalmostnoexception,
alltheexistingalgorithms[9,30,22,55,19,2]makeexclusiveuseofdatapartitioningparallelism,i.e.,theM subsets
are processed concurrently across the execution nodes and the partial results are merged together at the end of an
iteration. Tothebestofourknowledge, othertypesofparallelism[16], e.g., pipelining, super-scalarexecution, and
vectorization,arenotconsideredintheliterature.
4.1 DistributedBGD
ThestraightforwardstrategytoparallelizeBGDistooverlapgradientcomputation(line5inAlgorithm1)acrossthe
processingnodesstoringtheM datasubsets[9]. Thepartialgradientsaresubsequentlyaggregatedatacoordinator
node holding the current model w(cid:126)(k), where a single global update step is performed in order to generate the new
model w(cid:126)(k+1) based on Eq. (2). The update step requires completion of partial gradient computation across all the
nodes, i.e., update model is a synchronization barrier. Once the new model w(cid:126)(k+1) is computed – using line search
or a fixed step size – it is disseminated to the processing nodes for a new iteration over the data. Parallel BGD
works because the objective function Λ is linearly separable, i.e., the gradient of a sum is the sum of the gradient
(cid:104) (cid:105)
applied to each term: ∇ (cid:80)N f(w(cid:126),x(cid:126);y ) = (cid:80)N ∇f(w(cid:126),x(cid:126);y ). It provides linear processing speedup and
i=1 i i i=1 i i
logarithmic communication in the number of nodes since aggregation can be executed on a tree structure. In terms
of convergence though, there is no improvement with respect to the sequential algorithm—only faster iterations for
gradientcomputation.
An alternative parallel algorithm is given in [22]. Each node solves to convergence an independent BGD on its
corresponding data subset. The joint model is produced by averaging the partial models computed at each node.
As a result, communication between nodes is reduced to minimum. From a convergence standpoint, this ensemble
solution provides only variance reduction, but does not eliminate the bias when compared to the single node sub-
samplesolution[55]. Theexecutiontimeisdominatedbytheconvergencetimeoftheslowestnode.
4.2 DistributedIGD
Intuitively, parallelizing IGD cannot be any different from the batch version since the difference between the two
algorithms is minimal. It should be even simpler since no global gradient computation is required. Essentially, a
modelupdateisexecutedforeverydataexamplebasedsolelyonitsvalue. Atacloserlookthough,weobservethat
theupdatesinAlgorithm2arestrictlysequential,i.e.,theinputofthecurrentupdateistheoutputofthepreviousone.
Moreover,theorderinwhichupdatesareexecutedaffectsthefinalresult,i.e.,modelupdateisnotcommutative.These
reasonsseemtoprecludeanytypeofparallelismaltogether. Unlesswedroptherequirementthatallthedataexamples
havetobeusedtoupdatethemodelinoneiteration.Similartotheapproachin[22],thestandardmethodtoparallelize
IGD[55]istosolveaseparatemodelforeverydatapartitionandthentoaveragetheM modelsinordertogenerate
thefinalmodelateachiteration. Theresultingmodelisthenpassedasastartingpointtothesubsequentiterationand
theprocessisrepeateduntilconvergence,whichistheoreticallyachievedinalogarithmicnumberofiterations[55].
AnalternativethatminimizescommunicationistoexecuteIGDuntilconvergenceoneachpartitionandaverageonly
thefinalmodels.
Modelaveraging. ThefollowingtradeoffhastobeconsideredinthecaseofparallelIGD.Whenthenumberofdata
partitions M is large, so is the degree of parallelism. This results in higher speedup. It also results in more models
computedoverfewerexamplestobeaveraged. Thisisapotentialproblemsincethereisalowerboundonthenumber
ofexamplesrequiredforagivenmodeltoachievetheasymptoticregime[55]. Moreover,theexamplesseenbyevery
modelhavetorepresentarandomsamplefromtheentirepopulation.ThelargerthenumberofpartitionsM,thehigher
thecommunicationrequiredtogeneraterandomsamples. Thecombinedeffectofnon-asymptoticbehaviorandnon-
random examples is higher variance across the partial models, which finally results in slower convergence [19, 38].
Thus,choosingtheoptimalM valuehasdeeperimplicationsinthecaseofparallelIGD.
7
Shared-memory parallelization. Exactly because of this reason, the preferred solution to parallelize IGD in a
shared-memory setting is to discard model merging completely. A single model is shared across all the threads,
while concurrent updates to the model are serialized through locking. Unfortunately, this solution reduces speedup
dramaticallyevenwhenlightlockingmechanisms,e.g.,compare&swap,areused. Thesolutionproposedin[35]to
solvethisproblemistodiscardlockingaltogetherandallowforcontentiousupdatestothesharedmodel.Asexpected,
thisresultsinlinearspeedup. Whataboutconvergence? Aslongasthemodelishighly-dimensionalandthegradients
usedinupdatesaresparse,i.e.,onlyasmallnumberofmodeldimensionsareupdated,thedifferenceinconvergence
comparedtotheserialsolutionisminimal.
Hybridsolution. Inadistributedsettingwithmulti-corenodes,ahybridapproachisregardedastheoptimalsolu-
tion[39]. Asinglemodeliscreatedforeachnode. Itissharedacrossallthelocalthreadsorprocessesandupdated
usingoneoftheshared-memorystrategies,chosenbasedontheactualmodelproperties. Modelmergingacrossnodes
isexecutedusingthestandardaveragingtechniquealonganaggregationtree. Thehybridsolutionaimstomaximize
thedegreeofparallelismavailableinthesystemwithoutdramaticallyimpactingtheconvergence.
4.3 Summary
Parallelizinggradientdescentmethodsinadistributedenvironmentposesdifferentchallenges. Forthebatchmethod,
gradientcomputationistriviallyparallelizableduetotheobjectivefunctionbeinglinearlyseparable. Themaindiffi-
cultyisposedbymodelupdatewhichisastrictlysequentialoperation. ThemainissuewhenparallelizingIGDisthe
tradeoffbetweenthedegreeofparallelismandtheconvergencerate. Moreparallelismdoesnotautomaticallyresultin
betterconvergence. Ahybridapproachinwhichasharedmodeliscreatedforeachnode,whileaveragingisusedto
mergemodelsacrossnodes,islikelytobetheoptimalsolutioningeneral.
5 Speculative Iterations
In this section, we address two fundamental problems specific to model calibration—convergence detection and pa-
rametertuning.Aswithanyiterativemethod,gradientdescentconvergenceisachievedwhenthereisnomoredecrease
in the objective function, i.e., the loss, across consecutive iterations. While it is obvious that convergence detection
requires loss evaluation at every iteration, the standard practice, e.g., Vowpal Wabbit [2], MLLib [17], is to discard
detectionaltogetherandexecutethealgorithmforafixednumberofiterations.Thereasonissimple:losscomputation
requires a complete pass over the data, which doubles the execution time. This approach suffers from at least two
problems. First,itisimpossibletodetectconvergencebeforethespecifiednumberofiterationsfinishes. Andsecond,
it is impossible to identify bad parameter configurations, i.e., configurations that do not lead to model convergence.
RecallthatbothBGDandIGDdependonaseriesofparameters,themostimportantofwhichisthestepsize. Finding
theoptimalstepsizetypicallyrequiresmanytrials. Discardinglosscomputationincreasesboththenumberoftrialsas
wellasthedurationofeachtrial.
High-levelapproach. Weproposeaunifiedsolutionforconvergencedetectionandparametertuningbasedonspec-
ulativeprocessing. Themainideaistooverlapgradientandlosscomputationformultipleparameterconfigurations
across every data traversal. This allows for timely convergence detection and early bad configuration identification
since many trials are executed simultaneously. Overall, faster model training. The intuition behind our approach is
thatmodernCPUarchitecturesprovideextensiveparallelizationopportunities,e.g.,multi-core,hyper-threading,vec-
torization,thataredifficulttouseatfullpotentialindisk-basedworkloadsspecifictoBigDataanalyticsovermassive
datasets. Withtherightmixoftasksandjudiciousscheduling,thedegreeofparallelismsupportedinhardwarecanbe
fullyutilizedbyexecutingmultipletasksconcurrently. Moreover, theoverallexecutiontimeissimilartothetimeit
takestoexecuteeachtaskseparately.
Ourcontributionistodesignspeculativegradientdescentalgorithmsthattestmultiplestepsizessimultaneously
andoverlapgradientandlosscomputation. Thenumberofstepsizesusedateachiterationisdeterminedadaptively
anddynamicallyatruntime.Thestepsizesaredrawnfromaparametricdistributionthatiscontinuouslyupdatedusing
8
step size
1
step size G (k) L (k) step size
1 11 11 1
G (k) L (k) G (k) L (k)
1 1 Data1 step size 1 1
2
G (k) L (k)
21 21
w(k) w(k+1)
step size
1
G (k) L (k)
step size Data 1M 1M step size
2 M 2
G (k) L (k) step size G (k) L (k)
2 2 2 2 2
G (k) L (k)
2M 2M
Figure1: Speculativeparametertestingwithonlineaggregation.
aBayesianstatistics[23]approach. Onlythemodelwiththeminimumlosssurviveseachiteration,whiletheothers
are discarded (Figure 1). As long as the execution time does not dramatically increase when multiple step sizes are
tested, speculative execution leads to faster model training. To the best of our knowledge, this is the first solution
thatusesspeculativeexecutionforgradientdescentparametertuningandlosscomputation. Speculativeexecutionis
appliedinVowpalWabbit[2]andMLlib[17],butonlytodealwiththeproblemofslownodes. Nottotestmultiple
stepsizessimultaneously. VowpalWabbitalsosupportsprogressivelossestimationwhichoverlapsgradientandloss
computation.Progressivelossprovidesonlylocalestimates. Itnevercomputestheexactlossattheendofaniteration.
Thiscanbedoneonlywithanadditionalpassoverthedata. Oursolutionisexact–notapproximate–andworksina
distributedsetting.
5.1 SpeculativeBGD
Speculative BGD works as follows. Define a set of possible step sizes {α ,...,α }, s ≥ 1. Generate an up-
1 s
datedmodelw(cid:126)(k+1),1 ≤ i ≤ s,foreachofthesestepsizesandcomputethecorrespondingobjectivefunctionvalue
i
concurrently. Choosethemodelw(cid:126)(k+1)withtheminimumlossasthenewmodel. Repeattheprocedureforeveryiter-
i
ationuntilconvergence. Thepseudo-codeforthespeculativeBGDalgorithmisgiveninAlgorithm3. Thestatements
containedinsideaninparallelloopareexecutedconcurrentlyacrossiterations. Moreover,inthecaseofSpeculative
BGD, the two statements inside the loop (lines 8 and 9) are executed in parallel even for the same iteration. In the
following,wediscussthemaincomponentsofthealgorithm.
Concurrent step size evaluation. Intuitively, speculative processing replaces multiple passes over the data with a
single pass that requires O(s) times more computation. In SQL terms, this corresponds to merging s independent
aggregate queries into a single query with O(s) aggregates. This provides a significant reduction in execution time
evenfortraditionalI/O-bounddatabases. Moderndatabasesthattakeadvantageofstate-of-the-artparallelhardware
architecturescandomuchbetterandachievesimilarexecutiontimesasforasingleaggregatequery. Therearetwo
levelsofparallelismthatcanbeexploited—multi-threadedexecutiononmulti-coreCPUsandvectorizedinstructions.
Instead of updating all the s models in a single thread, each model is updated in a separate thread, initialized with
a different model. All the threads perform exactly the same processing over the same example data. Only their
initializationmodelsaredifferent. ThisisastandardinstanceofSIMDparallelismthatisdirectlysupportedbymulti-
core CPUs. Given that every example has to be processed by all the threads, scheduling example access requires
9
Algorithm3SpeculativeBGD
Input: {((cid:126)x ,y )} ,f,∇Λ,w(cid:126)(0),∇Λ(w(cid:126)(0)),s,Φ
j j 1≤j≤N
Output: w(cid:126)(k)
1. Initializew(cid:126)(0)and∇Λ(w(cid:126)(0))randomlyifnotprovided
2. Letk =1
3. while(true)do
4. Drawsstepsizes{α1,...,αs}fromdistributionΦ
5. Letw(cid:126)i =w(cid:126)(k−1)−αi∇Λ(w(cid:126)(k−1)),1≤i≤s
6. foreachexample((cid:126)xj,yj)do
7. foreachmodelw(cid:126)idoinparallel
8. Computegradient: ∇Λ(w(cid:126)i){((cid:126)xj,yj)}
9. Computeloss: f(w(cid:126)i,(cid:126)xj;yj)
10. endfor
11. endfor
12. Letw(cid:126)(k) =minf(w(cid:126)i){w(cid:126)i},1≤i≤s
13. ifmodel convergence({f(w(cid:126)(l))}0≤l<k)thenbreak
14. Updatestepdistrib: Φ=Bayes(Φ,{αi},{f(w(cid:126)i)})i≤s
15. Updatenumberofstepssbasedonnestedloops(lines6-11)executiontime
16. Letk =k+1
17. endwhile
18. return w(cid:126)(k)
discussion. One approach is to make every example available to all the threads. While the threads can process the
exampleconcurrently,sinceread-onlyaccessissufficient,synchronizationisstillrequiredinordertodeterminewhen
allthethreadsfinishprocessing.Anotherapproachistoorganizethethreadsintoapipeline.Thisallowsforoverlapped
execution across examples, i.e., each thread processes a different example, and eliminates barrier synchronization.
Vectorized processing, e.g., SSE and AVX instructions, can be applied in two places. A set of examples can be
grouped together into a chunk and processed as a unit inside a thread. As a side effect, chunking also reduces the
overheadofmovingexamplesthroughthepipeline. Thesecondinstanceforvectorizationisdot-productcomputation,
whichrepresentstheprimitiveoperationingradientandlossevaluation.
Gradientandlosscomputationoverlapping. Themodelthatminimizesthelossacrossthespossiblestepsizesis
taken as input by the subsequent iteration. The gradient has to be computed for this model. Remember though that
wehavealreadyusedthismodeltocomputethelossinthepreviousiteration. Unfortunately,wedidnotknowifthis
modelisselectedornot. Undertheassumptionthatdataaccessdominatestheexecutiontime,weargueforbundling
lossandgradientcomputationtogetherinordertosaveonepassoverthedata. Overall,thenumberofdatatraversals
is determined entirely by the number of gradient computations while loss evaluation piggybacks on the data access.
Wehavetoapplytheoverlappingstrategyforallthesmodelsthough,sincewedonotknowtheoneminimizingthe
loss.Althoughwemightexpectthistoincreasetheexecutiontimeofaniteration,theaggressiveuseofSIMDpipeline
parallelismacrossthesstepsizesminimizestheimpactofadditionalcomputationandhighermemoryusage,asthe
experimentalresultsinSection7show.
Howmanystepsizess? Thelargerthenumberofstepsizes,thehighertheprobabilitytofindamodelwithsmaller
loss. However, a too large degree of speculation might result in an unacceptable increase in iteration time. Thus,
choosingtherightnumberofstepsizesisimportantforoptimalspeculativeprocessing. Whatmakestheproblemeven
more difficult is the high variance in processing time as a function of model type and size. We propose an adaptive
solution that selects the number of step sizes dynamically at runtime based on available resources. Moreover, the
number of step sizes varies from one iteration to another. We start with a single step size in the first iteration and
measuretheexecutiontime. Foreachsubsequentiteration,weincreasethenumberofstepsizes,e.g.,exponentially,
aslongastheincreaseinexecutiontimeremainsbelowaspecifiedthreshold. Weusethisnumberofstepsforallthe
10