JetStream: Enabling High Performance Event Streaming across Cloud Data-Centers

13 Pages·2016·1.24 MB·English

View metadata, citation and similar papers at core.ac.uk brought to you by CORE
provided by HAL-Rennes 1
JetStream: Enabling High Performance Event
Streaming across Cloud Data-Centers
Radu Tudoran, Olivier Nano, Ivo Santos, Alexandru Costan, Hakan Soncu,
Luc Boug´e, Gabriel Antoniu
To cite this version:
Radu Tudoran, Olivier Nano, Ivo Santos, Alexandru Costan, Hakan Soncu, et al.. JetStream:
Enabling High Performance Event Streaming across Cloud Data-Centers. DEBS’14 - The 8th
ACMInternationalConferenceonDistributedEvent-BasedSystems,May2014,Mumbai,India.
ACM, pp.23 - 34, 2014, <10.1145/2611286.2611298>. <hal-01090281>
HAL Id: hal-01090281
https://hal.archives-ouvertes.fr/hal-01090281
Submitted on 19 Dec 2014
HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est
archive for the deposit and dissemination of sci- destin´ee au d´epˆot et `a la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publi´es ou non,
lished or not. The documents may come from ´emanant des ´etablissements d’enseignement et de
teaching and research institutions in France or recherche fran¸cais ou ´etrangers, des laboratoires
abroad, or from public or private research centers. publics ou priv´es.
JetStream: Enabling High Performance Event Streaming
across Cloud Data-Centers
Radu Tudoran Olivier Nano Ivo Santos
IRISA/ENSRennes,France MicrosoftResearch–ATL MicrosoftResearch–ATL
[email protected] Europe,Munich,Germany Europe,Munich,Germany
[email protected] [email protected]
Alexandru Costan Hakan Soncu Luc Bougé
IRISA/INSARennes,France MicrosoftResearch–ATL ENSRennes,France
[email protected] Europe,Munich,Germany [email protected]
[email protected]
Gabriel Antoniu
InriaRennes-Bretagne
Atlantique,Rennes,France
[email protected]
ABSTRACT rate can further be tripled thanks to the use of multi-route
streaming.
The easily-accessible computation power offered by cloud
infrastructures coupled with the revolution of Big Data are
CategoriesandSubjectDescriptors
expandingthescaleandspeedatwhichdataanalysisisper-
formed. In their quest for finding the Value in the 3 Vs of H.3.5[OnlineInformationServices]: Datasharing;D.2.4
Big Data, applications process larger data sets, within and [Distributed Systems]: Distributed applications
acrossclouds. Enablingfastdatatransfersacrossgeograph-
ically distributed sites becomes particularly important for Keywords
applications which manage continuous streams of events in
real time. Scientific applications (e.g. the Ocean Observa- Event Streaming, Cloud Computing, Multi Data-Centers,
tory Initiative or the ATLAS experiment) as well as com- High Performance Data Management
mercialones(e.g. Microsoft’sBingandOffice365large-scale
services) operate on tens of data-centers around the globe 1. INTRODUCTION
andfollowsimilarpatterns: theyaggregatemonitoringdata, On-demand resource provisioning, coupled with the pay-
assesstheQoSorrunglobaldataminingqueriesbasedonin- as-you-go model, have redefined the way data is processed
ter site event stream processing. In this paper, we propose nowadays. Scientific applications, simulations, monitoring,
a set of strategies for efficient transfers of events between networks of sensors, trading or commercial data mining are
clouddata-centersandweintroduceJetStream: aprototype allareaswhichhavebenefitedfromandsupportedtheemer-
implementingthesestrategiesasahighperformancebatch- gence of cloud computing infrastructures. Cloud providers
basedstreamingmiddleware. JetStreamisabletoself-adapt like Microsoft, Amazon, Google, Rackspace have built their
to the streaming conditions by modeling and monitoring a cloud solutions at a global scale with data-centers spread
setofcontextparameters. Itfurtheraggregatestheavailable across numerous geographical regions and continents. At
bandwidth by enabling multi-route streaming across cloud thesametime,theversatilityofdataanalysishasincreased
sites. The prototype was validated on tens of nodes from with applications running on multiple sites, which have to
US and Europe data-centers of the Windows Azure cloud process larger data sets coming from remote locations and
usingsyntheticbenchmarksandwithapplicationcodefrom distinct sources. The services dealing with such computa-
the context of the Alice experiment at CERN. The results tions have to face important challenges and issues regard-
showanincreaseintransferrateof250timesoverindividual ingthemanagementofdataacrossthesegeographicallydis-
event streaming. Besides, introducing an adaptive transfer tributed environments.
strategybringsanadditional25%gain. Finally,thetransfer Stream data processing is becoming one of the most sig-
nificant subclass of applications in the world of Big Data.
Vastamountsofstreamdataarecollectedatincreasingrates
frommultiplesources[20,29]inmanyareas: climatemoni-
toring,large-scalesensor-basedsystems,transactionalanal-
Permissiontomakedigitalorhardcopiesofallorpartofthisworkfor
personalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesare ysis, financial tickers, monitoring and maintenance systems
notmadeordistributedforprofitorcommercialadvantageandthatcopies forwebservices,large-scalescienceexperiments. Acquiring,
bearthisnoticeandthefullcitationonthefirstpage.Tocopyotherwise,to processing and managing this data efficiently raise impor-
republish,topostonserversortoredistributetolists,requirespriorspecific tantchallenges,especiallyiftheseoperationsarenotlimited
permissionand/orafee.
to a single geographical location. There are several scenar-
DEBS’14,May26-29,2014,Mumbai,India.
ios which created the need to geographically distribute the
Copyright2014ACM978-1-4503-2737-4...$15.00.
computation on clouds. The size of the data can be so big andinareallifescenariousingtheMonALISA[33]monitor-
that data have to be stored across multiple data-centers. It ing system of the CERN Alice experiment [1]. The experi-
isthecaseoftheATLASCERNexperimentwhichgenerates mentsshowperformanceimprovementsof250timesoverin-
40PBofdataperyear. Furthermore,evenincrementalpro- dividualeventstreamingand25%overstaticbatchstream-
cessingofsuchadatasetasastreamofeventswilloverpass ing,whilemulti-routestreamingcanfurthertriplethetrans-
the capacity of local scientific infrastructure, as it was the fer rate.
case of the Higgs boson discovery which had to extend the The rest of the paper is structured as follows. Section 2
computationtotheGooglecloudinfrastructure[2]. Another introducestheterminologyandpresenttheproblemswhich
scenario is given by the data sources which can be phys- appearinthecontextofstreamingacrossclouddata-centers.
ically distributed in wide geographical locations as in the Section 3 presents our approach for adapting the batch size
Ocean Observatory Initiative [5, 12] in which the collected by modeling the latency for clouds, while considering the
events are streamed to Nimbus [9] clouds. Finally, the na- cost of multi-routing. We continue with the description
ture of the analysis, for an increasing number of services, of the system architecture and its implementation in Sec-
requires aggregating streams of data from remote applica- tion 4 and discuss the evaluation of the approach across
tioninstances. Large-scaleserviceslikeMicrosoft’sBingand Azure data-centers from different continents (i.e. Europe
Office365operateontensofdata-centersaroundtheGlobe. andAmerica)inSection5. InSection6,wepresentanexten-
Maintenance,monitoring,assertingtheQoSofthesystemor sivestateoftheartinseveraldomainstangenttostreaming
global data mining queries all require (near) real-time inter anddatamanagementandpositionthecontributionsofour
siteeventstreamprocessing. Allsuchcomputationscarried approach with respect to these directions. Finally, Section
oncontinuousstreamsofeventsacrossresourcesfromdiffer- 7 concludes the paper.
ent geographical sites are highly sensitive to the efficiency
of the data management. 2. BACKGROUND:EVENTSTREAMINGON
An extended survey over thousands of commercial jobs
CLOUDS
and millions of machine hours of computation, presented
in [23], has revealed that the execution of queries is event- We start by defining the terminology related to stream
driven. Furthermoretheanalysisshowsthattheinputdata processing on clouds and discuss the main problems which
accountsonlyfor20%ofthetotalIO,therestcorresponding need to be addressed in this context.
tothereplicationofdatabetweenqueryservicesortothein-
2.1 Context
termediatedatapassedbetweenthem. Thisemphasizesthat
the processing services, be they distributed, exchange large
amounts of data. Additionally, the analysis highlights the Cloud computing is a recent paradigm which enables re-
sensitivity of the performance of the processing of streams sources (e.g. compute power, storage capacity, network
to the management and transfer of events. This idea is dis- bandwidth)tobeeasilyleasedon-demand,followingapay-
cussedalsoin[20],inwhichtheauthorsstresstheneedfora as-you-go pricing model. From the infrastructure point
highperformancetransfersystemforreal-timedata. Asim- of view, clouds are built across several data-centers dis-
ilarconclusionisdrawnin[10],wheretheissueswhichcome tributedaroundtheglobe. Theproximityofthegeograph-
fromthecommunicationoverheadandreplicationareexam- ical repartition of resources to users enables high QoS and
inedinthecontextofstate-basedparallelization. Finally,in fastqueryresponsesbymeansofcontentdeliverynetworks
[40],theauthorsemphasizetheimportanceofdatafreshness, (CDN). From the user point of view, all these resources
which improves the Quality of Service of a stream manage- are harvested transparently by means of virtualization. A
ment system and implicitly the quality of the data (QoD). typical cloud usage scenario consists of a user deployment
Alltheseresearcheffortssupportandmotivatethe growing through which an application is run within virtual ma-
need for a high performance system for event streaming. chines (VMs) on several cloud nodes from a data-center.
In this paper, we address challenges like latency, transfer Large-scaleapplicationsandwebservicestypicallyrunmul-
rate, throughput and performance variability for support- tiple deployments across several data-centers, enabling ge-
ing high performance event transfers in the context of ge- ographical proximity to users and load balancing. Besides
ographically distributed applications. Our solution, called the local computations answering to user requests, sev-
JetStream, is a novel approach for streaming events across eralglobaloperations(e.g.,monitoring,QoSenhancement,
cloud data-centers. In order to enable a high performance globaldatamining)requireinter-sitetransfers betweenge-
transfer solution, we leverage batch-based transfers. The ographically distant nodes, most often in (near) real-time.
size of the batches and the decision on when to stream the Thecloudsupportforsuchtransfersisratherrudimentary:
events are controlled by modeling the latency based on a it relies on the cloud storage (e.g., Azure Blobs, Queues,
set of parameters which characterize the streaming in the TablesorAmazonS3,SQS)asaglobalsharedpointwhich
context of clouds. To further improve performance, we ag- incurs high transfer latencies [24, 45].
gregate inter-data-center bandwidth as we extend our ap-
Stream computing referstotheprocessingofcontinuous
proachtoprovidemulti-routestreamingacrosscloudnodes.
sequences of relatively small data items [20], otherwise re-
Furthermore, we developed a prototype that can be used
ferred to as events. According to [17], the characteristics
on clouds, either public or private, which is environment-
ofdata streams are: 1)astreamisformedofevents,which
aware by means of light monitoring. The information gath-
aretuplesofrecords;2)itisapotentiallyinfinitesequence
eredandinferredabouttheenvironmentiscoupledwiththe
of events ordered by time (e.g., acquisition time, genera-
streaming model, allowing the system to adapt the transfer
tion time); and 3) the events can be produced by multi-
decisions to all context changes. The system was validated
ple external sources at variable rates, independent of the
ontheWindowsAzure[6]cloudusingsyntheticbenchmarks
constraints of the stream processing engine. The stream
a) Transfer Rate (throughput) b) Average Latency of an Event Sender - Site 1
2500 0.03 Time between events
2000 0.025 MTBE=Rate1Acquisition
Events/sec 11050000 Time(sec) 0.000..100512 Batch DestinationSP -rt roSecaietmses in2g
500 0.005 Serializer Network De-SerializerEngine
0 0
1 10 100 250 5001000 1 10 100 250 5001000
Batch Size (#Events) Batch Size (#Events)
Figure 1: The performance of transferring events in SEovuerncte LatencyEncoding LatencyDecoding
batchesbetweenNorthEuropeandNorthUSAzure LatencyBatching LatencyTransfer
data-centers. The measurements show the correla- Figure 2: Breaking down the latency to deliver an
tion between (a) the transfer rate and (b) the aver- eventfromsourcetostreamprocessingengineacross
age event latency for increasing batch sizes. cloud nodes
processingengines aresystemswhichtreattheeventsfrom Environment awareness - The cloud infrastructures are
thestreambyapplyingtheuser’squeries. Asitisgenerally subjecttoperformancevariationsduetomulti-tenancy
infeasible to store the streams entirely, the queries are ap- and network instability on the communication links
plied continuously and iteratively over windows of events. withinandacrosssites. Monitoringanddetectingsuch
Thecomputationisevent-driven,thereforemostofthedata performance changes allows the system to react ac-
management is done at the granularity of an event. Gen- cordinglyandschedulethetransferofeventsefficiently.
erally, the size of an event is small, in the range of bytes
to kilobytes, which allows to handle them in-memory, but Decoupling the transfer from processing - The event
also makes their transfer sensitive to any overhead. transfermoduleneedstobedesignedasastand-alone
component, decoupled from the stream processingen-
2.2 Problemstatement gine. We advocate this solution as it allows seamless
integration with any engine running in the cloud. At
Achievinghighperformanceeventstreamingacrossdata-
thesametime,itprovidessustainableandpredictable
centers requires new cloud-based solutions, since currently
performance, independent on the usage setup.
there are no adequate tools to support data streaming or
transfers across sites. Most of the existing stream process- Self-optimization - User configurations do not guarantee
ingenginesonlyfocusoneventprocessingandprovidelittle optimal performance, especially in dynamic environ-
ornosupportforefficienteventtransfers,simplydelegating ments. Moreover, when it comes to large-scale sys-
this functionality to the event source. Today, the typical tems, the tasks of configuring and tuning the service
way to transfer events is individually (i.e., event by event). tends to become complex and tedious. The alterna-
Thisishighlyinefficient,especiallyacrossWAN,duetothe tive is to design autonomic cloud middleware, able to
incurred latencies and overheads at various levels (e.g., ap- self-optimize. Coupledwithaneconomicmodel,these
plication, technology tools, virtualization, network). systems could also regulate the resource consumption
Abetteroptionistotransfereventsinbatches. Whilethis and enforce service-level agreements (SLAs).
improvesthetransferrate,italsointroducesanewproblem,
relatedtoselectingtheproperbatchsize. Figure1presents Generic solution - Building specific optimizations which
thedependencybetweenthebatchsizeandthetransferrate, target precise applications is efficient, but limits the
and the transfer latency per event, respectively. We notice applicabilityofthesolution. Instead,weproposeaset
thatthekeychallengehereisthechoiceofanoptimalbatch of techniques which can be applied in any cloud con-
size and the decision on when to trigger the batch sending. text, independent of the application semantics. Jet-
This choice strongly relies on the streaming scenario and Stream does not depend on the nature of the data,
on the sensed environment (i.e. the cloud). We aim to nor on the query types.
tackle these problems by proposing an environment-aware
3.2 Zoomontheeventdeliverylatency
solution, which enables optimum-sized batch streaming of
events in the clouds. To achieve this, we model the latency We express the latency of the events based on a set of
oftheeventtransferwithrespecttothecloudenvironment, cloudparameterswhichcanbemonitored. Suchatechnique
dynamically adapt the batch size to the context and enable allows to bind the batch size corresponding to the minimal
multi-route streaming across clouds nodes. eventlatencybothtothestreamandtheenvironmentinfor-
mation. We start by breaking down the end-to-end latency
3. MODELINGTHESTREAMINGOFDATA ofaneventinfourcomponents,depictedinFigure2: form-
ing the batch, encoding (e.g., serializing), transferring the
INTHECONTEXTOFCLOUDS
batchanddecoding(e.g.,de-serializing). Thesetofparam-
Thecloudvariabilityandthefluctuatingeventgeneration eters able to describe the context and define the latency is:
ratesrequireanappropriatemodelforeventstreaming,able the average acquisition rate (Rate ) or mean time
Acquisition
to capture the cloud specificities. In this section, we intro- between events (MTBE), the event size (Event ), the
SizeMB
ducesuchamodelandpresentthedecisionmechanismsfor serialization/deserializationtechnique,thethroughput(thr)
selecting the number of events to batch. andthenumberofeventstoputinthebatch(i.e.,batchsize
-batch ). Thegoalistodeterminethelatterdynamically.
3.1 DesignPrinciples Size
Inthefollowing,wewillmodelthelatencycomponentsusing
JetStream relies on the following set of design principles: these parameters.
(cid:20)(cid:8)(cid:21)(cid:22)(cid:18)(cid:23)(cid:13)(cid:11)(cid:2)(cid:3)(cid:7)(cid:24)(cid:18)(cid:14)(cid:2)(cid:11)(cid:25)(cid:15)(cid:5)(cid:26) a) Throughput b) Approximating the Throughput
(cid:23)(cid:15)(cid:25)(cid:18)(cid:23)(cid:13)(cid:11)(cid:2)(cid:3)(cid:7)(cid:24)(cid:18)(cid:14)(cid:2)(cid:11)(cid:25)(cid:15)(cid:5)(cid:26) 70 100
60 80
(cid:1)(cid:2)(cid:3)(cid:4)(cid:2)(cid:5) (cid:6)(cid:2)(cid:7)(cid:2)(cid:8)(cid:9)(cid:2)(cid:5) Throughput (MB/s) 23450000 Percentage (%) 246000
10
0
(cid:17)(cid:13)(cid:11)(cid:13)(cid:18) (cid:10)(cid:3)(cid:11)(cid:2)(cid:5)(cid:12)(cid:2)(cid:4)(cid:8)(cid:13)(cid:11)(cid:2) (cid:19)(cid:17)(cid:2)(cid:13)(cid:3)(cid:11)(cid:11)(cid:13)(cid:2)(cid:18)(cid:5) 0.00127030.1052554630.1205446312.054426310142.27531552510.29262018051.852417 0.0012730.1052554630.1205446312.05442631S0142i.z275e3 1(5M525B10).29262018051.852417
(cid:19)(cid:2)(cid:3)(cid:11)(cid:2)(cid:5) (cid:14)(cid:15)(cid:4)(cid:2)(cid:16) Size (MB) Number of channels
Number of channels 1 2
Figure 3: The proposed schema for multi-route 1 2 3 35 46
streaming across cloud nodes. 4 5 6 Approximation
Figure 4: a) The value of the throughput with re-
The batching latency correspondstothedelayintroduced spect to the number of routes across cloud Small
by an event waiting in the batch for other events to VMs for increasing size of the data chunk. b) Ap-
arrive,beforetheyareallsentasabatch. Theparam- proximatingwithonepatternthecloudthroughput,
eters which describe this latency are the average ac- independent of the number of routes. The approxi-
quisition rate of the events and the number of events mation is built from the averages of the normalized
in the batch. As the delay depends on the position throughput functions for each number of routes.
of the event in the batch, we chose to use the av-
erage latency per event. This can be computed by
averaging the sum of the delays of all events in the the intra-data-center throughput (between the nodes within
batch: Latency = batchSize . Intuitively, one’s single site deployment) is relatively stable and a pri-
batching 2×RateAcquisition ori known, as it is specified by the cloud provider within
this corresponds to the latency of the middle event.
the SLA (e.g., 100 Mbps for Small Azure VMs). On the
The transformation latency groupsthetimestoencode other hand, there are no guarantees when it comes to the
and to decode the batch using a specific serialization inter-data-center throughput (betweenthedata-centers). In
library. It depends on the used format (e.g. binary, fact,thehighlatencyandgenerallylowthroughputnetwork
JSON etc.), the number of bytes to convert and the connecting the sites is not owned by the cloud provider, as
numberofeventsinthebatch. Toexpressthis,werep- the packet routing is done across multiple internet service
resent the transformation operation as an affine func- providers (ISPs). This makes it more unstable, unknownin
tion (i.e. f(x) = ax+b) where a corresponds to the advance and subject to poor performance.
conversionrate(i.e. amountofbytesconvertedpersec- Inordertoaddresstheissueoflowinter-data-centerthrough-
ond-tDs),whilethebconstantgivesthetimetowrite put, we designed a transfer strategy suitable for a typical
themetadata(tHs). Thelatencypereventcanthenbe cloudsetup(i.e.,applicationrunningacrossseveralnodesin
expressedas: Latency = tHs+tDs×batchSizeMB, differentclouddata-centersdeployments),whichcanharvest
whichholdsbothfortthraensefonrcmoadtiionnganddecobdaticnhgSizoeper- extrabandwidthbyusingadditionalintermediatenodes. The
ations. Thisformulacanalsobeappliedtoexpressthe idea is to use multiple routes for streaming across sites, as
size of the data to be transferred by simply replacing depicted in Figure 3. We build this strategy on two obser-
the time constants with data related constants (i.e. vations: the virtual inter data-center routes do not map to
size of the metadata and the compression/extension identical physical paths and the latency of intra-site com-
ratio). munication is low (less than 10%) compared to inter site
communication [45]. These observations allow to aggregate
The transfer latency models the time required to trans- additionalbandwidthbysendingdatafromthesendernodes
fer an event between cloud nodes across data-centers. to intermediate nodes within the same deployment (i.e. be-
To express it, we consider both the amount of data in longing to the same application space), which will then for-
the batch as well as the overheads introduced by the ward it towards the destination. With this approach, the
transferprotocol(e.g. HTTP,TCP)-sOtandtheen- system aggregates extra physical bandwidth from the dif-
coding technique - sOe. Due to the potentially small ferent paths and routes established when connecting dif-
size of data transferred at a given time, the through- ferent nodes across the ISP infrastructures. This generic
put between geographically distant nodes cannot be method of aggregating inter-site bandwidth is further dis-
expressed as a constant value. It is rather a func- cussed in [45], in the context of bulk multi-hop data trans-
tion of the total batch size (SizeTotal =batchSizeMB× fers across multiple cloud data-centers. To integrate this
batchSize), since the impact of the high latency be- approachwithinourmodel,weextendthesetofparameters
tween data-centers depends on the batch size. The that are used to characterize the stream context with the
cloud inter data-center throughput - thr(Size) is dis- number of channels.
cussed in more detail in the following section. The Independent of the number of routes used for streaming,
average latency for transferring an event can then be the value of the throughput function has to be known at
expressed as Latencytransfer = sOt+tshOr(eS+izbeaTtocthaSli)zeMB runtime. In order to limit the number of network samples
that need to be performed to obtain it, and to reduce the
3.3 Multi-routestreaming
monitoring intrusiveness, we approximate this function. In
The throughput is one of the most peculiar aspects when Figure 4 a) we present measurements of the throughput for
consideringcommunicationintheclouds. Ontheonehand, a number of routes between North Central US and North
(cid:23)(cid:5)(cid:7)(cid:24)(cid:2)(cid:7)(cid:19)(cid:14)(cid:12)(cid:13) "(cid:9)(cid:13)(cid:9)(cid:12)(cid:19) Algorithm 1 The selection of the optimal batch size and
#$!(cid:11)(cid:16)(cid:3)(cid:14)(cid:10)(cid:2)(cid:3)(cid:9)!(cid:31)(cid:7)(cid:7)(cid:14)(cid:21)(cid:2)(cid:20)!(cid:22)(cid:2)(cid:3)(cid:9)!
(cid:25)(cid:5)(cid:19)(cid:9)(cid:16) the number of channels to be used
(cid:28)(cid:2)(cid:3)(cid:8)(cid:29) (cid:4)(cid:9)(cid:12)(cid:19)(cid:9)(cid:7) (cid:2)%(cid:12)$!(cid:19)(cid:18)!(cid:10)(cid:9)(cid:2)(cid:9)(cid:16)(cid:16)(cid:6)(cid:16)(cid:7)(cid:2)(cid:9)(cid:13)!(cid:9)(cid:15)!(cid:29)(cid:4)(cid:7)(cid:14)(cid:5)(cid:26)(cid:9)(cid:6)(cid:13)(cid:29) (cid:6)(cid:3) 1: procedureBatchAndChannelsSelection
(cid:30)(cid:7)(cid:2)(cid:8)(cid:20)(cid:9) (cid:25)(cid:5)(cid:19)(cid:9) (cid:15)(cid:7)(cid:2)(cid:12)(cid:16)(cid:17)(cid:9)(cid:7) &$!’(cid:12)((cid:6)(cid:14)(cid:7)(cid:9)!(cid:24)(cid:29)(cid:9)(cid:12)!(cid:3)(cid:5)!(cid:16)(cid:9)(cid:12)(cid:19) 2: getMonitoredContextParameters()
# & % (cid:18)(cid:5)(cid:19)(cid:6)(cid:20)(cid:9)
3: ESTIMATEMaxBatchedfrom[MaxTimeConstraint]
(cid:28)(cid:6)*(cid:9)(cid:7) (cid:11)(cid:21)(cid:9)(cid:12)(cid:3) (cid:15)(cid:7)(cid:2)(cid:12)(cid:16)(cid:17)(cid:9)(cid:7) (cid:15)(cid:7)(cid:2)(cid:12)(cid:16)(cid:17)(cid:9)(cid:7) (cid:11)(cid:21)(cid:9)(cid:12)(cid:3) (cid:28)(cid:6)*(cid:9)(cid:7) 4: whilechannels<MaxNodesConsidereddo
(cid:4)(cid:9)(cid:12)(cid:19)(cid:9)(cid:7) (cid:18)(cid:5)(cid:19)(cid:6)(cid:20)(cid:9) (cid:18)(cid:5)(cid:19)(cid:6)(cid:20)(cid:9) (cid:22)(cid:9)(cid:8)(cid:9)(cid:14)(cid:21)(cid:9)(cid:7) 5: whilebatchsize<MaxBatcheddo
(cid:22)(cid:9)(cid:8)(cid:9)(cid:14)(cid:21)(cid:9)(cid:7) 6: ESTIMATE latency from
(cid:4)(cid:9)(cid:7)(cid:14)(cid:2)(cid:20)(cid:14)(cid:26)(cid:9)(cid:7) )(cid:31)(cid:25) (cid:25)(cid:5)(cid:19)(cid:9) (cid:1)(cid:9)(cid:27)(cid:4)(cid:9)(cid:7)(cid:14)(cid:2)(cid:20)(cid:14)(cid:26)(cid:9)(cid:7) [RateAcquisition,batchsize] batching
7: ESTIMATE latency from
encoding
[overheads,batchsizeMB]
(cid:1)(cid:2)(cid:3)(cid:2) (cid:1)(cid:2)(cid:3)(cid:4)(cid:3)(cid:5)(cid:2)(cid:6)(cid:7) (cid:4)(cid:3)(cid:7)(cid:9)(cid:2)(cid:10) (cid:46)Estimatethetransferlatencyfor1channel
(cid:4)(cid:5)(cid:6)(cid:7)(cid:8)(cid:9) (cid:31)(cid:19)(cid:2) (cid:3)(cid:14)(cid:21)(cid:9)!(cid:28)(cid:2)(cid:3)(cid:8)(cid:29)!(cid:4)(cid:3)(cid:7)(cid:9)(cid:2)(cid:10)(cid:9)(cid:7) (cid:11)(cid:12)(cid:13)(cid:14)(cid:12)(cid:9) 8: ESTIMATE latencytransfer1 from
[batchsizeMB,thrRef,1channel]
Figure 5: The architecture and the usage setup of 9: COMPUTE RatioCPU from
[latency ,latency ,VM Cores]
encoding batching
the adaptive batch streamer (cid:46)Preventsidlechannels
Europe Azure data-centers. In Figure 4 b) we normalize 10: if RatioCPU ∗ latencybatching × channels <
latency +latency then
thesevalues(i.e.,%ofthecorrespondingstablethroughput) 11: transfer1ESTIMATEencoding latency from
decoding
and approximate them using a polynomial function, which
[overheads,batchsizeMB]
we determined empirically that it gives the most accurate 12: ESTIMATE latency from
transfer
approximation. Hence such a model can in fact represent [batchsizeMB,thrRef,channels]
the throughput pattern, with an error introduced by the 13: ESTIMATE latencyreordering from
cloudvariabilityoflessthan15%. Usingthisapproximation, 14: [channels,latenlactyetnracnysfer] =(cid:80)latency
the entire function can be extrapolated by measuring only perEvent ∗
15: if latency <bestLatencythen
the asymptotic stable throughput. This will be used as the perEvent
16: UPDATE[bestLatency,Obatch,Ochannels]
amplitude which multiplied with the normalized estimation
17: endif
will give the throughput for any size. 18: endif
Thedownsideofusingmultipleroutesforsendingbatches 19: endwhile
isthattheorderingguaranteesofferedbythecommunication 20: endwhile
protocol for one link do not hold anymore. This translates 21: endprocedure
into batches arriving slightly out of order due to changing
conditions on the communication links (e.g., packet drops,
congestions etc.). It is mandatory to maintain the integrity size, the maximal end-to-end event latency introduced by
ofthecommunicationandavoiddroppingdatajustbecause batching can be unsatisfactory for a user, as it might vio-
anotherlinkwasfastersothatbatchesneedtobereordered late application constraints, even if the system operates at
atthedestination. Weachievethisbybufferingthebatches optimaltransferrates. Hence,theuserscansetamaximum
at the destination until their turn to be delivered to the acceptable delay for an event, which will be converted in a
streamingenginearrives. Thisintroducesanewcomponent maximum size for the batch (Line 3). As for the number
for the latency: latency for reordering. It can be modeled ofchannelsselection,weestimatehowmanybatchescanbe
using the Poisson distribution (Poisson(k,λ) = λk×e−λ) to formed while one is being transferred (Lines 6-8).
k!
estimate the probability of having k number of batches ar- Once the transfer of a batch is completed, adding ad-
riving before the expected batch, which fixes λ to 1, as ditional channels to distribute the batches become useless.
we consider as reference the time to transfer one batch . The condition on Line 10 prevents the system from creat-
This probability can then be correlated with a penalty as- ing idle routes. Finally, the CPU usage needs to be also
signedtoeachunorderedbatch,givenbyitslatency. Finally, integrated in the decision process. Sending frequent small
summing these probabilities over the number of channels batches will increase the CPU consumption and artificially
and normalizing based on the events will give an estima- decrease the overall performance of the cloud node. We
tion of the worst case expected latency: Latency = therefore assign a penalty based on the ratio between the
reordering
(cid:80)ci=ha2nnels(cid:80)Lj Poisson(j,1)×j×Latency×batch. Here, L gives the time to form a batch and the time used by the CPU to en-
×batch×size×L code it, according to the formula RatioCPU =
maximum number of batches (e.g., 10) regarded as poten- latencyencoding . Tosumup,JetStream
tially arriving before the reference one through a channel, (latencybatching+latencyencoding)×VM Cores
collectsasetofcontextparameters(Line2)andusesthemto
givingtheupperlimitforiteratingthekPoissonparameter.
estimate the latency components according to the formulas
3.4 Adaptivecloudbatching presented above. Based on these estimations, it selects the
optimalbatchsizeandthenumberofchannelsforstreaming.
InAlgorithm1wewrapupeverythingandpresentthede-
cision mechanism in JetStream for selecting the number of
4. SYSTEMOVERVIEW
events to batch and the number of channels to use. The al-
gorithmestimatestheaveragelatencypereventforarange We designed JetStream as a proof of concept of the pre-
of batch sizes and channels, retaining the best one. As an vious approach. Its conceptual schema is presented in Fig-
optimization, a simulating annealing technique can be used ure5. Theeventsarefedintothesystematthesenderside,
toperformthesearchfaster,byprobingthespacewithlarge assoonastheyareproduced,andtheyarethendeliveredto
steps and performing exhaustive searches only around local the stream processing engine at the destination. The pro-
optima. Depending on the magnitude of the optimal batch totypewhichimplementsthisarchitecturewasdevelopedin
C#usingthe.NET4.5framework. Thedistributedmodules Latency components
and their functionality are described bellow: 0.014
0.012
The BfourffJeertSitsreuasmed. bTothheassenadneirnpauptplaicnadtioountpourttehnedepvoeinntt me (sec) 00..000.000681
source adds the events to be transferred, as they are Ti 0.004
produced,whilethereceiverapplication(i.e.,thestream 0.002
0
processing engine) pops (synchronously) the events or 1 5 10 25 50 100 200 400 500 750 1000
Batch Size (#Events)
it is notified (asynchronously) when the events have
Latency
been transferred. The module relies on two queues
Total Batching Transfer
for synchronization purposes in a producer-consumer Transformation
fashion. TheBufferisalsoinchargeofmonitoringthe Figure 6: The latency components per event with
input stream in order to assert in real time the acqui- respecttothebatchsizeforasinglestreamingchan-
sition rate of the events and their sizes. nel, for inter-sites event transfers
Avro (Microsoft HDInsight), but others modules can
The Batch Oracle staysatthecoreofJetStream,asiten-
be easily integrated. This module can host additional
forces the environment-aware decision mechanism for
functionalities (e.g., data compression or deduplica-
adaptively selecting the batch size and the amount of
tion) in the future.
channels to use. It implements Algorithm 1 and col-
lects the monitoring information from the Buffer and
the Transfer Module. It further controls the moni-
5. EVALUATION
toring intrusiveness by adjusting the frequency of the
monitorsamplesaccordingtotheobservedvariability. Thegoaloftheexperimentalevaluationpresentedinthis
section is to validate the JetStream system in a real cloud
The Transfer Module implementsthemulti-routestream- setup and discuss the main aspects that impact on its per-
ing. On the intermediate nodes, its role is to forward formance.
the packets towards the destination. Currently, the
5.1 Experimentalsetup
modulesisagnosticofanystreamingscenarioandcan
integrateanytransferprotocol. Itprovidesseveralim- The experiments were run in the Microsoft’s Windows
plementations on top of TCP: synchronous and asyn- Azure cloud in the North-Central US and the North Eu-
chronous,single-ormulti-threaded. Itisalsoincharge rope data-centers. We chose these sites in order to create a
ofprobingthenetworkandmeasuringthethroughput geographical-distributedsetupwhichpermitstoanalyzethe
and its variability. The batches are sent in a round- abilityofthesystemtoadapttochangingfactorsgenerated
robin manner across the channels, balancing the load. both by the cloud itself (i.e., multi-tenancy, virtualization)
As future work, the transfer performance can be im- andthewideareacommunicationacrossmultipleISPs. All
proved by integrating additional transfer techniques theexperimentswererunwithSmallWebRoleVMshaving
amongwhichthesystemcanautomaticallyswitchbased 1 CPU, 1.75 GB of memory, 225 GB local storage and 100
onthestreamingpatternandthecontextinformation. Mbps bandwidth. When using multi-route streaming, up
to 5 additional nodes were used within the sender deploy-
The Event Sender coordinatestheeventtransfersbyman-
ment, implementing the transfer scheme discussed in Sec-
aging the interaction between all modules. It queries
tion 3.3. Each experiment sends between 100,000 and 3.5
the Batch Oracle about when to start the transfer of
million events, which, given the size of an event, translates
the batch. Next, it setups the batch by getting the
intoatotalamountofdatarangingfromtensofMBsto3.5
eventsfromtheBuffer andaddingthemetadata(e.g.,
GB.Eachsampleiscomputedastheaverageofatleastten
batch ID, streams IDs etc.). The batch is then serial-
independent runs of the experiment performed at various
ized by the Serialization module and the data trans-
moments of the day (morning, afternoon and night).
ferred across data-centers by the Transfer Module.
The performance metrics considered are the transfer rate
The Event Receiver isthecounterpartoftheEventSender and the average latency of an event. The transfer rate is
module. The arriving batches are de-serialized, re- computed as the ratio between a number of events and the
ordered and delivered to the application as a stream time it takes to transfer them. More specifically, we mea-
of events, in a transparent fashion for the stream pro- sured,atthedestinationside,thetimetotransferafixedset
cessing engine. The module issues acknowledgements ofevents. Fortheaveragelatencyofanevent,wemeasured
to the sender or makes requests for re-sending lost or thenumberofeventsinthesenderbuffer, thetransfertime
delayed batches. Alternatively, based on users’ poli- andreportedthenormalizedaveragepereventbasedonthe
cies, it can decide to drop late batches, supporting latency formulas described in Section 3.2.
theprogressofthestreamprocessingdespitepotential Theevaluationisperformedusingasyntheticbenchmark
cloud-related failures. Users configure when such ac- and a real-life application. We created a synthetic bench-
tionsareperformedbymeansofwaitingtimes,number markinordertohavefullcontroloverthesizesandthegen-
ofbatches,markersorcurrenttimeincrements(CTI). eration rate of the events. The generation rates are varied
between hundred of events per second to tens of thousands
Serialization/De-Serialization has the role of convert- ofeventspersecond,asindicatedbythescaleofthetransfer
ing the batch to raw data, which can be afterwards rates. Such a predictable and configurable setup allows us
sent over the network. We integrate in our prototype to analyze and to understand the results better when test-
several libraries: Binary (native), JSON (scientific) or ing the behavior of JetStream in several scenarios. Finally,
Real Measured Model-based Estimation Low Events Arrival Rate High Events Arrival Rate
4 4 Transfer Rate Transfer Rate
1200
3 3 16000
Time (sec) 12 Time (sec) 12 Events/sec 1046800000000 Events/sec 128000000
4000
200
0 50 100 500 1000 2500 0 50 100 500 1000 2500 0 1 Route 3 Routes 5 Routes 0 1 Route 3 Routes 5 Routes
Batch Size (#Events) Batch Size (#Events)
Average Latency per Event Average Latency per Event
2 3 4 5 2 3 4 5
6 Number of channels 6 Number of channels 0.002 0.0002
Fcaigseurlaete7n:cCyo(mi.ep.,arthinegptehnealetyst)imofautinoonrdoefrethdebwatocrhsets- me (sec) 000...000000011826 me (sec) 00..800E00−00011526
with actual measurements, while increasing the Ti 0.0004 Ti 4E−05
number of channels used for the transfers 0 1 Route 3 Routes 5 Routes 0 1 Route 3 Routes 5 Routes
Batch Size Batch Size
1 10 100 1 10 100
we use the events collected by the MonALISA monitoring 1000 5000 JetStream 1000 5000 JetStream
systems from the Alice LHC experiment to assert the gains Figure8: Comparingtheperformance(transferrate
brought by our approach in a real setup. - top and average latency per event - bottom) of in-
dividualeventstreamingandstaticbatcheswiththe
5.2 Accuracy adaptivebatchofJetStreamfordifferentacquisition
rates, while keeping the size of an event fixed (224
We depict in Figure 6 the total latency and its compo-
bytes). Thelatencyandtransferratesforindividual
nents, per event, with respect to the number of batched
event streaming (i.e., batch of size 1) are between
events. Selecting the optimal size of the batch comes down
50 to 250 times worse than the ones of JetStream
to finding the value corresponding to the minimal latency
(e.g. ∼ 200 for the scenario illustrated in Figure 6). The Transfer Rate Average Latency per Event
searchforthebatchsizethatminimizesthelatencyperevent 12000 0.0002
0.00018
is at the core of the JetStream algorithm presented in Sec-
10000 0.00016
tomifoarnteipo3nr.4es.seaTnbtohauettiosetnlhefecotreiontnvhierooflnattmheenencoytp(tiseim.cgoa.mlnvpeautlwuteeodrfkrfrootmhmrtothuhigsehtepysupttie-, Events/sec 468000000000 Time (sec) 00..06800.EE000−−0000011055241
CPU),whichmaynotbeexact. Indeed,thecloudvariability 2000 4E−05
canleadtodeviationsaroundtheoptimalvalueintheselec- 2E−05
0 0
tion process of the amount of events to batch. Considering 1 Route 3 Routes 5 Routes 1 Route 3 Routes 5 Routes
thattheperformancearoundtheoptimumisratherflat(as Batch Size Batch Size
seen in Figure 6), JetStream delivers more than 95% of the 1 10 1 10
100 1000 100 1000
optimal performance even in the case of big and unrealistic 5000 JetStream 5000 JetStream
shifts from the optimum batch size (e.g. selecting a batch Figure 9: The performance (transfer rate – left and
sizeof150or250insteadof200inFigure6willdecreasethe averagelatencyperevent–right)ofindividualevent
performance with 3%). The solution for further increasing streaming, static batches and JetStream for events
the accuracy, when selecting the batch size, is to monitor of size 800 bytes. The latency and transfer rates for
the cloud more frequently. These observations show that individual event streaming (i.e. batch of size 1) are
accuracy can be traded for monitoring intrusiveness at the from 40 up to 150 times worse than JetStream
level of cloud resources (e.g. network, memory and CPU).
In order to validate the penalty model proposed for the
5.3 Individualvs. batcheventtransfers
latency of reordering when using multiple routes, we com-
pare the estimations of our approach with actual measure- The goal of this set of experiments is to analyze the per-
ments. TheresultsarepresentedinFigure7. Thedelayfor formance of individual event streaming compared to batch-
unordered batches was measured as the time between the based streaming, with static and adaptive (i.e., JetStream)
momentwhenanoutoforderbatch(nottheonenextinse- sizes. Inthestaticcase,thenumberofeventstobebatched
quence) arrives, and the moment when the actual expected is fixed a priori, with a batch of size 1 representing inde-
one arrives. The experiment shown in Figure 7 presents pendent event streaming. These setups are compared to
the average of this maximum unordered delay over tens of JetStream, which adapts the batch size to the context at
independent trials in which a fixed amount of events (i.e. runtime, for varying event sizes or generation rates.
1 million events of size 2.5 KB) is transferred. We notice The experiments presented in Figure 8 use an event of
thattheproposedmodelgivesagoodestimationofthisde- size 224 bytes and evaluate the transfer strategies consider-
lay, which can be used as a penalty latency for multi route ing low (left) and high (right) event generation rates. The
streaming. The accuracy in this case is 90%–95%, because experiments were repeated for different number of routes
the penalty model for multi-routing does not integrate the for streaming: 1, 3 and 5, and measure the transfer rates
variability of the cloud. Yet, such errors can be tolerated (top)andaveragelatencyperevent(bottom). Thefirstob-
as they are not determinant when selecting the number of servation is that batch-based transfers clearly outperform
channels to use. individual event transfers for all the configurations consid-
ered. The results confirm the high overhead and the low (cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:6)(cid:7)(cid:2)(cid:8)(cid:3)(cid:9)(cid:7) 35000
throughput with small sizes for the transfers between data- 30000
ctoenhtuernsd.rGedrooufptinimgetshe(uepvetnots2i5n0crteimaseesstfohreJtreatSntsrfeeramra)tewtheinles Ba51t00c00h00Size ents/sec 122505000000000
decreasing the average latency per event. Two aspects de- 1Je0t0Stream Ev 10000
EventArrivalRate 5000
terminetheperformance: thesizeofthebatchwithrespect
0
to the stream context and the performance changes of the 1 3 5 7 T9ime11int1e3rva15l(11T07imsee1c9oinnt2de1srv)a23l(1205se27con2d9s)
cloud. Static batches cannot handle these variations, mak-
30000 7000
i(aneng.dgc.heribgtaahticnehvbeeasnttochfacssqiizzueeiss1igt0iooogndivreiantpeoonaoenrdcpogneortfoeoxdrtmpaeanrndfcoebrmafdoarninc1eortfohouertr5es Events/sec 112205055000000000000000 Events/sec 123456000000000000000000
routes and low acquisition rates). Selecting the correct size 0 5 6 7 0 7 8 9
atruntimebringsanadditionalgainbetween25%and50% Timeinterval(10seconds) Timeinterval(10seconds)
over static batch configurations. We repeated the same set BatchSize BatchSize
5000 1000 100 JetStream 5000 1000 100 JetStream
of experiments for bigger event sizes, all showing the same
behavior. Due to space constraints, we only illustrate the Figure10: Theevolutionofthetransferrateintime
transfer rate and average event latency for an event size of for variable event rates with JetStream and static
800 bytes in Figure 9. batches transfer strategies
Transfer Rate
5.4 Adaptingtothecontextchanges
40000
Thedataacquisitionprocessinastreamingscenarioisnot
necessarilyuniform. Fluctuationsintheeventratesofanap- 30000
c
poflitchaetidonatrausnonuirncge,inthtehveirctluoualdizceadninafprpasetarrudctuueretoorthtehencaltouurde nts/se 20000
e
performancevariability[16]. ToanalyzethebehaviorofJet- Ev
10000
Stream in such scenarios, we performed an experiment in
which the event generation rate randomly changes in time.
0
For the sake of understanding, we present in Figure 10 a 1 3 5
Number of streaming routes
snapshot of the evolution of the transfer rate in which we
use fine grain intervals (10 seconds) containing substantial 100 Batch Size JetStream
rate changes. JetStream is able to handle these fluctua- Figure 11: The transfer rate for an increasing num-
tions by appropriately adapting the batch size. In contrast, ber of routes used for streaming
static batch transfers either are introducing huge latency
fromwaitingfortoomanyevents,especiallywhentheevent cisely, a larger bandwidth allows to send more data, and
acquisition rate is low (e.g., batches of size 1000 or 5000 at implicitly, the additional data carried with each batch does
time moment 9) or are falling behind the acquisition rate not throttle the network anymore. This brings the trans-
which leads to increasing amount of memory used to buffer ferrateofsmaller,andconsequentlymorefrequent,batches
the untransferred events (e.g., batch of size 100 at moment closertothemaximumpotentialeventthroughput. Itisthe
5). Reacting fast to such changes is crucial for delivering caseofthestaticbatchofsize100onFigure11,deliveringa
high performance in the context of clouds; this is also the throughput close to JetStream for higher number of routes.
reasonwhywechosetoconsidersuchsuddenrateschanges. With higher throughput and a lower overhead, the optimal
The number of used resources (e.g. the extra nodes which batch size can be decreased. When selecting it, JetStream
enable multiple route streaming) is reduced by 30% when isabletodecreasetheendtoendlatency. Weconcludethat
takingintoaccountthestreamcontext. Theseresourcesav- sustaininghightransferratesunderfixedtimeconstraintsis
ingsarejustifiedbythefactthatloweventacquisitionrates possiblebyimposingupperboundsforthebatchsizes,which
do not require multiple route streaming as higher ones do. enables JetStream to integrate users’ time constraints for
Additionally, as shown in [23], the fluctuations in the ap- maximum delay. As discussed in Algorithm 1, it does this
plication loads have certain patterns across the week days. by considering a maximum limit on the batch.
Hence,inlongrunningapplications,ourapproachwillmake
5.6 ExperimentingwithALICE:areal-life
substantialsavingsbydetectingthesedailyorhourlytrends
and using such knowledge to scale up/down the number of HighEnergyPhysicsapplication
additional nodes used for transfers. In a second phase, our goal was to assess the impact of
JetStream in a real-life application. We opted for ALICE
5.5 Benefitsofmultipleroutestreaming
(A Large Ion Collider Experiment) [1], one of four LHC
Figure 11 shows the gain obtained in transfer rate with (Large Hadron Collider) experiments at CERN (European
respecttothenumberofroutesusedforstreaming,forJet- OrganizationforNuclearResearch),asitsscale,volumeand
Stream and for a static batch of a relatively small size (i.e. geographical distribution of data require appropriate tools
100events). Multipleroutestreamingpaysoffwhenincreas- for efficient processing. Indeed, the ALICE collaboration,
ing the amount of data sent for both strategies (i.e. static consisting of more than 1,000 members from 29 countries
andadaptive). Infact,byaggregatingextrabandwidthfrom and 86 institutes, is strongly dependent on a distributed
the intermediate nodes, we are able to decrease the impact computingenvironmenttoperformitsphysicsprogram. The
of the overhead (batch metadata, communication headers, experiment collects data at a rate of up to four petabytes
serialization headers etc.) on smaller batches. More pre- per year, produce more than 109 data files per year, and
require tens of thousands of CPUs to process and analyze a) Latency b) Transfer Rate
them. TheCPUandstoragecapacitiesaredistributedover 2500 1600
morethan80computingcentersworldwide. Theseresources 2000 1400
1200
atoreOohpueertrefaortocinugegs,nseiynosuttsehmienseaanlsledarisbepasetcoctfhse,qxfurpoeeumriinmCgePnsUotsft,mwisoadroeen.lathnedmcoounnit- Time (sec) 11050000 Events/sec 1068000000
toringinformationcollectedinreal-timeaboutallALICEre- 500 400
200
sources. WeusedtheMonALISA[33]servicetoinstrument
0 0
timheenhtu.geMaomreoutnhtanof3m50onMitoorniAngLIdSaAtasiesrsvuiecdesfraorme rtuhnisneinxgpeart- 10 100 1000 1000J0etStream 1 10 100 1000 100J0et0Stream
sitesaroundtheworld,collectinginformationaboutALICE Batch Size (#Events) Batch Size (#Events)
computingfacilities,localandwideareanetworktraffic,and
Figure 12: The total latency (a) and the trans-
thestateandprogressofthemanythousandsofconcurrently
fer rate (b) measured when transferring 1.1 million
running jobs. This yields more than 1.1 million parameters
eventsfromMonALISAmonitoringtheAliceexper-
published in MonALISA, each with an update frequency of
iment. The latency for independent event transfer
one minute. Using ALICE-specific filters, these raw pa-
(i.e. batchofsize1)ina)isnotrepresentedbecause
rameters are aggregated to produce about 35,000 system-
itwouldmodifythescalegreatlydueitsmagnitude,
overview parameters in real time. These overviews are usu-
having a value of 30900 seconds as opposed to 80
ally sufficient to identify problems or to take global actions
seconds for JetStream
inthesystemandtheyarethefuelfortheJetStreamevent
streaming platform. The MonALISA framework and its
latencyoftheeventsasdepictedinFigure12a. Asourap-
highfrequencyupdatesforlargevolumesofmonitoringdata
proach selects the appropriate batch size at each moment,
matched closely with JetStream’s architecture so the pieces
itconsequentlyreducestheamountofeventswaitinginthe
fitnaturallytogether,butitalsoprovidedusamajoroppor-
senderqueueanddecreasestheoveralllatencyoftheevents.
tunity to fullfill the system’s initial goal: using the (moni-
Compared to the static batch strategies providing constant
toring)datatotakeautomateddecisionsinrealtime(inthis
transfer performance, the latency of JetStream is decreased
context, improve the observed system).
from2.2(100-eventbatches)downto17times(10000-event
Based on the monitoring data collected and stored by
batches).
MonALISA as of December 2013, we have replayed a se-
quenceof1.1millioneventsconsideringtheircreationtimes
at the rate they were generated by Alice. The measure- 6. RELATEDWORK
mentswereperformedusing2additionalVMsasintermedi-
Theexistingworkscanbegroupedinto5categoriesbased
ate nodes at the sender side (i.e. 3 streaming routes). Ini-
ontheirapplicationdomainsandfeatures. Wediscussthem
tially,theexperimentalsetupconsidered5streamingroutes.
in this section.
However,duringthetransferofthedatausingJetStream,we
observedthatmaximum3suchrouteswereused,asthesys-
Systemsfordatatransfers.
tem determined that the performance cannot be increased
Severalsolutionshavebeenproposedinthelastyearsfor
beyond this point. The same number of nodes is recom-
managing data transfers at large scale. StorkCloud [30] in-
mendedifwequerythesystembasedonthestreamcontext
tegrates multi-protocol transfers in order to optimize the
(i.e. event size and acquisition rate). The accuracy of this
end-to-end throughput based on a set of parameters and
decisionwasinfactvalidatedasouradaptiveapproachwas
policies. It adapts the parallel transfers based on the clus-
obtaining the same transfer performance using 3 nodes as
ter link capacity, disk rate and CPU capacity, using the
the static batch configurations which were using 5. Fur-
algorithm proposed in [47]. The communication between
thermore, the static configurations also obtained the same
StorkCloud components is done using textual data repre-
transfer performances when switched to 3 nodes, showing
sentation. Similarly, our system optimizes the streaming
that indeed this streaming context was not requiring more
betweendata-centersbymodelingasetofparameters. How-
than 3 streaming routes. This observation shows the im-
ever,JetStreamremainstransparenttothedataformatand
portance of an environment-aware adaptive approach, not
limits the annotation of events or batches with metadata,
subject to arbitrary human configuration choices.
reducing the overhead of the transfer, which becomes cru-
Figure 12 displays the total latency of the events at the
cial in the context of continuous streams of data. Other
sender(Figure12a)andthetransferrate(Figure12b)when
systems, like NetSticher [32], target bulk transfers between
comparing JetStream with static configurations for varying
data-centers, as we do. The authors exploit the day/night
batchsizes. Thetransferperformancesofstaticbatcheswith
patternpeaksofusageofadata-centerandleveragethere-
more than 100 events are similar with JetStream. Consid-
mainingunutilizedbandwidth. ThoughNetSticherisuseful
ering that the generation rate of the events varies from low
for backups and checkpoints, it does not work for real-time
to high, sub-optimal batch sizes will in fact lead to an ac-
systems. GlobusOnline[3]isanothersystemwhichprovides
cumulation of the events in the sender queue during the
data transfers over WAN, targeting data sharing between
peak rates. These buffered events will artificially increase
different infrastructures (e.g. grids, institution infrastruc-
the performance, at the expense of extra memory, during
tures,communityclouds). Unlikeoursystemwhichhandles
theperiodswhentheacquisitionrateofeventsislow. Allin
streams of events with a light resource fingerprint, Globu-
all,thisbehaviorwillproduceastabletransferperformance
sOnline manages only file transfers and has substantial re-
overawiderangeofstaticbatchsizes,asitcanbeobserved
sourcerequirements. Otherrecenteffortsforenablingparal-
in Figure 12 b; but on the other hand, it will increase the
leldatatransfersleadtothedevelopmentofthemulti-path