Table Of ContentOptimal Elephant Flow Detection
Ran Ben Basat Gil Einziger Roy Friedman Yaron Kassner
Computer Science Department Nokia Bell Labs Computer Science Department Computer Science Department
Technion [email protected] Technion Technion
[email protected] [email protected] [email protected]
Abstract—Monitoringthetrafficvolumesofelephantflows,in- SDN and NFV realizations, the ability to perform computa-
cludingthetotalbytecountperflow,isafundamentalcapability tions in a timely manner greatly depends on whether the data
for online network measurements. We present an asymptotically
structures fit in the hardware cache and whether the relevant
7
optimalalgorithmforsolvingthisproblemintermsofbothspace
1 memory pages can be pinned to avoid swapping.
and time complexity. This improves on previous approaches,
0
which can only count the number of packets in constant time. While the problem of identifying the top-K flows and
2
We evaluate our work on real packet traces, demonstrating an heavy-hitters in terms of the number of packets has been
n up to X2.5 speedup compared to the best alternative.
addressed by many previous works, e.g., [7], [19], [30], [37],
a
J I. INTRODUCTION detectionofelephantflowsintermsoftheirtrafficvolumehas
receivedfarlessattention.However,itisnon-trivialtotranslate
5 A. Background
1 a top-K or heavy-hitters algorithm to an efficient elephant
Network monitoring is at the core of many important
flowsolution,becausemanyoftheformermaintainorderedor
] networking protocols such as load balancing [6], [21], traffic semi-ordered data structures [7], [19], [30], [37]. Performing
S
engineering [9], [43], routing, fairness [29], intrusion and
fast updates to these data structures depends on the fact
D
anomaly detection [25], [39], [46], caching [23], policy en-
that each packet increments (or decrements) its corresponding
. forcement [41] and performance diagnostics [18]. Effective
s counter(s)by1.Hence,theperturbationcausedbyeachupdate
c network monitoring requires maintaining various levels of
isfairlycontained,predictableandtractable.Incontrast,when
[
traffic statistics, both as an aggregate and on a per-flow basis.
eachpacketmodifiesitscorrespondingcounter(s)byitsentire
1 This includes the number of distinct flows, also known as size,suchmaintenancebecomesmuchharder.Further,treating
v flow cardinality, the number of packets generated by each
an increment (or decrement) by the packet’s size S as a
1 flow, and the total traffic volume attributed to each flow.
sequenceofS increments(ordecrements)by1wouldmultiply
2
Each of these adds complementing capabilities for managing
0 the update time by a factor of S, rendering it too slow.
and protecting the network. For example, a large increase in
4
1) Contributions: In this paper, we introduce the first
0 cardinalitymayindicateaport-scanningattack,whilestatistics
. about the number of packets or the volume of traffic can elephant flow detection scheme that provides the following
1
benefits: (i) constant time updates, (ii) constant time point-
0 help perform load balancing, meet QoS guarantees and detect
queries, (iii) detection of elephant flows in linear time, which
7 denial-of-service (DoS) attacks. For the latter, identifying
1 the top-K flows, or the heavy-hitters and elephant flows is optimal, and (iv) asymptotically optimal space complexity.
: This is achieved with two hash-tables whose size is propor-
v are essential competencies. Similarly, traffic engineering [9]
tional to the number of elephant flows, and a single floating
i involves detecting high volume flows and ensuring that they
X point variable. We present two flavors of the algorithm for
are efficiently routed.
r maintaining these data structures. The first version Iterative
a Operation speed is of particular importance in network
Median SUMming (IM-SUM), which is easier to describe and
measurement. For example, a reduction in latency for par-
simpler to code, works in amortized O(1) time. The second
tition/aggregate workloads can be achieved if we are able
variant De-amortized Iterative Median SUMming (DIM-SUM)
to identify traffic bursts in near real time [18]. The main
de-amortizes the first, thereby obtaining worst case O(1)
challenges in addressing the above mentioned tasks come
execution time.
from the high line rates and large scale of modern networks.
Specifically, to keep up with ever growing line rates, update We also evaluate the performance of these variants on both
operations need to be extremely fast. As already mentioned, synthetic and real-world traces, and compare their execution
when near real-time decisions are expected, queries should time to other leading alternatives. We demonstrate that IM-
alsobeansweredquickly.Inaddition,duetothehugenumber SUM is up to 2.5 times faster than any of our competitors
of flows passing through a single network device, memory is on these traces, and up to an order of magnitude faster than
becomingamajorconcern.Inahardwareimplementation,the others.DIM-SUMisslowerthanIM-SUM,butstillfasterthan
datastructuresshouldfitinTCAMorSRAM,becauseDRAM all competitors for small values of ε. DIM-SUM is faster in
is too slow to keep up with line rate updates. Similarly, in a terms of worst case guarantee, ensuring constant update time.
software implementation, as can be envisioned in upcoming The latter is important when real-time behavior is required.
Algorithm Space QueryTime UpdateTime Deterministic
SpaceSaving[14] O(cid:0)1(cid:1) O(1) O(cid:0)log1(cid:1) (cid:51)
(cid:15) (cid:15)
CMSketch[16] O(cid:0)1 ·log1(cid:1) O(cid:0)log1(cid:1) O(cid:0)log1(cid:1) (cid:55)
(cid:15) δ δ δ
CountSketch[11] O(cid:16)1 ·log1(cid:17) O(cid:0)log1(cid:1) O(cid:0)log1(cid:1) (cid:55)
(cid:15)2 δ δ δ
ThisPaper-IM-SUM O(cid:0)1(cid:1) O(1) O(1)amortized (cid:51)
(cid:15)
ThisPaper-DIM-SUM O(cid:0)1(cid:1) O(1) O(1)worstcase (cid:51)
(cid:15)
Table I: Comparison of frequency estimation algorithms that support weights.
B. Related work counting, since all counter updates are +1, Space Saving and
FR can be implemented in O(1) [7], [37]. This makes Space
Network monitoring capabilities are needed in both hard-
SavingandFRasymptoticallyoptimalinbothspaceandtime.
ware and software [38]. In hardware, space is a critical
TheversionofSpaceSavingwecompareagainstinthispaper
constraint as there is no sufficient memory technology; while
uses aheap tomanage itscounters [14],[15], [35]; wedenote
SRAM is fast enough to operate at line speed, it is too small
it hereafter SSH. In the general case, packets have different
to accommodate all flows. On the contrary, DRAM is too
sizes so SSH requires a logarithmic runtime. This is preferred
slow to be read at line speed. Traditional network monitoring
overtheoriginalimplementationwithorderedlinkedlists[37],
approachesutilizedshortprobabilisticcounters[22],[42],[44]
whichrequiresaconstanttimetocountthenumberofpackets
to reduce the memory requirements.
but a linear time when considering packet sizes.
Counter arrays are managed by network devices [22], [42],
Various methods such as a sketches for tail items and a
[44]aswellassketchessuchastheCountSketch[11]andthe
randomized counter admission policy can make (empirically)
CountMinSketch(CM-Sketch)[16].Thesealgorithmscanbe
more efficient data structures [27], [8]. These improvements
extendedtosupportfindingfrequentitemsusinghierarchyand
are orthogonal to the one presented in this paper.
group testing [14], [16], [17]. As network line rates became
Alternatively, [5] implemented elephant flow identification
faster, sketches evolved into more complex algorithms that
with a Sample and Hold like technique [36]. That is, the
require significantly less memory at the expense of a long
controller receives sampled packets and periodically updates
decoding time. Such algorithms include Counter Braids [34],
the monitored flows to reflect the current state of the heavy
Randomized Counter Sharing [32] and Counter Tree [12].
hitters.Unfortunately,samplingdiscardssomeoftheinforma-
While these algorithms handle updates very fast, they incur a
tionaboutpacketsizesthatcanbeusedtoreducetheerror.In
long query time. Thus, queries can only be done off-line and
addition,sincesamplingdoesnottakeintoaccountthepacket
not in real time as required by some networking applications.
size, large packets can be missed, resulting in a large error.
Forsoftwareimplementations,counterbasedalgorithmsare In general, Ω(cid:0)1(cid:1) space is required to approximately count
(cid:15)
often the way to go [14]. These methods typically maintain a with an additive error bounded by ε · R, where R is the
flow table, where each monitored flow receives a table entry. total weight of all items in the stream [37]. The optimal
Counter algorithms differ from one another in the the flow runtime is O(1) for both update and query. Related works
table maintenance. Specifically, in Lossy Counting [36], new which are capable of estimating flow volume (rather than
flows are always added to the table. In order to keep the table packet counting) are summarized in Table I. As listed, our
size bounded, flow counters are periodically decremented and algorithms are the first to offer both (asymptotically) optimal
flows whose counters reach 0 are deleted. Lossy counting space consumption and optimal runtime.
is simple and effective, but its space consumption is not
optimal.LossyCounting’sspaceconsumptionwasempirically II. MODEL
improved with probabilistic eviction of entries [20] as well as We consider a stream (S) of tuples of the form, (a ,w ),
i i
statistical knowledge about the stream distribution [40] that where a denotes item’s id and w its (non-negative) weight.
i i
allows dropping excess table entries earlier. Ateachstep,anewtupleisaddedtothestreamandwedenote
Frequent (FR) [30] is a space optimal algorithm [31]. In by N the current number of tuples in the stream.
FR, instead of decrementing counters periodically, when a Given an identifier x, we denote the total weight of x as:
packet arrives for a non resident flow and the table is full, all
(cid:88)
countersaredecremented.Thisimprovesthespacecomplexity fxt (cid:44) wi.
to optimal, but flow counters are needlessly decremented and (ai,wi)∈S:
therefore the algorithm is less accurate. This is solved by the ai=x,i≤t
Space Saving algorithm [37]. In Space Saving, when a packet When t=N, we denote: f (cid:44)fN.
x x
that belongs to a flow that does not have a counter arrives, Also, the total weight of all ids in the stream at time t is:
andtheflowtableisfull,thealgorithmevictstheentrywhose
(cid:88)
R (cid:44) w .
counter is minimal. The rest of the counters are untouched t i
and their estimation is therefore more accurate. For packet (ai,wi)∈S:i≤t
Symbol Meaning Intuitively, our periodic maintenance process identifies the
(cid:6)1(cid:7)th largest counter, and evicts all flows whose counters
R Thetotalsumofallitemsinthestreamsofar. is(cid:15)smaller than that counter. This is obtained by internally
Rt Thetotalsumofallitemsinthestreamuntiltimet. splitting the table into two sub-tables named Active and
Passive. Each table entry contains both the flow’s id and a
fx Theweightedfrequencyforitemxinthestreamsofar. counter that represents the flow’s volume. The Active table is
fxt Theweightedfrequencyforitemx,attimet. mutable, while the Passive is read-only. That is, updates only
affect the Active table while queries consider both of them.
f(cid:98)xt Anestimateoftheweightedfrequencyofitemx,attimet. The maintenance process copies all counters that are larger
Aspeed-spacetradeoffparameter,affectsfrequencyof than the quantile of the (cid:6)1(cid:7)th largest flow from the Passive
γ (cid:15)
maintenanceoperationsandmemoryconsumption. table to the Active table, clears the Passive table, and then
Accuracyparameter,small(cid:15)meanslowerestimationerror switches between them. That way, the size of the Active and
(cid:15) butmorecounters. Passive tables remains bounded while the (cid:6)1(cid:7) largest flows
(cid:15)
(cid:108) (cid:109) are never evicted.
qt Theestimateofthetop 1+1γ quantileattimet. Eachtableisconfiguredtostoreatmost:T (cid:44)(cid:6)γ(cid:7)+(cid:6)1(cid:7)−1
(cid:15) (cid:15)
θ Thresholdforelephantflows. items,whereγ isapositiveconstantthataffectsaspeed-space
trade off as explained below. In addition, we keep a single
δ Theerrorprobabilityallowedinrandomizedalgorithms.
floating point variable q, which represents the value of the
Table II: Summary of notations used in this paper. last quantile calculated. q is the default estimation returned
when querying the frequency of non-resident flows. It ensures
that such estimates meet the maximum error guarantee.
A. Problem definitions: We then improve the complexity to worst case O(1) with a
de-amortizationprocess.Intuitively,thisisdonebyperforming
We now formally define the problems we address.
asmallpieceofthemaintenancepriortoeachupdate.Weneed
• (cid:15) - VOLUME ESTIMATION: We say that an algorithm to make sure that maintenance is always finished by the time
solves the (cid:15) - VOLUME ESTIMATION problem if at any
the Active table fills up. Once maintenance is over, we switch
time t, Query(x) returns an estimation f(cid:98)t that satisfies
x the Active table with the (now empty) Passive table.
fxt ≤f(cid:98)xt ≤fxt +Rt·(cid:15). B. Iterative Median Summing
• (θ,(cid:15))-ELEPHANT FLOWS: We say that an algorithm We describe our Iterative Median SUMming (IM-SUM)
solves the (θ,(cid:15)) - ELEPHANT FLOWS problem if at any algorithm and its deamortized variant Deamortized Iterative
time t, an Elephants() query returns a set of elements Median SUMming (DIM-SUM), by detailing four operations:
St, such that for every flow x, fxt >Rt·θ =⇒ x∈St, a query operation, which computes the volume of a flow; an
and fxt < Rt·(θ−(cid:15)) =⇒ x (cid:54)∈ St. We note that if the elephantsmethodthatreturnsallelephantflows;anupdateop-
streamisunweighted(allweightsare1),thisdegenerates eration,whichincreasesthevolumeofaflowcorrespondingto
totheHeavyHittersproblemdiscussedin[14],[37],[45]. a recently arrived packet; and a maintenance operation, which
periodically discards infrequent flows to prevent overflowing.
B. Notations
a) Query(x): The query procedure appears in Figure 1.
ThenotationsusedinthispaperaresummarizedinTableII.
As depicted, the estimated weighted frequency of element x,
III. SOLUTION ((f(cid:98)x), is Active[x] if x has an Active table entry, Passive[x]
if x has a Passive table entry and otherwise q.
A. Intuition
b) Elephants(): Returns the elephant flows. First, we
Ideally, when the table is full, we wish to evict the smallest set S ← ∅; then, we traverse both the Active and Passive
t
flowasinSpaceSaving.However,intheweightedcase,Space tables, adding to S any flow x for which Query(x)≥R ·θ.
t t
Saving runs in logarithmic time. We achieve constant runtime c) Update: The update procedure is illustrated in Fig-
by relaxing the memory constraint; instead of removing the ure 2. The procedure deals with an arrival of a pair (a ,w)
i
minimal flow upon arrival of a non resident flow, we peri- (flow id, weight), e.g., a may be a TCP five tuple and w
i
odically remove many small flows from the table at once. the byte size of its payload. In the update procedure, we first
TinhfirseqmuaeinnttleyneannoceugohpetorataicohnietavkeeasmlionretaizretdimOe,(1b)utriutnitsiminev.oTkhede performaqueryforai.Letf(cid:98)at−i 1 bethequery(cid:16)resultforai.W(cid:17)e
table size only increases by a constant factor, so our solution add the following entry to the Active table: ai,f(cid:98)at−i 1+w .
d) Maintenance: As mentioned above, we require peri-
remains (asymptotically) space optimal.
Inprinciple,anoptimalalgorithmrequiresO(1)space[10], odic maintenance to keep the Active table from overflowing.
(cid:15) We define generation in the stream; a generation starts/ends
and runs in constant time. The algorithm provides an approx-
every time the Active table fills up. Before a generation ends,
imation guarantee that the error is bounded by R(cid:15) where R is
the maintenance process needs to achieve the following:
the total weight of all flows in the stream.
(a) (b) (c)
Figure 1: An illustration of the query process: first we query the Active table, then if the item was not found, we query the
Passive table, and finally we return q if the item was not found in the Passive table as well.
(a) (b) (c) (d)
Figure 3: An illustration of the maintenance process. In 3a, we find a quantile for the Passive table (a median in this example
where γ =1). Next, in 3b, we copy all entries larger than the quantile to the Active table. Then, in 3c, we clear all remaining
entries from the Passive table. Finally, in 3d, we swap the Active and the Passive tables.
the switch, the Active table is empty and the Passive
contains all entries that were a part of the Active table.
The maintenance process is illustrated in Figure 3. Mainten-
ance operations are performed once per generation and we
needtoguaranteethatthenumberofitemsintheActivetable
never exceeds the allocated size of T (cid:44) (cid:6)γ(cid:7)+(cid:6)1(cid:7)−1. We
(cid:15) (cid:15)
later prove that the number of update operations between two
consecutive maintenance operations is at least F (cid:44) (cid:100)γ(cid:101), and
(cid:15)
showthatasaresulttheupdatetimeisconstantifγisconstant.
C. Maintenance Implementation
There are several ways to implement the maintenance steps
described in Section III-B, which provide different runtime
guarantees.SectionIII-C1explainsthatwhenthemaintenance
operation is run serially after the Active table fills up we
Figure 2: An illustration of the update process: we perform achieve an amortized O(1) runtime, while Subsection III-C2
a query, add w to the result and then update the Active table. explains how to improve the runtime to O(1) worst case.
1) SimpleMaintenance: Inthissuggestion,weperformthe
maintenance procedure serially at the end of each generation.
1) Set q to be the (cid:6)1(cid:7)th largest value in the Passive table. The most computationally intensive step is Step 1, where we
(cid:15)
2) Move or copy all items with frequency larger than q needtofindthe(cid:100)1(cid:101)th largestcounter.Whenγ =1,thisvalue
(cid:15)
fromthePassivetabletotheActivetable,exceptforthe is the median, and as long as γ is constant, it is a certain
items that already appear in the Active table. percentile.Itiswellknownthatmedian(andotherpercentiles)
3) Clear the Passive table. canbecalculatedinlineartimeinadeterministicmanner,e.g.,
4) When the Active Table has filled, switch the Active and byusingthemedianofmediansalgorithm.Therefore,thetime
Passive tables before a new generation begins. After complexity of Step 1 is linear with the table size of O(cid:0)1+γ(cid:1).
(cid:15)
In Step 2, we copy at most 1 entries from the Passive is usually performed with a recursive function. Therefore,
(cid:15)
table to the Active table. The complexity of this step is freezing the calculation and resuming it later requires saving
O(1). Step 3 can be performed in O(1) time or in O(cid:0)1+γ(cid:1) the trace of recursive calls. Second, the double-threading
(cid:15) (cid:15)
time, depending if the hash table supports constant time flush procedure enables running the update and the maintenance
operations. Finally, Step 4 requires O(1) time. Therefore, procedures in parallel, which can save time. The drawback of
since the number of update operations in a generation is this technique is the use of semaphore for synchronization,
(cid:100)γ(cid:101), as shown in Lemma 2, the average update time per which hurts performance.
(cid:15)
packet is O(1 + 1), which is O(1) when γ is constant. In our deamortized implementation, we used a combination
γ
Therefore, the simple maintenance procedure can be executed ofthetwotechniques.Weusedcontrolledexecutionforsteps2
withO(1)amortizedtime.Asmentionedabove,wenickname and3anddouble-threadingforStep1.Step4isexecutedonly
ouralgorithmwiththesimplemaintenanceprocedureIterative at the end of a generation.
Median SUMming (IM-SUM).
D. What is in a Hash Table?
2) De-amortized Implementation: In deamortization, we
canperformonlyasmallportionofthemaintenanceprocedure Theonlydatastructureusedbyoursolutionisahashtable.
on each packet arrival. The challenge is to evenly split the Thismeansthatthehashtable’spropertiesdirectlyimpactthe
maintenancetaskbetweenupdateoperations.Ourdeamortized performance of our solution. In particular, advances in hash
algorithm, nicknamed De-amortized Iterative Median SUM- tables (e.g., [24]) can directly improve our work.
ming (DIM-SUM), uses the following two techniques. Below, we survey some interesting possibilities.
a) Controlled Execution: Here, we execute O(1) main- 1) Do we really need two different tables?: In principle,
tenanceoperationsafterprocessingeachpacket.Sincethetotal onecouldimplementbothActiveandPassivetablesasasingle
maintenance time is O(cid:0)1+γ(cid:1), we divide the maintenance to tablewithabitthatdecidesiftheentrybelongstothePassive
(cid:16) (cid:17) (cid:15)
tasks of O 1+ 1 operations each and execute a single task or Active table. This approach can speed up the maintenance
γ
process, as moving an item from the Passive to the Active
for each update operation. Therefore, when γ is chosen to be
table can be done by flipping a bit, which can even be done
a constant, we achieve O(1) worst case complexity.
in an atomic manner.
Note that we do not know how many update operations
2) Cuckoo Hash Table: Cuckoo hashing is very attractive
will actually occur before the Active table fills, e.g., if a
for read intensive workloads as it offers O(1) worst case
flow with an Active table entry is updated, then the number
queries. The trade off is that insertions end in O(1) time only
of Active table entries does not increase. This motivates us
with high probability.
to dynamically adjust the size of the maintenance operations
Another interesting benefit of the Cuckoo hash table is fast
according to the workload, to avoid a large variance in the
delete operations; all entries below a certain threshold can be
length of update operations. If we use the static bound proven
instantly removed by treating them as empty [33]. A similar
above, the first update operations will be slow, and once
technique could also be used in our case and further reduce
maintenance is over, update operations will be fast. To avoid
the time required to clear the Passive table.
this situation, we suggest an approach to dynamically adjust
3) CompactHashTables: Whenspaceisverytight,wecan
the task size according to the actual workload.
leverage compact hash tables. These data structures avoid the
To do so, prior to each update operation, we recalculate the
use of pointers and instead employ bit indexes to encode the
minimum number U of remaining updates according to the
table structure [24], [28]. Clearly, leveraging such structures
actual number of entries in the Active table and the maximal
is expected to significantly reduce space consumption.
numberofentriesthatstillneedtobecopiedfromthePassive
4) Fast Hash Tables: Perhaps the most obvious benefit of
table. Next, we calculate the maximum number of operations
relyingonlyonstandardhashtablesisspeed.Sincewedonot
M leftinthecurrentmaintenanceiteration.Beforeeachupdate
maintain any proprietary data structures, we can potentially
operation, we execute (cid:100)M(cid:101) maintenance operations.
U improve our runtime as faster hash tables are developed. For
b) DoubleThreading: Asecondapproachistorunsome
example, Hop Scotch Hashing [26] is considered one of the
of the maintenance steps in a second thread in parallel to the
fastest parallel hash tables in practice.
updateoperations.Weshouldascertainthatasufficientnumber
of maintenance operations are executed with every update op-
IV. ANALYSIS
eration. This resembles a producer-consumer problem, where
A. Correctness
themaintenancethreadistheproducerandtheupdatethreadis
the consumer. It can be implemented with a semaphore. After Our goal is to prove that IM-SUM and DIM-SUM solve
a sufficient number of maintenance operations is executed, the (cid:15)-VOLUMEESTIMATION and (θ,(cid:15))-ELEPHANTFLOWS
the semaphore increments, and before an update operation is problems. To do so, we first bound the value of q .
t
executed, the semaphore decrements.
Lemma 1. q ≤R ·(cid:15) at all times t.
While the above technique requires synchronization t t
between the two threads, it has a few advantages. First, it Proof of Lemma 1: Let S be the sum of the (cid:6)1(cid:7) items
t (cid:15)
simplifies the implementation. The calculation of a quantile in the tables with the largest estimation value at time t. We
show by induction on i that S ≤R for any time i. Assume
i i
that S ≤R . Next,weprovethatouralgorithmisalsoabletoidentifythe
i−1 i−1
When an item x arrives at time i, if it is already one of the elephant flows. We note that the runtime is optimal as there
(cid:6)1(cid:7) largest items, then both R and ft increase by w , and could be Ω(1) elephant flows.
(cid:15) i−1 x i (cid:15)
consequently S =S +w ≤R +w =R .
i i−1 i i−1 i i Theorem 2. At any time t, the Elephants() query solves
If x was not one of the largest items at time i−1 and has
(θ,(cid:15))-ELEPHANTFLOWS in O(1) time.
not become one at time i, then S =S ≤R ≤R . (cid:15)
i i−1 i−1 i
If,however,xwasnotoneofthelargestitemsattimei−1 Proof: Queries are done by traversing the tables and
but became one at time i, mark the frequency of the (cid:6)1(cid:7)th observing all elements x such that f(cid:98)t > q. The tables are
largest item by Q . If there are fewer than (cid:6)1(cid:7) items in(cid:15)the of size O(1) when γ is constant and thxerefore the complexity
i−1 (cid:15) (cid:15)
tables, define Q = q . After adding w , the estimation is O(1).
i−1 i−1 i (cid:15)
of x (f(cid:98)xt) becomes at most Qi−1+wi. Therefore f(cid:98)xt adds to Letxbesuchthatfxt >Rt·θ.AccordingtoTheorem1and
Si−1 up to Qi−1 +wi. Fortunately, if Qi−1 > 0, then there Lemma1,f(cid:98)xt ≥fxt >Rt·θ ≥q.Hence,xmustappearinone
must be at least 1 items with a positive estimation, and since of the tables with frequency greater than R ·θ. Otherwise, it
(cid:15) t
x has become one of the largest items, it must have replaced wouldhaveanestimationoflessthanq.Consequently,x∈S .
t
an item with frequency at least Qi−1. Therefore, we subtract Next, let x be such that fxt < Rt ·(θ −(cid:15)). According to
atleastQi−1 fromSi−1.IncaseQi−1 was0,westillsubtract Theorem 1, f(cid:98)xt ≤fxt +Rt·(cid:15)<Rt·(θ−(cid:15))+Rt·(cid:15)=Rt·θ.
Qi−1 =0 from Si−1. Either way, Therefore, x will not be included in St.
S ≤S +Q +w −Q =S +w ≤R +w =R .
i i−1 i−1 i i−1 i−1 i i−1 i i B. Runtime
Thus S ≤R for every i.
i i We now prove that IM-SUM runs at O(1) amortized time
Wenextconsiderthetimegatthebeginningofageneration
and that DIM-SUM runs at O(1) worst case time. Our goal
beforeq wascomputed.Therecannotbe(cid:6)1(cid:7)itemslargerthan
t (cid:15) is to evaluate both the complexity of the maintenance and
(cid:100)R1g(cid:101) ≤ Rg(cid:15), or otherwise Sg would be larger than Rg. Since the number of update operations between two consecutive
(cid:15)
this is the beginning of the generation, all of the items are in maintenance operations. We start with evaluating the min-
the passive table, and the quantile will be computed on the imal number of update operations between two consecutive
current items. Therefore q will be smaller or equal to R (cid:15). generations, i.e., the number of unique flows that must be
t g
Since the sum of elements only grows, q ≤R (cid:15)≤R (cid:15). encountered before we switch to a new generation.
t g t
Theorem1. Ouralgorithmsolves(cid:15)-VOLUMEESTIMATION: Lemma 2. Denote by Gmin the minimal number of update
That is, at any step (t): fxt ≤f(cid:98)xt ≤fxt +Rt(cid:15). operations between subsequent generations: Gmin =(cid:6)γ(cid:15)(cid:7).
Proof: We prove the theorem by induction over the Proof: In the beginning of every generation the Active
number of elements seen t. table is empty. During step 2, the Active table receives up
Basis: At time t=0, the claim holds trivially. to (cid:6)1(cid:7)−1 items from the Passive table. Other than that, the
(cid:15)
Hypothesis: Suppose that at time t−1 some element x has Active table only grows due to update operations. Therefore,
an approximation there must be at least
ft−1 ≤f(cid:100)t−1 ≤ft−1+R (cid:15). (cid:18)(cid:24)1(cid:25) (cid:19)
x x x t−1 G =T − −1
min (cid:15)
Step:Assumethatattimetitema arriveswithweightw .If
t t (cid:108)γ(cid:109) (cid:24)1(cid:25) (cid:18)(cid:24)1(cid:25) (cid:19) (cid:108)γ(cid:109)
at =x,thenthenewestimationgrowsbyexactlywt,sameas = + −1− −1 =
the true weight of x. According to the induction hypothesis, (cid:15) (cid:15) (cid:15) (cid:15)
the estimation remains correct. Similarly, when another item
updates in each generation.
arrivesattimet,fxt−1 doesnotchange,andneitherdoesf(cid:100)xt−1. Next, we bound the number of hash table operations re-
Steps 1 , 2 and 4 do not change the estimation for any item. quired to perform the maintenance process.
During step 3, however, x could be removed from the
Passive table. If x is also in the Active table, f(cid:98)xt remains the Lemma 3. Themaintenanceprocessrequires:M =O(cid:0)1+(cid:15)γ(cid:1)
hash table operations.
same. Otherwise, f(cid:98)xt is changed from f(cid:100)xt−1 to q. However,
since x only appears in the Passive table, we deduce that it Proof: We calculate the complexity of all steps. For
wasnotmovedinstep2.Thismeansthattheestimationforx Step 1, it is well known that a median can be found in linear
at the beginning of the generation was smaller or equal to q. time, e.g., using the median of medians algorithm. The same
Therefore, it is immediate from the induction hypothesis that is true for any percentile, and therefore Step 1 requires
fxt−1 ≤q. After x is removed from the Passive table as well, (cid:18)1+γ(cid:19)
the estimation becomes f(cid:98)t =q. Hence, ft =ft−1 ≤q =f(cid:98)t. O
x x x x (cid:15)
Lemma 1 proves that q ≤ft +R (cid:15) and therefore we have
x t
fxt ≤f(cid:98)xt ≤fxt +Rt(cid:15). hash table operations to complete.
Step2goesoverthePassivetable,whichisofsizeO(cid:0)1+γ(cid:1), A. Datasets
(cid:15)
and copies or moves some of the items to the Active table.
Our evaluation includes the following datasets:
Therefore, its complexity is O(cid:0)1+γ(cid:1).
(cid:15) 1) The CAIDA Chicago Anonymized Internet Trace
Step 3 can be performed in O(1) and so does Step 4.
2015 [2] , denoted Chicago. The trace was collected by
Therefore, the total complexity of all steps is:
the‘equinix-chicago’high-speedmonitorandcontainsa
(cid:18)1+γ(cid:19) mix of 6.3M TCP, UDP and ICMP packets. The weight
O .
(cid:15) of each packed is defined as the size of its payload, not
including the header.
2) The CAIDA San Jose Anonymized Internet Trace
2014 [3] , denoted SanJose. The trace was collected by
Theorem3. Foranyfixedγ,IM-SUMrunsinO(1)amortized
the ‘equinix-sanjose’ high-speed monitor and contains a
and DIM-SUM in O(1) worst case hash table operations.
mixof20.2MTCP,UDPandICMPpackets.Theweight
Proof:First,weobservethatforanyconstantγ,Lemma2 of each packet is defined as the size of its payload, not
establishes that there are O(1) update operations between including the header.
(cid:15)
generations. Similarly, by Lemma 3, there are O(1) hash 3) The UCLA Computer Science department TCP packet
(cid:15)
function operations to do during each maintenance process. trace (denoted UCLA-TCP) [4]. This trace contains
In IM-SUM, maintenance is performed once per generation 16.5M TCP packets passed through the border router
and thus the amortized complexity is O(1). of the Computer Science Department, University of
In DIM-SUM, as the work is split between update opera- California, Los Angeles. The weight of each packet is
tions, each update has to perform O(1) hash table operations. defined as the size of its payload, not including the
Let U be the minimum number of update operations left in header.
t
the generation at time t and let M be the maximum number 4) UCLA-UDP [4]. This trace contains 18.5M UDP pack-
t
of maintenance operations left. We show by induction that ets passed through the border router of the Computer
Mt = O(1). At a t in the beginning of the generation, Science Department, UCLA. The weight of each packet
Ut
UtA=ssGummein,baynidndMuctt=ionM,th.aTthfoerreafnoyret,imMUettt=, MOt(=1).O(1).After ihseaddeefirn.ed as the size of its payload, not including the
Ut 5) YouTube Trace (referred to as YouTube) [47]. The trace
a single update, U ≥ U −1, because otherwise the min-
t+1 t
contains a sequence of 436K accesses to YouTube’s
imum number of updates at time t would be smaller than U .
(cid:16) (cid:17)t videos made from within the University of Massachu-
Mt decreases to at most Mt+1 ≤ Mt− MUtt = Mt 1− U1t . setts Amherst. The weight of each video is defined as
Therefore, the number of maintenance operations executed at
its length (in seconds). An example application for such
time t+1 is
measurement is the caching of videos according to their
(cid:16) (cid:17) bandwidth usage to reduce traffic.
(cid:18) (cid:19) M 1− 1 (cid:18) (cid:19)
O Mt+1 ≤O t Ut =O Mt =O(1). 6) (Unweighted) Zipf streams that contain a series of i.i.d
Ut+1 Ut−1 Ut elements sampled according to a Zipfian distribution of
a given skew. We denote a Zipf stream with skew X as
The query process requires 2 hash table operations and is ZipfX. Each packet is of weight 1.
therefore also O(1).
B. Implementation
C. Required Space We compare both IM-SUM and DIM-SUM [1] to Count
MinSketch(CM),aHierarchicalCM-sketch(CMH),whichis
Wenowshow thatforeachconstant γ,IM-SUMand DIM-
SUM require the (asymptotically) optimal O(1) table entries. an extended version of CM suited to find frequent items [13],
(cid:15) a robust implementation of the Count Sketch that has been
Theorem 4. For each constant γ, the space complexity of extended using Adaptive Group Testing by [14] to support
IM-SUM and DIM-SUM is O(1). finding Frequent Items (AGT), and a heap based implement-
ε
ation of Space Saving (denoted SSH) [15], [35]. The code
Proof: The proof follows from the selected table sizes.
for the competing algorithms is the one released by [14]. All
Each table is sized to contain a maximum of γ + 1 entries
(cid:15) (cid:15) implementationsareinC++andthemeasurementsweremade
and therefore the total number of table entries is O(1).
(cid:15) on a 64-bit laptop with a 2.30GHz CPU, 4.00 GB RAM, 2
cores and 4 logical cores. In our own algorithms, we used the
V. EVALUATION
same hash table that was used to implement SSH for fairness.
We now present an extensive evaluation for comparing our
C. The effect of γ
proposed algorithm with the leading alternatives. We only
consideredalternativealgorithmsthatareabletocountpackets Recall that γ is a performance parameter; increasing γ
of variable sizes and are able to answer queries on-line. increases our space requirement but also makes maintenance
(a)Chicago (b)SanJose (c)YouTube
(d)UCLA-TCP (e)Legend (f)U1CLA-UDP
(g)Zipf0.7 (h)Zipf1 (i)Zipf1.3
Figure 4: The effect of the parameter γ on the performance of the algorithms over different traces ((cid:15)=2−15).
operations infrequent and therefore potentially improves per- D. Effect of (cid:15)
formance. Figure 4 shows the performance of IM-SUM and
Since memory consumption is coupled to accuracy (cid:15), it is
DIM-SUM for values of γ ranging from 2−4 to 24 and
interesting to evaluate the operation speed as a function of
(cid:15) = 2−15. The YouTube trace seems to benefit from larger γ
(cid:15). As IM-SUM and DIM-SUM are asymptotically faster, we
untilγ =8,Chicagountilγ =4,andUCLAseemstosaturate
expectthemtoperformbetterthanknownapproachesforsmall
aroundγ =1.SanJosepeaksforIM-SUMatγ =4andγ =8
(cid:15)s (that imply a large number of counters). Figure 5 depicts
for DIM-SUM. Zipf1.3 increases with γ until γ = 8. The
the operation speed of the algorithms, as a function of (cid:15). As
Zipf0.7 and Zipf1 traces, however, slightly decrease with γ.
shown, IM-SUM is considerably faster than all alternatives in
Weexplainthedeclinebyanincreaseinmemoryconsumption,
alltestedworkloads,evenforlargevaluesof(cid:15).Theresultsfor
whichslowsdownmemoryaccess.Thedeclineisprobablynot
DIM-SUM are mixed. For large (cid:15) values, it is mostly slower
evident in other traces, because in high skew traces the effect
than the alternatives. However, for small (cid:15)s, it is consistently
of reducing the frequency of the maintenance procedure is
better than previous works.
larger than the effect of decreasing the memory requirement.
In order to balance speed and space efficiency, we continue
E. Effect of Skew on operation speed
our evaluation with γ = 4. We also recommend this setting
Intuitively, the higher the skew of the workload, the less
for traces with unknown characteristics. If additional memory
frequentmaintenanceoperationsoccur,becauseittakeslonger
is available, reducing (cid:15) is more effective than increasing γ.
before we see enough unique flows for the Active table to fill
It is also evident that our de-amortization process has up.Figure6demonstratestheeffectofthezipfskewparameter
significant overheads. We attribute this to the synchronization on performance. As can be observed, while CMH and AGT
overhead between the two threads of DIM-SUM. are quite indifferent to skew, IM-SUM and DIM-SUM benefit
(a)Chicago (b)SanJose (c)YouTube
(d)UCLA-TCP (e)legend (f)UCLA-UDP
Figure 5: Comparison of the number of updates/ms performed by the different algorithms for a given error bound.
(a)(cid:15)=2−10 (b)Legend (c)(cid:15)=2−20
Figure 6: Comparison of the number of updates/ms performed by the different algorithms for a given skew.
greatly from it and IM-SUM is faster than all alternatives. SUM. This would probably require redesigning our code
DIM-SUMisconsistentlyfasterthanCMHandAGT,andhas to eliminate synchronization, possibly by using an existing
similar speed to SSH. CM is slightly faster than DIM-SUM concurrent hash table implementation. A C++ based open
forlowskewworkloadsbutmorethanthreetimesslowerthan sourceimplementationofthisworkandallothercodeusedin
DIM-SUM when the skew is high. this paper is available at [1].
VI. DISCUSSION
REFERENCES
In this paper, we have shown two variants of the first
(asymptotically) space optimal algorithm that can estimate [1] IM-SUM and DIM-SUM - Open source implementation. ht-
tps://github.com/kassnery/dimsum.
the total traffic volume of every flow as well as identify the
[2] TheCAIDAUCSDAnon.ChichagoInternetTraces2015,Dec.17th.
elephant flows (in terms of their total byte count). The first [3] TheCAIDAUCSDAnon.SanJoseInternetTraces2014,June19th.
variant, IM-SUM, is faster on the average case but only en- [4] Unpublished,seehttp://www.lasr.cs.ucla.edu/ddos/traces/.
suresO(1)amortizedexecutiontime,whilethesecondvariant, [5] YehudaAfek,AnatBremler-Barr,ShirLandauFeibish,andLironSchiff.
Sampling and large flow detection in sdn. In ACM SIGCOMM, pages
DIM-SUM, offers O(1) worst case time guarantee. We have
345–346,2015.
benchmarked our algorithms on both synthetic and real-world [6] Mohammad Al-Fares, Sivasankar Radhakrishnan, Barath Raghavan,
traces and have demonstrated their superior performance. Nelson Huang, and Amin Vahdat. Hedera: Dynamic flow scheduling
fordatacenternetworks. InUSENIXNSDI,2010.
Looking into the future, we hope to further reduce the
[7] RanBen-Basat,GilEinziger,RoyFriedman,andYaronKassner.Heavy
actual running time of DIM-SUM to match the rate of IM- hittersinstreamsandslidingwindows. InIEEEINFOCOM,2016.
[8] RanBen-Basat,GilEinziger,RoyFriedman,andYaronKassner. Ran- [33] Yang Liu, Wenji Chen, and Yong Guan. Near-optimal approximate
domizedadmissionpolicyforefficienttop-kandfrequencyestimation. membership query over time-decaying windows. In IEEE INFOCOM,
CoRR,abs/1612.02962,2016. pages1447–1455,2013.
[9] Theophilus Benson, Ashok Anand, Aditya Akella, and Ming Zhang. [34] YiLu,AndreaMontanari,BalajiPrabhakar,SarangDharmapurikar,and
Microte: Fine grained traffic engineering for data centers. In ACM AbdulKabbani.Counterbraids:anovelcounterarchitectureforper-flow
CoNEXT,pages8:1–8:12,2011. measurement. InProc.oftheACMSIGMETRICS,2008.
[10] Radu Berinde, Graham Cormode, Piotr Indyk, and Martin Strauss. [35] Nishad Manerikar and Themis Palpanas. Frequent items in streaming
Space-optimalheavyhitterswithstrongerrorbounds. InACMPODS, data: An experimental evaluation of the state-of-the-art. Data Knowl.
2009. Eng.,pages415–430,2009.
[11] Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding [36] Gurmeet Singh Manku and Rajeev Motwani. Approximate frequency
frequent items in data streams. Theor. Comput. Sci., 312(1):3–15, countsoverdatastreams. InProc.ofVLDB,2002.
January2004. [37] Ahmed Metwally, Divyakant Agrawal, and Amr El Abbadi. Efficient
computationoffrequentandtop-kelementsindatastreams. InICDT,
[12] Min Chen and Shigang Chen. Counter tree: A scalable counter
2005.
architecture for per-flow traffic measurement. In IEEE ICNP, pages
[38] MasoudMoshref,MinlanYu,andRameshGovindan.Resource/accuracy
111–122,2015.
tradeoffs in software-defined measurement. In Proc. of the 2nd ACM
[13] Pascal Cheung-Mon-Chan and Fabrice Cle´rot. Finding hierarchical
SIGCOMM Workshop on Hot Topics in Software Defined Networking,
heavy hitters with the count min sketch. In Proceedings of 4th In-
HotSDN,pages73–78,2013.
ternationalWorkshoponInternetPerformance,Simulation,Monitoring
[39] B. Mukherjee, L.T. Heberlein, and K.N. Levitt. Network intrusion
andMeasurement,IPS-MOME,2006.
detection. Network,IEEE,8(3),May1994.
[14] GrahamCormodeandMariosHadjieleftheriou. Findingfrequentitems
[40] Qiong Rong, Guangxing Zhang, Gaogang Xie, and K. Salamatian.
in data streams. VLDB 2008., 1(2):1530–1541, August 2008. Code:
Mnemoniclossycounting:Anefficientandaccurateheavy-hittersiden-
http://hadjieleftheriou.com/frequent-items.
tificationalgorithm. InIEEEIPCCC,pages255–262,2010.
[15] Graham Cormode and Marios Hadjieleftheriou. Methods for finding [41] Joel Sommers, Paul Barford, Nick Duffield, and Amos Ron. Accurate
frequentitemsindatastreams. J.ofVLDB,19(1):3–20,2010. and efficient sla compliance monitoring. In Proc. of the Conf. on
[16] Graham Cormode and S. Muthukrishnan. An improved data stream Applications, Technologies, Architectures, and Protocols for Computer
summary: The count-min sketch and its applications. J. Algorithms, Communications,SIGCOMM.ACM,2007.
55(1):58–75,April2005. [42] Erez Tsidon, Iddo Hanniel, and Isaac Keslassy. Estimators also need
[17] GrahamCormodeandSMuthukrishnan.What’snew:findingsignificant sharedvaluestogrowtogether. InINFOCOM,pages1889–1897,2012.
differences in network data streams. IEEE/ACM Transactions on [43] VijayVasudevan,AmarPhanishayee,HiralShah,ElieKrevat,DavidG.
Networking(TON),13(6):1219–1232,2005. Andersen,GregoryR.Ganger,GarthA.Gibson,andBrianMueller.Safe
[18] AndrewR.Curtis,JeffreyC.Mogul,JeanTourrilhes,PraveenYalagan- andeffectivefine-grainedtcpretransmissionsfordatacentercommunic-
dula, Puneet Sharma, and Sujata Banerjee. Devoflow: Scaling flow ation. ACMSIGCOMM,pages303–314,2009.
managementforhigh-performancenetworks.InACMSIGCOMM,pages [44] Li Yang, Wu Hao, Pan Tian, Dai Huichen, Lu Jianyuan, and Liu
254–265,2011. Bin. Case:Cache-assistedstretchableestimatorforhighspeedper-flow
[19] ErikD.Demaine,AlejandroLo´pez-Ortiz,andJ.IanMunro. Frequency measurement. InIEEEINFOCOM,2016.
estimationofinternetpacketstreamswithlimitedspace. InProc.ofthe [45] YinZhang,SumeetSingh,SubhabrataSen,NickDuffield,andCarsten
10thAnnualEuropeanSymposiumonAlgorithms,ESA,2002. Lund. Online identification of hierarchical heavy hitters: Algorithms,
[20] XenofontasDimitropoulos,PaulHurley,andAndreasKind.Probabilistic evaluation,andapplications. ACMIMC,2004.
lossy counting: An efficient algorithm for finding heavy hitters. ACM [46] YingZhang. Anadaptiveflowcountingmethodforanomalydetection
SIGCOMMComput.Commun.Rev.,38,2008. insdn. InACMCoNEXT,pages25–30,2013.
[21] Gero Dittmann and Andreas Herkersdorf. Network processor load [47] MichaelZink,KyoungwonSuh,YuGu,andJimKurose. Watchglobal,
balancingforhigh-speedlinks.InProc.oftheInt.Symp.onPerformance cachelocal:Youtubenetworktrafficatacampusnetwork:measurements
EvaluationofComputerandTelecommunicationSystems,volume735, and implications. In Electronic Imaging 2008, pages 681805–681805,
2002. 2008.
[22] G.Einziger,B.Fellman,andY.Kassner.Independentcounterestimation
buckets. InIEEEINFOCOM,pages2560–2568,2015.
[23] G. Einziger and R. Friedman. TinyLFU: A highly efficient cache
admissionpolicy. InEuromicroPDP,pages146–153,2014.
[24] Gil Einziger and Roy Friedman. Counting with tinytable: Every bit
counts! InACMICDCN2016,pages27:1–27:10.
[25] PedroGarcia-Teodoro,JessE.Daz-Verdejo,GabrielMaci-Fernndez,and
E. Vzquez. Anomaly-based network intrusion detection: Techniques,
systemsandchallenges. ComputersandSecurity,pages18–28,2009.
[26] MauriceHerlihy,NirShavit,andMoranTzafrir. Hopscotchhashing. In
22ndIntl.Symp.onDistributedComputing,2008.
[27] NunoHomemandJoaoPauloCarvalho. Findingtop-kelementsindata
streams. Inf.Sci.,180(24):4958–4974,December2010.
[28] NanHua,Haiquan(Chuck)Zhao,BillLin,andJunXu. Rank-indexed
hashing:Acompactconstructionofbloomfiltersandvariants. InIEEE
ICNP2008.
[29] Abdul Kabbani, Mohammad Alizadeh, Masato Yasuda, Rong Pan,
and Balaji Prabhakar. Af-qcn: Approximate fairness with quantized
congestion notification for multi-tenanted data centers. In Proc. of the
18thSymposiumonHighPerformanceInterconnects,IEEEHOTI,2010.
[30] Richard M. Karp, Scott Shenker, and Christos H. Papadimitriou. A
simple algorithm for finding frequent elements in streams and bags.
ACMTrans.DatabaseSyst.,28(1),March2003.
[31] E. Kranakis, P. Morin, and Y. Tang. Bounds for frequency estimation
ofpacketstreams. InSIROCCO,pages33–42,2013.
[32] T. Li, S. Chen, and Y. Ling. Per-flow traffic measurement through
randomized counter sharing. IEEE/ACM Transactions on Networking,
20(5):1622–1634,Oct2012.
