Efficient Distributed Locality Sensitive Hashing

12 Pages·00.368 MB·English

by  Bahman Bahmani,Ashish Goel,Rajendra Shinde

Efficient Distributed Locality Sensitive Hashing
Bahman Bahmani Ashish Goel Rajendra Shinde
StanfordUnivesrity StanfordUnivesrity StanfordUnivesrity
Stanford,CA Stanford,CA Stanford,CA
[e-mail] [e-mail] [e-mail]
ABSTRACT thermore,sincetoday’smassivedatasetsaretypicallystored
(Key,Value)baseddistributedframeworks,suchasMapRe- andprocessed inadistributedfashion, wherenetwork com-
duce, Memcached, and Twitter Storm are gaining increas- munication is one of the most important bottlenecks, these
inglywidespreaduseinapplicationsthatprocesslargeamounts methodsneedtobenetworkefficient,asotherwise, thenet-
of data. One important example application is large scale work load would slow down thewhole scheme.
similaritysearch,forwhichLocalitySensitiveHashing(LSH)
hasemergedasthemethodofchoice,speciallywhenthedata There is a family of tree-based indexing methods for NN
ishigh-dimensional. Atitscore,LSHisbasedonhashingthe search,suchasK-Dtrees[5],covertrees[7],navigatingnets
datapointstoanumberof bucketssuchthat similar points [25], R-trees [20], and SR-trees [24]. But, these techniques
are more likely to map to the same buckets. To guarantee do not scale well with the data dimension, and it has been
high search quality, the LSH scheme needs a rather large empiricallyobserved[35]thatwithevenmodestdimensions,
number of hash tables. This entails a large space require- they are actually less efficient thanlinear scan of thedata.
ment,andinthedistributedsetting,witheachqueryrequir-
inga networkcall perhashbucketlook up,thisalso entails On the other hand, there is another family of ANN search
a big network load. The Entropy LSH scheme proposed by methods based on the notion of Locality Sensitive Hashing
Panigrahysignificantlyreducesthenumberofrequiredhash (LSH) [23]. At its core, LSH is based on hashing the (data
tables by looking up a number of query offsets in addition and query) points into a number of hash buckets such that
to the query itself. While this improves the LSH space re- similarpointshavehigherchancesofgettingmappedtothe
quirement, it does not help with (and in fact worsens) the same buckets. Then for each query, the nearest neighbor
searchnetworkefficiency,asnoweachqueryoffsetrequiresa among the data points mapped to a same bucket as the
networkcall. Inthispaper,focusingontheEuclidianspace querypoint is returnedas thesearch result.
under l norm and building up on Entropy LSH, we pro-
2
posethedistributedLayeredLSHscheme,andprovethatit Unlike tree-based methods, LSH has been shown to scale
exponentiallydecreasesthenetworkcost,whilemaintaining well with the data dimension [23, 27]. However, the main
a good load balance between dffeirent machines. Our ex- drawbackofconventionalLSHbasedschemesisthattoguar-
periments also verify that our scheme results in significant antee a good search quality, one needs a large number of
network traffic reductions which also bring about runtime hash tables. This entails a rather large space requirement
improvements. fortheindex,andalsointhedistributedsetting,alargenet-
work load, as each hash bucket look up requires a commu-
nication over the network. To mitigate the space efficiency
1. INTRODUCTION
issue, Panigrahy [32] proposed the Entropy LSH scheme,
Similarity search is the problem of retrieving data objects which significantly reduces thenumberof required hash ta-
similartoaqueryobject. Ithasbecomeanimportantcom- bles, bylooking upanumberof queryoffsets in additionto
ponent of modern data-mining systems, with applications thequeryitself. EventhoughthisschemeimprovestheLSH
rangingfromde-duplicationofwebdocuments,content-based space efficiency,it does nothelp with itsnetwork efficiency,
audio,video,andimagesearch[26,29,11],collaborativefil- as now each query offset lookup requires a network call. In
tering [13], large scale genomic sequencealignment [9], nat- fact, sincethenumberofrequiredoffsetsin EntropyLSHis
urallanguageprocessing[33],patternclassification[12],and larger than the number of required hash tables in conven-
clustering [6]. tional LSH,EntropyLSHamplifies thenetwork inefficiency
issue.
In these applications, objects are usually represented by a
high dimensional feature vector. Then, similarity search Inthispaper,focusingontheEuclidianspaceunderl norm
2
reduces to a variant of the Nearest Neighbor (NN) search andbuildingupontheEntropyLSHscheme,wedesignthe
problem, such as K-Nearest Neighbor (KNN) or Approxi- LayeredLSHmethodfordistributingthehashbucketsover
mate Nearest Neighbor (ANN) search. A scheme to solve a set of machines which leads to an extremely network ef-
the NN search problem constructs an index which, given a ficient distributed LSH scheme. We prove that, compared
querypoint,allowsforquicklyfindingthedatapointclosest to a straightforward distributed implementation of LSH or
to it. In addition to the query search procedure, the index EntropyLSH,our Layered LSHmethod results in an expo-
construction also needs to be time and space efficient. Fur-
nential improvement in the network load (from polynomial Panigrahy [32] introduced the Entropy LSH scheme. This
in n, the number of data points, to sub-logarithmic in n), scheme uses the same hash functions and indexing method
while maintaining a good load balance between the differ- asthebasicLSHscheme. However,itusesadifferentquery
ent machines. Our experiments also verify that our scheme time procedure: In addition to hashing the query point, it
results in large network traffic improvements which in turn hashes a number of query offsets as well and also looks up
results in significant runtimespeedups. thehash bucketsthatanyoftheseoffsetsmapto. Theidea
is that the close data points are very likely to be mapped
Intherestofthissection,wefirstprovidesomebackground toeitherthesamebucketasthequerypointortothesame
on the similarity search problem and the relevant methods, bucketasoneofthequeryoffsets. Thissignificantlyreduces
thendiscussLSHinthedistributedcomputationmodel,and the numberof hash tables required to guarantee the search
finally present an overview of our scheme as well as our re- quality and efficiency. Hence, this scheme significantly im-
sults. proves the index space requirement compared to the basic
LSHmethod. However,it unfortunately doesnot helpwith
1.1 Background thequerynetwork efficiency,as each queryoffset requiresa
network call. Indeed,since one can see that [23, 32, 29] the
Inthissection, webrieflyreviewthesimilarity search prob-
number of query offsets required by Entropy LSH is larger
lem,thebasicLSHandEntropyLSHapproachestosolving
than the numberof hash tables required by basic LSH, the
it, the distributed computation framework and its instanti-
querynetworkefficiencyofEntropyLSHisevenworsethan
ationssuchasMapReduceandActiveDHT,andastraight-
that of thebasic LSH.
forwardimplementationofLSHinthedistributedsettingas
well as its major drawback.
In this paper, we focus on the network efficiency of LSH
in distributedframeworks. Two main instantiationsofsuch
Similarity Search: The similarity search problem is that
frameworks are the batched processing system MapReduce
of finding data objects similar to a query object. In many
[16] (with its open source implementation Hadoop [1]), and
practical applications, the objects are represented by mul-
thereal-timeprocessingsystemsdenotedasActiveDistributed
tidimensional feature vectors, and hence the problem re-
Hash Tables (Active DHTs), such as Yahoo S4 [31] and
duces to finding objects closest to the query object under
Twitter Storm [2]. The common feature in all these sys-
the feature space distance metric. This latter problem is
tems is that they process data in the form of (Key, Value)
usually denotedastheNearestNeighbor(NN)search prob-
pairs, distributed over a set of machines. This distributed
lem. There are a few variations of the NN search problem:
(Key,Value) abstraction isall we need for bothour scheme
K-NearestNeighbor(KNN)searchproblemrequiresfinding
and analyses to apply. However, to make the later discus-
the K nearest data points to the query point, and the Ap-
sionsmoreconcrete,herewebrieflyoverviewthementioned
proximateNearestNeighbor(ANN)searchproblemrequires
distributed systems.
findingadatapointwhosedistancetothequerypointisal-
most equal to the distance of the closest data point. The
MapReduce: MapReduce [16] is a simple computation
goal in all these problems is to construct an index, which
modelforbatcheddistributedprocessingoflargescaledata,
given the query point, allows for quickly finding the search
usingalargenumberofcommoditymachines. Byautomat-
result. The index construction and the query search both
icallyhandlingthelowerlevelissuessuchasfaulttolerance,
need to bespace, time, and network efficient.
it providesasimple computationalabstraction, where com-
putations are done in three phases. The Map phase reads
Basic LSH:IndykandMotwani[23]proposedusingaspe-
acollection of (Key,Value)pairsfrom an inputsource, and
cifictypeof hashfunctions,namelyLocality SensitiveHash
by invoking a user defined Mapper function on each input
(LSH) functions, to solve the NN search problem over high
element independently and in parallel, emits zero or more
dimensionallargedatasets. AnLSHfunctionmapsthepoints
(Key,Value) pairs associated with that input element. The
in the feature space to a number of buckets in a way that
Shuffle phase then groups together all the Mapper-emitted
similar pointsmap tothesame bucketswith ahigh chance.
(Key, Value) pairs sharing the same Key, and outputs each
Then, a similarity search query can be answered by first
distinctgrouptothenextphase. TheReducephaseinvokes
hashing the query point and then finding the close data
a user-defined Reducer function on each distinct group, in-
points in the same bucket as the one the query is mapped
dependently and in parallel, and emits zero or more values
to. To guarantee both a good search quality and a good
toassociatewiththegroup’sKey. Theemitted(Key,Value)
search efficiency, one needs to use multiple LSH functions
pairscan thenbewritten on thediskorbetheinputofthe
and combine their results. Then, although this approach
Mapphaseinafollowingiteration. Theopensourcesystem
yields a significant improvement in the running time over
Hadoop [1] is a widely used implementation of this compu-
both the brute force linear scan and the space partitioning
tational framework.
approaches [19, 5, 7, 25, 20, 24], unfortunately the required
number of hash functions is usually large [9, 19], and since
Active DHT: A DHT (Distributed Hash Table) is a dis-
each hash table has the same size as the dataset, for large
tributed (Key, Value) store which allows Lookups, Inserts,
scale applications, this entails a very large space require-
andDeletesonthebasisoftheKey. ThetermActive refers
ment for the index. Also, in the distributed setting, since
to the fact that an arbitrary User Defined Function (UDF)
each hash table lookup at querytime corresponds to a net-
can be executed on a (Key, Value) pair in addition to In-
work call, this entails a large network load which is also
sert, Delete, and Lookup. Yahoo’s S4 [31] and Twitter’s
undesirable.
Storm[2]aretwoexamplesofActiveDHTswhicharegain-
ingwidespreaduse. TheActiveDHTmodelisbroadenough
Entropy LSH: To mitigate the space inefficiency of LSH,
to act as a distributed stream processing system and as a nearpointsarelikelytobeonthesamemachine(hencenet-
continuousversionofMapReduce[28]. Allthe(Key,Value) workefficiency)whilefarpointsarelikelytobeondifferent
pairsinanodeoftheactiveDHTareusuallystoredinmain machines (henceload balance).
memory. Thisrequiresthat thedistributeddeploymenthas
sufficient numberof nodes, and will allow for fast real-time Thisisachievedbyrehashingthebucketstowhichthedata
processing of data and queries. points and the offsets of query points map to, via an addi-
tional layer of LSH, and then using the hashed buckets as
Inadditiontothetypicalperformancemeasuresoftotalrun- Keys. More specifically, each data point is associated with
ning time and total space, two other measures are very im- a (Key,Value) pair where Key is themapped value of LSH
portant for both MapReduceand ActiveDHTs: bucket containing the point, and Value is the point’s hash
bucket concatenated with the point itself. Also, each query
point is associated with multiple (Key, Value) pairs where
1. total network traffic generated, that is theshuffle size Valueisthequeryitself andKeysarethemappedvaluesof
for MapReduce and the number of network calls for the buckets which need to be searched in order to answer
ActiveDHT this query.
2. the maximum number of values with the same key;
Use of an LSH to rehash the buckets not only allows using
a high value here can lead to the “curse of the last
theproximityofqueryoffsetstoboundthenumberof(Key,
reducer”in MapReduce or to one compute node be-
Value) pairs for each query (thus guaranteeing network ef-
coming a bottleneck in ActiveDHT.
ficiency), but also ensures that far points are unlikely to
behashed tothesame machine(thusmaintaining load bal-
ASimpleDistributedLSHImplementation: Webriefly ance).
discuss a simple implementation of LSH in the distributed
framework. Each hash table associates a (Key, Value) pair 1.3 OurResults
toeachdatapoint,wheretheKeyisthepoint’shashbucket, Here, we present a summary of our results in this paper:
and the Value is the point itself. These (Key, Value) pairs
arerandomlydistributedoverthesetofmachinessuchthat
all the pairs with the same Key are on the same machine. 1. We design a new scheme, called Layered LSH, to im-
This is doneimplicitly usingarandom hash functionof the plement Entropy LSH in the distributed (Key, Value)
Key. For each query, first a number of (Key, Value) pairs model, and provideits pseudo-codefor thetwo major
corresponding to the query point are generated. The Value distributedframeworks MapReduceand ActiveDHT.
in allofthesepairsisthequerypointitself. ForbasicLSH,
2. WeprovethatLayeredLSHincursonlyO(√logn)net-
per hash table, the Key is the hash bucket the query maps
work cost per query. This is an exponential improve-
to, and for Entropy LSH, per query offset, the Key is the
mentovertheO(nΘ(1))querynetworkcostofthesim-
hash bucket the offset maps to. Then, each of these (Key,
ple distributed implementation of both Entropy LSH
Value) pairs gets sent to and processed by the machine re-
and basic LSH.
sponsiblefor itsKey. Thismachinecontainsalldatapoints
mapping to the same query or offset hash bucket. Then, it 3. We prove that surprisingly the network efficiency of
canperformasearchwithinthedatapointswhichalsomap LayeredLSHisindependentofthesearchquality! This
to the same Key and report the close points. This search is in sharp contrast with both Entropy LSH and ba-
can be done using the UDF in Active DHT or the Reducer sicLSHin which increasingsearch qualitydirectlyin-
in MapReduce. creases the network cost. This offers a very large im-
provement in both network efficiency and hence over-
In the above implementation, the amount of network com- allruntimeinsettingswhichrequiresimilaritysearch
municationperqueryisdirectlyproportionaltothenumber withhighaccuracy. Wealsopresentexperimentswhich
of hash bucketsthat need to bechecked. However, as men- verify this observation on theMapReduceframework.
tioned earlier, this number is large for both basic LSH and
Entropy LSH. Hence, in large scale applications, where ei- 4. We prove that despite network efficiency (which re-
therthereisahugebatchof queriesorthequeriesarrivein quires collocating near points on the same machines),
real-timeatveryhighrates,thiswillrequirealotofcommu- Layered LSH sends points which are only Ω(1) apart
nication, which not only depletes the valuable network re- todifferentmachineswithhighlikelihood. Thisshows
sourcesinasharedenvironment,butalsosignificantlyslows Layered LSH hits the right tradeoff between network
down the query search process. In this paper, we propose efficiency and load balance across machines.
an alternative way, called Layered LSH, to implement the
5. WepresentexperimentalresultswithLayeredLSHon
Entropy LSH scheme in a distributed framework and prove
Hadoop,whichshowafactor50to100improvementin
that it exponentiallyreducesthenetworkload compared to
the“shufflesize”(i.e., theamountofdatatransmitted
the above implementation, while maintaining a good load
overthenetwork)and200to400percentimprovement
balance between differentmachines.
onaverageinruntime,overawiderangeofparameter
values, on both syntheticand real world data sets.
1.2 Overview ofOurScheme
Atitscore,LayeredLSHisacarefullydesignedimplementa-
tion ofEntropyLSHin thedistributed(Key,Value)model. Theorganizationofthispaperisasfollows. Insection2,we
The main idea is to distribute the hash buckets such that studytheBasicandEntropyLSHindexingmethods. Insec-
tion 3, we give the detailed description of Layered LSH,in- efficiency), but since it also makes it less likely (pk) for a
1
cludingitspseudocodefortheMapReduceandActiveDHT nearpointtomaptothesamebucket,weuseanumber,M,
frameworks, and also provide the theoretical analysis of its of hash tables to guarantee retrieving the near points with
networkcostandloadbalance. Wepresentexperimentswith a good chance (hencesearch quality).
SimpleandLayeredLSHonHadoop insection 4,studythe
related work in section 5, and concludein section 6. Toutilizethisindexingscheme,oneneedsanLSHfamily
H
tostartwith. Suchfamiliesareknownforavarietyofmetric
2. PRELIMINARIES spaces, including the Hamming distance, the Earth Mover
Distance,andtheJaccardmeasure[10]. Furthermore,Datar
Insection1.1,weprovidedthehigh-levelbackgroundneeded
etal. [15]proposedLSHfamiliesforl norms,with0 p
for thispaper. Here,wepresentthenecessary preliminaries p
≤ ≤
2,usingp-stabledistributions. ForanyW >0,theyconsider
in further detail. Specifically, we formally define the NN
searchproblem,thenotionofLSHfunctions,thebasicLSH a family of hash functions W : ha,b :Rd Z such that
H { → }
indexing,and EntropyLSH indexing. a v+b
ha,b(v)=& ·W ’
NN Search: Asmentionedinsection 1.1,similarity search
often reducestoaninstantiation oftheNNsearch problem. where a Rd is a d-dimensional vector each of whose en-
∈
Here,wepresenttheformaldefinitionofthislatterproblem. tries is chosen independently from a p-stable distribution,
Let T be the domain of all data and query objects, and ζ and b R is chosen uniformly from [0,W]. Further im-
∈
to be the distance metric between the objects. Given an provements have been obtained in various special settings
approximation ratio c > 1, the c-ANN problem is that of [4]. In this paper, we will focus on the most widely used p-
constructing an index that given any query point q T, stable distribution, i.e., the 2-stable, Gaussian distribution.
allowsforquicklyfindingadatapointp T whosedist∈ance For this case, Indykand Motwani [23] proved the following
to q, as measured by ζ, is at most c tim∈es larger than the theorem:
distance from q to its nearest data point. One can see that
[22, 23] this problem can be solved by reducing it to the
Theorem 2. With n data points, choosing k = O(logn)
(c,r)-NN problem, in which the goal is to return a data
point within distance cr of the query point q, given that and M = O(n1/c), the LSH indexing scheme above solves
there existsa data point within distancer of q. the (c,r)-NN problem with constant probability.
Basic LSH: To solve the (c,r)-NN problem, Indyk and
Motwani [23] introduced the following notion of LSH func- EntropyLSH: Asexplainedinsection1.1,themaindraw-
tions: back of thebasic LSH indexing scheme is the large number
(in practice, up to hundreds[9, 19]) of hash tables it needs
to get a good quality. This entails a large index space re-
Definition 1. For the space T with metric ζ, given dis- quirement,andinthedistributedsetting,alsoabignetwork
tance threshold r, approximation ratio c>1, and probabili- load. To mitigate the space inefficiency, Panigrahy [32] in-
ties p > p , a family of hash functions = h : T U troduced the Entropy LSH scheme. This scheme uses the
1 2
H { → }
is said to be a (r,cr,p ,p )-LSH family if for all x,y T, same indexing as in the basic LSH scheme, but a different
1 2
∈
querysearch procedure. Theideahereisthatforeachhash
if ζ(x,y) r then Pr [h(x)=h(y)] p ,
if ζ(x,y)≤cr then PrH [h(x)=h(y)]≥ p1 . (2.1) function H ∈H !, the data points close to the query point
≥ H ≤ 2 q are highly likely to hash either to thesame valueas H(q)
or to a value very close to that. Hence, it makes sense to
alsoconsiderascandidatesthepointsmappingtoclosehash
Hash functions drawn from have the property that near
H values. To do so, in this scheme, in addition to q, several
points (with distance at most r) have a high likelihood (at
“offsets”q+δ (1 i L), chosen randomly from the sur-
least p1) of being hashed to the same value, while far away face of B(q,ri), th≤e sp≤here of radius r centered at q, are
points(withdistanceat least cr)areless likely(probability
also hashed and the data points in their hash buckets are
atmostp )tobehashedtothesamevalue;hencethename
2 alsoconsideredassearchresultcandidates. Itisconceivable
locality sensitive.
that this may reduce the number of required hash tables,
andinfact,Panigrahy[32]showsthatwiththisschemeone
LSH families can be used to design an index for the (c,r)-
canuseasfewasO˜(1)hash tables. Theinstantiationofhis
NN problem as follows. First, for an integer k, let ! =
H : T Uk be a family of hash functions in whichHany result for thel2 norm is as follows:
{ → }
H ! is the concatenation of k functions in , i.e., H =
∈H H
(h ,h ,...,h ), where h (1 i k). Then, for an
1 2 k i ∈H ≤ ≤ Theorem 3. For n data points, choosing k logn
integer M, draw M hash functions from !, independently ≥ log(1/p2)
and uniformly at random, and use themHto construct the (with p2 as in Definition 1) and L = O(n2/c), as few as
indexconsistingofM hashtablesonthedatapoints. With O˜(1) hash tables suffice to solve the (c,r)-NN problem.
this index, given a query q, the similarity search is done
byfirstgeneratingtheset ofalldatapointsmappingtothe
samebucketasqinatleastonehashtable,andthenfinding Hence, this scheme in fact significantly reduces the num-
the closest point to q among those data points. The idea is ber of required hash tables (from O(n1/c) for basic LSH to
thatafunctiondrawnfromH! hasaverysmallchance(pk2) O˜(1)), and hence the space efficiency of LSH. However, in
to map far away points to the same bucket (hence search the distributed setting, it does not help with reducing the
network load of LSH queries. Actually, since for the basic terquerynetworkefficiencyisneeded. ThisiswhereLayered
LSH, one needs to look up M = O(n1/c) buckets but with LSH comes into thepicture.
this scheme, one needs to look up L = O(n2/c) offsets, it
makes thenetwork inefficiency issue evenmore severe. 3.1 Layered LSH
Inthissubsection,wepresenttheLayeredLSHscheme. The
3. DISTRIBUTED LSH mainideaistouseanotherlayeroflocalitysensitivehashing
to distribute the data and query points over the machines.
Inthissection,wewillpresenttheLayeredLSHschemeand
Morespecifically,givenaparametervalueD>0,wesample
theoretically analyzeit. Wewill focuson the d-dimensional
an LSHfunction G:Rk Z such that:
Euclidian space under l norm. As notation, we will let S
2 →
to be a set of n data points available a-priori, and Q to
be the set of query points, either given as a batch (in case
α v+β
of MapReduce) or arriving in real-time (in case of Active G(v)= · (3.1)
DHT). Parameters k,L,W and LSH families = and & D ’
W
H H
! = !W will be as defined in section 2. Since multiple
hHash tHables can be obviously implemented in parallel, for where α Rk is a k-dimensional vector whose individual
∈
the sake of clarity we will focus on a single hash table and entries are chosen from the standard Gaussian (0,1) dis-
use a randomly chosen hash function H ! as our LSH tribution,and β R is chosen uniformly from [0N,D].
∈H ∈
function throughoutthesection.
Then, denoting G(H()) by GH(), for each data point p
· · ∈
In (Key, Value) based distributed systems, a hash function S, we generate a (Key, Value) pair (GH(p),< H(p),p >
from thedomain ofall Keystothedomain ofavailable ma- ), which gets sent to machine GH(p). By breaking down
chinesisimplicitlyusedtodeterminethemachineresponsi- the Value part to its two pieces, H(p) and p, this machine
ble for each (Key, Value) pair. In this section, for the sake will then add p to the bucket H(p). This can be done by
of clarity, we will assume this mapping to be simply iden- theReducerin MapReduce,and byaUDFinActiveDHT.
tity. That is, the machine responsible for a (Key, Value) Similarly, for each query point q Q, after generating the
∈
data element is simply themachine with id equalto Key. offsets q+δ (1 i L),for each uniquevaluex in theset
i
≤ ≤
GH(q+δ ) 1 i L (3.2)
At the core, Layered LSH is a carefully distributed imple- { i | ≤ ≤ }
mentationofEntropyLSH.Hencebeforepresentingit,first we generate a (Key, Value) pair (x,q) which gets sent to
we further detail the simple distributed implementation of machine x. Then, machine x will have all the data points
Entropy LSH, described in section 1.1, and explain its ma- p such that GH(p)=x as well as the queries q Q one of
∈
jor drawback. Foranydatapointp S a(Key,Value)pair whoseoffsetsgetsmappedtoxbyGH(). Specifically,iffor
∈ ·
(H(p),p) is generated and sent to machine H(p). For each theoffsetq+δ ,wehaveGH(q+δ )=x,allthedatapointsp
i i
querypointq,aftergeneratingtheoffsets q+δ (1 i L), thatH(p)=H(q+δ )arealsolocatedonmachinex. Then,
i i
≤ ≤
for each offset q+δ a (Key, Value) pair (H(q+δ ),q) is thismachineregeneratestheoffsetsq+δ (1 i L),finds
i i i
≤ ≤
generatedandsenttomachineH(q+δ ). Hence,amachine their hash buckets H(q+δ ), and for any of these buckets
i i
x will have all the data points p S with H(p)=x as well such that GH(q+δ )= x, it performs a NN search among
i
∈
as all query points q Q such that H(q+δ )=x for some the data points in that bucket. Note that since q is sent to
i
∈
1 i L. Then, for any received query point q, this ma- thismachine,thereexistsatleastonesuchbucket. Alsonote
≤ ≤
chine retrieves all data points p with H(p) = x which are that,theoffsetregeneration,hash,andbucketsearchcanall
withindistancecrofq,ifanysuchdatapointsexist. Thisis be done by either a UDF in ActiveDHT or theReducer in
done via a UDF in Active DHT or the Reducer in MapRe- MapReducer. To make the exposition more concrete, we
duce,aspresentedinFigure3.1forthesakeofconcreteness have presented the pseudo code for both the MapReduce
of exposition. and Active DHT implementations of this scheme in Figure
3.2.
Inthisimplementation,thenetworkloadduetodatapoints
is not very significant. Not only just one (Key, Value) pair At an intuitive level, the main idea in Layered LSH is that
per data point is transmitted over the network, but also in since G is an LSH, and also for any query point q, we have
many real-time applications, data indexing is done offline H(q+δ ) H(q)foralloffsets q+δ (1 i L),thesetin
i i
( ≤ ≤
where efficiency and speed are not as critical. However,the equation 3.2 has avery small cardinality,which in turnim-
amount of data transmitted per query in this implementa- plies a small amount of network communication per query.
tionisO(Ld): L(Key,Value)pairs,oneperoffset,eachwith Ontheotherhand,sinceGandH arebothLSHfunctions,
thed-dimensional point q as Value. Both L and d are large iftwodatapointsp,p!arefarapart,GH(p)andGH(p!)are
in many practical applications with high-dimensional data highlylikelytobedifferent. Thismeansthat,whilelocating
(e.g., L can be in the hundreds, and d in the tens or hun- thenearbypointson thesame machines,Layered LSHpar-
dreds). Hence, this implementation needs a lot of network titionsthefarawaydatapointsondifferentmachines,which
communication perquery,andwithalargebatchofqueries in turn ensures a good load balance across the machines.
or with queries arriving in real-time at very high rates, this Note that this is critical, as without a good load balance,
will notonly puta lot of strain on thevaluableand usually thepoint in distributingtheimplementation would be lost.
shared network resources but also significantly slow down
thesearch process. In the next section, we present the formal analysis of this
scheme,andprovethatcomparedtothesimpleimplementa-
Therefore, adistributedLSHschemewith significantly bet- tion,itprovidesanexponentialimprovementinthenetwork
Algorithm 1 MapReduceImplementation Algorithm 2 ActiveDHTImplementation
Map: Preprocessing:
Input: Dataset S, queryset Q Input: Data set S
ChooseH from !W uniformlyatrandom,butconsis- for each data pointp S do
H ∈
tently across Mappers Computethe hash buckety=H(p)
for each datapoint p S do Send thepair(y,p) to machinewith id y
∈
Emit (H(p),p) At machiney add p tothein-memory buckety
endfor end for
for each querypoint q Q do
∈
for 1 i L do Query Time:
≤ ≤
Choosetheoffsetq+δ fromthesurfaceofB(q,r) Input: Querypoint q Q arrivingin real-time
i
∈
Emit (H(q+δ ),q) for 1 i L do
i
≤ ≤
endfor Generate theoffset q+δ
i
endfor Computethe hash bucketx=H(q+δ )
i
Send thepair(x,q) to machinewith id x
Reduce: At machinex, runSearchUDF(x,q)
Input: For a hash bucket x, all data points p S end for
∈
with H(p) = x, and all query points q Q one of
∈
whose offsets hashes tox. SearchUDF(x,q):
for each querypoint q among theinputpoints do for each data pointp with H(p)=x do
for each datapoint p among theinputpointsdo if p is within distance cr of q then
if p is within distancecr of q then Emit (q,p)
Emit (q,p) endif
endif end for
endfor
endfor
Figure 3.1: SimpleDistributed LSH
traffic, while maintaining a good load balance across the hence R(u) R(v) √k. Also,bydefinitionH =Γ R,
|| − ||≤ −
machines. and hance H(u) H(v) = (Γ( u) Γ(v))+(R(v) R(u)).
− − −
Then, theresult follows from triangle inequality.
3.2 Analysis
In this section, we analyze the Layered LSH scheme pre-
Our analysis also uses two well-known facts. The first is
sented in the previous section. We first fix some notation.
thesharpconcentrationofχ2-distributedrandomvariables,
As mentioned earlier in the paper, we are interested in the
whichisalsousedintheproofoftheJohnson-Lindenstrauss
(c,r)-NN problem. Without loss of generality and to sim-
lemma [23, 14]:
plify the notation, in this section we assume r = 1/c. This
can be achieved by a simple scaling. The LSH function
H ∈H !W that we use is H = (H1,...,Hk), where k is Fact 5. If ω Rm is a random m-dimensional vector
chosen as in Theorem 3and for each 1 i k: ∈
≤ ≤ each of whose entries is chosen from the standard Gaussian
a v+b (0,1)distribution,andm=Ω( logn),thenwithprobability
Hi(v)=& i·W i’ aNt least 1 1 , we have !2
− nΘ(1)
where a is a d-dimensional vector each of whose entries
i (1 ’)√m ω (1+’)√m
is chosen from the standard Gaussian (0,1) distribution, − ≤|| ||≤
andb Rischosenuniformlyfrom [0,NW]. Wewillalso let
i
Γ: Rd ∈ Rk beΓ=(Γ ,...,Γ ), where for 1 i k:
→ 1 k ≤ ≤ The second fact is the 2-stability property of Gaussian dis-
a v+b tribution:
Γi(v)= i·W i
hence, Hi() = Γi() . We will use the following small Fact 6. If θ is a vector each of whose entries is chosen
· & · ’
lemma in our analysis: from the standard Gaussian (0,1) distribution, then for
N
any vector v of the same dimension, the random variable
θ v has Gaussian (0, v ) distribution.
Lemma 4. For any two vectors u,v Rd, we have: · N || ||
∈
Γ(u) Γ(v) √k H(u) H(v) Γ(u) Γ(v) +√k
|| − ||− ≤|| − ||≤|| − || Theplanfortheanalysis isasfollows. Wewill firstanalyze
(intheorem8)thenetworktrafficofLayeredLSHandderive
Proof. Denoting R =Γ H (1 i k) and R = a formula for it based on D, the bin size of LSH function
i i i
− ≤ ≤
(R ,...,R ),wehave0 R (u),R (v) 1(1 i k),and G. We will see that as expected, increasing D reduces the
1 k i i
≤ ≤ ≤ ≤
Algorithm 3 MapReduceImplementation Algorithm 4 ActiveDHTImplementation
Map: Preprocessing:
Input: Dataset S, queryset Q Input: Data set S
ChoosehashfunctionsH,Grandomlybutconsistently for each data pointp S do
∈
across mappers ComputethehashbucketH(p)andmachineidy=
for each datapoint p S do GH(p)
∈
Emit (GH(p),<H(p),p>) Sendthepair(y,<H(p),p>)tomachinewith idy
endfor Atmachiney,add ptothein-memorybucketH(p)
for each querypoint q Q do end for
∈
for 1 i L do
≤ ≤
Generate theoffsetq+δ Query Time:
i
Emit (GH(q+δ ),q) Input: Querypoint q Q arrivingin real-time
i
∈
endfor for 1 i L do
≤ ≤
endfor Generatetheoffsetq+δ ,computex=GH(q+δ )
i i
if GH(q+δ )=x( j <i) then
j
* ∀
Reduce: Sendthepair (x,q) to machinewith id x
Input: For a hash bucket x, all pairs < H(p),p > At machinex,run SearchUDF(x,q)
for data points p S with GH(p) = x, and all query endif
∈
points q Q one of whose offsets is mapped to x by end for
∈
GH.
for each datapoint p among theinputpoints do SearchUDF(x,q):
Addp to bucketH(p) for 1 i L do
≤ ≤
endfor Generateoffsetq+δ ,computeH(q+δ ),GH(q+δ )
i i i
for each querypoint q among theinputpoints do if GH(q+δ ) = x,H(q+δ ) = H(q+δ )( j < i)
i i j
* ∀
for 1 i L do then
≤ ≤
Generate theoffsetq+δ , and findH(q+δ ) for each datapoint p in bucketH(q+δ ) do
i i i
if GH(q+δ )=x,H(q+δ )=H(q+δ )( j<i) if p is within distancecr from q then
i i j
* ∀
then Emit (q,p)
for each data pointp in bucketH(q+δ ) do endif
i
if (p is within distance cr of q) then endfor
Emit (q,p) endif
endif end for
endfor
endif
endfor
endfor
Figure 3.2: Layered LSH
network traffic, and our formula will show the exact rela- Sinceq isd-dimensional,thenetworkload duetoqueryq is
tionbetweenthetwo. Wewillnextanalyze(intheorem11) O(df ). Hence,toanalyzethenetwork efficiencyof Layered
q
the load balance of Layered LSH and derive a formula for LSH, it suffices to analyze f . This is done in thefollowing
q
it, again based on D. Intuitively speaking, a large value of theorem:
D tends to put all points on one or few machines, which is
undesirable from the load balance perspective. Our analy- Theorem 8. Foranyquerypointq,withhighprobability,
sis will formulate this dependenceand show its exact form. that is probability at least 1 1 , we have:
These two results together will then show the exact trade- − nΘ(1)
off governing the choice of D, which we will use to prove k
f =O( )
(inCorollary12)thatwithanappropriatechoiceofD,Lay- q D
eredLSHachievesbothnetworkefficiencyandloadbalance.
Before proceeding to theanalysis, we givea definition: Proof. Sincefor any offset q+δi, thevalueGH(q+δi)
is an integer, we have:
f max GH(q+δ ) GH(q+δ ) (3.3)
q i j
≤1 i,j L{ − }
Definition 7. HavingchosenLSHfunctionsG,H,fora ≤ ≤
For any vectorv, we have:
query point q Q, with offsets q+δ (1 i L), define
i
∈ ≤ ≤
α.v+β α.v+β
1 G(v)
f = GH(q+δ )1 i L D − ≤ ≤ D
q i
|{ | ≤ ≤ }|
hencefor any1 i,j L:
≤ ≤
tobethenumberof(Key,Value)pairssentoverthenetwork α (H(q+δ ) H(q+δ ))
for query q. GH(q+δi)−GH(q+δj)≤ · iD− j +1
Thus, from equation 3.3, we get: LSH, to increase the search quality one needs to increase
the number of offsets, which will then directly increase the
1
fq ≤ D1mi,ajxL{α·(H(q+δi)−H(q+δj))}+1 network load. Similarly, with basic LSH, to increase the
≤ ≤ search quality one needs to increase the number of hash ta-
From Cauchy-Schwartz inequality for inner products, we bles, which again directly increases the network load. How-
havefor any 1 i,j L: ever, with Layered LSH the network efficiency is achieved
≤ ≤
independently of the level of search quality. Hence, search
α (H(q+δ ) H(q+δ )) α H(q+δ ) H(q+δ )
· i − j ≤|| ||·|| i − j || quality can be increased without any effect on the network
Hence, we get: load!
α
fq ≤ ||D|| ·1mi,ajxL{||H(q+δi)−H(q+δj)||}+1 (3.4) Next, we proceed to analyzing the load balance of Layered
≤ ≤
LSH.First, recalling theclassic definitionof error function:
For any 1≤i,j ≤L, we know from lemma 4: erf(z)= 2 ze−τ2dτ
√π !
H(q+δ ) H(q+δ ) Γ(q+δ ) Γ(q+δ ) +√k (3.5) 0
i j i j
|| − ||≤|| − || we definethefunction P():
·
Furthermorefor any 1≤t≤k, since P(z)=erf(z)− √1πz(1−e−z2) (3.8)
a (δ δ )
Γt(q+δi)−Γt(q+δj)= t· Wi− j and provethefollowing lemma:
we know, using Fact 6, that Γ (q+δ ) Γ (q+δ ) is dis-
t i t j
tributed as Gaussian (0,||δi−δj||). Now−, recall from theo- Lemma 10. For any two points u,v∈Rk with ||u−v||=
N W λ, we have:
rem3thatforourLSHfunction,k logn . Hence,there
≥ log(1/p2) D
is a constant’=’(p2)<1 for which we have,using Fact 5: Pr[G(u)=G(v)]=P( )
√2λ
δ δ
||Γ(q+δi)−Γ(q+δj)||≤(1+’)√k|| iW− j|| Proof. Since β is uniformly distributed over [0,D], we
with probabilityat least 1 1/nΘ(1). Since,asexplainedin have:
−
section2,alloffsetsarechosenfromthesurfaceofthesphere l
B(q,1/c) of radius 1/c centered at q, we have: δ δ Pr[G(u)=G(v) α (u v)=l]=max 0,1 ||
|| i− j|| ≤ | · − " − D#
2/c. Henceoverall, for any 1 i,j k:
≤ ≤
Then,sincebyFact 6,α (u v) isdistributedasGaussian
√k √k (0,λ), we have: · −
||Γ(q+δi)−Γ(q+δj)||≤2(1+’)cW ≤4cW N
D l 1 l2
Pr[G(u)=G(v)]= (1 ||) e−2λ2dl
! − D √2πλ
withhighprobability. Then,sincethereareonlyL2different −D
choicesofi,j,andLisonlypolynomiallylargeinn,weget: D l 1 l2
=2 (1 ) e−2λ2dl
! − D √2πλ
√k 0
1≤mi,aj≤xL{||Γ(q+δi)−Γ(q+δj)||}≤4cW =! D √√π2λe−2lλ22dl−! D Dlλ√√2πe−2lλ22dl
withhighprobability. Then,usingequation3.5,wegetwith 0 0
high probability: D 2 λ D2
=erf(√2λ)−$πD(1−e−2λ2)
4
1mi,ajxL{||H(q+δi)−H(q+δj)||}≤(1+ cW)√k (3.6) =P( D )
≤ ≤ √2λ
Furthermore, since each entry of α Rk is distributed as
∈
(0,1),another application ofFact 5gives(again with ’=
N
’(p )):
2
One can easily see that P() is a monotonically increasing
α (1+’)√k 2√k (3.7) ·
|| ||≤ ≤ function,andforany0<ξ< 1thereexistsanumberz=zξ
with high probability. Then, equations 3.4, 3.6, and 3.7 such that P(zξ) = ξ. Using this notation and the previous
together give: lemma, we provethefollowing theorem:
4 k
f 2(1+ ) +1
q ≤ cW D Theorem 11. For any constant 0<ξ< 1, there is a λξ
such that
which finishestheproof.
D
λ /W =O(1+ )
ξ √k
Remark 9. AsurprisingpropertyofLayeredLSHdemon-
and for any two points u,v with u v λ , we have:
strated by theorem 8 is that the network load is independent || − ||≥ ξ
of the number of query offsets, L. Note that with Entropy Pr[GH(u)=GH(v)] ξ+o(1)
≤
where o(1) is polynomially small in n. Remark 13. Corollary 12 shows that, compared to the
simpledistributedimplementationofEntropyLSHandbasic
Proof. Letu,v Rdbetwopointsanddenote u v = LSH,Layered LSH exponentiallyimprovesthenetwork load,
∈ || − || fromO(nΘ(1))toO(√logn),whilemaintainingtheloadbal-
λ. Then bylemma 4, we have:
ance across the different machines.
H(u) H(v) Γ(u) Γ(v) √k
|| − ||≥|| − ||− 4. EXPERIMENTS
Asintheproofoftheorem8,onecansee,usingk logn
≥ log(1/p2) In this section we present an experimental comparison of
(fromtheorem3)andFact5,thatthereexistsan’=’(p )=
2 Simple and Layered LSH via the MapReduce framework
Θ(1) such that with probability at least 1 1 we have:
− nΘ(1) whichshowafactor50to100reductioninthenetworkcost
λ√k (shufflesize)forLayeredLSH,andonaverage,a200to400%
Γ(u) Γ(v) (1 ’) improvement in the“wall-clock”run time.
|| − ||≥ − W
Hence, with probability at least 1 1 , we have: 4.1 Datasets
− nΘ(1)
We use two data sets, an artificially generated “Random”
λ
H(u) H(v) λ!=((1 ’) 1)√k datasetandan“Image”datasetobtainedfromTinyImages
|| − ||≥ − W −
[3], as described below.
Now, letting:
λ = 1 (1+ D )W Random: Forthed-dimensionalEuclideanspace Rd,
ξ 1−’ zξ√2k • letNd(0,r)denotethenormaldistributionaroundthe
origin,0,wherethei-thcoordinateofarandomlycho-
we haveif λ≥λξ then λ!≥ √D2zξ, and hencebylemma 10: senpointhasthedistribution (0,r/√d), i 1...d.
N ∀ ∈
D Wecreatea dataset by sampling1 million (1M) data
Pr[GH(u)=GH(v) H(u) H(v) λ!] P( ) ξ pointsfromNd(0,1)withd=100andchoosehundred
||| − ||≥ ≤ √2λ ≤
! thousand (100K) query points by randomly selecting
a data point and adding a small perturbation drawn
which finishes the proof by recalling Pr[ H(u) H(v) < from Nd(0,r)toit,wherer=0.3. Wepresentthere-
|| − ||
λ!]=o(1). sultsonsolvingthe(c,r)-NNproblemonthisdataset
with c = 2. The choice of the distribution as well as
parameters(c,r)issuchthatforeachquerypoint,the
Theorems8,11showthetradeoffgoverningthechoiceofpa-
expected distance to its nearest neighbor in the data
rameterD. IncreasingD reducesnetworktrafficatthecost
set is r and that with high probability only the data
ofmoreskewedloaddistribution. WeneedtochooseDsuch
point to which it was chosen to benear to, by design,
thattheloadisbalancedyetthenetworktrafficislow. The-
will be at a distance at most cr from it. The data
orem 11 shows that choosing D = o(√k) does not asymp-
set and query set were chosen large enough to ensure
totically help with the distance threshold at which points
thatanontrivialshufflecomponentintheMapReduce
becomelikelytobesenttodifferentmachines. Ontheother
implementation of EntropyLSH.
hand, theorem 11 also shows that choosing D = ω(√k) is
undesirable, as it unnecessarily skews theload distribution. Image [3]: The Tiny Image Data set consists of al-
•
Toobservethismoreclearly,recallthatintuitivelyspeaking, most 80M“tiny”images of size 32 32 pixels [3]. We
×
thegoalinLayeredLSHisthatiftwodatapointp ,p hash extracta64-dimensionalcolorhistogramfromeachim-
1 2
tothesamevaluesastwooftheoffsetsq+δ ,q+δ (forsome ageinthisdatasetusingtheextractcolorhistogramtool
i j
1 i,j L) of a query point q (i.e., H(p ) = H(q+δ ) in the FIRE image search engine, as described in [29,
1 i
≤ ≤
and H(p )=H(q+δ )),then p ,p are likely tobe sent to 17] and normalize it to unit norm in the preprocess-
2 j 1 2
the same machine. Since H has a bin size of W, such pair ing step. Next, we randomly sample 1M data set and
of points p ,p most likely have distance O(W). Hence, 200Kquerysetpointsensuringnooverlapamongdata
1 2
D should be only large enough to make points which are and query sets. Since d = 64, we chose the query set
O(W)awaylikelytobesenttothesamemachine. Theorem tobetwiceaslargeastheRandomdatasetinorderto
11 shows that to doso, we need tochoose D such that: ensureapproximatelythesamenetworkcostperquery
offset for theEntropyLSHmethod. Sincetheaverage
D
O(1+ )=O(1) distanceofaquerypointtoitsnearestneighborinthis
√k data set was 0.08 (with standard deviation 0.07), we
that is D = O(√k). Then, by theorem 8, to minimize present the results of solving the (c,R)-NN problem
the network traffic, we choose D =Θ( √k), and get f = onthisdatasetwithparameters(r=0.08,c=2)and
q
(r=0.16,c=2),notingthatweobtainsimilarresults
O(√k) = O(√logn). This is summarized in the following
for many other parameter choices.
corollary:
4.2 Metrics
Corollary 12. ChoosingD=Θ( √k),LayeredLSHguar- WecomparetheoverallnetworkefficiencyoftheSimpleand
antees that the number of (Key,Value) pairs sent over the LayeredLSHusingtheshufflesizei.e.,totalamountofdata
network per query is O(√logn) with high probability, and shuffledoverthenetworkintheShufflephase(“Reduceshuf-
yet points which are Ω(W) away get sent to different ma- fle bytes”in Hadoop [1]) and also compare the“wall-clock”
chines with constant probability. runningtimeof thetwo schemes.
1 1000
Recall Layered LSH
0.98 Simple LSH
0.96
0.94
100
0.92 GB)
0 0.8.98 Shuffle (
10
0.86
0.84
0.82
0.8 1000 1200 1400 1600 1800 2000
1000 1200 1400 1600 1800 2000 L
(a) Random: recall (b) Random: shufflesize
100 100
1.02 rr == 00..0186 LaSyimerpelde LLSSHH LaSyimerpelde LLSSHH
1
0.98
GB) 10 GB) 10
00..9946 Shuffle ( Shuffle (
0.92
1 1
0.9
0.88 200 400 600 800 1000 1200 200 400 600 800 1000 1200
0 200 400 600 800 1000 1200 1400 1600 L L
(c) Image: recall (d) Image: shufflesize, r=0.08 (e) Image: shufflesize, r=0.16
Figure 4.1: Variation in recall and shuffle size (log scale) with L for the Random (Figures (a), (b)) and the Image (Figures
(c), (d), (e)) data sets. Layered LSH yields a factor 100 improvement for the Random data set and on average a factor 50
improvement on the Imagedata set.
AsdescribedinSection3,eventhoughtheSimpleandLay- 4.4 Results
ered LSH have different distributed implementations, they NextwepresentperformancevariationexperimentsforSim-
output the same result. Consequently, evaluating the accu- pleandLayeredLSHbyscalingL,thenumberofqueryoff-
racyoftheresults(i.e. recall) isnotanimportantaspectof sets. In addition to this, we also study the sensitivity of
this paper, since the effectiveness of LSH has been verified performance with variation in D, the parameter of Layered
by many earlier studies [15, 19, 29]. Still, for completeness, LSH.
(and to make sure our Layered LSH implementation does
not have implementation errors,) we measure the accuracy
Scaling L:
of search results by computing the recall, i.e. the fraction
Figure 4.1 describes the results of scaling L on the shuf-
of querypoints,havingatleast asingle datapoint within a
fle size and recall for both the data sets, where the shuffle
distance cr, returnedin theoutput.
sizes are plotted in a logarithmic scale. We observe that
although increasing L improves the recall, it also results in
a linear increase in the shuffle size for Simple LSH since a
4.3 Implementation Details (Key, Value) pair is emitted per query and offset in Simple
We perform experiments on a small cluster of 16 compute LSH. In contrast, the shuffle size for Layered LSH remains
nodesusingHadoopversion0.20.2with512MBheapsizefor almost constant with increasing L and offers a factor 100
individual JVMs. Consistency in the choice of hash func- improvementoverSimpleLSHfortheRandomdatasetand
tions H,G for Simple and Layered LSH (figures 3.1, 3.2) on average a factor 50 improvementfor theImage dataset.
across mappers is ensured by setting the seed of the ran- ThisobservationverifiesTheorem8,aswellasthesurprising
dom number generator(RNG) appropriately. Consistency property that network load of Layered LSH is independent
in choice of offsets for each query point across mappers of L, as described in Remark 9.
(and reducers in Layered LSH) is ensured by using a RNG
with thequery point itself as seed. Wechoose the LSH pa- Although minimizing the“wall-clock”running time for the
rameters (W = 0.5,k = 10) for the Random data set and MapReduceimplementationwasnotoneofourprimaryob-
(W =0.3,k=16)fortheImagedatasetaspercalculations jectives,weobservethatimprovementintheshufflesizealso
in theEntropyLSHwork andpreviouswork on image data leads to a factor 4 improvement on average in run time for
sets [32, 29]. We choose D via a binary search to minimize the Random data set and a factor 2 improvement on aver-
the“wall-clock”runtimeofLayeredLSHandandreportper- age for theImagedataset, evenwith acrudebinarysearch
formance variation with increasing values of L, the number for D, as shown in Figure 4.2. Hence,this verifies load bal-
ofqueryoffsets. Wenotethatasystematicunderstandingof ancingproperties of Layered LSH(Theorem 11). Notethat
the choice of parameters for Layered LSH (as well as Basic since Hadoop offers checkpointing guarantees where (Key,
LSH) is an important research direction. Value) pairs output by mappers and reducers may be writ-