Update Consistency for Wait-free Concurrent Objects

0.19 MB·English

by  Matthieu Perrin

Update Consistency for Wait-free Concurrent Objects
Matthieu Perrin, Achour Mostefaoui, and Claude Jard
LINA – University of Nantes, Nantes, France
Email: [firstname.lastname]@univ-nantes.fr
Abstract—In large scale systems such as the Internet, worst-case response time must be proportional to the
replicating data is an essential feature in order to provide latency of the network either for the reads or for the
5
availability and fault-tolerance. Attiya and Welch proved
1 writes to be sequentially consistent [1] and for all
that using strong consistency criteria such as atomicity is
0 the operations for linearizability [2]. This generalizes
costlyaseachoperationmayneedanexecutiontimelinear
2 with the latency of the communication network. Weaker to many objects [2]. Moreover, the availability of the

n consistencycriterialikecausalconsistencyandPRAMcon- shared object cannot be ensured in asynchronous sys-
a sistency do not ensure convergence. The different replicas tems where more than a minority of the processes of
J are not guaranteed to converge towards a unique state.
a system may crash [3]. In large modern distributed
9 Eventualconsistencyguaranteesthatallreplicaseventually systems such as Amazon’s cloud, partitions do occur
convergewhentheparticipantsstopupdating.However,it
] fails to fully specify the semantics of the operations on between data centers, as well as inside data centers
C shared objects and requires additional non-intuitive and [4]. Moreover, it is economically unacceptable to sac-
D error-prone distributed specification techniques. rifice availability. The only solution is then to provide
This paper introduces and formalizes a new consis- weaker consistency criteria. Several weak consistency
.
s tency criterion, called update consistency, that requires criteria have been considered for modeling shared
c
the state of a replicated object to be consistent with a
[ memorysuchasPRAM[1]orcausality[5].Theyexpect
linearization of all the updates. In other words, whereas
1 atomicity imposes a linearization of all of the operations, the local histories observed by each process to be
v this criterion imposes this only on updates. Consequently plausible, regardless of the other processes. However,
5 somereadoperationsmayreturnout-datedvalues.Update these criteria do not impose that the data eventually
6 consistency is stronger than eventual consistency, so we convergestoaconsistentstate.Eventualconsistency[4]
1 can replace eventually consistentobjects with updatecon- is another weak consistency criterion which requires
2 sistent onesinany program.Finally,weprovethatupdate
that when all the processes stop updating then all
0 consistencyisuniversal,inthesensethatanyobjectcanbe
. implemented under this criterion in a distributed system replicas eventually converge to the same state.
1 where any number of nodes may crash.
Thispaperfollowsthelongquestofthe(a)strongest
0
5 consistency criterion (there may exist several incom-
Keywords-Abstract Data Types; Consistency Criteria;
1 parable criteria) implementable for different types of
Eventual Consistency; Replicated Object; Sequential Con-
v: sistency; Shared Set; Update Consistency; objects in an asynchronous system where all but one
i process may crash (wait-free systems [6]). A contribu-
X
tionofthispaperconsistsinprovingthatweakconsis-
I. INTRODUCTION
r tency criteria such as eventual consistency and causal
a
Reliability of large scale systems is a big challenge consistency cannot be combined is such systems. This
when building massive distributed applications over paper chooses to explore the enforcement of eventual
the Internet. At this scale, data replication is essential consistency. The relevance of eventual consistency has
to ensure availability and fault-tolerance. In a perfect been illustrated many times. It is used in practice
world, distributed objects should behave as if there is in many large scale applications such as Amazon’s
a unique physical shared object that evolves following Dynamo highly available key-value store [7]. It has
the atomic operationsissued by the participants1. This been widely studied and many algorithms have been
is the aim of strong consistency criteria such as lin- proposed to implement eventually consistent shared
earizability and sequential consistency. These criteria object. Conflict-free replicated data types (CRDT) [8]
serialize all the operations so that they look as if they givesufficientconditionsonthespecificationofobjects
happened sequentially, but they are costly to imple- so that they can be implemented. More specifically, if
ment in message-passing systems. If one considers a all the updates made on the object commute or if the
distributed implementation of a shared register, the reachable states of the object form a semi-lattice then
theobjecthasaneventuallyconsistentimplementation
1We use indifferently participant or process to designate the [8]. Unfortunately,many usefulobjectsarenot CRDTs.
computingentitiesthatinvokethedistributedobject.
The limitations of eventual consistency led to the • It introduces updateconsistency, a new consistency
study of stronger criteria such as strong eventual con- criterion stronger than eventual consistency and
sistency [9]. Indeed, eventual consistency requires the for which the converging state must be consistent
convergencetowardsacommonstatewithoutspecifying with a linearization of the updates.
whichstatesarelegal.Inordertoprovethecorrectness • Finally,itprovesthatforanyUQ-ADTobjectwith
of a program, it is necessary to fully specify which a sequential specification there exists an update
behaviors are accepted for an object. The meaning of consistent implementation by providing a generic
an operation often depends on the context in which it construction.
is executed. The notion of intention is widely used to The remainderof this paperis organizedasfollows.
specifycollaborativeediting[10],[11].Theintentionof Section II formalizes the notion of consistency crite-
an operation not only depends on the operation and ria and the type of objects we target in this paper.
thestateonwhichitisdone,butalsoontheintentions Section III recalls the definition of (strong) eventual
of the concurrent operations. In another solution [12], consistency. Section IV proves that eventual consis-
it is claimed that, it is sufficient to specify what the tency cannot be combined with causal consistency in
concurrent execution of all pairs of non-commutative wait-freesystems.SectionVintroduces(strong)update
operationsshould give(e.g.anerrorstate).Thisresult, consistency and compares it with (strong) eventual
acceptable for the shared set, cannot be extended to consistency. Section VI compares, through the exam-
othermorecomplicatedobjects.Inthiscase,anypartial ple of the set, the expressiveness of strong update
order of updates can lead to a different result. This consistency and strong eventual consistency. Section
approachwasformalizedin[13],wheretheconcurrent VII presents a generic construction for any UQ-ADT
specification of an object is defined as a function object with a sequential specification. Finally, Section
of partially ordered sets of updates to a consistent VIII concludes the paper.
state leading to specifications as complicated as the
implementations themselves. Moreover, a concurrent II. ABSTRACT DATATYPES ANDCONSISTENCY
specification of an object uses the notion of concurrent CRITERIA
Before introducing the new consistency criterion,
events. In message-passing systems, two events are
this section formalizes the notion of object and how a
concurrent if they are produced by differentprocesses
consistencycriterionisdefined.Indistributedsystems,
and each process produced its event before it received
sharing objects is a way to abstract message-passing
the notification message from the other process. In
communication between processes. The abstract type
other words, the notion of concurrency depends on
of these objects has a sequential specification, defined
the implementation of an object not on its specifica-
in this paper by a transition system that character-
tion. Consequently, the final user may not know if
izes the sequential histories allowed for this object.
two events are concurrent without explicitly tracking
However, shared objects are implemented in a dis-
the underlying messages. A specification should be
tributed system using replication and the events of
independentofthesystemonwhichitisimplemented.
the distributed history generatedby the executionof a
Contributions of the paper: for not restricting this
distributed programis a partialorder[14]. The consis-
work to a given data structure, this paper first de-
tency criterion makes the link between the sequential
fines a class of data types called UQ-ADT for update-
specification of an object and a distributed execution
query abstractdatatype.Thisclassencompassesalldata
that invokes it. This is done by characterizing the
structureswhereanoperationeither modifiesthestate
partially ordered histories of the distributed program
of the object (update) or returns a function on the
that are acceptable. The formalization used in this
current state of the object (query). This class excludes
paper is explained with more details in [15].
data types such as a stack where the pop operation
An abstract data type is specified using a transition
removes the top of the stack and returns it (update
system very close to Mealy machines [16] except that
andqueryatthesametime).However,suchoperations
infinite transition systemsareallowed asmanyobjects
can always be separated into a query and an update
have an unbounded specification. As stated in the
(lookup top and delete top in the case of the stack)
Introduction, this paper focuses on ”update-query”
whichisnotaproblemas,inweakconsistencymodels,
objects. On the one hand, the updates have a side-
itisimpossibletoensureatomicityanyway.Thispaper
effect that usually affects the state of the object (hence
has three main contributions.
all processes), but return no value. They correspond
• It provesthatin a wait-freeasynchronous system, to transitions between abstract states in the transition
itisnotpossibletoimplementeventualandcausal system. On the other hand, the queries are read-only
consistency for all UQ-ADTs. operations. They produce an output that depends on
thestateoftheobject.Consequently,theinputalphabet valuevintothesetobject.R/sdenotesareadoperation
of the transition system is separated into two classes thatreturnsthesetsrepresentingthecontentoftheset.
of operations (updates and queries).
During an execution, the participants invoke an
Definition 1 (Update-query abstract data type). An object instance of anabstractdatatype using the asso-
update-query abstract data type (UQ-ADT) is a tuple ciatedoperations(queriesandupdates).Thisexecution
O =(U,Q ,Q ,S,s ,T,G) such that: produces a set of partially ordered events labelled by
i o 0
the operations of the abstract data type. This repre-
• U is a countable set of update operations; sentation of a distributed history is generic enough
• Qi and Qo are countable sets called input and to model a large number of distributed systems. For
output alphabets; Q = Q × Q is the set of
i o example,inthe caseofcommunicating sequentialpro-
query operations. A query operation (q ,q ) ∈ Q
i o cesses, an event a precedes an event b in the program
is denoted q /q (query q returns value q ).
i o i o order if they are executed by the same process in that
• S is a countable set of states; sequential order. It is also possible to model more
• s0 ∈S is the initial state; complex modern systems in which new threads are
• T :S×U →S is the transition function; created and destroyed dynamically, or peer-to-peer
• G:S×Qi →Qo is the output function. systems where peers may join and leave.
A sequential history is a sequence of operations. An Definition 2 (Distributed History). A distributed his-
infinite sequence of operations (wi)i∈N ∈ (U ∪Q)ω is tory is a tuple H =(U,Q,E,Λ,7→):
recognized by O if there exists an infinite sequence of
states (si)i≥1 ∈ Sω (note that s0 is the initial state) • U and Q are disjoint countable sets of update and
such that for all i ∈ N, T(s ,w ) = s if w ∈ U or query operations, and all queries q ∈ Q are in the
i i i+1 i
si =si+1andG(si,qi)=qoifwi =qi/qo ∈Q.Thesetof form q =qi/qo;
all infinite sequences recognized by O and their finite • E is a countable set of events;
prefixes is denoted by L(O). Said differently, L(O) is • Λ:E →U ∪Q is a labelling function;
the set of all the sequential histories allowed for O. • 7→⊂E×E is a partial order called program order,
such that for all e∈E, {e′ ∈E :e′ 7→e} is finite.
Along the paper, replicated sets are used as the key
example. Three kinds of operations are possible: two
Let H = (U,Q,E,Λ,7→) be a history. The sets U =
H
update operation by element, namely insertion (I) and
{e ∈ E : Λ(e) ∈ U} and Q = {e ∈ E : Λ(e) ∈ Q}
H
deletion(D)andaqueryoperationread(R)thatreturns
denoteitssetsofupdateandqueryeventsrespectively.
thevaluesthatbelongtotheset.LetVal bethesupport
We also define some projections on the histories. The
of the replicated set (it contains the values that can be
first one allows to withdraw some events: for F ⊂ E,
inserted/deleted). At the beginning, the set is empty
H = (U,Q,F,Λ| ,7→ ∩(F × F)) is the history that
F F
and when an element is inserted, it becomes present
contains only the events of F. The second one allows
until it is deleted. More formally, it corresponds to the
to substitute the order relation: if → is a partial order
UQ-ADT given in Example 1.
that respects the definition of a program order (7→),
Example 1 (Specification of the set). Let Val be a H→ =(U,Q,E,Λ,→∩(E×E)) is the history in which
countable set, called support. The setobject SVal is the the events areorderedby →. Note thatthe projections
UQ-ADT (U,Qi,Qo,S,∅,T,G) with: commute, which allows the notation HF→.
• U ={I(v),D(v):v ∈Val}; Definition 3 (Linearizations). Let H = (U,Q,E,Λ,7→)
• Qi ={R},andQo =S =P<∞(Val)containallthe be a distributed history. A linearization of H corre-
finite subsets of Val; sponds to a sequential history that contains the same
• for all s∈S and v ∈Val, G(s,R)=s, events as H in an order consistent with the program
T(s,I(v))=s∪{v} and T(s,D(v))=s\{v}. order. More precisely, it is a word Λ(e )...Λ(e )...
0 n
such that {e ,...,e ,...} = E and for all i and j, if
The set U of updates is the set of all insertions and 0 n
i < j, e 67→ e . We denote by lin(H) the set of all
deletions of any value of Val. The set of queries Q j i
i linearizations of H.
contains a unique operation R, a read operation with
no parameter. A read operation may return any value Definition 4 (Consistency criterion). A consistency
in Qo, the set of all finite subsets of Val. The set S of criterion C characterizes which histories are allowed
the possible states is the same as the set of possible for a given data type. It is a function C that associates
returned values Q as the read query returns the with any UQ-ADT O, a set of distributed histories
o
content ofthe setobject. I(v) (resp.D(v)) with v ∈Val C(O). A shared object (instance of an UQ-ADT O) is
denotes an insertion (resp. a deletion) operation of the C-consistent if all the histories it allows are in C(O).
III. EVENTUALCONSISTENCY • Growth: if an event has been viewed once by a
In this section, we recall the definitions of eventual process, it will remain visible forever:
consistency [4] and strong eventual consistency [9]. ∀e,e′,e′′ ∈E,(e−v−i→s e′∧e′ 7→e′′)⇒(e−v−i→s e′′);
Fig.1illustratesthesetwoconsistencycriteriaonsmall • Strong convergence: if two query operations view
examples. In the remaining of this article, we consider the same past of updatesV, they can be issued in
an UQ-ADT O = (U,Qi,Qo,S,s0,T,G) and a history the same state s: ∀V ⊂UH,∃s∈S,∀qi/qo ∈QH,
H =(U,Q,E,Λ,7→). V ={u∈U :u−v−i→s q /q }⇒G(s,q )=q .
H i o i o
Eventualconsistency: eventualconsistencyrequires
The history of Fig. 1a is not strong eventually con-
that, if all the participants stop updating, all the repli-
sistent because the I(1) must be visible by all the
cas eventually converge to the same state. In other
queriesof the firstprocess(byreflexivityand growth),
word, H is eventually consistent if it contains an infi-
so there are only two possible sets of visible updates
nitenumberofupdates(i.e.theparticipantsneverstop
({I(1)}and{I(1),I(2)})fortheseevents,butthequeries
writing) or if there exists a state (the consistent state)
are done in three different states ({1}, {2} and ∅);
compatible with all but a finite number of queries.
consequently,atleasttwoofthesequeriesseethesame
Definition 5 (Eventual consistency). A history H is setofupdatesandthus needtoreturnthesame value.
eventually consistent (EC) if UH is infinite or there Fig.1c,onthecontrary,isstrongeventuallyconsistent:
exists a state s ∈ S such that the set of queries the replicasthatsee {I(1)}areinstate∅and those that
that return non consistent values while in the state s, see {I(1),I(2)} are in state {1,2}.
{q /q ∈Q :G(s,q )6=q }, is finite.
i o H i o IV. PIPELINED CONVERGENCE
All the histories presented in Fig. 1 are eventu- A straightforward way to strengthen eventual con-
allyconsistent. Theexecutionsrepresenttwoprocesses sistency is to compose it with another consistency
sharing a set of integers. In Fig. 1a, the first process criterion that imposes restrictions on the values that
inserts value 1 and then reads twice the set and gets canbereturnedbya readoperation. Causalityisoften
respectively {2} and {1}; afterwards, it executes an cited as a possible candidate to play this role [10]. As
infinity of read operations that return the empty set causalconsistencyiswellformalizedonlyformemory,
(ω in superscript denotes the operation is executed an we will instead consider Pipelined Random Access
infinity of times). In the meantime, the second process Memory (PRAM) [1], a weaker consistency criterion.
insertsa2thenreadsthesetaninfinityoftimes.Itgets As the name suggests, PRAM was initially defined for
respectively{1}and{2}thetwofirsttimes,andempty memory.However,itcanbeeasilyextendedtoallUQ-
set aninfinity of times. Both processesconverge to the ADTs.Let’scallthisnewconsistencycriterionpipelined
same state (∅), so the history is eventually consistent. consistency(PC).Inapipelinedconsistentcomputation,
However, before converging, the processes can read each process must have a consistent view of its local
anything a finite but unbounded number of times. history with all the updates of the computation. More
Strongeventualconsistency: strongeventualconsis- formally,itcorrespondstoDef.7.Pipelinedconsistency
tencyrequiresthattworeplicasofthesameobjectcon- is local to each process, as different processes can see
vergeassoon asthey havereceivedthe same updates. concurrent updates in a different order.
The problem with that definition is that the notions Definition 7. A history H is pipelined consistent (PC)
of replica and message reception are inherent to the if, for all maximal chains (i.e. sets of totally ordered
implementation, andarehiddenfromtheprogrammer events) p of H, lin(H )∩L(O)6=∅.
UH∪p
that uses the object, so they should not be used in
Pipelined consistency can be implemented at a very
its specification. A visibility relation is introduced to
low cost in wait-free systems. Indeed, it only requires
model the notion of message delivery. This relation is
FIFO reception. However, it does not imply conver-
not an order since it is not required to be transitive.
gence. For example, the history given in Figure 2 is
Definition 6 (Strong eventual consistency). A history
pipelined consistent but not eventually consistent. In
H is strong eventually consistent (SEC) if there exists
this history, two processes p and p share a set of
vis 1 2
an acyclic and reflexive relation −−→ (called visibility
integers. Process p first inserts 1 and then 3 in the set
1
relation) that contains 7→ and such that:
and then reads the set forever. Meanwhile, process p
2
• Eventual delivery: when an update is viewed by inserts2,deletes3andreadsthesetforever.Thewords
areplica,itiseventuallyviewedbyallreplicas,so w andw arecorrectlinearizationsforbothprocesses,
1 2
therecanbe atmost afinite number of operations with regard to Definition 7 so the history is pipelined
that do not view it: consistent, but after stabilization, p sees the element
2
vis
∀u∈UH,{e∈E,u−−6 →e} is finite; 3 whereas p1 does not.
I(1) R/{2} R/{1} R/∅ω I(1) D(2) R/{1,2}ω I(1) R/∅ R/{1,2}ω I(1) R/{1} I(2) R/{1,2}ω
• • • • • • • • • • • • • •
• • • • • • • • • • •
I(2) R/{1} R/{2} R/∅ω I(2) D(1) R/{1,2}ω I(2) R/{1,2}ω R/{2} R/{1,2}ω
(a) EC but not SEC nor UC (b) SEC but not UC (c) SEC and UC but not SUC (d) SUC but not PC
Figure 1: Four histories for an instance of SN (cf. example 1), with different consistency criteria. The arrows
represent the program order, and an event labeled ω is repeated an infinite number of times.
I(1) I(3) R/{1,3} R/{1,2,3} R/{1,2}ω V. UPDATECONSISTENCY
• • • • •
a b c d e
•f •g •h •i •j
I(2) D(3) R/{2} R/{1,2} R/{1,2,3}ω In this section, we introduce two new consistency
criteria: update consistency and strong update consis-
w =I(1)·I(3)·R/{1,3}·I(2)·R/{1,2,3}·D(3)·R/{1,2}ω tency2, and we compare them to eventual consistency
1
andstrongeventualconsistency. Fig.1illustratesthese
w2 =I(2)·D(3)·R/{2}·I(1)·R/{1,2}·I(3)·R/{1,2,3}ω four consistency criteria on four small examples.
Figure 2: PC but not EC Update consistency: eventual consistency and
strongeventualconsistencyarenotinterestedindefin-
ingthestatesthatarereachedduringthehistories(the
Proposition 1 (Implementation). Pipelined convergence, same updates have to lead to the same state whatever
that imposes both pipelined consistency and eventual con- is the state). They do not depend on the sequential
sistency, cannot be implemented in a wait-free system. specification of the object, so they give very little
constraintsonthehistories.Forexample,animplemen-
Proof: We consider the same program as in Figure tation that ignores all the updates is strong eventually
2, and we suppose the shared set is pipelined conver- consistent, as all the queries return the initial state.
gent. By the same argument as developed in [2], it is In update consistency, we impose the existence of a
not possible to prevent the processes from not seeing total order on the updates, that contains the program
each other’s first update at their first reads. Indeed, order and that leads to the consistent state according
if p did not receive any message from process p , it to the abstract data type. Another equivalent way to
1 2
is impossible for p to make the difference between approach update consistency is that, if the number
1
thecasewherep crashedbeforesendinganymessage of updates is finite, it is possible to remove a finite
2
and the case where all its messages were delayed. To number of queriessuchthatthe history issequentially
achieve availability, p must compute the return value consistent.
1
basedsolelyonitslocalknowledge,soitreturns{1,3}.
Definition 8 (Update consistency). A history H is
Similarly, p returns {2}. To circumvent this impossi-
2 update consistent (UC) if U is infinite or if there
bility, it is necessary to make synchrony assumption H
exists a finite set of queries Q′ ⊂ Q such that
on the system (e.g. bounds on transmission delays) or H
to assume the correctness of a majority of processes. lin(cid:0)HE\Q′(cid:1)∩L(O)6=∅.
If the first read of p1 returns {1,3}, as the set is The history of Fig. 1c is update consistent because
pipelined consistent, there must exist a linearization the sequence of operations I(1)I(2) is a possible ex-
for p1 that contains all the updates, R/{1,3} and planation for the state {1,2}. The history of Fig. 1b is
an infinity of queries. As 2 6∈ {1,3}, the possible not update consistent because any linearization of the
linearizations are defined by the ω-regular language
updates would position a deletion as the last event.
I(1)·I(3)·R/{1,3}+·I(2)·R/{1,2,3}⋆·D(3)·R/{1,2}ω,so Only three consistent states are actually possible: state
any history must contain an infinity of events labelled ∅, e.g. for the linearization I(1)·I(2)·D(1)·D(2), state
R/{1,2}ω. Similarly, if p2 starts by reading {2}, it {1}for the linearizationI(2)·D(1)·I(1)·D(2) and state
will eventually read {1,2,3} an infinity of times. This {2} for the linearization I(1)·D(2)·I(2)·D(1). Update
impliesthatpipelinedconvergencecannotbeprovided consistency is incomparable with strong eventual con-
in wait-free systems. sistency.
Consequently causal consistency, that is stronger
thanpipelinedconsistency,cannotbesatisfiedtogether
2These consistency criteria were previously presented as a brief
with eventual consistency in a wait-free system. announcementinDISC2014[17].
Strong update consistency: strong update consis- Now, suppose H is strong update consistent. Strong
tencyisastrengtheningofbothupdateconsistencyand update consistency respects both eventual delivery
strong eventual consistency. The relationship between and growth properties. Let V ⊂ U . As the rela-
H
update consistency and strong update consistency is tion ≤ is a total order, there is a unique word w in
analogoustotherelationbetweeneventualconsistency lin H≤ ∩ L(O). Let us denote s the state obtained
(cid:16) V(cid:17)
and strong eventual consistency.
after the execution of w. For all q ∈Q such that V =
H
Definition9 (Strongupdateconsistency). A history H {u∈U :u−v−i→s q}, lin H≤ ∩L(O)={w·Λ(q)},
is strong update consistent (SUC) if there exists (1) an H (cid:16) Eq∪{q}(cid:17)
so q = q /q with G(s,q ) = q . Consequently, H is
vis i o i o
acyclic and reflexive relation −−→ that contains 7→ and strong eventually consistent.
vis
(2) a total order ≤ that contains −−→ such that:
• Eventual delivery: VI. EXPRESSIVENESS OF UPDATECONSISTENCY: A
vis CASESTUDY
∀u∈U ,{e∈E,u−−6 →e} is finite;
H
• Growth: The set is one of the most studied eventually con-
∀e,e′,e′′ ∈E, e−v−i→s e′∧e′ 7→e′′ ⇒ e−v−i→s e′′ ;
(cid:16) (cid:17) (cid:16) (cid:17) sistent data structures. Different types of sets have
• Strong sequential convergence: A query views an been proposed as extensions to CRDTs to implement
vis
updateifthisupdateprecedesitaccordingto−−→. eventually consistent sets even though the insert and
Each query is the result of the ordered execution, delete operations do not commute. The simplest set
according to ≤, of the updates it views: is the Grow-Only Set (G-Set) [9], in which it is only
∀q ∈Q ,lin H≤ ∩L(O)6=∅ possible to insert elements. As the insertion of two
H (cid:16) V(q)∪{q}(cid:17)
elements commute, G-Setis a CRDT. Using two G-Set,
vis
where V(q)={u∈U :u−−→q}.
H a white list for inserted elements and a black list for
the deleted ones, it is possible to build a Two-Phases
Fig. 1d shows an example of strong update con-
Set (2P-Set, a.k.a. U-Set, for Unique Set) [18], in which
sistent history: nothing prevents the second process
it is possible to insert and remove elements, but never
from seeing the insertion of 2 before that of 1. Strong
insert again an element that has already been deleted.
eventual consistency and update consistency does not
Other implementations such as C-Set [19] and PN-Set,
imply strong update consistency: in the history of Fig.
add counters on the elements to determine if they
1c, after executing event I(1), the only three possible
should be present or not. The Observe-Remove Set
update linearizations are I(1), I(1)·I(2) and I(2)·I(1)
(OR-Set)[9], [20] isthe bestdocumented algorithm for
and none of them can lead to the state ∅ according
theset.Itisveryclosetothe2P-Setinitsprinciples,but
to the sequential specification of a set object. So the
eachinsertion istimestamped withaunique identifier,
historyofFig.1cisnotstrongupdateconsistent,while
and the deletion only black-lists the identifiers that
itisupdateconsistentandstrongeventuallyconsistent.
it observes. It guaranties that, if an insertion and
Proposition 2 (Comparison of consistency criteria). If
a deletion of the same element are concurrent, the
a history H is update consistent, then it is eventually
insertion will win and the element will be added
consistent. If H is strong update consistent, then it is both
to the set. Finally, the last-writer-wins element set
strong eventually consistent and update consistent.
(LWW-element-Set) [9] attaches a timestamp to each
Proof: Suppose H is update consistent. If H con- element to decide which operation should win in case
tains an infinite number of updates, then it is even- ofconflict.Allthesesets,andtheeventuallyconsistent
tually consistent. Otherwise, there exists a finite set objectsingeneral,haveadifferentbehaviorwhenthey
Q′ ⊂ QH and a word w ∈ lin(cid:0)HEH\Q′(cid:1) ∩ L(O). As are used in distributed programs.
the number of updates is finite, there is a finite prefix
The above mentioned implementations are even-
v of w that contains them all. v ∈ L(O), so it labels
tually consistent. However, as eventual consistency
a path between s and a state s in the UQ-ADT. All
0 does not impose a semantic link between updatesand
the queries that are in w but not in v return the same
queries, it is hazardousto say anything on the confor-
states,andthe numberof queriesinQ′ andv isfinite.
mance to the specification of the object. Burckhardt et
Hence, H is eventually consistent.
al.[13]proposetospecifythesemanticsofaquerybya
Suppose H is strong update consis- function on its concurrent history, called visibility, that
tent with a finite number of updates. correspondstothevisibilityrelationinstrongeventual
Q′ = {q∈Q ,q ≤u} is finite, and lin(E \Q′) consistency, and a linearization of this history, called
Su∈UH H H
contains only one word that is also contained into arbitration. In comparison, sequential specifications are
L(O). Obviously, H is update consistent restricted to the arbitration relation. It implies that
fewerupdateconsistentobjectsthaneventuallyconsis- Proof: Suppose H is strong update consistent for
tent objects can be specified. Although the variety of SVal. We define the new relation −I−W→ such that for all
objects with a distributed specification seems to be a e,e′ ∈ E, e −I−W→ e′ if one of the following conditions
chance thatcompensates the lower levelof abstraction
holds:
itallows,animportantbiasmustbetakenintoaccount:
from the point of view of the user, the visibility of an • e−v−i→s e′;
operation is not an a priori propertyof the system, but • eande′ aretwoupdatesonthesameelementand
an a posteriori way to explain what happened. If one e≤e′;
onlyfocusesonthefinalstate,anupdateconsistentob- • e′ is a query, and there is an update e′′ such that
ject is appropriate to be used instead of an eventually e−I−W→e′′ and e′′ −I−W→e′.
consistent object, since the final state is the same as if
IW
no operations were concurrent. The relation −−→ is acyclic because it is included in
≤, its growth and eventual delivery properties are
By adding further constraints on the histories, con- vis
ensured by the fact that it contains −−→. Moreover,
current specifications strengthen the consistency cri-
no two updates for the same element are concurrent
teria. Even if strong update consistency is stronger
IW
accordingto−−→and the last updatesarealso the last
than strong eventual consistency, we cannot say in
forthe≤relation,consequently H isstrongeventually
general that a strong update consistent object can al-
consistent for the Insert-wins set.
waysbeusedinsteadofitsstrongeventuallyconsistent
counterpart. We claim that this is true in practice for This result implies that an OR-set can always be
reasonable objects, and we prove this in the case of replacedbyanupdateconsistentset,becausetheguar-
the Insert-wins set (the concurrent specification of the anties it ensures are weaker than those of the update
OR-set). The arbitration relation is not used for the consistent set. It does not mean that the OR-set is
OR-set, and the visibility relation has already been worthless. It can be seen as a cache consistent set [21]
definedforstrongeventualconsistency.Theconcurrent that,insomecasesmayhaveabetterspacecomplexity
specification only adds one more constraint on this than update consistency.
relation: an element is present in the set if and only
if it was inserted and is not yet deleted. VII. GENERIC CONSTRUCTION OF STRONG UPDATE
CONSISTENT OBJECTS
Definition 10 (Strong eventual consistency for the
Insert-wins set). A history H is strong eventu- In this section, we give a generic construction of
ally consistent for the Insert-wins set on a sup- strong update consistent objects in crash-prone asyn-
port Val if it is strong eventually consistent for chronous message-passing systems. This construction
vis is not the most efficient ever as it is intended to
the set SVal and the visibility relation −−→ veri-
work for any UQ-ADT object in order to prove the
fies the following additional property. For all x ∈
Val and q ∈ Q , with Λ(q) = R/s, x ∈ s ⇔ universalityofupdateconsistency.Foraspecificobject
H
∃u∈vis(q,I(x)),∀u′ ∈vis(q,D(x)),u −v−6i→s u′ , where anadhocimplementationonaspecificsystemmaybe
(cid:16) (cid:17) more suitable.
vis
for all o∈U, vis(q,o)={u∈U :u−−→q∧Λ(u)=o}.
H
A. System Model
The OR-Set implementation of a set is not update We consideramessage-passingsystemcomposed of
consistent. The history on Fig. 1b is not updateconsis- finitesetofsequentialprocessesthatmayfailbyhalting.
tent,asthelastoperationmustbeadeletion.However, A faulty process simply stops operating. A process
if the updates made by a process are not viewed by that does not crash during an execution is correct. We
the other process before it makes its own updates, make no assumption on the number of failures that
the insertions will win and the OR-set will converge can occur during an execution. Processes communi-
to {1,2}. On the contrary, a strong update consistent cate by exchanging messages using a communication
implementation of a set can always be used instead of network complete and reliable. A message sent by a
an Insert-wins set, as it only forbids more histories. correctprocesstoanother correctprocessiseventually
received. The system is asynchronous; there is no
Proposition 3 (Comparison with Insert-wins set). Let
bound on the relative speed of processes nor on the
H = (U,Q,E,Λ,7→) be a history that is strong update
message transfer delays. In such a situation a process
consistent for SVal. Then H is strong eventually consistent
cannotwaitfortheparticipationofanyaprioriknown
for the Insert-wins set.
number of processes as they can fail. Consequently,
when an operation on a replicated object is invoked
Algorithm 1: a generic UQ-ADT (code for p ) (process ids are assumed unique and totally ordered).
i
The algorithm actions performed by a process p are
1 object (U,Qi,Qo,S,s0,T,G) i
2 var clocki ∈N←0; atomic and totally ordered by an order 7→i. The union
3 var updatei ⊂(N×N×U)←∅; o7→f.these orders for all processes is the program order
4 fun update (u∈U)
5 clocki ←clocki+1; At the application level, a history is composed of
6 broadcast message (clocki,i,u); update and query operations. In order to allow only
7 end strong update consistent histories, Algorithm 1 pro-
8 on receive message (cl∈N,j∈N,u∈Q) poses a procedure update() and a function query().
9 clocki ←max(clocki,cl); A history H is allowed by the algorithm if update(u)
10 updatei ←updatei∪{(cl,j,u)}; is called each time a process performs an update u,
11 end and query(qi) is called and returns qo when the event
12 fun query (q∈Qi)∈Qo qi/qo appears in the history. The code of Algorithm 1
13 clocki ←clocki+1; is given for process pi. Each process pi manages its
14 var statei ∈S ←s0; view clocki of the logical clock and a list updatesi of
15 for (cl,j,u)∈updatei sorted on (cl,j) do all timestamped update events process pi is aware of.
16 statei ←T(statei,u); The list updatesi contains triplets (cl,j,u) where u is
17 end an update event and (cl,j) the associated timestamp.
18 return G(statei,q); This list is sorted according to the timestamps of the
19 end updates: (cl,j) < (cl′,j′) if (cl < cl′) or (cl = cl′ and
20 end j <j′).
The algorithm timestamps all events (updates and
queries). When an update is issued locally, process
p informs all the other processes by reliably broad-
locallyatsomeprocess,itneedstobecompletedbased i
casting a message to all other processes (including
solely on the local knowledge of the process. We call
itself). Hence, all processes will eventually be aware
this kind of systems wait-free asynchronous message-
of all updates. When a message(cl,j,u) is received,
passing system.
p updates its clock and inserts the event to the list
i
We modelexecutionsashistoriesmadeupof these-
updates . When a query isissued, the function query()
i
quencesofeventsgeneratedbythedifferentprocesses.
replays locally the whole list of update events p is
i
As we focus on shared objects and their implemen-
aware of starting from the initial state then it executes
tation, only two kinds of actions are considered: the
the query on the state it obtains.
operationsonsharedobjects, thatareseenaseventsin
Whenever an operation is issued, its is completed
the distributed history, and message receptions.
without waiting for any other process. This corre-
B. A universal implementation sponds to wait-free executions in shared memory dis-
Now, we prove that strong update consistency is tributed systems and implies fault-tolerance.
universal, in the sense that every UQ-ADT has a Proposition4(Strongupdateconsistency). Allhistories
strong update consistent implementation in a wait- allowed by Algorithm 1 are strong update consistent.
free asynchronous system. Algorithm 1 presents an
implementation of a generic UQ-ADT. The principle Proof: Let H = (U,Q,E,Λ,7→) be a distributed
is to build a total order on the updates on which all history allowed by Algorithm 1. Let e,e′ ∈ E be
H
the participants agree, and then to rewrite the history two operations invoked by processes pi and pi′, on
a posteriori so that every replica of the object eventu- the states (update,clock) and (update′,clock′),
ally reaches the state corresponding to the common respectively. We pose:
sequentialhistory.Anystrategytobuildthetotalorder
ontheupdateswouldwork.InAlgorithm1,thisorder
is built from a Lamport’s clock [14] that contains the • e −v−i→s e′ if e ∈ UH and pi′ received the message
happened-before precedence relation. Process order is sent during the execution of e before it starts
hence respected. A logical Lamport’s clock is a pre- executing e′, or e ∈ QH and e 7→ e′. As the mes-
total order as some events may be associated with the sages are received instantaneously by the sender,
vis
same logical time. In order to have a total order, the −−→ contains 7→. It is growing because the set of
events are timestamped with a pair composed of the messages received by a process is growing with
logical time and the id of the process that produced it time.
• e ≤ e′ if c < c′ or c = c′ and i < i′. This Algorithm 2: the shared memory (code for pi)
lexical order is total because two operations on
1 object UC mem(X,V,v0)
the same provciessshave a differentclock. Moreover 2 var clocki ∈N←0;
it contains −−→ because when pi′ received the 3 var memi ∈mem(X,(N2×V),(0,0,v0));
message sent by e, it executed line 9 and when 4 fun write (x∈X,v∈V)
it executed e′, it executed line 5, so c′ ≥ c + 1. 5 clocki ←clocki+1;
Moreover,thehistoryofecontainsatmostc×n+i 6 broadcast msg (clocki,i,x,v);
events, where n is the number of processes, so it 7 end
is finite. 8 on receive msg (cl∈N,j∈N,x∈X,v∈V)
vis
Let q ∈ QH and Eq = {u ∈ UH : u −−→ q}. Lines 9 clocki ←max(clocki,cl);
15 to 18 build an explicit sequential execution, that is 10 var (cl′,j′,v′)∈N2×V ←memi.read(x);
in lin(cid:16)HE≤q∪{q}(cid:17) by definition of ≤ and in L(O) by 11 if (cl′,j′)<(cl,j) then
definition of O. 12 memi.write(x,(cl,j,v))
13 end
C. Complexity
14 end
Algorithm 1 is very efficient in terms of network 15 fun read (x∈X)∈V
communication. A unique message is broadcast for 16 var (cl,j,v)∈N2×V ←memi.read(x);
eachupdateandeachmessageonlycontainstheinfor- 17 return v;
mation to identify the update and a timestamp com- 18 end
posedoftwointegervalues,thatonlygrowlogarithmi- 19 end
cally with the number of processes and the number of
operations. Moreover, this algorithm is wait-free and
its execution does not depend on the latency of the
network.
Beaudouin-Lafon propose an algorithm to implement
Thisalgorithmre-executesallpastupdateseachtime objects such that each update operation u contains
a new query is issued. In an effective implementation, an undo u−1 such that for all s, T(T(s,u),u−1) = s.
a process can keep intermediate states. These interme- This algorithm is very close to ours as it builds the
diate states are re-computed only if very late message convergent state from a linearization of the updates
arrive.Thealgorithmdoesnotlookspaceefficientalso stored by each replica. They use the undo operations
asthewhole historymustbekeptinordertorebuilda topositionnewlyknownupdatesattheircorrectplace,
sequentialhistory.Becausedataspaceischeapandfast whichsavescomputationtime.Asitisaveryfrequent
nowadays,comparedtobandwidth,manyapplications example in distributed systems, we now focus on the
can afford this complexity and would keep this infor- shared memory object.
mation anyway. For example, banks keep track of all
Algorithm 2 shows an update consistent implemen-
the operations made on an account for years for legal
tation of the shared memory object. A shared memory
reasons. In databases systems, it is usual to record all
offers a set X of registers that contain values taken
the events in log files. Moreover, asynchrony is used
from a set V. The query operation read(x), where
asaconvenientabstractionforsystemsinwhichtrans-
x ∈ X, returns the last value v ∈ V written by
mission delaysare actuallybounded, but the bound is
the update operation write(x,v), or the initial value
too large to be used in practice. This means that after
v ∈V if x was never written. Algorithm 2 orders the
some time old messages can be garbage collected. 0
updatesexactlylikeAlgorithm1.Astheoldvaluescan
The proposed algorithmis atheoreticalworkwhose never be read again, it is not necessary to store them
goal is to prove that any update-query object has a forever,sothealgorithmonlykeepsinmemorythelast
strong update consistent implementation. This gener- known value of each register and its timestamp in a
icity prevents an effective implementation that may local memory mem , implemented with an associative
i
take benefit from the nature and the specificity of array. When a process receives a notification for a
the actual object. The best example of this are pure write, it updates its local state if the value is newer
CRDTslikethecounterandthegrow-onlyset.Ifallthe that the current one, and the read operations just
update operations commute in the sequential specifi- return the current value. This implementation only
cation, all linearizations would lead to the same state needs constant computation time for both the reads
so a naive implementation, that applies the updates and the writes, and the complexity in memory only
on a replica as soon as the notification is received, grows logarithmically with time and the number of
achieves update consistency. In [22], Karsenty and participants.
VIII. CONCLUSION [8] M. Shapiro, N. Preguic¸a, C. Baquero, and M. Zawirski,
This paper proposes a new consistency criterion, “Conflict-free replicated data types,” in Stabilization,
Safety,andSecurityofDistributedSystems. Springer,2011,
update consistency, thatis stronger than eventual con-
pp. 386–400.
sistency and weaker than sequential consistency. Our
approachformalizesthe intuitivenotions of sequential [9] M.Shapiro,N.Preguic¸a,C.Baquero,M.Zawirskietal.,
specification for an abstract data type and distributed “A comprehensive study of convergent and commuta-
tive replicated data types,”INRIA, Tech. Rep., 2011.
history. This formalization first allowed to prove that
eventualconsistencywhenassociatedwithcausalcon- [10] C. Sun, X. Jia, Y. Zhang, Y. Yang, and D. Chen,
sistency or PRAM consistency can no more be imple- “Achieving convergence, causality preservation, and
mented in an asynchronous distributed system where intention preservation in real-time cooperative editing
systems,”ACMTransactionsonComputer-HumanInterac-
all but one process may crash.
tion (TOCHI),vol. 5, no. 1, pp. 63–108, 1998.
This paper formalizes the new consistency criterion
and proves that (1) it is strictly stronger than eventual [11] D. Li, L. Zhou, and R. R. Muntz, “A new paradigm
of user intention preservation in realtime collaborative
consistency and (2) that it is universal in the sense
editing systems,” in International Conference on Parallel
that allowed any update consistent object can be im-
And DistributedSystems. IEEE, 2000, pp. 401–408.
plemented in wait-free systems. The latter has been
proved through a generic construction that implement [12] A. Bieniusa, M. Zawirski, N. Preguic¸a, M. Shapiro,
C. Baquero, V. Balegas, and S. Duarte, “An op-
all considered data types.
timized conflict-free replicated set,” arXiv preprint
arXiv:1210.3368, 2012.
ACKNOWLEDGMENT
This work has been partially supported by a French [13] S. Burckhardt, A. Gotsman, H. Yang, and M. Zawirski,
government support granted to the CominLabs ex- “Replicated data types: specification, verification, opti-
cellence laboratory (Project DeSceNt: Plug-based Decen- mality,”inProceedingsofthe41stsymposiumonPrinciples
of programming languages. ACM, 2014, pp. 271–284.
tralized Social Network) and managed by the French
NationalAgencyforResearch(ANR)inthe ”Investing [14] L. Lamport, “Time, clocks, and the ordering of events
for the Future”programunder referenceNb. ANR-10- in a distributed system,” Communications of the ACM,
LABX-07-01. This work was partially funded by the vol. 21, no. 7, pp. 558–565, 1978.
French ANR project SocioPlug (ANR-13-INFR-0003).
[15] M. Perrin, M. Petrolia, C. Jard, and A. Moste´faoui,
“Consistentshareddatatypes:Beyondmemory,”LINA,
REFERENCES
Universite´ de Nantes, Tech. Rep., 2014.
[1] R. J.Liptonand J. S. Sandberg,PRAM:A scalableshared
[16] G. H. Mealy, “A method for synthesizing sequential
memory. Princeton University, Department of Com-
circuits,”BellSystemTechnicalJournal,vol.34, no.5, pp.
puter Science, 1988.
1045–1079, 1955.
[2] H. Attiya and J. L. Welch, “Sequential consistency
[17] M.Perrin,A.Moste´faoui,andC.Jard,“Briefannounce-
versus linearizability,” ACM Transactions on Computer
ment: Update consistency in partitionable systems,” in
Systems (TOCS), vol. 12, no. 2, pp. 91–122, 1994.
Proceedings of the 28th International Symposium on Dis-
tributedComputing. Springer, 2014, p. 546.
[3] H. Attiya,A. Bar-Noy,and D.Dolev,“Sharing memory
robustly in message-passing systems,” J. ACM, vol. 42, [18] G. T. Wuu and A. J. Bernstein, “Efficient solutions to
no. 1, pp. 124–142, 1995. the replicated log and dictionary problems,” Operating
systems review, vol. 20, no. 1, pp. 57–66, 1986.
[4] W.Vogels,“Eventually consistent,”Queue,vol.6,no. 6,
pp. 14–19, 2008. [19] K.Aslan,P.Molli,H.Skaf-Molli,S.Weissetal.,“C-set:a
commutative replicated data type for semantic stores,”
[5] M. Ahamad, G. Neiger, J. E. Burns, P. Kohli, and P. W. in RED: Fourth International Workshop on REsource Dis-
Hutto, “Causal memory: Definitions, implementation, covery, 2011.
andprogramming,”DistributedComputing,vol.9,no.1,
pp. 37–49, 1995. [20] M. Mukund, G. Shenoy, and S. Suresh, “Optimized or-
sets without ordering constraints,” in Distributed Com-
[6] M.Herlihy,“Wait-freesynchronization,”inACMTrans- puting and Networking. Springer, 2014, pp. 227–241.
actionsonProgrammingLanguagesandSystems,1991, pp.
124–149. [21] J. R. Goodman, Cache consistency and sequential consis-
tency. University of Wisconsin-Madison, Computer
[7] G. DeCandia, D. Hastorun, M. Jampani, G. Kakula- Sciences Department, 1991.
pati, A. Lakshman, A. Pilchin, S. Sivasubramanian,
P. Vosshall, and W. Vogels, “Dynamo: amazon’s highly [22] A. Karsenty and M. Beaudouin-Lafon, “An algorithm
available key-value store,” in ACM SIGOPS Operating for distributed groupware applications,” in ICDCS,
Systems Review, vol. 41. ACM, 2007, pp. 205–220. 1993.