DW 6 aao

17 Pages·2015·1.62 MB·English

US 20140310240A1
(19) United States
(12) Patent Application Publication (10) Pub. No.: US 2014/0310240 A1
MENG et al. (43) Pub. Date: Oct. 16, 2014
(54) EXECUTING DISTRIBUTED Publication Classi?cation
GLOBALLY-ORDERED TRANSACTIONAL
WORKLOADS IN REPLICATED STATE (51) Int- Cl
MACHINES G06F 17/30 (2006.01)
(52) US. Cl.
(71) Applicant: International Business Machines CPC .............................. .. G06F 17/305 75 (2013.01)
Corporation, Armonk, NY (US) USPC ........................................................ .. 707/615
(72) Inventors: SHICONG MENG, Elmsford, NY
NY (U S) Yorktown
(73) ASSlgnee: Internati‘fnal Busmess MaChines A method of transaction replication includes transmitting at
corporatlon’ Armonk’ NY (Us) least one transaction received during an epoch from a local
(21) APPL NO; 14/023,921 node to remote nodes of a domain of 2N+1 nodes at the end of
an epoch (N is an integer greater than or equal to 1). The
(22) Filed: SeP- 11, 2013 remote nodes log receipt of the at least one transaction, notify
. . the local node of the recei t of the at least one transaction,
Related U's' Apphcatlon Data transmit the at least one trarpisaction to all of the 2N+1 nodes,
(63) Continuation of application No. 13/863,065, ?led on and add the atleast one transaction to an execution orderupon
Apr. 15, 2013. receiving at least N+1 copies of the at least one transaction.
Datacenter
101 LQA
Datacenter
1%
K Client Library
Datacenter
K K192.
Transaction
[ Leader )[AceeptorH Learner H
Replicator r
102
l® T®
DW 6 a a o
Patent Application Publication Oct. 16, 2014 Sheet 1 0f 7 US 2014/0310240 A1
@EEQQ Ema
S@A
@5223 \H/
@QQ
$@bo#3?6w655omw33mr:2 m:m?
\/\/@ @
92$ng
:26meth
j)
\522%6
Patent Application Publication Oct. 16, 2014 Sheet 2 0f7 US 2014/0310240 A1
3E6mm3mwmwm s
g_2$m_3mmm e
wmEw@n9mo3c8 5
22smw56a.2g3 Haw 2.8H; Nam mD5266:302 mam $$$853:3 5 gm .mQI
5232m23e2
a a %
:
=wh5oammuo m
@$25203
+
wHmgowommcmg
220&83m 3=6882m
.NOE
5E.8m\%
2256m685?2%. HUN Nam 222.60:H¢o5 n mam mam
a % a
Patent Application Publication Oct. 16, 2014 Sheet 3 0f7 US 2014/0310240 A1
6529,m8 2 285
a
3E6225533$2% 22m52625
$232m3e
52:8m6%m3 g 2#32 B$5855:8 wow E5m9so;v Mg,
E
_9£0_58mv a
5Ewg0?0mw% cm : E5$26EmeOth
22:B2628;:¢68 395a3nm
.B$$5258 52382m8582:2
s22wm66a.2g: 3% N9“ 3_82_28m02 §
HEmm?
Patent Application Publication Oct. 16, 2014 Sheet 4 0f 7 US 2014/0310240 A1
Transmit transaction by local node to remote nodes
5.91
1
Log receipt of transaction by each remote node
$2
l
Transmit notification from remote nodes to local node
5%
l
Transmit transaction from remote nodes to all other nodes
other than local node
5.9.4.
1
Add transaction to execution list
5&5
FIG. 5
Patent Application Publication Oct. 16, 2014 Sheet 5 0f 7 US 2014/0310240 A1
Receive local transaction at iocai node during epoch
59.1
i
Receive transaction at remote node during epoch
£02
1
Assign timestamp to each transaction upon. receiving
each transaction
603
i
Assign epochtd to each transaction upon receiving
each transaction
5%
i
Combine transactions into transaction set
525
i
Assign global order to transactions in transaction set
5%
FIG. 6
Patent Application Publication Oct. 16, 2014 Sheet 6 0f 7 US 2014/0310240 A1
Generate local signature at locai node
ZQJ.
i
Transmit local signature to all nodes in domain
E
i
Generate remote signatures
ZDQ
i
Transmit remote signatures to all nodes in domain
29A
i
Transmit signature mismatch notifications to local node
19.5
i
Disconnect local node upon receiving at least N+1
signature mismatch notifications
1%
i
Restore disconnected local node using a remote signature
Ztll
FIG. 7
Patent Application Publication Oct. 16, 2014 Sheet 7 0f 7 US 2014/0310240 A1
f804
8021 I805
CPU Display
8071 806
Input 5
8031 Memory <—><-W Devices
Signal
Source
FIG. 8
US 2014/0310240 A1 Oct. 16, 2014
EXECUTING DISTRIBUTED method further includes notifying the local datastore node of
GLOBALLY-ORDERED TRANSACTIONAL the receipt of the at least one transaction, by the at least one
WORKLOADS IN REPLICATED STATE remote datastore node, upon receiving the at least one trans
MACHINES action at the at least one remote datastore node. The method
further includes transmitting the at least one transaction to all
CROSS-REFERENCE TO RELATED datastore nodes in the domain other than the local datastore
APPLICATIONS node and the at least one remote datastore node, by the at least
one remote datastore node, upon receiving the at least one
[0001] This application is a Continuation application of
transaction at the at least one remote datastore node. The
US. application Ser. No. 13/863,065, ?led on Apr. 15, 2013,
method further includes adding the at least one transaction to
the disclosure of which is incorporated by reference herein in
its entirety. an execution order, at each of the datastore nodes of the
domain, upon receiving at least N+1 copies of the at least one
transaction at each of the datastore nodes.
BACKGROUND
[0009] In an exemplary embodiment the method further
[0002] 1. Technical Field includes measuring an elapsed time, by the at least one remote
[0003] The present disclosure relates to transaction pro datastore node, since last logging the receipt of the at least one
cessing, and more particularly, to a system and method for transaction, transmitting a timeout message to all datastore
executing distributed globally-ordered transactional work
nodes in the domain, by the at least one remote datastore
loads in replicated state machines node, upon the elapsed time exceeding a prede?ned timeout
[0004] 2. Discussion of RelatedArt threshold, transmitting a timeout con?rmation message to all
[0005] The rapid development of cloud computing and net datastore nodes in the domain, by each of the datastore nodes
working infrastructure has improved the availability of busi of the domain, upon receiving at least N+1 timeout con?rma
ness information systems across the globe. As a popular tion messages regarding the local datastore node at each of the
deployment paradigm, business applications often run in dis datastore nodes, and removing the local datastore node from
tributed datacenters. Clients connect to their local datacenters a participating datastore node list, by each of the datastore
to minimize data access latency. To ensure business applica nodes of the domain, upon receiving at least N+1 timeout
tions running in different datacenters share the same view of con?rmation messages regarding the local datastore node at
data, application developers often rely on cross-datacenter each of the datastore nodes.
data replication services or global transaction support. [0010] In an exemplary embodiment, the method further
[0006] Some cloud systems adopt a primary-secondary includes discarding un?nished processes that have occurred
deployment for data replication over multiple datacenters, in prior to removal from the participating datastore node list, by
which one primary datastore has the most recent version of the local datastore node, upon being removed from the par
the data, and one or more secondary datastores asynchro ticipating datastore node list, transmitting a recovery noti?
nously replicate updates from the primary datastore to local cation message, to each of the datastore nodes in the domain,
datastores. In such a system, since all write operations can by the local datastore node, and adding the local datastore
only be performed on the primary datastore to ensure data node to the participating datastore node list, by each of the
consistency, data access of clients remote to the primary datastore nodes of the domain, upon each of the datastore
datastore may be delayed. Further, read operations on sec nodes receiving at least N+1 copies of the recovery noti?ca
ondary datastores may yield stale results due to asynchronous tion message.
replication, which may not be acceptable for business appli
[0011] In an exemplary embodiment, the local datastore
cations with high consistency requirements.
node is added to the participating datastore node list by each
[0007] In addition, some cloud systems adopt a primary of the datastore nodes of the domain during different epochs.
primary deployment for data replication over multiple data
[0012] In an exemplary embodiment, the method further
centers, in which any participating datastore can perform read includes generating a local signature corresponding to the
and write operations while still maintaining a globally con local datastore node, by the local datastore node, transmitting
sistent data copy. In such a system, a transaction is either the local signature, by the local datastore node, to all data
committed or aborted globally. That is, if one datastore fails to store nodes in the domain, generating a remote signature
execute (e. g., as the result of deadlocks), an entire transaction corresponding to the at least one remote datastore node, by
is aborted.
the at least one remote datastore node, transmitting the remote
signature, by the at least one remote datastore node, to all
BRIEF SUMMARY
datastore nodes in the domain, transmitting a signature mis
[0008] According to an exemplary embodiment of the match noti?cation linked to the remote signature, by the at
present invention, a method of transaction replication least one remote datastore node, to the local datastore node
includes transmitting at least one transaction received during upon determining that the remote signature does not match
the local signature, and disconnecting the local datastore node
an epoch, by a local datastore node, to at least one remote
datastore node of a plurality of remote datastore nodes upon from the domain of datastore nodes upon the local datastore
an end of the epoch, wherein the epoch corresponds to a node receiving at least N+1 signature mismatch noti?cations
prede?ned time period having a ?xed length, and the local linked to the remote signature.
datastore node and the plurality of remote datastore nodes are [0013] In an exemplary embodiment, the method further
part of a domain of 2N+1 datastore nodes, wherein N is an includes restoring the disconnected local datastore node
integer greater than or equal to 1. The method further includes using the remote signature.
logging a receipt of the at least one transaction, by the at least [0014] According to an exemplary embodiment of the
one remote datastore node, upon receiving the at least one present invention, a method of establishing a global consis
transaction at the at least one remote datastore node. The tent execution order of transactions includes receiving at least
US 2014/0310240 A1 Oct. 16, 2014
one local transaction during an epoch at a local datastore DETAILED DESCRIPTION
node. The method further includes receiving at least one
[0027] Exemplary embodiments of the present invention
remote transaction during the epoch, at a remote datastore
now will be described more fully hereinafter with reference to
node, wherein the epoch corresponds to a prede?ned time
the accompanying drawings. This disclosure, may however,
period having a ?xed length, and the local and remote datas
be embodied in many different forms and should not be
tore nodes are part of a domain of 2N+1 data store nodes,
construed as limited to embodiments set forth herein.
wherein N is an integer greater than or equal to 1 . The method
[0028] Exemplary embodiments of the present invention
further includes assigning a timestamp to each of the at least
are directed to a transaction service framework that imple
one local and remote transactions upon respectively receiving
ments a transaction replication process, establishes a global
the at least one local and remote transactions, wherein the
consistent execution order of workloads on all nodes without
timestamps correspond to a respective arrival time of the at
utilizing clock synchronization, and detects inconsistencies
least one local and remote transactions. The method further
among distributed nodes. The transaction service framework
includes assigning an epochId to each of the at least one local
according to exemplary embodiments may ensure consis
and remote transactions upon respectively receiving the at
tency of replicated data across distributed nodes in a distrib
least one local and remote transactions, wherein the epochId
uted system. Examples of distributed systems that the trans
corresponds to the epoch in which the at least one local and
action service framework may be used with include, but are
remote transactions were respectively received. The method
not limited to, distributed databases, distributed application
further includes combining the at least one local transaction
servers, distributed storage, distributed publish and subscribe
and the at least one remote transaction into a transaction set.
systems, etc.
The method further includes assigning a global order to all
[0029] For ease of description, exemplary embodiments are
transactions within the transaction set.
described herein with reference to distributed databases that
[0015] In an exemplary embodiment, assigning the global
are replicated across geographically distributed datacenters
order is based on respective timestamps and epochIds of all
and that process transaction workloads, however, exemplary
transactions within the transaction set.
embodiments are not limited thereto. For example, business
[0016] In an exemplary embodiment, assigning the global applications may be deployed in a cloud computing environ
order is further based on respective node IDs of all transac ment and may be executed across distributed datacenters. In
tions within the transaction set, and the respective node IDs this setting, clients may connect to their local datacenters, as
correspond to a node at which the transactions were received. opposed to remote datacenters, to reduce or minimize data
access latency. Exemplary embodiments of the present inven
[0017] In an exemplary embodiment, at least one local
transaction is a read-only transaction, and the read-only trans tion are directed to a transaction replication process that
action is not combined into the transaction set. results in business applications running in different data
centers sharing the same consistent view of data.
[0030] Business applications, as well as various other types
BRIEF DESCRIPTION OF THE SEVERAL
of applications, as discussed above, rely on a set of replicated
VIEWS OF THE DRAWINGS
datastores running in different datacenters. Herein, a group of
[0018] Exemplary embodiments of the present disclosure replicated datastores that collectively execute transactions
will be described below in more detail, with reference to the may be referred to as a domain. A single datacenter may host
accompanying drawings, in which: different datastores for different applications, and thus, may
be included in multiple domains. A transaction service may
[0019] FIG. 1 shows a distributed globally-ordered trans
actional workload system, according to an exemplary be utilized to orchestrate the execution of a transaction across
all datastores within the same domain, and may provide an
embodiment of the present invention.
abstraction of a single global domain datastore image to cli
[0020] FIG. 2 is a ?owchart showing operations of a node
ents. Clients are connected to the transaction service running
functioning as a leader, according to an exemplary embodi
within their respective local datacenters, which may reduce
ment of the present invention. read/write latency.
[0021] FIG. 3 is a ?owchart showing operations of a node [0031] Datacenters may be interconnected with each other
functioning as an acceptor, according to an exemplary through dedicated communication links, which are utilized
embodiment of the present invention. for distributed transactions occurring amongst the data
[0022] FIG. 4 is a ?owchart showing operations of a node centers. The round-trip latency of the dedicated communica
functioning as a learner, according to an exemplary embodi tion links may typically be between about 50 ms and about
ment of the present invention. 200 ms, and may increase when a link is congested. For
example, the round-trip latency of a communication link may
[0023] FIG. 5 is a ?owchart showing a transaction replica
tion process, according to an exemplary embodiment of the increase to about 10 seconds when the amount of data to be
present invention. transmitted is large, or when the link is undergoing device
failures or malfunction. Device failures and/or malfunctions
[0024] FIG. 6 is a ?owchart showing a method of establish
may be the result of common errors such as, for example,
ing a global consistent execution order of workloads, accord
software crashes, detected or undetected hardware failures,
ing to an exemplary embodiment of the present invention.
network failures, viruses, site disasters, etc.
[0025] FIG. 7 is a ?owchart showing a method of detecting [0032] The transaction service framework, according to
inconsistency among replicated datastore nodes, according to exemplary embodiments of the present invention, is indepen
an exemplary embodiment of the present invention. dent of the underlying datastore implementation. For
[0026] FIG. 8 illustrates a computer system for implement example, the transaction service framework may be utilized
ing aspects of exemplary embodiments of the present inven with any datastore implementation in which the datastores
tion. provide a table abstract where each row is uniquely identi?ed