Table Of ContentLIVE DISTRIBUTED OBJECTS
A Dissertation
Presented to the Faculty of the Graduate School
of Cornell University
in Partial Fulfillment of the Requirements for the Degree of
Doctor of Philosophy
by
Krzysztof Jan Ostrowski
August 2008
⃝c 2008 Krzysztof Jan Ostrowski
ALL RIGHTS RESERVED
LIVE DISTRIBUTED OBJECTS
Krzysztof Jan Ostrowski, Ph.D.
Cornell University 2008
Distributed multiparty protocols such as multicast, atomic commit, or gossip are
currently underutilized, but we envision that they could be used pervasively, and
that developers could work with such protocols similarly to how they work with
CORBA/COM/.NET/Java objects. We have created a new programming model and a
platform in which protocol instances are represented as objects of a new type called live
distributed objects: strongly-typed building blocks that can be composed in a type-safe
manner through a drag and drop interface. Unlike most prior object-oriented distributed
protocol embeddings, our model appears to be flexible enough to accommodate most
popular protocols, and to be applied uniformly to any part of a distributed system, to
build not only front-end, but also back-end components, such as multicast channels,
naming, or membership services.
While the platform is not limited to applications based on multicast, it is replication-
centric, and reliable multicast protocols are important building blocks that can be used to
create a variety of scalable components, from shared documents to fault-tolerant storage
or scalable role delegation. We propose a new multicast architecture compatible with
the model and designed in accordance with object-oriented principles such as modu-
larity and encapsulation. The architecture represents multicast objects as compositions
of smaller objects, glued using a decentralized membership mechanism. The benefits
include support for heterogeneous networks by allowing a single multicast protocol in-
stance to simultaneously leverage different mechanisms in different parts of the Internet.
Finally, we describe and evaluate a scalable reliable multicast prototype that incor-
porates a novel approach to scaling with the number of protocol instances by leveraging
regular patterns of overlap. The system achieves exceptional performance, scalability,
and stability in the presence of perturbations, in part thanks to a novel application of
techniques such as priority scheduling or pull-based processing. We describe a previ-
ously unnoticed relationship between memory overheads and scheduling and the per-
formance and scalability of a reliable multicast protocol. Our results demonstrate that
to build a new global Web operating system that the live objects vision leads to, the
distributed protocol and the local system architecture cannot be treated in isolation.
BIOGRAPHICAL SKETCH
Krzysztof Ostrowski is originally from Poland. He received his Bachelor of Science in
Mathematics and Computer Science from the University of Warsaw in Poland in 1998,
and his Master of Science in Computer Science from the same university in 2001. In
August, 2008 he received his Ph.D. from Cornell University.
iii
To my beloved Yejin.
iv
ACKNOWLEDGEMENTS
First and foremost, I am deeply grateful to Ken Birman for being an extremely support-
ive advisor, open to my ideas and encouraging, and for enabling me to develop and to
pursue my vision. Ken is a great teacher of assertiveness, optimism, and confidence in
one’s work, and has motivated me to stay firmly on the course that I have taken, and
to be determined in pursuing my goals despite the difficulties and obstacles I met along
the way. Ken has created countless opportunities for my work to be noticed and recog-
nized by people in the research community and in the industry. His critical feedback and
guidance were invaluable in helping me learn how to efficiently communicate my ideas.
Ken has also very much inspired me with his passion for building and measuring real
systems and with his commitment to tackling concrete real-world problems, even if hard
and intimidating. I especially appreciate having been able to benefit from Ken’s tremen-
dous practical knowledge and experience while working together on understanding the
performance of QSM and refining the design of the system.
I am also profoundly grateful to Danny Dolev, whose deep insight and wisdom at
both academic and personal level not only enriched my research, but also energized me
with high morale. Danny has been a fantastic collaborator with an extraordinary ability
to grasp the essence of one’s ideas and to quickly point out the subtleties involved in
realizing them. While working together on the foundations of Properties Framework,
he challenged and guided my thinking to help me steer around pitfalls; the architecture
would not have been possible without him. I am tremendously lucky to have been able
to work with Danny.
I am also much indebted to Jong-Hoon Ahnn for all the great work he has done for
the live objects platform. John has provided a great amount of feedback that has been
extremely helpful in improving the stability and usability of the platform, and his help
was essential to the success of our demos. He has also developed many components,
without which the platform would be incomplete.
I am sincerely grateful to Paul Francis, Andrew Myers, Anil Nerode, and Robbert
van Renesse, who have been members of my special committee, for their many in-
sightful comments and suggestions regarding my dissertation, research vision, style of
presentation, and career in general. I would like to especially thank Andrew and Robbert
for reading my dissertation so thoroughly; their detailed feedback has been very helpful
in improving the structure and style of my writing.
I would like to also thank Hussam Abu-Libdeh, Mahesh Balakrishnan, Lars Brenna,
Daniel Freedman, Lakshmi Ganesh, Maya Haridasan, Chi Ho, Ingrid Jansch-Porto, Tu-
dor Marian, Amar Phanishayee, Stefan Pleisch, Robbert van Renesse, Michael Siegen-
thaler, Yee Jiun Song, Ymir Vifgusson, Werner Vogels, Einar Vollset, and Hakim Weath-
erspoon, who currently are, or who have once been members of our group, collaborators,
and visitors, for listening to my practice talks, reading drafts of my papers, and for their
many helpful comments and suggestions. They have been the greatest colleagues I ever
worked with, and I am proud to have been a part of the team.
I would like to thank Lars Brenna, Dmitri Chmelev, Adam Davis, Richard Grandy,
Ankur Khetrapal, Daniel Margo, Chuck Sakoda, Weijia Song, Zhen Xiao, as well as all
v
the students in CS514 in Spring 2007 and 2008, for being the users of QSM and live
objects. I am especially indebted to Lars for pointing out the bugs in QSM’s support for
unmanaged applications.
I am grateful to Ernie Davis for believing in our technology and convincing Cornell
to file patent applications for QSM and live objects, and to Kathleen Chapman for the
hard work we went through together preparing these patent applications.
I would like to also thank Brian Bershad, Marin Bertier, Jason Bryantt, Antonio
Carzaniga, Michael Caspersen, Ranveer Chandra, Wei Chen, Gregory Chockler, Jon
Currey, Alan Demers, Kathleen Fisher, Davide Frey, Johannes Gehrke, Rachid Guer-
raoui, Chang Heng, Robert Hillman, Mingsheng Hong, Ray O. Johnson, Idith Kei-
dar, Anne-Marie Kermarrec, Annie Liu, John Manferdelli, Milan Milenkovic, Benjamin
Pierce, Josh Rattner, Fred Schneider, Gennady Staskevich, Daniel Sturman, Eric Suss,
Marvin Theimer, Toste Wallmark, Calvin Wong, Zheng Zhang, and all the anonymous
COMSWARE, DEPSA, ECOOP, HotOs, ICDCS, ICWS, Internet Computing, IPTPS,
JWSR, NCA, NSDI, OSDI, SOSP, and STC reviewers for comments and suggestions. I
would like to thank Ranveer Chandra for the opportunity to give a talk and to meet his
colleagues researchers in Microsoft Research labs at Redmond.
I would like to present my special gratitude to Bill Hogan for all his help and advice
in administrative matters. I would like to also very much thank Ben Atkin for being my
mentor at Cornell, and a great friend. Ben has been like an older brother, and made me
feel at home in Ithaca.
I am grateful to my parents Tadeusz and Maria for their love and sacrifice, and to
my brother Piotr for being a wise, supportive and loving brother, and for being there for
me whenever I needed him. Last, but not least, I would like to thank my wife Yejin for
her support, for the time she spent reading early drafts of my papers and listening to my
practice talks to help me improve my presentation and writing, for the many valuable
technical comments she made, and most importantly, for being so wise and charming at
the same time.
This work was supported by DARPA/IPTO under the SRS program, by the Rome Air
Force Research Laboratory, AFRL/IF, and by AFOSR (under AF TRUST). Additional
support was provided by Intel and NSF. Any opinions, findings, or recommendations
presented in the following pages, however, are my own, and do not necessarily reflect
the views of DARPA/IPTO, AFRL, AFOSR, Intel, or NSF.
vi
TABLE OF CONTENTS
Biographical Sketch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Dedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
List of Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
List of Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
1 Introduction 1
1.1 The Limited Adoption of Distributed Computing . . . . . . . . . . . . 4
1.2 The Active Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 The Emerging Paradigm . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Prior Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.4.1 Embedding . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.4.2 Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.4.3 Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2 Programming 32
2.1 Programming Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.1.1 Objects and their Interactions . . . . . . . . . . . . . . . . . . 32
2.1.2 Defining Distributed Types . . . . . . . . . . . . . . . . . . . 38
2.1.3 Constraint Formalisms . . . . . . . . . . . . . . . . . . . . . . 42
2.1.4 Language and Type System Embeddings . . . . . . . . . . . . 45
2.1.5 Construction and Composition . . . . . . . . . . . . . . . . . 49
2.1.6 Deployment Considerations . . . . . . . . . . . . . . . . . . . 55
2.2 Prototype Implementation . . . . . . . . . . . . . . . . . . . . . . . . 56
2.2.1 OS Embedding via Drag and Drop . . . . . . . . . . . . . . . 57
2.2.2 Language Embedding via Reflection . . . . . . . . . . . . . . 59
3 Extensibility 67
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.1.2 Technical Challenges . . . . . . . . . . . . . . . . . . . . . . 68
3.1.3 Design Principles . . . . . . . . . . . . . . . . . . . . . . . . 69
3.1.4 Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.2.1 The Hierarchy of Scopes . . . . . . . . . . . . . . . . . . . . . 73
3.2.2 The Anatomy of a Scope . . . . . . . . . . . . . . . . . . . . . 75
3.2.3 Hierarchical Composition of Policies . . . . . . . . . . . . . . 77
3.2.4 Communication Channels . . . . . . . . . . . . . . . . . . . . 79
3.2.5 Constructing the Dissemination Structure . . . . . . . . . . . . 82
vii
3.2.6 Local Architecture of a Dissemination Scope . . . . . . . . . . 85
3.2.7 Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.2.8 Incorporating Reliability and Other Strong Properties . . . . . 89
3.2.9 Hierarchical Approach to Reliability . . . . . . . . . . . . . . 92
3.2.10 Building the Hierarchy of Recovery Domains . . . . . . . . . . 95
3.2.11 Recovery Agents . . . . . . . . . . . . . . . . . . . . . . . . . 99
3.2.12 Modeling Recovery Protocols . . . . . . . . . . . . . . . . . . 100
3.2.13 Implementing Recovery Domains with Agents . . . . . . . . . 107
3.2.14 Reconfiguration . . . . . . . . . . . . . . . . . . . . . . . . . 109
3.2.15 Reliability and Consistency . . . . . . . . . . . . . . . . . . . 110
4 Performance 113
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.1.1 Application Scenario . . . . . . . . . . . . . . . . . . . . . . . 113
4.1.2 Exploiting Overlap . . . . . . . . . . . . . . . . . . . . . . . . 115
4.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.2.1 Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.2.2 Protocol Stack . . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.3.1 Memory Overheads on the Sender . . . . . . . . . . . . . . . . 132
4.3.2 Memory Overheads on the Receiver . . . . . . . . . . . . . . . 137
4.3.3 Overheads in a Perturbed System . . . . . . . . . . . . . . . . 139
4.3.4 Overheads in a Lightly-Loaded System . . . . . . . . . . . . . 142
4.3.5 Per-Group Memory Consumption . . . . . . . . . . . . . . . . 144
4.3.6 Extrapolating the Results . . . . . . . . . . . . . . . . . . . . 148
4.3.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
4.3.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
5 Future Work: Properties Framework 163
5.1 Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
5.2 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
5.3 Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
5.4 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
6 Conclusions 170
6.1 Component Integration . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.1.1 Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Formalisms to Express Reliability and Security Properties . . . 174
Ontology and Inheritance for Constraint Formalisms . . . . . . 175
Proof-Carrying Annotations for Provable Composition . . . . . 175
Incorporating Theorem Provers in a Scalable Manner . . . . . . 176
6.1.2 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Automatically Finding and Downloading Libraries . . . . . . . 177
Managing Multiple Versions of Objects and Types . . . . . . . 177
viii
Conversions Between Binary-Incompatible Values . . . . . . . 177
Synchronous vs. Asynchronous Object Interactions . . . . . . . 178
6.1.3 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Assigning Proxies to Isolated Application Domains . . . . . . . 179
Scalable Components for Implementing Security . . . . . . . . 180
Integration with Existing Security Infrastructure . . . . . . . . 180
Leveraging Dynamic Composition and Reflection . . . . . . . . 181
Managing Distributed Object-Object Connections . . . . . . . . 181
Just-in-Time Typing Support for Library Objects . . . . . . . . 182
6.1.4 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Intelligent Activation and Deactivation of Proxies . . . . . . . . 182
Proxies with Different “Degrees” of Connectivity . . . . . . . . 183
Managing Resources Required for Correct Operation . . . . . . 184
6.1.5 Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Object Specifications Embedded in Object References . . . . . 185
High-Level Constructs in our Composition Language . . . . . . 185
Controlling Inflation of the Size of Object References . . . . . . 186
6.2 Multicast Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . 187
6.2.1 State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Limiting Dependence on Costly Flavors of Multicast . . . . . . 192
Backup and State Persistence for Replicated Objects . . . . . . 193
Using Locally Cached Configuration State in Proxies . . . . . . 194
Synchronizing Replicated State Stored in Documents . . . . . . 194
Modeling Nested Transactions as Object Composition . . . . . 195
6.2.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Making Systems Regular through Clustering of Objects . . . . 196
Indexing Schemes for Amortizing Work Across Objects . . . . 196
6.2.3 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Connecting Clients Separated with NATs and Firewalls . . . . . 197
Incorporating Mechanisms for Privacy and Anonymity . . . . . 197
Embedding Content-Based Publish-Subscribe Objects . . . . . 198
Infrastructure Objects in Support of Ad-Hoc Networks . . . . . 198
6.2.4 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Application Model Based On Distributed Membership . . . . . 198
General-Purpose Scalable Role Delegation Framework . . . . . 199
Self-Discovery and Bootstrapping with Gossip Objects . . . . . 200
6.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Bibliography 202
Glossary 230
ix
