DISSERTATION APPROVAL The abstract and dissertation of Bill Howe for the Doctor of ...

292 Pages·2006·5.45 MB·English

DISSERTATION APPROVAL
The abstract and dissertation of Bill Howe for the Doctor of Philosophy in Com-
puterSciencewerepresentedonDecember8,2006,andacceptedbythedissertation
committee and the doctoral program.
COMMITTEE APPROVALS:
David Maier, Chair
Lois Delcambre
Leonard Shapiro
Timothy Sheard
Gerald Recktenwald
Representative of the Office of Graduate Studies
DOCTORAL PROGRAM
APPROVAL:
Cynthia Brown, Director
Computer Science Ph.D. Program
ABSTRACT
An abstract of the dissertation of Bill Howe for the Doctor of Philosophy in Com-
puter Science presented December 8, 2006.
Title: Gridfields: Model-Driven Data Transformation in the Physical Sciences
Scientists’ ability to generate and store simulation results is outpacing their
ability to analyze them via ad hoc programs. We observe that these programs
exhibit an algebraic structure that can be used to facilitate reasoning and improve
performance. Inthisdissertation, wepresentaformaldatamodelthatexposesthis
algebraic structure, then implement the model, evaluate it, and use it to express,
optimize, and reason about data transformations in a variety of scientific domains.
Simulation results are defined over a logical grid structure that allows a con-
tinuous domain to be represented discretely in the computer. Existing approaches
for manipulating these gridded datasets are incomplete. The performance of SQL
queries that manipulate large numeric datasets is not competitive with that of spe-
cializedtools,andtheup-fronteffortrequiredtodeployarelationaldatabasemakes
them unpopular for dynamic scientific applications. Tools for processing multidi-
mensional arrays can only capture regular, rectilinear grids. Visualization libraries
accommodate arbitrary grids, but no algebra has been developed to simplify their
2
use and afford optimization. Further, these libraries are data dependent—physical
changes to data characteristics break user programs.
We adopt the grid as a first-class citizen, separating topology from geometry
and separating structure from data. Our model is agnostic with respect to di-
mension, uniformly capturing, for example, particle trajectories (1-D), sea-surface
temperatures (2-D), and blood flow in the heart (3-D). Equipped with data, a
grid becomes a gridfield. We provide operators for constructing, transforming, and
aggregating gridfields that admit algebraic laws useful for optimization. We imple-
ment the model by analyzing several candidate data structures and incorporating
their best features. We then show how to deploy gridfields in practice by injecting
the model as middleware between heterogeneous, ad hoc file formats and a popular
visualization library.
Inthisdissertation, wedefine, develop, implement, evaluateanddeployamodel
of gridded datasets that accommodates a variety of complex grid structures and a
variety of complex data products. We evaluate the applicability and performance
of the model using datasets from oceanography, seismology, and medicine and
conclude that our model-driven approach offers significant advantages over the
status quo.
GRIDFIELDS:
MODEL-DRIVEN DATA TRANSFORMATION
IN THE PHYSICAL SCIENCES
by
BILL HOWE
A dissertation submitted in partial fulfillment of the
requirements for the degree of
DOCTOR OF PHILOSOPHY
in
COMPUTER SCIENCE
Portland State University
2007
i
DEDICATION
For my father, Dr. Gerry Howe, whose kindness, wisdom, honor, and patience are
the model for my conscious behavior, however well I implement it.
ii
ACKNOWLEDGEMENTS
Were it not for convention, the work represented by this dissertation would be
attributed to a whole host of indirect co-authors, each of whom has had an effect
deserving more than just “acknowledgement.” Formatting conventions being what
they are, however, and not wishing for these individuals to share any blame that
may accompany any credit, I have settled for using the plural first person in the
body of this dissertation as a little anonymous symbol of my gratitude.
IofferlovingthankstomywifeandbestfriendTammi. Herapparentlylimitless
patience and support have always intensified precisely when I least deserved it.
I am deeply grateful to David Maier, my thesis advisor, for continual support
and guidance as an advisor, a colleague, and a friend. His ability to maintain a
calm demeanor while slogging through early drafts of papers and this dissertation
amazesmeinhindsight. IalsomadefrequentuseofDave’sabilitytolistencarefully
to an hour of wild, tenuous ideas, yet identify and articulate the five-minute kernel
of significance when it (occasionally) existed. Finally, Dave deserves special thanks
for retaining me as a student after I ran off with his wife’s umbrella in a hail storm.
I thank the rest of my committee for not only their helpful comments and
questions, but for tolerating my unusual scheduling demands: Lois Delcambe, Len
Shapiro, Tim Sheard and Gerald Recktenwald.
I am grateful to Antonio Baptista and his staff for providing such fascinating
data and such significant challenges in the context of the CORIE project. Dr.
Baptista’s advice and support were crucial in the development of these ideas. I am
iii
especially grateful to Paul Turner for his feedback as my prototypes evolved.
I thank my parents, who managed to provide me an excellent education and
even instill in me a passable work ethic despite my best efforts to the contrary.
They have my love, respect, and heartfelt thanks.
I was extremely fortunate to have ready access to excellent personal and profes-
sional advice as a student. I appreciate all the faculty I have had the opportunity
to chat with. I am indebted to Lois Delcambre for several crucial and inspirational
conversations along with the way. I thank Wu-chi Feng for contagious energy and
plying me my only healthy hobby.
I owe my gratitude to the excellent database group at OGI and Portland State,
past and present: Vassilis Papadimos, Kristin Tufte, Sun Murthy, Shawn Bowers,
Pete Tucker, Mat Weaver, Laura Bright, Susan Price, James Terwilliger, Jin Li,
and Nick Rayner. All were consistently available to offer help, feedback, and
general conversation; their collective impact on the successful completion of this
work cannot be underestimated.
I am in debt to Laura Bright for her excellent work developing the curriculum
for our Summer course at a time when this dissertation kept me rather distracted.
I wish to also thank the students of the Systems Group: Francis Chang, who
is clearly the better pool player; and Ed Kaiser and Chris Chambers, whom had I
not known, I might have graduated three months earlier.
I thank the excellent staff at OGI and Portland State for cheerfully tolerating
my continual requests and occasional complaints: Lorie Gookin, Dana Director,
Jo Ann Binkerd, Cindy Pfaltzgraff, Renee Remillard, Kathi Lee, and Leai Rose.
I owe thanks to the the National Science Foundation, whose successful Infor-
mation Technology Research program funded this work.
Finally, Iextendmythankstomyfriends, currentlyscatteredaroundtheworld:
iv
John Bowen, Curtis Atkins, Andy Verras, Gary Moss, and Chuck Petrakopolous.
I look forward to returning your favors.
v
CONTENTS
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Computational Science . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Source Data Organization . . . . . . . . . . . . . . . . . . . . . . . 10
1.3 Data Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4 Example Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.5 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.6 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2 Data Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.1 Model Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.2 The Gridfield Model . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.2.1 Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.2.2 Grid Properties . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.2.3 Grid Relations and Operations . . . . . . . . . . . . . . . . 42
2.2.4 Gridfields . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.3 Grid Computation Requirements . . . . . . . . . . . . . . . . . . . 51
2.3.1 Grid Serialization . . . . . . . . . . . . . . . . . . . . . . . . 52
2.3.2 Direct Rendering . . . . . . . . . . . . . . . . . . . . . . . . 53
2.3.3 Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.3.4 Streamlines . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.3.5 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . 59
2.3.6 From Algorithms to Recipes . . . . . . . . . . . . . . . . . . 59
2.4 Gridfield Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
2.4.1 Bind . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
2.4.2 Restrict . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
2.4.3 Accrete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
vi
2.4.4 Merge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
2.4.5 Cross Product . . . . . . . . . . . . . . . . . . . . . . . . . . 70
2.4.6 Regrid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
2.4.7 Abstract Syntax and the Fixpoint Operator . . . . . . . . . 79
2.4.8 Feature Summary . . . . . . . . . . . . . . . . . . . . . . . . 81
2.5 Examples and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 82
2.5.1 Example: Wetgrid . . . . . . . . . . . . . . . . . . . . . . . 83
2.5.2 Example: Plume Volume . . . . . . . . . . . . . . . . . . . . 85
2.5.3 Example: Minimum Cell Diameter . . . . . . . . . . . . . . 87
2.5.4 Example: Feature Extraction . . . . . . . . . . . . . . . . . 88
2.5.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3 Data Structures for Gridfields . . . . . . . . . . . . . . . . . . . . . 91
3.1 Characterizing Grid Representations . . . . . . . . . . . . . . . . . 95
3.2 Review of Grid Data Structures . . . . . . . . . . . . . . . . . . . . 99
3.2.1 Incidence Graph . . . . . . . . . . . . . . . . . . . . . . . . 99
3.2.2 Indexed Cell Set . . . . . . . . . . . . . . . . . . . . . . . . 101
3.2.3 Edge-based Structures . . . . . . . . . . . . . . . . . . . . . 106
3.2.4 Cell Tuples . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
3.2.5 Generalized Combinatorial Maps . . . . . . . . . . . . . . . 110
3.2.6 A Relational Implementation . . . . . . . . . . . . . . . . . 113
3.2.7 A Data Structure for Visualization . . . . . . . . . . . . . . 117
3.3 Implementing and Testing Gridfield Structures . . . . . . . . . . . . 119
3.3.1 Bound Attributes . . . . . . . . . . . . . . . . . . . . . . . . 119
3.3.2 Cells and Grids . . . . . . . . . . . . . . . . . . . . . . . . . 122
3.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
3.4.1 Restrict Operator over ICS, NGMap, and TIGraph . . . . . 131
3.4.2 Neighborhood Operator over NGMap and TIGraph . . . . . 134
3.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4 Retrofitting Gridfields onto Existing Data . . . . . . . . . . . . . . 139
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
4.3 Running Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
4.4 Native Data Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
4.4.1 Data From Directory Structures . . . . . . . . . . . . . . . . 152