Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

13 Pages·2017·1.24 MB·English

Omnisc’IO: A Grammar-Based Approach to Spatial and
Temporal I/O Patterns Prediction
Matthieu Dorier, Shadi Ibrahim, Gabriel Antoniu, Robert Ross
To cite this version:
Matthieu Dorier, Shadi Ibrahim, Gabriel Antoniu, Robert Ross. Omnisc’IO: A Grammar-Based Ap-
proach to Spatial and Temporal I/O Patterns Prediction. SC14 - International Conference for High
Performance Computing, Networking, Storage and Analysis, IEEE, ACM, Nov 2014, New Orleans,
United States. ￿hal-01025670￿
HAL Id: hal-01025670
https://hal.inria.fr/hal-01025670
Submitted on 24 Jul 2014
HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est
archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non,
lished or not. The documents may come from émanant des établissements d’enseignement et de
teaching and research institutions in France or recherche français ou étrangers, des laboratoires
abroad, or from public or private research centers. publics ou privés.
Omnisc’IO: A Grammar-Based Approach to
Spatial and Temporal I/O Patterns Prediction
Matthieu Dorier,∗ Shadi Ibrahim,† Gabriel Antoniu,† Rob Ross‡
∗ENS Rennes, IRISA, Rennes, France, [email protected]
†Inria, Centre de Rennes Bretagne-Atlanique, France, {shadi.ibrahim,gabriel.antoniu}@inria.fr
‡Argonne National Laboratory, Argonne, IL 60439, USA, [email protected]
Abstract—The increasing gap between the computation per- long been an important goal in large-scale supercomputers,
formanceofpost-petascalemachinesandtheperformanceoftheir researchers have focused mainly on statistical methods (e.g.,
I/O subsystem has motivated many I/O optimizations including hidden Markov models (HMM) [8] and ARIMA models [6]).
prefetching, caching, and scheduling techniques. In order to These approaches often focus exclusively on either spatial
furtherimprovethesetechniques,modelingandpredictingspatial
or temporal I/O behaviors. Furthermore, they require a large
and temporal I/O patterns of HPC applications as they run has
numberof observationsto accomplishgoodprediction;hence,
became crucial.
they either need long execution time (several runs in some
InthispaperwepresentOmnisc’IO,anapproachthatbuilds
cases) [9] or are doomed to offline trace-based training [10]
agrammar-basedmodeloftheI/ObehaviorofHPCapplications
in order to converge.
and uses it to predict when future I/O operations will occur,
and where and how much data will be accessed. Omnisc’IO is Our work addresses the limitations of current prediction
transparently integrated into the POSIX and MPI I/O stacks systemsandtakesasteptowardintelligentI/Omanagementof
and does not requireany modification in applicationsor higher- HPC applicationsin next-generationpost-petascale supercom-
level I/O libraries. It works without any prior knowledge of puters [11] that is capable of run-timeanalysis and adaptation
the application and converges to accurate predictions within a totheI/Obehaviorofapplications.Tothisend,wepresentthe
couple of iterations only. Its implementation is efficient in both
design and implementation of Omnisc’IO, a grammar-based
computation time and memory footprint.
approachfor modelingand predicting the I/O behavior of any
Keywords—Exascale,HPC,Storage,I/O,Prediction,Grammar, HPC application.
Omnisc’IO
The intuition behind Omnisc’IO is that, on the one hand,
while statistical modelsare appropriatemostlyforphenomena
I. INTRODUCTION
that exhibit a random behavior, the (mostly) deterministic be-
Existing petascale computing platforms often fail to meet
haviorofHPCapplications,inherentfromtheircodestructure,
the I/O performance requirements of applications running at
makesotherrepresentationsof theirI/O behaviorpossible.On
scale. This weakness of the I/O system relative to computing
the other hand, formalgrammars, as naturalmodels to form a
capabilities is part of a trend that is worsening as we develop
sequenceofsymbols,havebeenwidelyappliedtoareasoftext
ever more capable computing platforms, and we expect the
compression, natural language processing, music processing,
next generation of systems to have an even larger gap [1].
and macromolecular sequence modeling [12] [13]. Therefore,
Moreover,most HPC applicationsexhibita periodic behavior,
an approach based on formal grammars is suitable for I/O
alternating between computation, communication, and I/O
behavior modeling, since it detects the hierarchical nature of
phases. The I/O phases are commonly used for coordinated
the code that produced the I/O patterns, with its nested loops
checkpointing and/or analysis or visualization output. They
and stacks of function calls. To the best of our knowledge,
produceburstsofactivityintheunderlyingstoragesystem[2],
grammar-based models have never been used in the context
[3] that further limit the overall scalability of the HPC appli-
of HPC applications. Omnisc’IO is the first prediction system
cationandpotentiallyinterferewithotherapplicationsrunning
that adopts this appealing approach.
concurrently on the platform [4].
Contributions: Omnisc’IO builds a grammar-based model
To alleviate the performance penalty of the I/O bottleneck
of the I/O behavior of any HPC application and uses it to
in petascale systems, researchers have explored techniques
predict when future I/O operations will occur (i.e., predict
such as prefetching, caching, and scheduling [4], [5], [6] to
the interarrival time between I/O requests), as well as where
improveI/Operformance.Theeffectivenessofsuchtechniques
and how much data will be accessed (i.e., predict the file
strongly depends on a certain level of knowledge of the I/O
being accessed as well as the location –offset and size– of
access patterns: prefetching and caching indeed require the
the data within this file). It learns its model at run time using
location of future accesses (i.e., spatial behavior), while I/O
an algorithm inspired by Sequitur [14]. Sequitur is designed
scheduling leverages estimations of I/O requests interarrival
to build a grammar from a sequence of symbols and has
time (i.e., temporal behavior). The key challenges inherent in
been used mainly in the area of text compression; one of the
these techniques include the proper comprehension and ex-
main challenges in our work was to turn it into an algorithm
ploitationoftheapplication’sI/ObehaviorwithintheI/Ostack
capableof makingpredictions.Inthis respect,ouralgorithmic
itself[7],[5].Hence,modelingandpredictingtheapplications
contribution is much more general than its application to I/O
I/O behavior are of utmost importance.
pattern prediction.
While predicting the I/O patterns of HPC applications has
Omnisc’IO is transparentlyintegrated into the POSIX and
MPI I/O stacks and does not require any modification in
applications or higher-level I/O libraries. It works without
any prior knowledge of the application, and it converges to
accurate predictions within a couple of iterations. Its im-
plementation is efficient both in computation time (less than
a few microseconds to update the model on a recent x86
hardware) and in memory footprint. Omnisc’IO is evaluated
with four real HPC applications –CM1, Nek5000, GTC, and
LAMMPS– using a variety of I/O backends ranging from
simple POSIX to Parallel HDF5 on top of MPI-I/O. Our
experiments show that Omnisc’IO achieves from 79.5% to
100% accuracy in spatial prediction (percentage of matching
between the predicted segment and the effectively accessed
(a) Architecture ofOmnisc’IO
segment) and an average precision of temporal predictions
(absolute differencebetween the predicted and the actual date
of the next accesses) ranging from 0.2 seconds to less than a
millisecond.
Goals:The primarygoalforOmnisc’IO,andthe focusof this
paper, is modeling the I/O behavior of any HPC application
and using this model to accurately predict the spatial and
temporal characteristics of future I/O operations. Omnisc’IO (b) Integration ofOmnisc’IO intheI/Ostack
canthereforebeappliedatthecoreofmanyI/Ooptimizations,
including prefetching, caching, or scheduling. Fig. 1: Overview of the Omnisc’IO approach.
In order not to undermine the generality of our approach,
thispaperdoesnotpresenttheuseofOmnisc’IOinaparticular context symbols. The application of Sequitur to the field of
context(i.e.,prefetching,caching,or scheduling).Othershave application behavior modeling is noveland constitutes part of
alreadydemonstratedthebenefitsofapplyingI/Opredications our contribution.
to enhance the performance in each of these techniques [8], 3) Spatial (size, offset, file) and temporal (interarrival time)
[6],[9],[10].Wefocusourstudyonthepredictioncapabilities accesspatternsarerecordedintablesassociatingcontextsym-
of Omnisc’IO. bolsortransitionsamongsymbolswithaccesspatterns(lower-
The paper is organized as follows. Section II presents left part of Figure 1(a)). The intuition is (for the example
our contribution, followed by an extensive evaluation of its of the access size) that a given context symbol will often be
predictioncapabilitiesin SectionIII.Relatedworkthathasat- associatedwiththesameaccesssizeorwithareducednumber
temptedspatialand/ortemporalI/OaccesspredictionsinHPC of sizes whose sequence can also be learned.
applicationsispresentedinSectionIV.SectionVpresentsour 4) WeimprovedontheSequituralgorithmtomakepredictions
conclusions and suggestions for future work. of future context symbols. It then becomes easy to predict
the characteristics of future accesses by looking up the access
II. OMNISC’IO APPROACH patterns associated to the predicted context symbols in the
aforementionedtables(lower-rightpartofFigure1(a))Turning
This section first gives an overview of Omnisc’IO, then
Sequitur into a prediction system is a challenge that, to the
dives into its technical and algorithmic details.
best of our knowledge, has never been addressed before.
The algorithmic details of our prediction model therefore
A. Overview of Omnisc’IO
constitutes another important part of our contribution.
Figure 1 presents an overview of Omnisc’IO. Omnisc’IO
captures each atomic request to the file system (open, close, B. Algorithmic and technical description
read, write) in a transparent manner, without requiring any As shown in Figure 1(b), Omnisc’IO is integrated within
change in the application or I/O libraries. At each operation, the POSIX I/O layer and the ADIO layer in MPI-I/O. In the
Omnisc’IO operates as follows. following,weprovidemoredetailsonthefourstepsdescribed
1) Thecontext inwhichthe operationis executedis extracted above.
by recording the call stack of the program (upper-left part of 1)Tracking applications behavior: To give a context to
Figure1(a)).Thisisaknowntechnique[15]thathelpscapture each atomic I/O operation, we use the libc backtrace
the structure of the code that issues the I/O operations. The functionto retrievethe list ofstacked programcounters(array
context is abstracted as a context symbol (a in the figure). of void* pointers). When called within a function f, this
2) A grammar-based model of the stream of context symbols list of addresses characterizes the series of return addresses
(upper-right part of Figure 1(a)) is updated by using the that leads from f back to main. Different calls to f in
Sequitur algorithm. Sequitur has been applied to text com- distinctplacesintheprogram(orlibraries)leadtodifferentcall
pressioninthepast[12]becauseofitsabilitytodetectseveral stack traces. Omnisc’IO calls backtrace within wrappers
occurrences of substrings in a text and to store them into of I/O functions, stores the returned array in a dictionary,
grammar rules. We have adapted it to model the repetitive and associates it with a unique integer. In the following, such
behavior of an HPC application, represented as a stream of integers are called context symbols and represent the context
in which an I/O operation occurs.
TABLE I: Examples of context-free grammars. Lowercase
Omnisc’IOisbasedontheobservationthat(1)aparticular letters represent terminal symbols, while uppercase letters
context is likely to be associated with fixed parameters (e.g., represent rules and their instances. Example 1 is correct from
two calls to write within the same context are likely to a Sequitur perspective. Example 2 violates the rule utility
involvethe same amountof data);(2) transitionsbetween two (rule A is used only once; it should be deleted an its only
contextscanalsobeassociatedwithfixedparameters(tracking instanceshouldbereplacedbyitscontent).Example3violates
the evolution of the offset can be done by tracking transitions the digram uniqueness (digram ab appears twice; a new rule
betweencontexts)andwithlittle-varyingtransitiontimes;and, B →ab can be created to replace it).
mostimportant,(3)thestreamofcontextsymbolsiseventually
predictable, and a model of it can be built at run time.
Example 1 Example 2 Example 3
In our prototype, we overloaded the POSIX I/O functions S →abAAe S →abAe S →ababAAe
(write,read...)andthelibcfunctions(fwrite,fread...) A→cd A→cd A→cd
using a preloaded shared library. In MPI-I/O we added an
intermediatelayerwithintheADIOlayerinROMIO,apopular
complexity both in space and in time.
implementation of MPI-I/O [16], to track the lowest-level I/O
functionsthataccessfilesmetadata(open,close...)andatomic b)Predictions using the grammar model: Sequitur
functions that access contiguous blocks of data and are used builds a grammar from a stream of symbols, but it does not
by more elaborate I/O algorithms. predict the next incoming symbols from past observations.
Therefore,we enriched the algorithm to be able to make such
While working at the lowest level of the I/O stack is
a prediction.
necessary to capture the I/O behavior at a fine grain (i.e.,
a series of atomic requests to the file system), the use of This improvementworksby markingsome of the terminal
backtraceslets Omnisc’IOhave informationalso on the upper symbols in the grammar as predictors. This predictor char-
layers thatissued the I/O, includingI/O middleware,libraries, acteristic is extended to nonterminal symbols by using the
and the application itself. following constraints:
a)Learning the grammar of the application: While Predictor nesting: A nonterminalcan be a predictoronly if
capturing a stream of symbols representing the behavior of at least one of its nested symbols is a predictor.
the application,we aim to predictthe nextsymbolsgivenpast
observations. Omnisc’IO models the stream of symbols using
a context-free grammar. This grammar is learned at run time Predictorutility:Ifsymbolx(terminalornot)isapredictor
using an algorithm inspired by Nevill-Manning and Witten’s in rule Y, then there exists at least one rule Z such that an
Sequitur algorithm [14]. instance of Y is a predictor in Z.
As background, a context-free grammar G is a quadruple
(Σ,V,R,S), where Σ is a finite set of terminal symbols (in These constraints enforce that (1) if the grammar contains
our case the symbols defined by the call stack traces); V is a at least one predictor, then rule S contains at least one
finitesetofnonterminalsymbolsdisjointfromΣ;Risafinite predictor, and (2) all the terminal predictors of the grammar
relation from V to (V∪Σ)∗, usually written as a set of rules canbereachedfromapredictorinS (proofsoftheseproperties
in the form A → x1...xk, where xi ∈ (V∪Σ); and S ∈ V are trivial). The relations linking predictors together form a
is a starting symbol. In the following, we call x the nested direct acyclic graph within the tree structure of the grammar.
i
symbols of A. These two structural properties have to be carefully preserved
Sequitur builds a context-free grammar from a stream of when updating the grammar.
symbols by updating the grammar at each input. It starts with In order to make predictions from the set of predictors,
a single rule S. At each new input x, it appends x to the end two operations are defined, respectively, to update the set of
of rule S and recursively enforces two constraints: predictors and to find new ones.
Updating predictors: We call incrementing a predictor the
Digram uniqueness: Any sequence of two symbols ab ∈
operation that consists of unmarking a symbol previously
(V∪Σ)2 (digram)cannotappearmorethanonceinallrules.
marked as a predictor and marking as a predictor the
Ifonedoes,anewruleR→abiscreated,andtheconstraints
symbol that immediately follows it in the rule where it
are enforced recursively.
appears. Updating predictors consists of first unmarking
allterminalpredictorsthatdidnotcorrectlypredictthelast
input,enforcingthe predictor’sconstraints,andthen incre-
Rule utility: All rules should be instantiated at least twice. menting all remainingterminalpredictors.If a predictoris
If a rule appears only once, its instance is replaced by the the last symbol of a rule, then nonterminal predictors that
content of the rule, the rule is deleted, and the constraints reference it are incremented recursively. Examples of this
are enforced recursively. operation are shown in Table II, where predictor symbols
are marked in red and underlined.
Examples of context-free grammars are given in Table I, Discovering predictors: If all predictors have been removed
some of which violate the Sequitur constraints. In the follow- because none of them correctly predicted the last input, a
ing, the grammar built from the context symbols is called the new step is necessary to rebuild a set of predictors. This
maingrammarofOmnisc’IO.Sequiturhasalinearworst-case step is completed by navigating through the grammar and
execution, making it easy to predict the exact size of the next
TABLE II: Predictorsincrementationmatchinga giveninput.
accessesgivenacorrectpredictionofthenextcontextsymbols.
The predictors are marked in red and underlined. In the
first input, a does not match and disappears from the set of Forthe minorityofsymbolsassociated withseveralaccess
predictors, c matches and is incremented to d, and A stays sizes, Omnisc’IO keeps track of all access sizes encountered
a predictor. In the second example, d matches but has no and builds a grammar from this sequence of sizes. The sizes
successor in rule A; thus A is incremented to e in rule S. constitute the terminal symbols of this grammar, which we
The resulting models correspond to the grammars before the call a local size grammar. The local size grammar associated
input is appended and Sequitur’s constraints are applied. withacontextsymbolisupdatedwheneverthecontextsymbol
is encountered, and it evolves independently of the main
grammar and independently of local size grammars attached
Before Update Input After Update
to other symbols. It can then be used to make predictions of
S →abAAe c S →abAAe
the size.
A→cd A→cd
S →abAAe d S →abAAe If the number of different access sizes observed for a
A→cd A→cd given symbolis too large (typically larger than a configurable
constant N), the local size grammar is replaced by a simple
average,minimumvalueandmaximumvaluethatareupdated
TABLE III: Discovery of new predictors matching the last whenever the context symbol is encountered. For our experi-
input (b, appended at the end of rule S). The predictors ments,afteranalyzingthedistributionsofdifferentaccesssizes
are marked in red and underlined. The symbol b becomes a per symbol, we choose N =24.
predictor wherever it appears, and recursively any rule that
More elaborate methods could be implemented to predict
leadstoanoccurrenceofbbecomesapredictor.Thepredictors
the access sizes for context symbols that exhibit apparently
are then updated to predict the next expected input (here c
random sizes. We show in Section III, however, that the
or e).
three cases presented above have been sufficient to cover the
behavior of all our applications.
Before Discovery After Discovery After Update
b)Tracking offsets: Many prefetchingsystems, includ-
S →abAAeb S →abAAeb S →abAAeb
ing those implemented in the Linux kernel [17], are based
A→cdb A→cdb A→cdb
on the assumptions of consecutive accesses; that is, the next
operation is likely to start from the offset where the previous
setting as predictors all symbols matching with the last
one ended. As we will show in our experiments,this assump-
symbol of rule S (after insertion of the last input). Parent
tion is held for the POSIX-based applications that we tested,
rules are also set as predictors recursively wherever they
but it fails for applicationsthat use a higher-levellibrary such
appear. Note that the last symbol of rule S may be a
as HDF5. Indeed such libraries often move the offset pointer
nonterminal, which forces new predictors to be searched
backward or forward to write headers, footers, and metadata.
only within the context of its corresponding rule and thus
To predict the offset of the next operation, we define the
reduces the number of predictors and narrows down the
notion of offset transformation. An offset transformation can
prediction. An example of this operation is shown in
(1) leave the offset as it was at the end of the previous
Table III. After the discovery of these new predictors, an
operation (consecutive access transformation), (2) set it to a
update is necessary.
specific absolute value (absolute transformation), or (3) set it
The prediction of the model corresponds to the set of to a value relative to the offset after the previous operation
terminal symbols marked as predictors after inserting a new (relative transformation). Since it is not possible at low level
input, updating the predictors, and enforcing the constraints. to distinguish between absolute and relative transformations,
Although statistical methods could be used to weight each Omnisc’IO uses absolute transformations only for operations
predictedsymbolwithaprobabilityofappearance,considering that reset the offset to 0 (open and close). All other
equal probability for all predicted symbols appeared to be nonconsecutive offset transformations are considered relative
sufficient to achieve good results in our experiments. to the previous offset.
To implement our algorithm, we reused the simple C++ Omnisc’IO associates transitionsbetween contextsymbols
codeprovidedbytheauthorsofSequitur.1Forcomparison,the with offsettransformationsthe same way it associates context
originalcodehas358lines,whereasourimprovedversionhas symbols with access sizes. For instance, if symbol B follows
982 (without counting the I/O wrappers and the code related A in the execution and A left the offset at a value from
to access tables, which is explained later). which B starts, then the transition A → B is associated
2)Context-aware access behavior: The final step in Om- withasequentialtransformation.Whenatransitionencounters
nisc’IOistheactualbookkeepingofper-contextaccessbehav- different types of offset transformations, Omnisc’IO builds a
ior. This is done differently for each type of tracked metrics. local offset grammar for the transition. Local offset gram-
mars are the counterpart of local size grammars for offset
a)Tracking access sizes: Access sizes are tracked on
transformations. If the grammar associated with a transition
a per-contextsymbolbasis, so that predictingthe nextcontext
grows too large (more than 24 symbols in our experiments),
symbol helps in predicting the next access size. As shown in
Omnisc’IO switches back to always predicting a sequential
Section III, most context symbols are always associated with
offset transformation for this transition of context symbols.
the same access size each time they are encountered in an
c) Tracking files pointers: For the prediction of off-
1http://www.sequitur.info/sequitur simple.cc sets to work properly, Omnisc’IO needs to know that two
consecutive operations work on the same file or that an
TABLE IV: List of applicationsused and their I/O backends.
operation works on a new file or a file that has already been
accessed earlier. This is particularly important when accesses
Application Field I/O Method(s)
to multiple files interleave. The prediction of files accessed
HDF5+POSIX
is done by recording opened file pointers and associating
CM1 Climate HDF5+MPI-I/O
transitions between symbols to changes of file pointers. Since
HDF5+Gzip
inourexperimentsthecaseofinterleavedaccessestodifferent
GTC Fusion POSIX
files did not appear, we will not study this particular aspect
further. Nek5000 Fluid Dynamics POSIX
LAMMPS Molecular Dynamics POSIX
d)Tracking interarrival times: To keep track of the
time between the end of an operation and the beginning of
the next one, Omnisc’IO uses a table associating transitions with a set of predicted request structures (including the size,
between context symbols to statistics on the measured time. offset, and date) representing the expected next I/O accesses.
Thesestatisticsincludetheminimumandmaximumobserved,
the average, and the variance. We prefer these statistics rather III. EVALUATION
than keeping only the average because they represent the Inthissection,weevaluateOmnisc’IOonrealapplications.
minimum required to answer (1) whether an operation will We first assess its capability to predict the next context
immediately follow (maximum, minimum, average, and vari- symbols, and we show how the grammar grows in size as the
ance close to 0); (2) whether the next operation will follow applicationcontinuestorun.We thenevaluateitsperformance
in a predictable amount of time (maximum, minimum, and in predicting the spatial and temporal characteristics of the
average close to each other, small variance); or (3) whether next operations.
the time before the next operation is more unpredictable or
depends on parameters that are not captured by our system
A. Platform and applications
(large minimum-maximum interval, large variance). Thus,
All our experiments are carried out at the Nancy site of
these statistics, while minimal, give us confidence in the
the French Grid’5000 testbed [18]. The applications run on
predicted interarrival time, which may be important in the
a Linux cluster consisting of 92 Intel Xeon L5420 nodes
context of scheduling, for example.
(8 cores per node, 736 cores in total), using MPICH 3.0.4.
To quickly react to changes in interarrival times, Om-
The OrangeFS 2.8.7 parallel file system [19] is deployed on
nisc’IO also keeps a weighed interarrival average time, up-
a separate set of 12 Intel Xeon X3440 nodes. All nodes,
datedeverytimethetransitionbetweensymbolsisencountered
including the file system’s, are interconnected through a 20G
by using the following formula,
InfiniBand network.
ThelistofapplicationsusedispresentedinTableIV.These
Tweighed ← Txw→eiyghed+T, (1) applicationsarereal-worldcodesrepresentativeofapplications
x→y b 2 running on current supercomputers. They have been used on
b NCSI’s Kraken2 and NCSA’s Blue Waters3 for CM1, ORNL’s
where x → y is the observed transition between context Titan4 forGTCandLAMMPS,andANL’s IntrepidandMira5
symbolsx andy andT is themeasuredinterarrivaltime.This for Nek5000.
weighed average is more efficient at making predictions of
For the experiments in this paper, the applications are
interarrivaltime, especiallyin a contextwhere thisinterarrival
run on 512 cores of Grid5000, except for Nek5000, which
time varies a lot between different observations of the same
runs on 32 cores. These applications are written in Fortran
transition.
except for LAMMPS (C++). Most of them use a POSIX
3)Overall prediction process and API: At each operation, I/O interface. To show the generality of our approach with
Omnisc’IO updates its models (the main grammar and the respect to higher-level I/O libraries, CM1 [20], [21] uses the
tables of access sizes, offset transformations and interarrival HDF5 [22] I/O library over the default (POSIX) I/O driver,
times). It then updates its predictors and builds the set of as well as the MPI-I/O driver offered by pHDF5, and Gzip
possible next context symbols (this set often consists of a compression over the default POSIX driver. CM1 writes one
single prediction). From these possible next symbols, a set file per process per I/O phase. The domain decomposition
of triplets (size, offset, date) is formed that can then be used in CM1 is such that the amount of data remains the same
by scheduling, prefetching,or caching systems. Although this over time and across processes. The use of compression
kind of prediction can easily be extended to the series of N exemplifies the case of varying data size in a nonvarying
next I/O operations, our experiments will focus only on the domain decomposition. GTC [23] writes one file per node
capability of Omnisc’IO to predict the next one. per iteration, but the amount of data varies between files as
To use Omnisc’IO, any software aiming at optimizing I/O particles move from one process to another. Like CM1, the
simply needs to be linked against the Omnisc’IO library and domain decomposition in Nek5000 [24] does not vary over
to call the following function: time, but the I/O phase is executed only by the rank 0 after a
reducephase.LAMMPS[25], [26] also sendsdata tothe rank
int omniscio_next( 2www.nics.tennessee.edu
omniscio_req** prediction, int* n) 3www.ncsa.illinois.edu
4www.olcf.ornl.gov
This function allocates the prediction array and fills it 5www.alcf.anl.gov
Omnisc’IO seems to learn GTC’s behavior (Figure 2(e))
100 100
% 80 % 80 fast: the reason is that GTC’s I/O phase consists of a loop
Prediction 4600 Prediction 4600 othveerfiarlsltptawrtoicpleasr,tiwclheischariesewarsiitltyenmooduet.leTdhienpthreedgircatimonmaqruaafltietyr
20 20
0 0 drops at the end of the first iteration when Omnisc’IO does
0 200 400 600 800 1000 1200 1400 1600 0 200 400 600 800 1000 1200 1400 not predictthe end of this loop and the file being closed. This
Operations Operations
mistake is never repeated in later iterations. The same pattern
(a) CM1+POSIX (b) CM1+Gzip
appears in LAMMPS.
100 100 ThecaseofNek5000ismoreinteresting:althoughitwrites
Prediction % 24680000 Prediction % 24680000 pmoetohrideoredliacpdapollleyiscatnthiooetnsec.xoaIncnvtveerssgatemigeaatsianmgfaosttuhneatncdoofdaedsaotpafe,rNftehecektl5yg0r0aa0ms,mtwhaeer
0 0 1000 2000 3000 4000 5000 6000 7000 8000 0 0 2000 4000 6000 8000 10000 12000 14000 16000 finouandditfhfeartenthtiswaiys dduepeentodincgodoenbirtasnccohnestenthtaatnpdrothceensswdraittae
Operations Operations
(c) CM1+MPI-I/O (d) Nek5000 it in an identical manner, leading to the creation of several
symbols that are actually interchangeable in the grammar.
100 100 Moving the write call outside the branches would help
Prediction % 24680000 Prediction % 24680000 rwtheomerikonvsveowltivhtehismnieonndpterotieforrtmhkeinnaoipswmplle.idcBageteicooanfusdteheevwealepoppcleliarc,iamwtieoedndoiadunrdnoswotilatuhptopioulynt
0 0 10000 20000 30000 40000 50000 60000 70000 80000 9000 0100000 0 0 5000 10000 15000 20000 25000 this code modification.
Operations Operations We also observe a drop in prediction quality at the end of
(e) GTC (f) LAMMPS theLAMMPSandNek5000runs.Thisdropisduetothe final
resultsbeingoutputin a sectionof thecodedifferentfromthe
Fig. 2: Context prediction capability of Omnisc’IO over the
one used for the periodic checkpoints. Thus these symbols,
runofeachapplication.Configurations(a),(b),(c),(e),and(f)
which appear the first time at the end of the execution could
exhibit a clear learning phase after which Omnisc’IO makes
not have been predicted by any model.
perfect predictions ((e) and (f) exhibit a drop of prediction at
Cost of a failed prediction: A failed prediction leads to
the end of the first iteration).
searchingnewpredictorswithinthegrammar,insteadofsimply
updating existing ones. This operation is linear in the size of
0 process only. This process then writes each set of particles the grammar (number of symbols). A failed prediction also
(of potentially different sizes) contiguously in a single file. has an effect on the system that leverages the prediction. For
instance,a prefetchingalgorithmwouldreadunnecessarydata
Although all processes write data in CM1 and GTC, we
and/or fail to read the data that is actually needed by the
consider the results of Omnisc’IO only on processrank 0 (for
program. The real cost would therefore depend on how much
applicationsthatissue I/O fromallprocesses,these resultsare
the incorrectoperationconsumesresourcesthat couldbe used
in fact identical in all processes since they exhibit the same
more productively.
behavior). We first evaluate how well our algorithm predicts
future context symbols based on past observations. We then
C. Grammar size
evaluatetheabilityofOmnisc’IOtopredictthelocation(offset
Figure 3 shows the evolution of the size of the main
andsizeinthefile)ofthenextI/Ooperations.Wealsoevaluate
grammar as a function of the number of operations. One
its ability to predict when future accesses will happen.
can clearly distinguish a first learning phase during which
Omnisc’IO discovers the behavior of the application. This
B. Context prediction
phasecorrespondstothefirstiteration(potentiallyprecededby
To measure context prediction capabilities, we run each an input phase). It is followed by a stationary regime during
application, and at each I/O operation we use Omnisc’IO to which the model is updated in a mostly logarithmic manner.
predict the context symbol associated with the next one. We All the applications considered here exhibit this logarithmic
use a sliding windowoften operationsandreportthe percent- growth of the grammar size after the learning phase. GTC’s
age of correct predictions. When Omnisc’IO predicts several grammar growth is logarithmic as well, but it exhibits a
possiblenextsymbols,theyareweighedas 1 .For staircase pattern. This is due to a variable number of particles
#predictedsymbols
instance, if Omnisc’IO predicts that the next symbol will be written at each checkpoint, which leads to a variable number
eitheraorbandtherealnextsymbolisb,thenthisprediction ofwritesandthuspreventsOmnisc’IOfromgroupingthese
is weighed 1. For CM1 and GTC, which run for long periods writes into large rules. That said, after 100,000 accesses the
2
of time, we show only several iterations starting from the grammar has only 450 symbols.
beginning of the run. The memory footprint is directly linked to the size of the
Results are shown in Figure 2. For all three configurations main grammar (a symbol in our implementationis a 100-byte
of CM1 as well as for LAMMPS and GTC, Omnisc’IO C++ object, making the grammar consume 26 KB in the case
converges to a perfect (steady 100%) prediction of symbols ofCM1+POSIX,forexample),andthenumberofentriesinthe
after the first iteration. The variation observed during the tables (oneentryper symbolor pertransition,accountingalso
first iteration corresponds to the moment the grammar starts forafewbytes.CM1+POSIXuses198symbols,forexample).
detecting the innermost loops. This part of the memoryfootprintdoes not increases after the
300 250 450 5 5
Grammar Size 112205050000 Grammar Size 112050000 Grammar Size 122334505050000000 Relative Size Error 01234 Relative Size Error 01234
5 00 0 200 400 600 800 1000 1200 1400 1600 5 00 0 200 400 600 800 1000 1200 1400 1 05 000 0 1000 2000 3000 4000 5000 6000 7000 8000 -1 0 200 400 600Op e80r0atio 1n0s00 1200 1400 1600 -1 0 200 400 6O00pera t8io00ns 1000 1200 1400
Operations Operations Operations (a) CM1+POSIX (b) CM1+Gzip
(a) CM1+POSIX (b) CM1+Gzip (c) CM1+MPI-I/O
5 5
Grammar Size 112233440505050500000000 Grammar Size 112233445050505050000000000 Grammar Size 11111 0246846800000000 Relative Size Error - 101234 0 1000 2000 3000 4000 5000 6000 7000 8000 Relative Size Error - 101234 0 2000 4000 6000 8000 10000 12000 14000 16000
50 50 20 Operations Operations
0 0 2000 4000 6000 800 01000 01200 01400 016000 0 1000 02000 03000 04000 05000 06000 07000 08000 0900 010000000 0 0 5000 10000 15000 20000 25000 (c) CM1+MPI-I/O (d) Nek5000
Operations Operations Operations
5 5
Feaicgh. 3r(:udl)EeN,voeinklu50nti0uo0mnboefrmoafinsygm(reab)moGlmsT)Ca.rsize (sum(fo)fLtAhMeMlePnSgth of Relative Size Error 01234 Relative Size Error 01234
-1 0 10000 20000 30000 40000 50000 60000 70000 80000 9000 0100000 -1 0 5000 10000 15000 20000 25000
learning phase. The memory footprint of Omnisc’IO is thus Operations Operations
correlated mainly with the grammar size and does not exceed (e) GTC (f) LAMMPS
a few hundreds of kilobytes.
Fig. 4: Relative error in the prediction of access sizes in all
D. Spatial prediction simulations.
In this section, we first analyze how well our solution
predicts the size and the offset of the next operation. We then
TABLE V: Statistics on access sizes by each application.
combine these two predictions to compute a hit ratio.
1)Prediction of sizes: We analyzed how many different
Application Min Max Average Std. dev.
access sizes were associated with each context symbol. We
CM1 (POSIX) 4 B 562.5 KB 212.9 KB 268.2 KB
found that the vast majority of symbols were associated
CM1 (Gzip) 4 B 28.1 KB 5.5 KB 10.0 KB
with just one size, potentially different for each symbol (171
CM1 (MPI-I/O) 4 B 562.5 KB 109.5 KB 219.8 KB
symbols out of 183 for CM1 using HDF5 are associated with
Nek5000 4 B 96.0 KB 20.1 KB 29.7 KB
onesize,andsimilarnumberswithGZIPandpHDF5,12outof
GTC 4 B 8.0 KB 7.8 KB 100.0 B
17 for GTC, and all 38 of them for Nek5000).LAMMPS had
LAMMPS 3 B 2.70 MB 851.1 KB 1.24 MB
the most interesting distribution, with 123 symbols associated
with a unique size (yet potentially different for each symbol),
and one unique symbol associated with a different size at the number of particles written (and thus the size of each
almost every appearance. This distribution is due to the fact write) varies slightly from one write to another. Thus, after
that all n processes send their set of particles to the rank 0 tryingtobuildalocalsizegrammaroutofthoserandomsizes,
process, which writes them into a file in n successive write Omnisc’IO falls back to keeping track of the average only.
calls.Asthenumberofparticlesvariesbetweenprocessesand
Note that the graphs are cut down to a maximum relative
between checkpoints, this leads to the variation in observed
errorof5,whereasthe observederrorscanbeof uptoseveral
sizes.
thousands. For instance, if Omnisc’IO predicts a write or
To evaluate the prediction of sizes, we use the following 5,000 bytes while the application actually writes only 2, the
relative error as a metric: relative error is 2,499.
Toputtheserelativeerrorsinperspective,TableVprovides
|size −size |
E = p o , (2) statistics on the sizes accessed by each operation. Note that
size
size
o the standard deviationis often close to or even largerthan the
where size is the predicted size and size is the observed average access size, making this average a poor estimator if
p o
size.Intuitively,ifthepredictionsarealwayssuchthatE ≤ we were to use it in a prediction.
size
ǫ, then allocating 1+ǫ times the predicted size (in a caching 2)Prediction of offsets: We consider that an offset pre-
system, for example)will always be enoughto coverthe need diction is either correct or incorrect. When our algorithm
for the next operation. makes several predictions for the next context symbol (and
Figure4showstherelativeerrorobservedforallsixcases. therefore several predictions of offset), correct predictions
In all but Nek5000, the error goes to 0 or close to 0 after are weighed accordingly. We compare our solution with the
the learning phase. Errors observed in Nek5000 match the classical contiguous access estimation [17], which consists of
incorrect predictions of context symbols. In LAMMPS, the always predicting that the next offset will follow the previous
prediction is very close but not equal to 0. The reason is that access. Table VI shows the proportion of contiguous accesses
TABLE VI: Proportion of correct offset prediction using 100 100
raounnadiveedctoonctliogsueosuts0.o1f%fse.ts approach, and using Omnisc’IO, Ratio (%) 468000 Ratio (%) 468000
Hit 20 Hit 20
0 0
Application Contiguous Accesses Omnisc’IO 0 200 400 600 800 1000 1200 1400 1600 0 200 400 600 800 1000 1200 1400
CM1 (POSIX) 47.4% 92.2% Operations Operations
CM1 (Gzip) 53.2% 83.0% (a) CM1+POSIX (b) CM1+Gzip
CM1 (MPI-I/O) 72.7% 98.0%
100 100
NGLAeTkCM50M0P0S 999999...499%%% 911900.007%%% Hit Ratio (%) 24680000 Hit Ratio (%) 24680000
0 0
in our set of applications as well as the proportion of correct 0 1000 2000 3000 4000 5000 6000 7000 8000 0 2000 4000 6000 8000 10000 12000 14000 16000
Operations Operations
predictions made by Omnisc’IO. In all cases, Omnisc’IO
(c) CM1+MPI-I/O (d) Nek5000
achieves a better prediction of offsets than does the naive
approximation based on contiguous accesses. It is especially 100 100
iabncectCteeMsrsse1us,itstehindactweahirteenmnuaosnniancgogenastihtgoiugomhu-osled.veIelnlaIpn/aOdrtpliicrbeurdlaaircry,tstthhueechpparoesrdtHiiocDntiFoo5nf Hit Ratio (%) 24680000 Hit Ratio (%) 24680000
oafcoofnftsiegtuoinusCaMcc1esussiensgtimHaDtiFo5n,gtooes92f.r2o%mw47it.h4%O,mwnhisecn’IuOs.ing 0 0 10000 20000 30000 40000 50000 60000 70000 80000 9000 0100000 0 0 5000 10000 15000 20000 25000
Operations Operations
3)Hit ratio: We also combine the prediction of sizes and
(e) GTC (f) LAMMPS
offsetsto measurehow accuratelyoursolution can predictthe
locationofthenextaccess.Thisinformationformsapredicted Fig. 5: Measurement of the hit ratio using Omnisc’IO to
segmentS =Jxstart,xendK. The segmenteffectivelyaccessed predictthelocationofthenextaccessedsegment,asafunction
by the next I/O operation is denoted S0 = Jystart,yendK. of the number of the number of operations completed.
The hit ratio of S with respect to S0, denoted H(S|S0), is
computed by
TABLE VII: Average hit ratio achieved by Omnisc’IO,
rounded to closest 0.1%.
100×|S∩S0|

H(S|S0)=max(xend,yend)−min(xstart,ystart) (3) Application Hit Ratio
100 if S =S0 =∅ CM1 (POSIX) 84.6%

CM1 (Gzip) 79.5%
This metrics yields the percentage of overlap between
CM1 (MPI-I/O) 96.0%
the two segments with respect to the distance between their
Nek5000 98.6%
extrema: H(S|S0) = 100 ⇐⇒ S = S0. Since our approach GTC 100%
may propose several potential next locations, this formula is
LAMMPS 99.4%
extended to multiple segments S1...Sn by considering the
average of H(Si|S0) for i ∈ J1,nK. Figure 5 shows the
results obtained with our simulations, and Table VII presents note that Omnisc’IO is efficient at discriminating immediate
the average hit ratio over the course of the entire run for transitions (low transition times, which can be used as a hint
each application. Note that for CM1+POSIX and CM1+MPI- thattwo consecutiveoperationsbelongto the sameI/O phase)
I/O, Omnisc’IO holds a perfect hit ratio after the learning from distant transitions (corresponding to computation and
phase. Although the hit ratio in LAMMPS also seems to communication phases that last much longer).
be perfect, it is actually slightly lower than 100% because Figure7presentstheabsolutedifferencebetweenobserved
of the small error made in the prediction of the size (see and predicted transition times on a logarithmic scale. For
explanation in Section III-D1). The lowest hit ratio achieved readability, we consider only the 1,000 last operations of
in our experiments was that of CM1+Gzip (79.5%), which, each run, that is, during the stationary regime. Table VIII
since the study made on the prediction of offsets and sizes in reports the average of absolute difference over the course of
earlier sections, is explained mainly by incorrect predictions each run (in its entirety, and not restricted to the stationary
of offsets. Our guess is that HDF5 writes compressed data by regime). We also compare the performance of Omnisc’IO
blocks of predictable size but jumps back and forth in a more with the immediate reaccess estimation used by some I/O
unpredictable manner to update metadata. schedulers (e.g., [27]), which consists of assuming that the
next I/O operation is likely to immediately follow the current
E. Temporal prediction one (i.e., interarrival time are always estimated to 0) and use
Temporal prediction involves estimating the time between a time window during which a potential new operation is
the end of an I/O operation and the beginningof the next one expected).Inallsituations,Omnisc’IOappearstobeverygood
(interarrival time). For qualitative analysis, Figure 6 presents atpredictingtheinterarrivaltimeofI/Oaccesses.Inparticular,
the series of observed interarrival times between consecutive the average difference between the predicted and observed
operations,alongwiththepredictionsmadebyOmnisc’IO.We interarrivaltimeisbelowa microsecondforLAMMPS, andat
porbesdeircvteedd opbresdeircvteedd 10 10
Inter-arrival time 0 0.1 0 0.1 000 0.100000 .10010111 0 100 200 300 400 500 600 700 800 900 1000 Inter-arrival time 0 0.1 0. 00 010 0.100000. 00111011 0 100 200 300 400 500 600 700 800 900 1000 Difference (sec) 0 1.0 0e0. 00-0.0000. 115111 0 100 200 300 400 500 600 700 800 900 1000 Difference (sec) 0 110. 0ee.0 00--0.00000. 1116511 0 100 200 300 400 500 600 700 800 900 1000
Operations Operations Operations Operations
(a) CM1+POSIX (b) CM1+Gzip (a) CM1+POSIX (b) CM1+Gzip
porbesdeircvteedd opbresdeircvteedd 1 01 1 01
Inter-arrival time 0 1 0.1 0e 0.1 000- 0.1000000 .100105111 0 100 200 300 400 500 600 700 800 900 1000 Inter-arrival time 0 0. 0.0 00 0.1000. 011111 0 100 200 300 400 500 600 700 800 900 1000 Difference (sec) 0 1111.0 0eeee0. 00----0.0000000.18765111 0 100 200 300 400 500 600 700 800 900 1000 Difference (sec) 0 1110. 0eee.0 00---0.000000.1176511 0 100 200 300 400 500 600 700 800 900 1000
Operations Operations Operations Operations
(c) CM1+MPI-I/O (d) Nek5000 (c) CM1+MPI-I/O (d) Nek5000
porbesdeircvteedd opbresdeircvteedd 0.001 0.01
Inter-arrival time 0 0. 00.00.000111 0 100 200 300 400 500 600 700 800 900 1000 Inter-arrival time 0 0. 0.0 00 0.1000. 011111 0 100 200 300 400 500 600 700 800 900 1000 Difference (sec) 0 111.0eee0---00007651 0 100 200 300 400 500 600 700 800 900 1000 Difference (sec) 0 11110.0eeee.00----000000187651 0 100 200 300 400 500 600 700 800 900 1000
Operations Operations Operations Operations
(e) GTC (f) LAMMPS (e) GTC (f) LAMMPS
Fig. 6: Matching between observed and predicted interarrival Fig. 7: Differencebetweenpredictedandobservedinterarrival
times of I/O events. times of I/O events.
TABLE VIII: Averagetime differencebetween predictedand TABLE IX: Overhead of Omnisc’IO in the run time of each
observed interarrival times (rounded to closest millisecond), application (in microseconds per operation).
and comparison with an immediate re-access estimation.
Application Average Overhead Std. dev.
Application Time Difference Immediate Reaccess CM1 (POSIX) 20.51 µsec 18.27 µsec
CM1 (POSIX) 0.197 sec 0.735 sec CM1 (Gzip) 20.20 µsec 15.56 µsec
CM1 (Gzip) 0.199 sec 0.791 sec CM1 (MPI-I/O) 19.95 µsec 14.50 µsec
CM1 (MPI-I/O) 0.060 sec 0.406 sec Nek5000 23.44 µsec 18.96 µsec
Nek5000 0.012 sec 0.049 sec GTC 19.03 µsec 27.79 µsec
GTC 0.001 sec 0.006 sec LAMMPS 22.10 µsec 14.72 µsec
LAMMPS 0.000 sec 0.003 sec
requireOmnisc’IOtoknowonwhichspecificpartoftheentire
worst 0.199 seconds for CM1+Gzip, as opposed to 0.003 and simulation’s data the branch depends.
0.791 seconds, respectively, when considering an immediate
reaccess estimation. G. Run-time overhead
Note thatby combiningthe predictionof interarrivaltimes The run-time overhead on a commodity hardware is pre-
andcontextsymbols,wecanestimatehowmanyaccesseswill sented in Table IX. This overhead of a few microseconds is
happenwithinagiventimewindowandhowmanyconsecutive negligiblecomparedwiththe time takenbythe I/Ooperations
operationswilloccurbeforetheendoftheI/Ophase.Because themselves (a few milliseconds to several seconds). However,
of space constraints, these studies are not included in this since Omnisc’IO works at the level of atomic, contiguous
paper. operations, these I/O operations can be made asynchronous
to hide the overhead of Omnisc’IO behind the I/O time.
F. Limitations of our approach
Likeallsystems,Omnisc’IOhaslimitations.Asitleanson IV. RELATEDWORK
therepetitivenessofI/Opatterns,anynonperiodicapplications This section presents the related work in the context of
(e.g., applications that write their results only once at the end grammar-based modeling as well as spatial and temporal I/O
of their run) will make Omnisc’IO incapable of discovering prediction.
repetitive structures in the I/O pattern. To deal with such
applications,however,Omnisc’IOcansaveitsmodelintofiles A. Grammar-based modeling
and reload it before the next run. The first work related to ours is Sequitur [14]. Sequitur is
As noted in Section III-B, Omnisc’IO is sensitive to designed to build a grammarfrom a sequence of symbolsand
branches in the code that depend on the content of the data. has been used mainly in the area of text compression [12],
Solvingthisproblemis arguablymoredifficult,since itwould but also natural language processing, music processing, and