Nonparametric Bayesian Storyline Detection from Microtexts

0.07 MB·

by  Vinodh Krishnan

Nonparametric Bayesian Storyline Detection from Microtexts
Vinodh Krishnan JacobEisenstein
GeorgiaInstituteofTechnology GeorgiaInstituteofTechnology
Atlanta,GA 30308 Atlanta,GA30308
[email protected] [email protected]
6
1
0
2

p
e
S Abstract A major technical challenge is in fusing infor-
mation from two heterogeneous data sources: tex-
4
2 News events and social media are com- tual content and time. Two different documents
posed of evolving storylines, which cap- aboutasingle eventmightuseverydifferent vocab-

] ture public attention for a limited period of
L ulary, particularly in sparse social media data such
time. Identifyingstorylinesrequiresintegrat-
C as microblogs; conversely, two different sporting
ing temporal and linguistic information, and
. events might be described in nearly identical lan-
s priorworktakesa largelyheuristicapproach.
c We present a novel online non-parametric guage, with differences only in the numerical out-
[
Bayesian framework for storyline detection, come. Temporalinformation istherefore critical: in

2 usingthedistance-dependentChineseRestau- the first case, to find the commonalities across dis-
v rant Process (dd-CRP). To ensure efficient
paratewritingstyles,andinthesecondcase,toiden-
0
linear-time inference, we employ a fixed-lag
8 tify the differences. A further challenge is that un-
Gibbssamplingprocedure,whichisnovelfor
5
likeinstandarddocumentclusteringtasks,thenum-
4 the dd-CRP. We evaluate on the TREC Twit-
ber of events in a data stream is typically unknown
0 terTimelineGeneration(TTG),obtainingen-
1. couragingresults: despiteusingaweakbase- inadvance. Finally,thereisahighpremiumonscal-
0 lineretrievalmodel,thedd-CRPstorycluster- ability, sinceonlinetextisproduced atahighrate.
6 ingmethodiscompetitivewiththebestentries
1 inthe2014TTGtask. Due to these challenges, existing approaches for
:
v combining these modalities have been somewhat
i
X heuristic, relying on tunable parameters to control
1 Introduction
r the tradeoff between textual and temporal similar-
a
A long-standing goal for information retrieval and ity. In contrast, the Bayesian setting provides ele-
extractionistoidentifyandgrouptextualreferences gantformalismsforreasoningaboutlatentstructures
to ongoing events in the world (Allan, 2002). Suc- (e.g., events) and their stochastically-generated re-
cess on this task would have applications in per- alizations across text and time. In this paper, we
sonalized news portals (Gabrilovich et al., 2004), describe one such model, based on the distance-
intelligence analysis, disaster relief (Vieweg et al., dependent Chinese Restaurant Process (dd-CRP;
2010), and in understanding the properties of the BleiandFrazier,2011). Thismodelisdistinguished
news cycle (Leskovec et al., 2009). This task at- by the neat separation that it draws between tex-
tains a new importance in the era of social media, tual content, which is treated as a stochastic emis-
where citizen journalists can document events as sionfromanunknownMultinomialdistribution,and
they unfold (Lotan et al., 2011), but where repe- time, which is modeled as a prior on graphs over
tition and untrustworthy information can make the documents, through an arbitrary distance function.
reader’s task especially challenging (Becker et al., However, straightforward implementations of the
2011;Marcusetal.,2011;Petrovic´ etal.,2010). dd-CRPareinsufficientlyscalable,andsothemodel
has been relatively underutilized in the NLP liter- ihasthefollowingform,
ature (Titov and Klementiev, 2011; Kim and Oh,
2011;Sirtsetal.,2014). Wedescribe improvements f(di,j), i 6= j
Pr(c = j) ∝ (1)
i
to Bayesian inference that make the application of (α, i = j,
thismodelfeasible, andpresent encouraging empir-
whered isthedistance betweenunitsiandj,and
ical results on the Tweet Timeline Generation task i,j
α > 0 is a parameter of the model. Large values
fromTREC2014(Linetal.,2014).
ofαinducemoreself-linksandthereforemorefine-
grained partitionings. Since we are concerned with
temporalcovariates, wedefinethedistance function
2 Model
asfollows:
−|ti−tj|
The basic task that we address is to group short f(di,j) = e a . (2)
text documents into an unknown number of story- Thus, the likelihood of document i following doc-
lines,basedontheirtextualcontentandtheirtempo- ument j decreases exponentially as the time gap
ralsignature. Thetextual content maybeextremely |ti−tj|increases.
sparse — the typical Tweet is on the order of ten The text of each document i is represented by a
vector of word counts w . The likelihood distribu-
words long — so leveraging temporal information i
tionismultinomial,conditionedonaparameterθas-
is crucial. Moreover, the temporal signal is multi-
sociated with the partition to which document i be-
scale: in the 24-hour news cycle, some storylines
longs. Byplacing aDirichlet prior onθ,wecanan-
last for less than an hour, while others, like the dis- alytically integrate it out. Writing z(c) for the clus-
appearance of the Malaysian Airlines 370 plane in termembershipinducedbythefollowergraphc,we
2014, continue forweeksormonths. Insomecases, have:
thetemporaldistribution ofreferences toastoryline
willbeunimodalandwell-describedbyaparametric P(w |c;η)= P({wi :zi(c) =k};η) (3)
k
model (Marcus et al., 2011); in other cases, it may Y
be irregular, with bursts of activity followed by pe- = P({wi :zi(c) =k}|θ)P(θ;η)dθ
riods of silence (He et al., 2007). Finally, it will be Yk Zθ
(4)
crucial to produce an implementation that scales to
largecorpora.
Given a multinomial likelihood P(w | θ) and
The distance-dependent Chinese Restaurant Pro- a (symmetric) Dirichlet prior P(θ | η), this in-
cess (dd-CRP) meets many of these criteria (Blei tegral has a closed-form solution as the Dirichlet-
and Frazier, 2011). In this model, the key idea is Multinomial distribution (also known as the mul-
that each instance (document) i “follows” another tivariate Polya distribution). The joint probability
instance ci (where it is possible that ci = i), induc- is therefore equal to the product of Equation 1 and
ingagraph. Wecancomputeapartitioning overin- Equation4,
stancesbyconsideringtheconnectedcomponentsin
the undirected version of the follower graph; these P(w,c)= P(ci;α,a) P({wi : zi(c) = k};η).
partitions correspond to“tables”intheconventional i k
Y Y
(5)
“Chinese Restaurant” analogy (Aldous, 1985), orto
clusters. The advantage of this approach is that it
The model has three hyperparameters: α, which
is fundamentally non-parametric, and it introduces
controls thelikelihood ofself-linking, andtherefore
a clean separation between the textual data and the
affectsthenumberofclusters; a,whichcontrols the
covariates: thetextisgenerated byadistribution as-
timescaleofthedistancefunction, andthereforeaf-
sociated with the partition, while the covariates are
fects the importance of the temporal dimension to
associated with the following links, which are con-
theresulting clusters; andη,whichcontrols thepre-
ditioned onadistance function.
cisionoftheDirichletprior,andthereforetheimpor-
Thedistribution overfollowerlinksfordocument tanceofrarewordsinthetextuallikelihoodfunction.
Estimation ofthese hyperparameters isdescribed in (this growth is logarithmic in the Chinese Restau-
§3.2. rant Process). Scalability to the streaming setting
therefore requiresmoreaggressive optimizations.
3 Inference
To get back to linear time complexity, we em-
ploy a fixed-lag sampling procedure (Doucet et al.,
The key sampling equation for the dd-CRP is the
2000). Afterreceivinginstancei,weperformGibbs
posterior likelihood,
sampling only within the fixed window [t − τ,t ],
i i
Pr(ci = j |c−i,w)∝Pr(ci = j)P(w | c). leaving cj fixed if tj < ti − τ. This approximate
sampling procedure implicitly changes the underly-
The prior is defined in Equation 1. Let ℓ represent ingmodel, because thereisnopossibility oflinking
the likelihood under the partitioning induced when itoalatermessagej ifthetimegaptj −ti > τ.
the link c is cut. Now, the likelihood term has two Sinceweareonly interested inobtaining asingle
i
cases: inthefirstcase, j isalready inthesamecon- storyline clustering — rather than a full Bayesian
nected component as i (even after cutting the link distribution over clusterings — we perform anneal-
c ),sonocomponents aremerged bysetting c = j. ing for samples towards the end of the sampling
i i
Inthiscase, thelikelihood P(w | c = j)isexactly window. Specifically, we set the temperature to
i
equal toℓ. Inthe second case, setting c = j causes γ = 2.0 and exponentiate the sampling likelihood
i
twoclusterstobemerged. Thisgivesthelikelihood, by the inverse temperature (Geman and Geman,
1984). This has the effect of interpolating between
P(w |ci = j,c−i) probabilistically-correct Gibbs sampling and a hard
P({w : z(c) = z(c)∨z(c) = z(c)}) coordinate-ascent procedure.
∝ k k j k i ,
P({w : z(c) = z(c)})P({w : z(c) = z(c)}) 3.2 Hyperparameterestimation
k k i k k j
Themodelhasthreeparameters toestimate:
where the constant of proportionality is exactly
equaltoℓ. Eachofthetermsinthelikelihoodratiois • α,theconcentration parameterofthedd-CRP
aDirichletCompoundMultinomiallikelihood. This • a,theoffsetofthedistancefunction
likelihood function is itself a ratio of gamma func- • η,thescaleofthesymmetricDirichletprior.
tions; by eliminating constant terms and exploiting
We interleave maximization-based updates to these
the identity Γ(x +1) = xΓ(x), we can reduce the
parameters with sampling, in a procedure inspired
number of Gamma function evaluations required to
byMonteCarloExpectationMaximization(Weiand
computethisratiotothenumberofwordswhichap-
(c) (c) Tanner, 1990). Specifically, we compute gradients
pear in both clusters z and z . Words that oc-
i j on the likelihood P(c)with respect to αand a, and
cur in neither cluster can safely be ignored, and the
takegradient stepsaftereveryfixednumberofsam-
gamma functions for words which occur in exactly
ples. For the symmetric Dirichlet parameter η, we
one of the two clusters cancel in the numerator and
employ the heuristic from Minka (2012) by setting
denominator of the ratio. Note also that we only
theparametertoη = (K−1)/2,whereK isthenum-
need compute the likelihood for ci with respect to Pklogpk
ber of words that appear exactly once, and p is the
eachcluster, notforeverypossible followerlink. k
probability of choosing the kth word from the vo-
3.1 Onlineinference cabulary under the unigram distribution for the en-
tirecorpus.
Whilewemakeeveryefforttoacceleratethecompu-
tation of individual Gibbs samples, the complexity
4 TREC Evaluation
of the basic algorithm is superlinear in the number
of instances. This is due to the fact that each sam- Totesttheefficacyofthisapproach, weevaluate on
ple requires computing the probability of instance i the Twitter Timeline Generation (TTG) task in the
joining every possible cluster, while the number of Microblog track of TREC 2014. It involves taking
clusters itself grows with the number of instances tweets based on a query Q at time T and returning
a summary that captures relevant information. We 2014)andandhltcoe(Xuetal.,2014)teams(lines5
perform the task on55queries with different times- and 6) used retrieval models with MAP scores of at
tamps and compare our results with 13 groups that least0.5. Bayesiandd-CRPstoryline clustering was
submitted50runsforthistaskin2014. competitive with these timeline generation systems
Weconsiderthefollowingsystems: despite employing a far worse retrieval model, so
improvingtheretrievalmodeltoachieveparitywith
Baseline We replace the distance-dependent prior
thesealternativesystemsseemsthemoststraightfor-
with a standard Dirichlet prior. The number
wardpathtowardsbetteroverallperformance.
of clusters is heuristically set to 20. Annealed
Gibbssamplingisemployed forinference.
5 Related work
Offlineinference The dd-CRP model with offline
inference procedure (described in§3). Topic tracking and first-story detection are very
well-studiedtasks;spacedoesnotpermitacomplete
Onlineinference The dd-CRP model with online
analysis of the related work, but see (Allan, 2002)
inference procedure (described in§3.1).
for a summary of “first generation” research. More
For the online inference implementation, we set recent non-Bayesian approaches have focused on
the size of window and number of iterations to five string overlap (Suen et al., 2013), submodular opti-
days and 500 respectively. For the baseline, the pa- mization(Shahafetal.,2012),andlocality-sensitive
rameter of the Dirichlet prior was set to a vector hashing (Petrovic´ et al., 2010). In Bayesian story-
of 0.5 for each cluster. These values were chosen line analysis, the seminal models are Topics-Over-
through 10-foldcrossvalidation. Time (Wang and McCallum, 2006), which asso-
Tomeasurethequalityoftheclusteringsobtained ciates a parametric distribution over time with each
by these models, we compare the average weighted topic (Ihler et al., 2006), and the Dynamic Topic
and unweighted F-measures for 55 TREC topics, Model(BleiandLafferty,2006),whichmodelstopic
using the evaluation scripts from the TREC TTG evolution asalinear dynamical system (Nallapati et
task. Overall results areshown inTable1. The ON- al., 2007). Later work by Diao et al. (2012) offers
LINE MODEL has the best weighted F1 score, out- a model for identifying “bursty” topics, with infer-
performing the offline version of the same model, encerequiring dynamicprogramming. Alltheseap-
even though its inference procedure is an approxi- proaches require the number of topics to be iden-
mation to the OFFLINE MODEL. It may be that its tified in advance. Kim and Oh (2011) apply a
approximate inference procedure discourages long- distance-dependent Chinese Restaurant Franchise
range linkages, thus placing a greater emphasis on for temporal topic modeling; they evaluate using
the temporal dimension. Both models were trained predictive likelihood rather than comparing against
over 500 iterations, and the ONLINE MODEL was groundtruth,anddonotconsider onlineinference.
30%fastertotrainthantheofflinemodel. The Infinite Topic-Cluster model (Ahmed et al.,
Comparedtotheother2014TRECTTGsystems, 2011a) is non-parametric over the number of sto-
our dd-CRP models are competitive. Both mod- rylines, through the use of the recurrent Chinese
els outperform all but one of the fourteen submis- Restaurant Process (rCRP). The model is substan-
sions onthe unweighted F metric, and would have tially more complex than our approach. Unlike the
1
placed fourth ontheweighted Fw metric. Notethat dd-CRP, the rCRP is Markovian in nature, so that
1
the TREC evaluation scores both clustering qual- the topic distribution at each point in time is con-
ity and retrieval. We use only the baseline retrieval ditioned on the previous epoch (or, at best, the pre-
model, which achieved ameanaverage precision of vious K epochs, with complexity of inference in-
0.31. The competing systems shown in Table 1 all creasing with K). This Markovian assumption cre-
use retrieval models that are far superior: the re- ates probabilistic dependencies between the topic
trieval model for top-ranked PKUICST team (line assignment for a given document and the docu-
4) achieved a mean average precision (MAP) of ments that follow in subsequent epochs, necessitat-
0.59 (Lv et al., 2014), and the QCRI(Magdy et al., inganinference procedurethatcombinessequential
Model Rec. Rec.w Prec. F Fw
1 1
dd-CRPclustering models
1. BASELINE 0.14 0.27 0.33 0.20 0.30
2. OFFLINE 0.32 0.47 0.27 0.29 0.34
3. ONLINE 0.34 0.55 0.26 0.29 0.35
TopsystemsfromTrec-2014TTG
4. TTGPKUICST2(Lvetal.,2014) 0.37 0.58 0.46 0.35 0.46
5. EM50(Magdyetal.,2014) 0.29 0.48 0.42 0.25 0.38
6. hltcoeTTG1(Xuetal.,2014) 0.40 0.59 0.34 0.28 0.37
Table1:PerformanceofModelsintheTREC2014TTGTask.WeightedrecallandF1areindicatedasRec.wandF1w.
MonteCarloandMetropolisHastings,andacustom Aldous,1985 David J Aldous. 1985. Exchangeability
datastructure;thisinferenceprocedurewascomplex andrelatedtopics. Springer.
enoughtowarrantacompanionpaper(Ahmedetal., Allan,2002 James Allan. 2002. Topic detection and
tracking: event-based information organization, vol-
2011b). The rCRP is also employed by Diao and
ume12. SpringerScience&BusinessMedia.
Jiang (2013, 2014). In contrast, the dd-CRP makes
Beckeretal.,2011 Hila Becker, Mor Naaman, and Luis
no Markovian assumptions, and efficient inference
Gravano. 2011. Beyondtrendingtopics: Real-world
is possible through relatively straightforward Gibbs
event identification on twitter. In Proceedings of the
samplinginafixedwindow.
International Conference on Web and Social Media
(ICWSM),pages438–441.
6 Conclusion
BleiandFrazier,2011 David M Blei and Peter I Frazier.
2011. Distance dependent chinese restaurant pro-
Wepresentasimplenon-parametric modelforclus-
cesses. The Journal of Machine Learning Research,
tering short documents (such as tweets) into sto-
12:2461–2488.
rylines, which are conceptually coherent and tem-
BleiandLafferty,2006 David M Blei and John D Laf-
porally focused. Future work may consider learn- ferty. 2006. Dynamictopicmodels. InProceedingsof
ingmoreflexibletemporaldistancefunctions,which the 23rd international conference on Machine learn-
could potentially represent temporal periodicity or ing,pages113–120.ACM.
parametricmodelsofcontent popularity. DiaoandJiang,2013 QimingDiaoandJingJiang. 2013.
A unified modelfor topics, events and users on twit-
Acknowledgments We thank the reviewers for ter. InProceedingsofEmpiricalMethodsforNatural
their helpful feedback. Thisresearch wassupported LanguageProcessing(EMNLP).
by an award from the National Institutes for Health DiaoandJiang,2014 QimingDiaoandJingJiang. 2014.
(R01GM112697-01), and by Google, through a Fo- Recurrentchinese restaurantprocess with a duration-
baseddiscountforeventidentificationfromtwitter. In
cused Research Award for Computational Journal-
The18thPacific-AsiaConferenceonKnowledgeDis-
ism.
coveryandDataMining(SDM’14).
Diaoetal.,2012 QimingDiao,JingJiang,FeidaZhu,and
Ee-Peng Lim. 2012. Finding bursty topics from mi-
References
croblogs. InProceedingsoftheAssociationforCom-
Ahmedetal.,2011a Amr Ahmed, Qirong Ho, Jacob putational Linguistics (ACL), pages 536–544, Jeju,
Eisenstein, Eric Xing, Alexander J. Smola, and Korea.
ChoonH. Teo. 2011a. Unifiedanalysisof streaming Doucetetal.,2000 Arnaud Doucet, Simon Godsill, and
news. InWWW,pages267–276,Hyderabad,India. ChristopheAndrieu. 2000. Onsequentialmontecarlo
Ahmedetal.,2011b Amr Ahmed, Qirong Ho, Choon H samplingmethodsforbayesianfiltering. Statisticsand
Teo,JacobEisenstein,EricPXing,andAlexJSmola. computing,10(3):197–208.
2011b. Online inference for the infinite topic-cluster Gabrilovichetal.,2004 Evgeniy Gabrilovich, Susan Du-
model: Storylines from streaming text. In AISTATS, mais, and Eric Horvitz. 2004. Newsjunkie: provid-
pages101–109,FortLauderdale,FL. ingpersonalizednewsfeedsviaanalysisofinformation
novelty. InProceedingsofthe13thinternationalcon- Nallapatietal.,2007 Ramesh M Nallapati, Susan Dit-
ferenceonWorldWideWeb,pages482–490.ACM. more,JohnDLafferty,andKinUng. 2007. Multiscale
GemanandGeman,1984 Stuart Geman and Donald Ge- topictomography. InKDD,pages520–529.ACM.
man. 1984. Stochasticrelaxation,gibbsdistributions, Petrovic´etal.,2010 Sasˇa Petrovic´, Miles Osborne, and
and the bayesian restoration of images. IEEE Trans- Victor Lavrenko. 2010. Streaming first story detec-
actionsonPatternAnalysisandMachineIntelligence, tionwithapplicationtotwitter. InProceedingsofthe
pages721–741. North AmericanChapteroftheAssociationforCom-
putationalLinguistics (NAACL), pages 181–189, Los
Heetal.,2007 Qi He, Kuiyu Chang, and Ee-Peng Lim.
Angeles,CA.
2007. Analyzing feature trajectories for event detec-
Shahafetal.,2012 Dafna Shahaf, Carlos Guestrin, and
tion. InSIGIR,pages207–214.ACM.
Eric Horvitz. 2012. Trains of thought: Generating
Ihleretal.,2006 Alexander Ihler, Jon Hutchins, and
informationmaps. In Proceedingsof the Conference
PadhraicSmyth. 2006. Adaptiveeventdetectionwith
on World-Wide Web (WWW), pages 899–908, Lyon,
time-varyingpoissonprocesses. InKDD,pages207–
France.ACM.
216.ACM.
Sirtsetal.,2014 KairitSirts,JacobEisenstein,MichaEl-
KimandOh,2011 Dongwoo Kim and Alice Oh. 2011.
sner, and Sharon Goldwater. 2014. Pos induction
Accountingfor data dependencieswithin a hierarchi-
withdistributionalandmorphologicalinformationus-
cal dirichlet process mixture model. In cikm, pages
ing a distance-dependent chinese restaurant process.
873–878.
In Proceedings of the Association for Computational
Leskovecetal.,2009 Jure Leskovec, Lars Backstrom, Linguistics(ACL),Baltimore,MD.
andJonKleinberg. 2009. Meme-trackingandthedy- Suenetal.,2013 Caroline Suen, Sandy Huang, Chantat
namics of the news cycle. In Proceedings of Knowl- Eksombatchai, Rok Sosic, and Jure Leskovec. 2013.
edgeDiscoveryandDataMining(KDD),pages497– Nifty: asystemforlargescaleinformationflowtrack-
506. ing andclustering. InProceedingsofthe Conference
Linetal.,2014 JimmyLin,MilesEfron,YuluWang,and onWorld-WideWeb(WWW),pages1237–1248.
Garrick Sherman. 2014. Overview of the trec-2014 TitovandKlementiev,2011 Ivan Titov and Alexandre
microblogtrack. In Proceedingsof the Twenty-Third Klementiev. 2011. A bayesian model for unsuper-
TextREtrievalConference. vised semantic parsing. In Proceedings of the 49th
Lotanetal.,2011 Gilad Lotan, Erhardt Graeff, Mike AnnualMeetingoftheAssociationforComputational
Ananny,Devin Gaffney,Ian Pearce, and danahboyd. Linguistics: Human Language Technologies-Volume
2011. Thearabspring—therevolutionsweretweeted: 1, pages 1445–1455. Association for Computational
Informationflowsduringthe2011tunisianandegyp- Linguistics.
tianrevolutions. Internationaljournalofcommunica- Viewegetal.,2010 Sarah Vieweg, Amanda L. Hughes,
tion,5:31. Kate Starbird, and Leysia Palen. 2010. Microblog-
ging during two natural hazardsevents: What twitter
Lvetal.,2014 Chao Lv, Feifan Fan, RunweiQiang, Yue
may contribute to situational awareness. In Proceed-
Fei, and Jianwu Yang. 2014. PKUICST at TREC
ingsoftheSIGCHIConferenceonHumanFactorsin
2014MicroblogTrack:featureextractionforeffective
ComputingSystems,CHI’10,pages1079–1088,New
microblog search and adaptive clustering algorithms
York,NY,USA.ACM.
forTTG. Technicalreport,DTICDocument.
WangandMcCallum,2006 Xuerui Wang and Andrew
Magdyetal.,2014 Walid Magdy, Wei Gao, Tarek El-
McCallum. 2006. Topics over time: a non-markov
ganainy,andZhongyuWei. 2014. Qcriattrec 2014:
continuous-timemodeloftopicaltrends. InProceed-
applying the kiss principle for the ttg task in the mi-
ings of the 12th ACM SIGKDD international confer-
croblogtrack. Technicalreport,DTICDocument.
enceonKnowledgediscoveryanddatamining,pages
Marcusetal.,2011 Adam Marcus, Michael S Bernstein,
424–433.ACM.
Osama Badar, David R Karger, Samuel Madden, and
WeiandTanner,1990 Greg CG Wei and Martin A Tan-
Robert C Miller. 2011. Twitinfo: aggregating and
ner. 1990. A monte carlo implementation of the em
visualizing microblogs for event exploration. In chi,
algorithmandthepoorman’sdataaugmentationalgo-
pages227–236.ACM.
rithms. Journal of the American Statistical Associa-
Minka,2012 Thomas Minka. 2012. Estimating a tion,85(411):699–704.
dirichlet distribution. http://research.
Xuetal.,2014 Tan Xu, Paul McNamee, and Douglas W
microsoft.com/en-us/um/people/minka/
Oard. 2014. Hltcoeattrec2014:Microblogandclini-
papers/dirichlet/minka-dirichlet.
caldecisionsupport.
pdf.