Retrieval of Spoken Documents

45 Pages·2014·0.54 MB·English

This is a repository copy of Retrieval of Spoken Documents: First Experiences (Research
Report TR-1997-34).
White Rose Research Online URL for this paper:
http://eprints.whiterose.ac.uk/4529/
Monograph:
Crestani, F. and Sanderson, M. (1997) Retrieval of Spoken Documents: First Experiences
(Research Report TR-1997-34). Technical Report. Department of Computing Science at
the University of Glasgow
Reuse
Unless indicated otherwise, fulltext items are protected by copyright with all rights reserved. The copyright
exception in section 29 of the Copyright, Designs and Patents Act 1988 allows the making of a single copy
solely for the purpose of non-commercial research or private study within the limits of fair dealing. The
publisher or other rights-holder may allow further reproduction and re-use of this version - refer to the White
Rose Research Online record for this item. Where records identify the publisher as the copyright holder,
users can verify any specific terms of use on the publisher’s website.
Takedown
If you consider content in White Rose Research Online to be in breach of UK law, please notify us by
emailing [email protected] including the URL of the record and the reason for the withdrawal request.
[email protected]
https://eprints.whiterose.ac.uk/
Retrieval of Spoken Documents:
First Experiences
(cid:3)
Fabio Crestani and Mark Sanderson
Department of Computing Science
University of Glasgow
Glasgow G12 8QQ, Scotland
9th October 1997
Abstract
We report on our (cid:12)rst experiences in dealing with the retrieval of spo-
ken documents. While lacking the tools and the know-how for performing
speech recognition on the spoken documents, we tried to use in the best
possible way our knowledge of probabilistic indexing and retrieval of tex-
tual documents. The techniques we used and the results we obtained are
encouraging, motivating our future involvement in other further experi-
mentation in this new area of research.
(cid:3)
Supported by a \Marie Curie" Research Fellowship from the European Community.
1
Contents
1 Introduction 3
2 Probabilistic Information Retrieval 3
2.1 The binary independence retrieval model . . . . . . . . . . . . . . 4
2.2 Term weighting schemas . . . . . . . . . . . . . . . . . . . . . . . 9
3 The SIRE Information Retrieval system 11
4 The Abbot speech recognition system 12
5 Spoken document retrieval at TREC-6 14
6 The SDR TREC-6 data set 15
7 Experimenting probabilistic retrieval of spoken documents 18
7.1 The PFT weighting schema . . . . . . . . . . . . . . . . . . . . . 18
7.2 Generating a weighting schema by merging di(cid:11)erent transcriptions 21
8 Results and discussion 22
8.1 Results from the training data set . . . . . . . . . . . . . . . . . . 23
8.2 Results from the test data set . . . . . . . . . . . . . . . . . . . . 30
8.3 O(cid:14)cial SDR results for TREC-6 . . . . . . . . . . . . . . . . . . . 32
9 Related work 37
10 Conclusions and future works 37
2
1 Introduction
Retrieval of spoken documents is a fast emerging area of research in Multimedia
InformationRetrieval. It involves the e(cid:11)ective combinationof the most advanced
techniques used in speech recognition and Information Retrieval (IR). The in-
creasing interest in this area of research is con(cid:12)rmed by the inclusion, for the
(cid:12)rst time, of a retrieval of spoken documents track in the TREC initiative [7].
Despite our lack of know-how and toolsfor speech recognition, we decided to par-
ticipatein the retrievalof spoken documents track of TREC-6, con(cid:12)dent that our
knowledge of natural language processing and probabilistic information retrieval
would compensate. Because of the way the retrieval of spoken documents track
of TREC-6 is set up, groups with no speech recognition tools can participate all
the same. However, thanks to our research contacts, we were able to team up
with the Speech and Hearing Research Group of the Department of Computing
Science of the University of She(cid:14)eld. They provided us with the transcripts of
theirspeech recognitionsystem, Abbot, thatweusedformostofourexperiments.
This paper reports our (cid:12)rst experiences in retrieval of spoken documents. The
resultsweobtainedareencouragingtowardscontinuingworkinginthisinteresting
area of research.
Thepaperisstructuredasfollows. InSection2wedescribeindetailthenecessary
background on probabilistic IR so that some of the indexing choices taken in
the rest of the paper will be better understood. Sections 3 and 4 describe the
tools used for the experiments reported in this paper. Section 5 describes the
experimental framework in which our investigation took place: the data sets
used, the evaluation procedure, etc. Section 7 describes the two indexing schema
we experimented with. The results of a set of experiments with these indexing
schemas are reported and discussed in Section 8.
2 Probabilistic Information Retrieval
InIR,probabilisticmodellingconcernstheuseofamodelthatranksdocumentsin
decreasingorderoftheirevaluatedprobabilityofrelevancetoauser's information
need (also known as RetrievalStatus Value). Past and present research has made
much use of formal theories of probability and of statistics in order to evaluate,
or at least estimate, these probabilities of relevance. These attempts are to be
distinguished from looser ones like, for example, the \vector space model" [14]
in which documents are ranked according to a measure of similarity with the
query. A measure of similarity cannot be directly interpretable as a probability.
In addition, similarity based models generally lack the theoretical soundness of
3
probabilistic models.
The (cid:12)rst attempts to develop a probabilistic theory of retrieval were made over
thirty years ago [9]. Since then, there has been a steady development of the
approach, so that there are already several operational IR systems based upon
probabilistic or semi-probabilistic models.
One major obstacle with probabilistic or semi-probabilistic IR models is that of
(cid:12)nding methods for estimating the probabilities used to evaluate the probability
of relevance that are both theoretically sound and computationallye(cid:14)cient. The
problem of estimating these probabilities is di(cid:14)cult to tackle unless some sim-
plifying assumptions are made. In the early stages of the study of probabilistic
modelling in IR, assumptions related to event independence were employed in
order to facilitate the computations. One of the (cid:12)rst models to be based upon
such assumptions was the binary independence retrieval model.
2.1 The binary independence retrieval model
As in most IR models, it is assumed that queries and documents are described
by sets of index terms. Let T = ft1;:::;tng denote the set of terms used in
the collection of documents. We represent the query qk with terms belonging
to T. Similarly, we represent a document dj as the set of terms occurring in
it. If we use a binary representation then dj is represented as the binary vector
~x = (x1;:::;xn) with xi = 1 if ti 2 dj and xi = 0 otherwise. The query qk is
represented in the same manner.
The basic assumption, common to many other models, is that the distribution
of terms within the document collection provides information concerning the
relevance of a document to a given query. This is because it is assumed that
terms are distributed di(cid:11)erently in relevant and non-relevant documents. This is
known as the cluster hypothesis (see [18] pp. 45-47). If the term distribution was
thesamewithinthesetsofrelevantandnon-relevantdocumentsthenitwouldnot
be possible to devise a discrimination criterion between them. In which case, a
di(cid:11)erent representation of the document informationcontent would be necessary.
The term distribution provides information about the probability of relevance of
a document to a query. If we assume binary relevance judgements, then the term
distribution provides information about P(R j qk;dj).
The quantity P(R j qk;~x), with ~x as a binary document representation, cannot
be estimated directly. Instead, Bayes' theorem is applied [10]:
P(R j qk)(cid:1)P(~x j R;qk)
P(R j qk;~x) =
P(~x j qk)
4
Cj(R;dec) retrieved not retrieved
relevant document 0 (cid:21)1
non relevant document (cid:21)2 0
Table 1: The cost of retrieving and not retrieving a relevant and non relevant
document
To simplify notation, we omit the qk on the understanding that evaluations are
with respect to a given query qk. The previous relation becomes:
P(R)(cid:1)P(~x j R)
P(R j ~x) =
P(~x)
where P(R) is the prior probability of relevance, P(~x j R) is the probability of
observing the description ~x conditioned upon relevance having been observed,
and P(~x) is the probability that ~x is observed. The latter is determined as the
joint probability distribution of the n terms within the collection. The above
formula evaluates the \posterior" probability of relevance conditioned upon the
information provided in the vector ~x.
The provision of a ranking of documents by the Probability Ranking Principle
[12] can be extended to provide an \optimal threshold" value. This can be used
to set a cut-o(cid:11) point in the ranking to distinguish between those documents that
are worth retrieving and those that are not. This threshold is determined by
means of a decision strategy, whose associated cost function Cj(R;dec) for each
document dj is described in Table 1.
The decision strategy can be described simply as one that minimises the average
cost resulting from any decision. This strategy is equivalent to minimising the
following risk function:
X
R(R;dec) = Cj(R;dec)(cid:1)P(dj j R)
dj2D
It can be shown (see [18], pp. 115-117) that the minimisation of that function
brings about an optimalpartitioningof the document collection. This is achieved
by retrieving only those documents for which the following relation holds:
P(dj j R)
> (cid:21)
P(dj j R)
5
where
(cid:21)2 (cid:1)P(R)
(cid:21) =
(cid:21)1 (cid:1)P(R)
It remains now necessary to estimate the joint probabilities P(dj j R) and P(dj j
R), that is P(~x j R) and P(~x j R) if we consider the binary vector document
representation ~x.
In order to simplify the estimation process, the components of the vector ~x are
assumed to be stochastically independent when conditionally dependent upon R
or R. That is, the joint probability distribution of the terms in the document dj
is given by the following product of marginal probability distributions:
Yn
P(dj j R) = P(~x j R) = P(xi j R)
i=1
and
Yn
P(dj j R) = P(~x j R) = P(xi j R)
i=1
This binary independence assumption, is the basis of a model (cid:12)rst proposed by
RobertsonandSparkJonesin1976[13]: theBinaryIndependenceRetrievalmodel
(BIR). The assumption has always been recognised as unrealistic.
Nevertheless, as pointed out by Cooper [1], the assumption that actually under-
pins the BIR model is not that of binary independence, but that of the weaker
assumption of linked dependence:
P(~x j R) Yn P(xi j R)
=
P(~x j R) i=1 P(xi j R)
This states that the ratio between the probabilitiesof~x occurring in relevant and
non relevant documents is equal to the product of the corresponding ratios of the
single terms.
Considering the decision strategy of the previous section, it is now possible to
devise adecisionstrategyby usingalogarithmictransformationtoobtainalinear
decision function:
P(dj j R)
g(dj) = log > log(cid:21)
P(dj j R)
6
To simplify notation, we de(cid:12)ne the following quantities: pj = P(xj = 1 j R), and
qj = P(xj = 1 j R) which represent the probability of the jth term appearing
in a relevant, and in a non relevant document, respectively. Clearly: 1 (cid:0) pj =
P(xj = 0 j R), and 1(cid:0)qj = P(xj = 0 j R). This gives:
Yn
xj 1(cid:0)xj
P(~x j R) = pj (cid:1)(1(cid:0)pj)
j=1
and
Yn
xj 1(cid:0)xj
P(~x j R) = qj (cid:1)(1(cid:0)qj)
j=1
Substituting the above, gives:
Pn pj 1(cid:0)pj
g(di) = j=1(xj (cid:1)log qj +(1(cid:0)xj)(cid:1)log 1(cid:0)qj)
P
n
= j=1cj xj +C
where:
pj (cid:1)(1(cid:0)qj)
cj = log
qj (cid:1)(1(cid:0)pj)
and
Xn 1(cid:0)pj
C = log
j=1 1(cid:0)qj
This formula gives the Retrieval Status Value (RSV) of document dj for the
query under consideration. Documents are ranked according to their RSV and
presented to the user. The cut-o(cid:11) value (cid:21) can be used to determine the point at
which the displaying of the retrieved documents to the user should be stopped,
although, the RSV is generally used to rank the entire collection of documents.
In a real IR system, the presentation of documents ordered on their estimated
probability of relevance to a query matters more than the actual value of those
probabilities. Therefore, since the value of C is constant for a speci(cid:12)c query, we
need only consider the value of cj. This value, or more often the value exp(cj),
is called the term relevance weight (TRW), and indicates the term's capability
7
to discriminate relevant from non relevant documents. Because of the binary
independence assumption, in the BIR model term relevance weights contribute
\independently" to the relevance of a document.
To apply the BIR model, it is necessary to estimate the parameters pj and qj
for each term used in the query. This is performed in various ways, depending
upontheamountofinformationavailable. Theestimationcanberetrospectiveor
predictive. The (cid:12)rst is used on test collections where the relevance assessments
are known. The second is used with normal collection where parameters are
estimated by means of relevance feedback from the user.
Relevance feedback is a technique that allows a user to interactively express his
information requirement by adapting his original query formulation with further
information[6]. This additionalinformationis often provided by indicating some
relevant documents among the documents retrieved by the system.
Let us assume that the IR system has already retrieved some documents for the
query qk. The user is asked to give relevance assessments for those documents,
from which the parameters of the BIR are estimated. If we also assume to
be working in the retrospective case, then we know the relevance value of all
individual documents in the collection. Let a collection have N documents, R
of which are relevant to the query. Let nj denote the number of documents in
which the term xj appears, amongst which, only rj are relevant to the query.
The parameters pj and qj can then be estimated as follows:
rj
p^j =
R
and
nj (cid:0)rj
q^j =
N (cid:0)R
These give:
rj
R(cid:0)rj
TRWj = nj(cid:0)rj
N(cid:0)nj(cid:0)R+rj
This approach is possible only if we have relevance assessments for all documents
in the collection, i.e. where we know R and rj. According to Croft and Harper,
given that the only information concerning the relevance of documents is that
provided by a user through relevance feedback, predictive estimations should
8
be used. Let R~ denote the number of documents judged relevant by the user.
Further, let r~j be the number of those documents in which the term xj occurs.
We can then combine this with the estimation technique of [2].
r~j+0:5
~ R~(cid:0)r~j+0:5
TRWj = nj(cid:0)r~j+0:5
N(cid:0)nj(cid:0)R~+r~j+0:5
Usually, the relevance informationgivenby auser islimitedandisnotsu(cid:14)ciently
representative of the entire collection. Consequently, the resulting estimates tend
to lack precision. As a partial solution, one generally simpli(cid:12)es by assuming pj
to be constant for all the terms in the indexing vocabulary. The value pj = 0:5
is often used, which gives a TRW that can be evaluated easily:
TR~~Wj = N (cid:0)nj
nj
Work up to this point requires the use of at least a few relevant documents,
making this model more closely related to retrieval as an interactive precess
between the user and the IR system. This is exploited by the relevance feedback
technique. However, we need to be able to use this model even in the absence of
relevance information from the user, as in the predictive case, when the user (cid:12)rst
submit his query. In the next section we will see how probabilistic retrieval work
in this case.
2.2 Term weighting schemas
In the previous section we have described the probabilisticretrieval model known
as Binary Independence Retrieval model. In that model there are two important
quantities that need to be explicated in order to use it in the predictive case:
1. the component xj of the document vector representation ~xj;
2. the term relevance weight (TRW).
Intheprevioussectionwehaveassumedabinarydocumentvectorrepresentation,
that is, a document is represented by a vector whose values are zeros and ones
depending if the feature represented by the position of the element of the vector
is present or absent from the document. If document features are terms (or
keywords), this indexing schema is very simplistic, since it does not take into
9