Musical Information Dynamics as Models of Auditory Anticipation

42 Pages·2009·3.69 MB·English

by  Shlomo Dubnov

Musical Information Dynamics as Models of Auditory Anticipation
Shlomo Dubnov
Department of Music, University of California in San Diego
[email protected]

General:
The research on modeling musical cognition involves multiple perception modalities, with
expectancies playing one of the central roles in shaping the experience of musical
structure. An idea put forward many years ago by music theorists such as Meyer and
Narmour states that listening to music consists of forming expectations and continual
fulfillment or denial thereof (Narmour 1990, Meyer 1956). Recently several information
theoretic measures of audio structure have been proposed in attempt to characterize
musical contents according to its predictive structure (Dubnov, 2006; Dubnov et al. 2006;
Potter et al., 2007; Abdallah and Plumbley 2009). These measures consider statistical
relations between past, present and future in signals, such as predictive information that
measures the mutual information between limited past and the complete future,
information rate that measures information between unlimited past and the immediate
present and predictive information rate that tries to combine past and future in view of a
known present. Additional models of musical expectancy structure build upon short-long
term memory neural networks and predictive properties that are local to a single piece
versus broader knowledge collected through corpus-based analysis.
The underlying assumption in investigation of Musical Information Dynamics is that the
changes in information content, that could be measured in terms of statistical properties
such as entropy and mutual information, correlate with musically significant events,
which in parallel could be captured by cognitive processes related to music perception
and acquired through exposure and learning of regularities present in a corpus of music
in a certain style. These models may provide an explanation for the “inverted-U”
relationship often found between simple measures of randomness (e.g. entropy rate)
and judgments of aesthetic value (Rigau et al.).

In this chapter we will explore possible relations between cognitive measures of musical
structure and statistical signal properties that are revealed through such information
dynamics analysis. The questions we will try to address are: 1) description of music as
an information source, 2) modeling of music – listener relations in terms of
communication channel, 3) choice of musical features and dealing with their
dependencies, 4) survey of different information measures for description of musical
structure and measures of shared information between listener and the music, and 5)
suggestion of new approach to characterization of listening experience in terms of
different combinations of musical surface and form expectancies.

Structure of this chapter
After a brief introduction to the theories of expectancy in music and some historical as
well as modern musicological and computational background, we shall address the
question of modeling music as an information source and listener as information
receiver. From this approach we will develop a model of listening that is based on mutual
information between the past and the present called Information Rate (IR) (Dubnov,
2006; Dubnov et al. 2006). This model will be extended to include predictive information
(Bialek et al.) and predictive information rate (Abdallah and Plumbley, 2009).

We introduce a new notion of information gap as a measure of the salience of a present
musical segment (instance in time) with respect to future and past of a musical signal.
This measure combines notions of predictive information with a notion of momentary
forgetfulness, determining saliency of the present instance in terms of how detrimental
forgetfulness is on the ability to make predictions. We will show that the information gap
unites the three notions of information dynamics through simple algebraic relations. Next
we will consider application of IR for simple Markov chains (Cover and Thomas, 2006),
and consider actual musical data from MIDI representation and using cepstral audio
features from recordings. Dealing with multiple features requires factorization, which
establishes the basis for vector IR.

Information theoretic, statistical or corpus based approaches rely on adaptive
capabilities of the listening apparatus to extract rules from data rather then being
explicitly coded by an expert. We will discuss the question of symbolic rules versus
learning algorithms in the context of neural network model (Potter et al., 2007) that
creates melodic pitch expectancies for minimalist music using Short and Long Term
models. Short term affective and long term cognitive familiarity features will be discussed
in relation to spectral feature expectancies and spectral centroids repetitions (Dubnov
McAdams and Reynolds, 2006) for a contemporary large orchestral piece. We will
discuss model-based Bayesian approaches to surprise and salience detection in non
musical cases, such as Bayesian surprise in images (Itti and Baldi, 2005).

These experiments suggest that a principled approach is needed for combination of
expectancies from surface and structural predictions. Surface predictions are finding
regularities in the data in a stationary regime using information in the immediate past.
When used in a sliding window method, an anticipation profile captures changes in
surface expectancy over a complete musical piece. Clustering of musical features over
larger periods leads to measure of expectancies for structural repetition. This creates an
anticipation profile of expectancies for model change, capturing novelty and surprise in a
musical form. We will show how both aspects of expectancy can be derived from
Information Rate, depending on the underlying assumptions about the type of data
distribution (type of musical source) and choice of the listening mechanism that is
employed by the listener when creating these expectancies.

Expectancy in Music

Modeling musical listening in terms of information processing has to deal with modeling
of cognitive functions that try to ``make sense" of incoming auditory data using various
cognitive processes as well as relying on natural and learned schemata (Temperley,
2004; Huron, 2006; Narmour, 1990; Meyer, 1956)

One of salient functions is forming anticipations on different time scales and structural
levels of music information. The structure of music can be seen as a complex network of
short and long-time relations between different music parameters, established through
schemata related to repetitions and variations. In order to evoke a musical experience, a
listening model needs to be able to recognize and predict such different structures in the
audio signal. Moreover, response to music is unique in the sense that in addition to
operating on specific perceptual categories such as sense of loudness, pitch,
dissonance versus consonance, recognition of rhythm, tonality, and so on, an important
aspect of musical “meaning” is related to the process of activating memories and forming
expectations in an attempt to capture temporal structural information in the musical data.
So understanding of music is ultimately linked with a specification of the listening
mechanism, such as a computer audition system, that encapsulates a set of listening
operations. The process of allocating resources during listening act considers music
itself as an organization of sounds that creates an experience through a process of
active listening and appraisal of the success in discovering that organization. Following
this line of thought, we distinguish here between two levels of listening: one of forming
expectations about the immediate musical events, which, depending on the type of
musical representation, could be notes in a musical score or spectral features in a
recording. We will term this type of information as Musical Surface, to distinguish it from
Musical Form or larger musical structure that creates another set of expectations related
to likelihood of repetition of larger blocks of sonic materials during a musical piece.

Musical theorists suggest that meaning in music is related to emotions that arise through
the process of implication-realization (Narmour 1990). In that sense, past musical
material is framing our appraisal of what comes next, such as expecting a resolution
after a melodic leap in a musical phrase. These expectations could be “natural”, i.e.
described by simple mathematical relations, or be learned from cultural conventions,
requiring modeling of complex syntax structures and be style specific. Going beyond
melodic expectations, larger scale expectancy framing can be established through
conventions of style, known to the listener through familiarity with typical patterns
(schematic expectations), large scale repetition structure typical to certain musical forms,
and even expectations framed as recognizable references to other musical works called
veridical expectations (Huron, 2006).

Communicative Listening Act

The model of musical listening that we develop comprises of a pair music-listener that
are connected through a virtual time-channel where the present or the next (future)
musical observations enters the channel at the transmitter end and the past appears on
the receiving side. This model is shown in Figure 1, and is described using the following
variables: X , X , Xfuture.
PAST PRESENT

Time “Channel” –
! ! XPRESENT Mutual Information X
PAST

Music Listener

! ! X | X X | X
FUTURE PAST ! PRESENT PAST

Figure 1. Information Dynamics are represented in terms of different mutual information
measures between Past, Present and Future.
!
!

The idea is that the receiver (listener) holds the past information, which represents both
the ability of the listener to use the information in earlier heard materials belonging to the
current musical piece and possibly employing prior musical information acquired through
training or exposure to other works in the same genre. Using the past, present and future
we define three measures of anticipation: information rate (IR) (Dubnov 2006),
predictive information (PI) (Bialek et al. 1999) and predictive information rate (PIR)
(Abdallah and Plumbley, 2009).
IR: I(X ,X )= H(X )"H(X | X )
PRESENT PAST PRESENT PRESENT PAST
PI: I (T)= lim I(X (T1),X (T))
pred FUTURE PAST
T1#$
PIR: I(X ,X | X )= H(X | X )"H(X | X ,X )
PRESENT FUTURE PAST FUTURE PAST FUTURE PRESENT PAST

It should be noted that all three measures use slightly different definitions of what Past,
! Present and Future are. In case of IR, the present is a single observation and the past is
of arbitrary length, which could be infinite. If we are observing samples
x = x ,x ,...,x and the current time is n, then the present is X = x , and the
[1:n] 1 2 n PRESENT n
past is X = x . Since the beginning of time can be earlier then the start of the
PAST [1:n"1]
present musical piece, the past can extend to minus infinity. Another extension of IR that
! !
will be considered later is allowing a buffer of samples to exit in the present. This will be
!
discussed later.
In the case of PI, there is no clear notion of present, or one might say that present is
included in the future as the first sample. What is special about PI is that it explicitly
defines the duration of the past and the future, where T1 is the extent of the future and T
is the duration of the past, relative to some zero point where the two meet. Since PI was
originally defined for continuous time, we will modify it for discrete time and define it as
! !
follows. If we assume that the present (zero time) point is n, then the past and future
become X (T)= x and X (T1)= x .
PAST [n"T:n"1] FUTURE [n:n+T1"1]
In the PIR approach, an explicit distinction is made between Past, Present and Future. In
this case the definitions of the different random variables become X = x ,
! ! PAST [1:n"1]
X = x , and X = x . It should be noted that the duration of past or
PRESENT n FUTURE [n+1:K]
future are not defined, leaving it open to interpretation of the reader, or defined according
!
to the problem at hand. Let us discuss these three measures in more detail.
! !

In the first approach the past is used to predict the present. In this model there is no
further guessing beyond the immediate sample and the overall listening act is measured
in terms of its ability to predict the present given the past over some larger time period.
Since the predictability of the present based on the past varies in time, the measure is
only segment-wise stationary. Modeling of this situation requires two variables X
PRESENT
and X that establish a sort of communication channel, where the mutual information
PAST
between X and X represents the amount of information transferred from the
PRESENT PAST
!
past to the present. The listener tries to reduce the uncertainty of X by using
PRESENT
!
information in X , and his achievement in doing so is measured in terms of relative
! ! PAST
reduction in uncertainty, calculated as a difference between the entropy of X and
! PRESENT
conditional entropy of X given X . We call this method “Information Rate” (IR)
PRESENT PAST
!
(Dubnov 2006). It should be noted that in order to account for mutual information
!
between past and the present, the measure has to take into account the listener
! !
expectations about both X and X . In other words, IR measures the average
PRESENT PAST
reduction of uncertainty between the past and the present for all realizations of both, i.e.
it takes into account the joint probability of past and present over ensemble of
! !
observations rather then being a function of a particular occurrence of the past. In
practice this means that the effect of prediction done over some short frames is
averaged over a larger macro-frame where multiple occurrences of past and present are
available. The issue concerning the differences between comparing probability
distributions of future given a particular past and measuring mutual information between
the two will be discussed again in the section about PIR below and also it will reappear
in a discussion about the difference between Bayesian surprise and average Bayesian
surprise.

In a second scenario, the listener tries to predict the entire future based on the past. It is
a more demanding task since the prediction quality quickly deteriorates as we try to
guess more then just a few next observations. One of the reasons for considering this
scenario is that it alleviates the need to define what the present is. In some sense, the
future is as long as the correlation between the past and the future allows it to be. Since
in principle the uncertainty about a big segment of sound grows proportionally with the
size of the segment, it can be shown that this view, called “Predictive Information” (PI)
(Bialek, 1999), actually reveals the so called “intensive” part of the entropy of the
available past, where the difference between intensive and extensive quantities is that
extensive quantity grows with the size of the system (like mass for instance) while
intensive properties remain constant with size (like heat). So if we write the entropy as a
function of time, H(T)= #P(x )log [P(x )], then H(T) can be expressed
[1:n+T"1] 2 [1:n+T"1]
Allrealizationsof
x ,x ,...,x
1 2 n+T"1
in terms of a combination of extensive and intensive parts as follows
H(T)=H T+H (T). Using this expression, PI can be written in terms of relation
0 1
!
between entropies of the past, present and the combined time interval
I(X (T1),X (T))=H(T)+H(T1)"H(T+T1). Since the intensive part does not
! FUTURE PAST
H (T)
grow in time, lim 1 =0, for sufficiently long future T1 the intensive parts of H(T1)
T"# T
!
and H(T+T1) cancel out I (T)= lim I(X (T1),X (T))=H (T) .
pred FUTURE PAST 1
T1"#
!
!

!
So by looking into infinite future, not only the extensive parts cancel out, but also the
!
!
intensive parts of the future disappear, leaving only the intensive part related to the past
at hand. Looking into larger blocks of future is important in the case of long correlations.
In the following we will show that in the case of low order Markov models, predictive
information in fact equals information rate, where the present is defined by the size of the
memory of the Markov process. PI measure is described in Figure 2

I(X ,X )=H(T)+H(T1)"H(T+T1).
PAST FUTURE

H(T) H(T1)
|------------------------->|-------------------------->|

!

"T 0 T1
! !
H(T+T1)

|------------------------------------------------------>|

!
!

!
Figure 2: PI reveals the intensive part of the entropy by taking the future to infinity. See
text for more detail.

A third scenario assumes that the present can be distinguished from both the past and
the future. In such a case we might think of a listener who holds past instance of the
music information in his memory. The question now becomes one about information
contents of the present segment, where it is judged in terms of how well it can predict the
entire future, considering that the past is already known. To explain this notion, we would
need to use the following relation between relative entropy, also called Kullback-Liebler
(KL) distance, and mutual information. Mutual information between two random variables
X and Y can be written in terms of KL distance as

I(X,Y)=D[P(X,Y)||P(X)P(Y)].

This can be shown to arise directly from the definitions of KL distance and definitions of
!
P(X)
entropy and conditional entropy D(X,Y)="P(X)log , H(X)="#P(X)logP(X),
P(Y)
H(X |Y)="#P(X,Y)logP(X |Y)respectively. It should be noted that conditional
entropy averages log-conditional probability of X giv!en Y on both X and Y. If instead of
!
considering every possible event Y we assume a specific outcome Y = y, conditional
!
entropy can be considered as averaging of the entropy of X for this every outcome of Y
!
H(X |Y)="#P(Y)[#P(X |Y = y)logP(X |Y = y)]= E[H(X |Y = y)] . This also
P(Y)
y x
allows writing mutual information in terms of averaging of KL between the probability of X
for specific outcome of Y and probability of X without seeing that outcome, as follows
!

P(X,Y)
I(X,Y)=D[P(X,Y)||P(X)P(Y)]="P(X,Y)log =
P(X)P(Y)

P(X |Y = y)
="P(Y)"P(X |Y = y)log = E{D[P(X |Y = y)||P(X)]}
P(X) P(Y)
y x

This can be extended also for the case of X and Y being conditional on a third variable Z.
!
Assigning times to three variables as X = X ,Y = X ,Z = X , such measure
PRESENT FUTURE PAST
becomes the instantaneous predictive information rate that measures the distortion (in
KL sense) between the future values when specific outcomes of present X and past Z
!
are known, compared to knowing only the past

I(X = x,Y |Z =z)=D[P(Y |X = x,Z =z)||P(Y |Z =z)],

PIR, defined as mutual information between present and future given the past, can be
!
obtained by averaging of instantaneous predictive information over all possible outcomes
of past and present can be obtain by averaging

I(X,Y |Z)= E{D[P(Y |X = x,Z =z)||P(Y |Z =z)]} .
P(X,Z)

It can be shown that in this scenario PIR actually represents a difference between two
!
predictive information situations – one where the past and future are adjacent to each
other, and the other where there is a gap between past and future that we call present.
We will discuss that in detail below. By doing that, we realize that PIR is a measure of
difference between two types of prediction, one that uses the present and one that skips
it to predict the future. We will use a notion of information gap to show how detrimental a
forgetfulness or absentmindedness of a listener during the period of time (the present) is
on the information that is transmitted from the past to the present. It can be shown that in
the case of simple Markov processes, there is a “sweet spot” in terms of correspondence
between predictive information rate and the entropy rate of the process, where entropy
rate is actually the lower bound on the entropy that exploits all possible knowledge about
the process that is available through knowledge of the Markov transition matrix. So in
cases where the structure is very loose or very strong, i.e. when entropy rate is close to
zero or close to maximum, PIR is low, and a peak occurs at some intermediate values
where many but not all continuations are possible for the different steps. The intuition
behind this measure is that forgetting one step in such scenario is detrimental in the mid-
range, while for totally random or totally predictive processes being absent minded for a
moment does not have much effect on the ability to predict the future (zero in one and
perfect prediction in the second extreme case).

Informational Aesthetics

Eighty years ago, Birkhoff formalized the notion of beauty by introducing the aesthetic
measure (M), defined as the ratio between order and complexity. According to this
measure, aesthetic feelings originate from discovery of some harmonious interrelations
inside an object, dependent on its initial complexity. He identified three successive
phases in the aesthetic experience related to a preliminary effort of attention, which
increases proportionally to the objectʼs complexity, and eventually the feeling of value or
aesthetic measure coming from this effort through discovery of structures such as
harmony, symmetry, or order factors which seems to be necessary for evoking of an
aesthetic effect. This measure was later formulated in terms of information theory as
follows (Bense 1969) using the notion of compressed versus uncompressed
representation. Complexity is measured in terms of uncompressed data size, or
unconditional entropy, while order becomes the difference between this and size of data
when compression is applied to it. We will modify this idea and write instead of
compression an expression for entropy conditional on some past knowledge. In such a
case the past is used to “compress” the present, or render it into more compact
representation. This can be written as follows, with X representing the present and Z the
prior knowledge.