Ami Pro - TITLEPG.SAM

257 Pages·1998·0.69 MB·English

A Probabilistic Method for Tracking a Vocalist
Lorin V. Grubb
September 1998
CMU-CS-98-166
School of Computer Science
Computer Science Department
Carnegie Mellon University
Pittsburgh, PA
Thesis Committee
Roger B. Dannenberg, Chair
Tom Mitchell
Jack Mostow
Shuji Hashimoto, Department of Applied Physics, Waseda University, Tokyo
Submitted in partial fulfillment of the requirements
for the Degree of Doctor of Philosophy
© Copyright 1998 Lorin V. Grubb
Keywords: Computer Music, Stochastic Processes, Signal Processing, Singing, Musical
Accompaniment, Statistical Modeling, Automated Accompaniment
Abstract
When a musician gives a recital or concert, the music performed generally includes
accompaniment. To render a good performance, the soloist and the accompanist must know the musical
score and must follow the other musician's performance. Both performing and rehearsing are limited by
constraints on the time and money available for bringing musicians together. Computer systems that
automatically provide musical accompaniment offer an inexpensive, readily available alternative.
Effective computer accompaniment requires software that can listen to live performers and follow along in
a musical score.
This work presents an implemented system and method for automatically accompanying a singer
given a musical score. Specifically, I offer a method for robust, real-time detection of a singer's score
position and tempo. Robust score following requires combining information obtained both from
analyzing a complex signal (the singer's performance) and from processing symbolic notation (the score).
Unfortunately, the mapping from the available information to score position does not define a function.
Consequently, this work investigated a statistical characterization of a singer's score position and a model
that combines the available musical information to produce a probabilistic position estimate. By making
careful assumptions and estimating statistics from a set of actual vocal performances, a useful
approximation of this model can be implemented in software and executed in real time during a musical
performance.
As part of this project, a metric was defined for evaluating the system's ability to follow a singer.
This metric was used to assess the system's ability to track vocal performances. The presented evaluation
includes a characterization of how tracking ability can be improved by using several different
measurements from the sound signal rather than only one type of measurement. Examined measurements
of the sound signal include fundamental pitch, spectral features dependent upon the score's phonetic
content, and amplitude changes correlated with the start of a musical note. The evaluation results
demonstrate how incorporating multiple measurements of the same signal can improve the accuracy of
performance tracking, for individual performances as well as on average. Overall improvement of the
performance tracking system through incremental specification, development, and evaluation is facilitated
by the formal statistical approach to the problem.
Acknowledgments
First and foremost, I would like to thank my advisor, Roger Dannenberg, for both his assistance
and his support, as well as for his great patience throughout the life span of this project. Thanks are also
due to the members of my thesis committee—Shuji Hashimoto, Jack Mostow and Tom Mitchell—for their
numerous insights and helpful comments as well as their general interest in this work. I am also grateful
for the many helpful discussions with other faculty and students in the Department of Computer Science
including John Lafferty, Rich Stern, Lonnie Chrisman, Sven Koenig, Ravi Mosur, Eric Thayer and
Belinda Thom.
I am indebted to numerous faculty and students in the Department of Music at Carnegie Mellon
University. My efforts would have been far less fruitful without their support for this project and their
willingness to devote generous amounts of time to participate in the conducted studies. Special thanks are
due to Anne Elgar Kopta, Robert Fire and Geeta Bhatnagar.
Finally, I wish to thank my parents for their encouragement and support during my extensive
time as a student in the areas of both science and music.
Contents
List of Figures ................................................................................. vii
List of Tables ................................................................................... ix
Chapter 1: The Problem of Vocal Performance Tracking
1.1 Musical Performance and Computers ....................................................... 1
1.2 The Problem of Automated Musical Accompaniment ....................................... 4
1.3 Summary of Previous Work ................................................................ 7
1.4 Limitations of Previous Score Following Approaches ...................................... 10
1.5 Problems with Tracking Vocal Performances .............................................. 12
1.6 Stochastic Score Following for Automated Accompaniment ................................ 15
Chapter 2: A Model for Stochastic Score Following
2.1 Motivation ................................................................................ 17
2.2 Definition of a Score ...................................................................... 18
2.3 Definitions and Notation ................................................................... 20
2.4 A Probabilistic Description of Score Position .............................................. 22
2.5 A General Model for Stochastic Score Following .......................................... 24
2.6 An Independence Assumption ............................................................. 27
2.7 An Assumption of Fixed Conditioning Variables ........................................... 30
2.8 A Convolution Assumption ................................................................ 31
2.9 A Discrete Approximation and Implementation ............................................ 33
2.10 A General Error Analysis of the Models .................................................. 43
2.11 The Process of Determining Distributions ................................................ 52
2.12 Comments on the Validity of the Model .................................................. 53
Chapter 3: A Stochastic Model of Motion under Positive-valued Rate
3.1 Motivation ................................................................................ 55
3.2 Properties of Motion under Positive-Valued Rate .......................................... 55
3.3 Estimating the Distance Density Using Actual Vocal Performances ........................ 61
3.4 A Model Providing Consistent Estimation When Elapsed Time Varies ..................... 70
3.5 Implementing the Model of Motion ........................................................ 77
vi
3.6 Comments on Assumptions of the Score-following Model ................................. 85
Chapter 4: Events Based on Observing Fundamental Pitch
4.1 Defining Events and Modeling Observations ............................................... 88
4.2 Fundamental Pitch in Vocal Performances ................................................. 91
4.3 Methods of Pitch Detection ................................................................ 94
4.4 A Model of Observed Fundamental Pitch ................................................. 102
4.5 Summary of Observations Based on Fundamental Pitch ................................... 107
Chapter 5: Events Based on Observing Spectral Envelope
5.1 Extending the Tracking Model for Multiple, Simultaneous Observations .................. 113
5.2 Vowel Detection and Measuring Spectral Envelope ...................................... 115
5.3 Spectral Envelope in Vocal Performances ................................................ 123
5.4 A Model of Spectral Envelope ........................................................... 126
5.5 Summary of Observations Based on Spectral Envelope ................................... 140
Chapter 6: Events Based on Observing Note Onsets
6.1 Measuring Changes in Amplitude and Detecting Note Onsets ............................. 143
6.2 Onset Detection for Vocal Performances .................................................. 146
6.3 A Model of Detected Note Onsets ........................................................ 148
6.4 Improved Position Estimates Using Multiple Observation Types .......................... 153
Chapter 7: Application and Evaluation of the Stochastic Score-following Model
7.1 Overview ................................................................................. 167
7.2 The Automated Accompaniment System .................................................. 168
7.3 The Objectives of Evaluation and the Metrics Applied .................................... 176
7.4 Evaluation Using Recorded Performances ................................................ 179
7.5 Evaluation Using Live Performances ..................................................... 193
7.6 Comparison of Expected and Actual Tracking Ability ..................................... 203
7.7 Summary and Conclusions ................................................................ 211
Chapter 8: Related Techniques and Conclusions
8.1 Summary of the Stochastic Approach to Score Following ................................. 213
8.2 Related Statistical Techniques ............................................................ 214
8.3 Conclusions and Contribtions ............................................................. 231
Bibliography ................................................................................... 235
Appendix: Musical Works Included in this Study .................................... 246
Figures
1-1 Excerpt from a musical score of a piece for voice and piano ..................................... 2
1-2 Simple block diagram of an automated accompaniment system .................................. 3
1-3 Performed pitch versus score pitch for a vocal performance .................................... 12
1-4 Sound waveform during a single phrase from a vocal performance ............................. 14
2-1 Example of a density function characterizing the score position of a performer ................. 22
2-2 Odds that the performer is in one region of the score versus another region of the score ......... 23
2-3 Depiction of the convolution assumption ....................................................... 32
2-4 Depiction of a single application of the final stochastic score-following model .................. 33
2-5 Sampling a density function that characterizes the score position of a performer ................ 34
2-6 Rectangular integration of a density function ................................................... 35
2-7 Flowchart for direct implementation of the discrete model ...................................... 36
2-8 Flowchart for implementing the discrete model by using the fast Fourier transform ............. 37
2-9 Cyclic convolution as results from executing discrete convolution via the Fourier transform .... 38
2-10 Extending f with zero-valued samples ...................................................... 39
Prior
2-11 Using a window size of less than 2S for convolution via the Fourier transform ................ 40
2-12 Depiction of windowing the score ............................................................ 42
2-13 Effecting the window movement by shifting one function prior to convolution ................ 42
2-14 Possible sources of error introduced by transitioning between the various models ............. 43
2-15 Sampling the same function using two different sample intervals .............................. 45
2-16 How sharp observation densities can dominate ................................................ 51
3-1 Calculating multiple actual rates for each predicted rate that is calculated ....................... 64
3-2 Graph of the means of the lognormal models fit to the rate pair data ............................ 66
3-3 Two graphs examining a simple regression line fit to the inverse variances ..................... 67
3-4 Normal curves fit to rate ratio data sets using the revised regression line ........................ 78
viii
4-1 Graphs of a periodic signal and the amplitude from the associated spectrum .................... 95
4-2 A bank of filters whose cutoff frequencies are spaced by half-octave intervals .................. 99
4-3 Block diagram of the pitch detector used for this project ....................................... 102
4-4 Distribution of the difference between detected pitch and the pitch in the musical score ........ 105
4-5 Stochastic score-following model using pitch, rate, elapsed time, and source position ......... 109
5-1 Spectra of two vowels, [U] and [a], sung with a fundamental frequency around 300 Hz ........ 116
5-2 Two log spectra calculated from the same 33 ms of signal from the vowel [U] ................. 121
5-3 Three graphs showing different versions of the spectrum for the same portion of signal ........ 121
5-4 Block diagram of the spectral envelope detector used for this project .......................... 122
5-5 Average log power spectra for the vowel [u] sung with four different fundamentals ............ 125
5-6 Distributions over the vector quantization codebook entries for all vowels ..................... 133
5-7 Distributions over the vector quantization codebook entries for all consonants ................. 135
5-8 Stochastic score-following model using fundamental pitch, spectral envelope, rate, etc. ........ 142
6-1 Probability of observing an onset based on transition class and number of voiced consonants .. 151
6-2 The observation density function, f(v| i), based on observing "E", [ ]e and an onset ............. 156
6-3 The observation density function, f(v| i), based on observing "Eb", [ ]e and no onset ............ 156
6-4 The observation density function, f(v| i), based on observing "D", [ ]e and no onset ............. 157
7-1 Diagram of signal and data flow in the automated accompaniment system ..................... 169
7-2 Stochastic score-following model using pitch, spectral envelope, note onsets, rate, etc. ........ 170
7-3 Possible sources of error introduced by transitioning between models ......................... 177
7-4 Histograms of the time differences used to assess tracking accuracy ........................... 182
7-5 Time differences for estimated score positions in two trials with recorded performances ....... 187
7-6 Histograms of synchronization and tracking time differences for the same performance ........ 188
7-7 Exponential regression curves based on E[k ] for the recorded performances ................... 208
7-8 Exponential regression curves based on E[k ] for the live performances ........................ 209
7-9 Exponential regression curves based on E[k ] for the live and recorded performances .......... 210
Tables
1-1 Summary statistics for syllable, word, and melodic interval content of vocal scores ............. 13
3-1 Statistics for samples of ln R – ln R .......................................................... 65
D T C
3-2 K-S statistics for normal curves compared to samples of ln R – ln R .......................... 68
D T C
3-3 Results of numerical convolution using the lognormal density for D T = 600 and R = 1 ......... 74
C
3-4 Results of numerical convolution using the lognormal density from the revised regression line .. 75
3-5 Results of numerical convolution using the lognormal density for D T = 100 and R = 1 ......... 76
C
3-6 S.D. estimates from results from numerical convolution ........................................ 76
3-7 Relative errors for numerical self-convolution of the lognormal density for D T=100 and R =1 .. 81
C
3-8 Results of numerical integration of the lognormal density for D T = 100 and R = 1 .............. 82
C
4-1 Probability for Offset = v – ScoredPitch (i) approximated from detected pitch sequences ..... 106
5-1 Vowels and diphthongs that were modeled for each language .................................. 128
5-2 Consonants that were modeled for each language .............................................. 129
5-3 Estimated likelihood (percent) of confusing the seven vowels in Italian ........................ 136
5-4 Estimated likelihood (percent) of confusing the monophthongs in English ..................... 136
5-5 Average duration of individual phonemes in the recorded performances ....................... 139
6-1 Probability of observing an onset based on transition class and number of voiced consonants .. 150
6-2 Results of calculating equation 6-1 for various values of n and p ............................... 159
7-1 Mean and SD of time differences for recorded performances .................................. 183
7-2 Mean and SD of time differences for recorded performances, 5% outliers discarded ........... 185
7-3 Results of paired comparisons tests for recorded performances ................................ 186
7-4 Mean and SD of synchronization time differences for recorded performances .................. 189
7-5 Causes of extreme time differences in the trials using all observations ......................... 190
7-6 SD of tracking time differences using different point estimates of score position ............... 192
x
7-7 Ordering of the three trials for each performer when tracking live performances ............... 196
7-8 Mean and SD of tracking time differences for live performances .............................. 197
7-9 Results of paired comparisons tests for live performances ..................................... 198
7-10 Mean and SD of synchronization time differences for live performances ..................... 199
7-11 Performers' subjective assessments of best and worst accompaniment performances .......... 200
7-12 SD of tracking time differences for recordings of live performances .......................... 202
7-13 Correlations for SD and expected likelihood of confusion .................................... 205
7-14 Correlations for SD and expected likelihood of confusion, per observation types used ........ 206