Computational Methods for the Alignment and Score-Informed Transcription of Piano Music

152 Pages·2017·8.48 MB·English

1
Computational Methods for the Alignment
and Score-Informed Transcription of Piano
Music
By
Siying WANG
Submittedinpartialfulfilmentoftherequirements
oftheDegreeofDoctorofPhilosophy
London,UK
November6,2017
AUTHOR’S DECLARATION
I,SiyingWang,confirmthattheresearchincludedwithinthisthesisismyown
workorthatwhereithasbeencarriedoutincollaborationwith,orsupported
by others, that this is duly acknowledged below and my contribution indi-
cated.Previouslypublishedmaterialisalsoacknowledgedbelow.
IattestthatIhaveexercisedreasonablecaretoensurethattheworkisorig-
inal, and does not to the best of my knowledge break any UK law, infringe
anythirdparty’scopyrightorotherIntellectualPropertyRight,orcontainany
confidentialmaterial.
IacceptthattheCollegehastherighttouseplagiarismdetectionsoftwareto
checktheelectronicversionofthethesis.
Iconfirmthatthisthesishasnotbeenpreviouslysubmittedfortheawardofa
degreebythisoranyotheruniversity.
The copyright of this thesis rests with the author and no quotation from it
or information derived from it may be published without the prior written
consentoftheauthor.
SIGNATURE: .......................................... DATE: ..........................................
Detailsofcollaborationandpublications:SeeSection1.3.
2
ABSTRACT
Thisthesisisconcernedwithcomputationalmethodsforalignmentandscore-informed
transcriptionofpianomusic. Firstly,severalmethodsareproposedtoimprovethealign-
mentrobustnessandaccuracywhenvariousversionsofonepieceofmusicshowcomplex
differenceswithrespecttoacousticconditionsormusicalinterpretation.Secondly,score
toperformancealignmentisappliedtoenablescore-informedtranscription.
Although music alignment methods have considerably improved in accuracy in re-
centyears,thetaskremainschallenging. Theresearchinthisthesisaimstoimprovethe
robustnessforsomecaseswheretherearesubstantialdifferencesbetweenversionsand
state-of-the-artmethodsmayfailinidentifyingacorrectalignment. Thisthesisfirstex-
ploitstheavailabilityofmultipleversionsofthepiecetobealigned. Byprocessingthese
jointly,thealignmentprocesscanbestabilisedbyexploitingadditionalexamplesofhow
asectionmightbeinterpretedorwhichacousticconditionsmayarise. Twomethodsare
proposed, progressivealignmentandprofileHMM,bothadaptedfromthemultiplebi-
ologicalsequencealignmenttask. Experimentsdemonstratethatthesemethodscanin-
deedimprovethealignmentaccuracyandrobustnessovercomparablepairwisemethods.
Secondly,thisthesispresentsascoretoperformancealignmentmethodthatcanimprove
therobustnessincaseswheresomemusicalvoices,suchasthemelody,areplayedasyn-
chronouslytoothers–astylisticdeviceusedinmusicalexpression.Theasynchroniesbe-
tweenthemelodyandtheaccompanimentarehandledbytreatingthevoicesasseparate
timelinesinamulti-dimensionalvariantofdynamictimewarping(DTW).Themethod
measurably improves the alignment accuracy for pieces with asynchronous voices and
preservestheaccuracyotherwise.
Onceanaccuratealignmentbetweenascoreandanaudiorecordingisavailable,the
scoreinformationcanbeexploitedaspriorknowledgeinautomaticmusictranscription
(AMT),forscenarioswherescoreisavailable,suchasmusictutoring.Score-informeddic-
tionarylearningisusedtolearnthespectralpatternofeachpitchthatdescribestheenergy
distributionoftheassociatednotesintherecording.Moreprecisely,thedictionarylearn-
ingprocessinnon-negativematrixfactorization(NMF)isconstrainedusingthealigned
score. Thisway, byadaptingthedictionarytoagivenrecording, theproposedmethod
improvestheaccuracyoverthestate-of-the-art.
3
ACKNOWLEDGEMENTS
Firstandforemost,IwouldliketothankmytwowonderfulPhDsupervisors-Sebastian
EwertandSimonDixon.Ihavebeenfeelingextremelyluckysincethefirstdaytheyagreed
tosuperviseme. Backthen,Iwasconfusedwithmyresearchtopicandhadstrongself-
doubt. I simply would not be able to accomplish the work in this thesis without their
invaluableguidanceandencouragement,supportandfaithinme. Theyshapedmyun-
derstandingofgoodresearchandshowedmethespiritofscientists. Iespeciallywantto
thankSebastianforalwaysmakingthetimetodiscusswithme,despitebeingverybusy
himself.
I want to express my gratitude to Professor Elaine Chew, for being my female role
model. IwasinspiredbyElaine’sattitudetowardresearchandlifeingeneralandheref-
fortsingenderequality.
ThefouryearsIspentinQueenMaryhavealwaysbeenenjoyable,thankstomydear
friendshere. EspeciallyYading,Chunyang,ChrisandShan. Iwillneverforgetthelaugh
andtearswesharetogether,thedaysandeveningswespenttogetherworkingandchat-
ting.IreallyappreciatethatyouarealwaystherewheneverIneed,offeringhelpandsug-
gestionsorsimplylistening. IwouldalsoliketothankMaria, Veronica, Mi, Tian, Beici,
Simin,Katerina,Sid,Peter,Di,Mattias,Janis,Luwei,Matt,Eitaforbeinggreatlab-mates
andfriends.
ThanksforStellaSickandWernerGoeblforkindlyprovidingthedataset.
Lastbutnotleast, mydeepestthanksgotomyfamilyfortheirunwaveringsupport
andtrust. Thankstomyparentsforunderstandingandsupportingmeinpursuingmy
dreamfarfromhome.ThankstoFranforyourenduringtrust,encouragementandcaring
forme, forkeepingmephysicallyandmentallyhealthy, sometimessimplybydragging
meoutfromworktoawalk,alsoforproofreadingmythesis,discussingwithmeaboutmy
researchfromtimetotimeandalwaysprovidinggoodadvices.
ThisworkwasfundedbytheChinaScholarshipCouncil.
4
TABLE OF CONTENTS
Author’sDeclaration 2
Abstract 3
Acknowledgements 4
TableofContents 8
ListofTables 9
ListofFigures 10
ListofSymbols 13
1 Introduction 15
1.1 MotivationandGoals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2 ThesisStructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.3 AssociatedPublications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2 Background 22
2.1 MusicTerminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.1.1 MusicalScore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.1.2 MusicalPerformancesandExpression . . . . . . . . . . . . . . . . . 24
2.1.3 MIDINotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2 FeatureRepresentationinMusicAlignment. . . . . . . . . . . . . . . . . . . 25
2.2.1 Chroma-basedFeature . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5
TABLEOFCONTENTS 6
2.2.2 Decaying Locally adaptive Normalised Chroma Onset (DLNCO)
Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.3 AlignmentMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.3.1 DynamicProgramming . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.3.2 ProbabilisticModelling . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.3.3 Similarity between Dynamic Time Warping and Hidden Markov
Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.3.4 OtherTechniquesusedinMusicAlignment . . . . . . . . . . . . . . 44
2.4 ExploitingScoreInformationusingAlignmentTechniquesinMIR . . . . . 44
2.4.1 Score-informedExpressiveParameterExtraction . . . . . . . . . . . 44
2.4.2 Score-informedSourceSeparation . . . . . . . . . . . . . . . . . . . 45
2.4.3 Score-informedTranscription . . . . . . . . . . . . . . . . . . . . . . 46
2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3 Robustjointalignmentofmultipleperformances 50
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3 BackgroundonMultipleSequenceAlignment . . . . . . . . . . . . . . . . . 52
3.4 MethodsforJointMusicAlignment . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.1 ProgressiveAlignment . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.2 ProbabilisticProfile . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.4.3 AcceleratingAlignmentsUsingMulti-ScaleDynamicProgramming 66
3.5 ComparingPairwise,ProgressiveandProfile-HMMBasedAlignment . . . 67
3.5.1 DatasetandSettings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.5.2 ComparisonBetweenthePairwiseandJointAlignments . . . . . . 70
3.5.3 ComparisonoftheTwoJointAlignmentMethods. . . . . . . . . . . 73
3.6 FurtherInvestigationsoftheJointAlignmentMethod . . . . . . . . . . . . . 74
3.6.1 SubsetExperiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.6.2 GapPenalty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.6.3 AlignmentOrder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.6.4 ViterbiTraining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.6.5 IterativeAlignment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
TABLEOFCONTENTS 7
3.6.6 Furtherevaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4 Compensating For Asynchronies Between Musical Voices In Score-
PerformanceAlignment 84
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.3 AlignmentMethod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.3.1 ComputingFeaturesforIndividualVoices . . . . . . . . . . . . . . . 88
4.3.2 Three-DimensionalDynamicTimeWarping. . . . . . . . . . . . . . 88
4.3.3 PathConstraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.4.1 DataSet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.4.2 GroundTruthGeneration . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.4.3 EvaluationMeasure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.5 ConclusionandFutureWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5 Identifying Missing and Extra Notes in Piano Recordings Using Score-
InformedDictionaryLearning 100
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.3 BaselineMethod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.4 AnalysisandExtensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.4.1 ExampleofFailureUsingtheBaselineMethod . . . . . . . . . . . . 111
5.4.2 FromSpectralTemplatestoTime-FrequencyPatterns . . . . . . . . 113
5.4.3 FromHardtoSoftConstraintRegions. . . . . . . . . . . . . . . . . . 114
5.4.4 EncouragingTemporalContinuityinA . . . . . . . . . . . . . . . . . 115
5.4.5 EncouragingEnergyDecayintheTemplateMatrix . . . . . . . . . . 116
5.4.6 ParameterEstimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.5.1 Dataset&EvaluationMeasure . . . . . . . . . . . . . . . . . . . . . . 118
TABLEOFCONTENTS 8
5.5.2 InfluenceofIndividualParameters . . . . . . . . . . . . . . . . . . . 120
5.5.3 ComparisonBetweentheBaselineMethodandExtendedMethod. 124
5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6 Conclusion 127
6.1 SummaryofContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.2 FutureDirections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.3 ClosingRemarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
References 133
LIST OF TABLES
2.1 AnexampleofLCSalignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1 ChopinMazurkasandtheiridentifiersusedintheexperiments . . . . . . . . . 69
3.2 Alignmenterror(meanandstandarddeviationofaveragebeatdeviationinmil-
liseconds)forfourtypesofalignmentmethodsandarandombaseline. . . . . 71
3.3 Comparing the Pairwise II alignment method (Ewert et al., 2009b)), Profile
HMMandProgressiveAlignmentmethodsintermsofaveragenoteonsetde-
viation(inmilliseconds) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.1 Experimentalresultsforthreeexcerptsplayedwithstrongasynchrony(upper)
andthreepieceswithoutasynchrony(lower). Thistableshowsthenumberof
performancesavailableandstatisticsofthealignmenterrorinmillisecondsfor
therespectivepieces.Bothresultsforthe2D-DTW(Ewertetal.,2009b)andour
3D-DTWalignmentmethodarecomputedseparatelyforthemelody(Mel)and
accompaniment(Acc). Theerrorvaluesofthesetwovoicesareaveragedover
the number of notes to get the overall (OA) alignment error. The change in
alignmenterrorachievedby3D-DTWisshowninparentheses. . . . . . . . . . 96
5.1 Piecesforevaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.2 AverageEvaluationResultsofthreeMethodsforCorrectlyPlayedNotes(C),Ex-
traNotes(E)andMissingNotes(M). . . . . . . . . . . . . . . . . . . . . . . . . . 125
9
LIST OF FIGURES
2.1 Anexampleofamusicalscore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2 Generalframeworkofmusicalignment . . . . . . . . . . . . . . . . . . . . . . . 26
2.3 SolvingtheDynamicProgrammingproblemwithalook-uptable . . . . . . . . 31
2.4 AnexampleofDTWalignmentfortwoperformanceexcerptsofChopinOp.24
No.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.5 MultiscaleDTW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.6 AnexampleofHMMmodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.7 Non-NegativeMatrixFactorisation(NMF) . . . . . . . . . . . . . . . . . . . . . . 45
3.1 Areal-worldexampleofpairwisemethodfailingtocomputeracorrectalignment 51
3.2 Anexampleofprogressivealignment . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.3 Anexampleofreplacingcopiesofelementswithgapsymbolsafteraligninga
newsequenceX tothetemplateF,whereC(3,2)<C(2,2)andC(5,5)<C(5,4). 59
3.4 TopologyofaProfileHMM(Durbinetal.,1999),showingrowsofdeletestates(top),
insertstates(middle)andmatchstates(bottom) . . . . . . . . . . . . . . . . . . . . . 60
3.5 AnexampleofProfileHMM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.6 ComparisonofthePairwiseIIalignmentmethod(Ewertetal.,2009b)withthe
proposedprogressivealignmentmethodandprofileHMMmethod. . . . . . . 71
3.7 HistogramsofbeatdeviationusingthePairwiseIIalignmentmethod(Ewert
etal.,2009b),theprogressivealignmentandprofileHMMmethod. . . . . . . . 73
3.8 Comparison between the Pairwise II alignment method (Ewert et al., 2009b)
andtwojointalignmentmethodsforsubsetexperiments. . . . . . . . . . . . . 75
10