Visual Perception of Music Notation: On-Line and Off-Line Recognition

373 Pages·2004·14.208 MB·English

by  Susan Ella George, David Sammon

Visual Perception of
Music Notation:
On-Line and Off-Line
Recognition
Susan E. George
University of South Australia, Australia
IDEA GROUP PUBLISHING
VVVVViiiiisssssuuuuuaaaaalllll PPPPPeeeeerrrrrccccceeeeeppppptttttiiiiiooooonnnnn ooooofffff
MMMMMuuuuusssssiiiiiccccc NNNNNoooootttttaaaaatttttiiiiiooooonnnnn:::::
OOOOOnnnnn-----LLLLLiiiiinnnnneeeee aaaaannnnnddddd OOOOOffffffffff-----LLLLLiiiiinnnnneeeee
RRRRReeeeecccccooooogggggnnnnniiiiitttttiiiiiooooonnnnn
Susan E. George
University of South Australia, Australia
IRM Press
Publisher of innovative scholarly and professional information
technology titles in the cyberage
Hershey • London (cid:127) Melbourne (cid:127) Singapore
Acquisitions Editor: Mehdi Khosrow-Pour
Senior Managing Editor: Jan Travers
Managing Editor: Amanda Appicello
Development Editor: Michele Rossi
Copy Editor: Michelle Wilgenburg
Typesetter: Amanda Appicello
Cover Design: Lisa Tosheff
Printed at: Integrated Book Technology
Published in the United States of America by
IRM Press (an imprint of Idea Group Inc.)
701 E. Chocolate Avenue, Suite 200
Hershey PA 17033-1240
Tel: [phone]
Fax: [phone]
E-mail: [email protected]
Web site: http://www.irm-press.com
and in the United Kingdom by
IRM Press (an imprint of Idea Group Inc.)
3 Henrietta Street
Covent Garden
London WC2E 8LU
Tel: [phone]
Fax: [phone]
Web site: http://www.eurospan.co.uk
Copyright © 2005 by IRM Press. All rights reserved. No part of this book may be
reproduced in any form or by any means, electronic or mechanical, including
photocopying, without written permission from the publisher.
Library of Congress Cataloging-in-Publication Data
George, Susan Ella.
Visual perception of music notation : on-line and off-line recognition
/ Susan Ella George.
p. cm.
Includes bibliographical references and index.
ISBN 1-931777-94-2 (pbk.) -- ISBN 1-931777-95-0 (ebook)
1. Musical notation--Data processing. 2. Artificial
intelligence--Musical applications. I. Title.
ML73.G46 2005
780'.1'48028564--dc21
2003008875
ISBN 1-59140-298-0 (h/c)
British Cataloguing in Publication Data
A Cataloguing in Publication record for this book is available from the British Library.
All work contributed to this book is new, previously-unpublished material. The views
expressed in this book are those of the authors, but not necessarily of the publisher.
New Releases from IRM Press
• Visual Perception of Music Notation: On-Line and Off-Line Recognition
Susan Ella George
ISBN: 1-931777-94-2; eISBN: 1-931777-95-0 / © 2004
(cid:127) 3D Modeling and Animation: Synthesis and Analysis Techniques for the
Human Body
Nikos Sarris & Michael G. Strintzis
ISBN: 1-931777-98-5; eISBN: 1-931777-99-3 / © 2004
(cid:127) Innovations of Knowledge Management
Bonnie Montano
ISBN: 1-59140-229-8; eISBN: 1-59140-230-1 / © 2004
(cid:127) e-Collaborations and Virtual Organizations
Michelle W. L. Fong
ISBN: 1-59140-231-X; eISBN: 1-59140-232-8 / © 2004
(cid:127) Information Security and Ethics: Social and Organizational Issues
Marian Quigley
ISBN: 1-59140-233-6; eISBN: 1-59140-234-4 / © 2004
(cid:127) Issues of Human Computer Interaction
Anabela Sarmento
ISBN: 1-59140-235-2; eISBN: 1-59140-236-0 / © 2004
(cid:127) Instructional Technologies: Cognitive Aspects of Online Programs
Paul Darbyshire
ISBN: 1-59140-237-9; eISBN: 1-59140-238-7 / © 2004
(cid:127) E-Commerce and M-Commerce Technologies
P. Candace Deans
ISBN: 1-59140-239-5; eISBN: 1-59140-240-9 / © 2004
Excellent additions to your institution’s library!
Recommend these titles to your Librarian!
To receive a copy of the IRM Press catalog, please contact
1/717-533-8845, fax [phone],
or visit the IRM Press Online Bookstore at: [http://www.irm-press.com]!
Note: All IRM Press books are also available as ebooks on netlibrary.com as well as other
ebook sources. Contact Ms. Carrie Skovrinskie at [[email protected]] to receive
a complete list of sources where you can obtain ebook information or
IRM Press titles.
VVVVViiiiisssssuuuuuaaaaalllll PPPPPeeeeerrrrrccccceeeeeppppptttttiiiiiooooonnnnn ooooofffff
MMMMMuuuuusssssiiiiiccccc NNNNNoooootttttaaaaatttttiiiiiooooonnnnn:::::
OOOOOnnnnn-----LLLLLiiiiinnnnneeeee aaaaannnnnddddd OOOOOffffffffff-----LLLLLiiiiinnnnneeeee RRRRReeeeecccccooooogggggnnnnniiiiitttttiiiiiooooonnnnn
TTTTTaaaaabbbbbllllleeeee ooooofffff CCCCCooooonnnnnttttteeeeennnnntttttsssss
Preface.......................................................................................vi
Susan E. George, University of South Australia, Australia
SSSSSeeeeeccccctttttiiiiiooooonnnnn 11111::::: OOOOOffffffffff-----LLLLLiiiiinnnnneeeee MMMMMuuuuusssssiiiiiccccc PPPPPrrrrroooooccccceeeeessssssssssiiiiinnnnnggggg
Chapter 1
Staff Detection and Removal .......................................................1
Ichiro Fujinaga, McGill University, Canada
Chapter 2
An Off-Line Optical Music Sheet Recognition ...............................40
Pierfrancesco Bellini, University of Florence, Italy
Ivan Bruno, University of Florence, Italy
Paolo Nesi, University of Florence, Italy
Chapter 3
Wavelets for Dealing with Super-Imposed Objects in Recognition of Music
Notation.....................................................................................78
Susan E. George, University of South Australia, Australia
SSSSSeeeeeccccctttttiiiiiooooonnnnn 22222::::: HHHHHaaaaannnnndddddwwwwwrrrrriiiiitttttttttteeeeennnnn MMMMMuuuuusssssiiiiiccccc RRRRReeeeecccccooooogggggnnnnniiiiitttttiiiiiooooonnnnn
Chapter 4
Optical Music Analysis for Printed Music Score and Handwritten Music
Manuscript..................................................................................108
Kia Ng, University of Leeds, United Kingdom
Chapter 5
Pen-Based Input for On-Line Handwritten Music Notation ............128
Susan E. George, University of South Australia, Australia
SSSSSeeeeeccccctttttiiiiiooooonnnnn 33333::::: LLLLLyyyyyrrrrriiiiiccccc RRRRReeeeecccccooooogggggnnnnniiiiitttttiiiiiooooonnnnn
Chapter 6
Multilingual Lyric Modeling and Management...............................162
Pierfrancesco Bellini, University of Florence, Italy
Ivan Bruno, University of Florence, Italy
Paolo Nesi, University of Florence, Italy
Chapter 7
Lyric Recognition and Christian Music ..........................................198
Susan E. George, University of South Australia, Australia
SSSSSeeeeeccccctttttiiiiiooooonnnnn 44444::::: MMMMMuuuuusssssiiiiiccccc DDDDDeeeeessssscccccrrrrriiiiippppptttttiiiiiooooonnnnn aaaaannnnnddddd iiiiitttttsssss AAAAAppppppppppllllliiiiicccccaaaaatttttiiiiiooooonnnnnsssss
Chapter 8
Towards Constructing Emotional Landscapes with Music ................227
Dave Billinge, University of Portsmouth, United Kingdom
Tom Addis, University of Portsmouth, United Kingdom and University of
Bath, United Kingdom
Chapter 9
Modeling Music Notation in the Internet Multimedia Age.............272
Pierfrancesco Bellini, University of Florence, Italy
Paolo Nesi, University of Florence, Italy
SSSSSeeeeeccccctttttiiiiiooooonnnnn 55555::::: EEEEEvvvvvaaaaallllluuuuuaaaaatttttiiiiiooooonnnnn
Chapter 10
Evaluation in the Visual Perception of Music.................................304
Susan E. George, University of South Australia, Australia
About the Editor .........................................................................350
About the Authors.......................................................................351
Index..........................................................................................354
vi
PPPPPrrrrreeeeefffffaaaaaccccceeeee
Overview of Subject Matter and Topic Context
The computer recognition of music notation, its interpretation and use within
various applications, raises many challenges and questions with regards to the
appropriate algorithms, techniques and methods with which to automatically
understand music notation. Modern day music notation is one of the most
widely recognised international languages of all time. It has developed over
many years as requirements of consistency and precision led to the develop-
ment of both music theory and representation. Graphic forms of notation are
first known from the 7th Century, with the modern system for notes developed
in Europe during the 14th Century. This volume consolidates the successes,
challenges and questions raised by the computer perception of this music
notation language.
The computer perception of music notation began with the field of Optical
Music Recognition (OMR) as researchers tackled the problem of recognising and
interpreting the symbols of printed music notation from a scanned image. More
recently, interest in automatic perception has extended to all components of
song lyric, melody and other symbols, even broadening to multi-lingual
handwritten components. With the advent of pen-based input systems,
automatic recognition of notation has also extended into the on-line context
— moving away from processing static scanned images, to recognising
dynamically constructed pen strokes. New applications, including concert-
planning systems sensitive to the emotional content of music, have placed new
demands upon description, representation and recognition.
Summary of Sections and Chapters
This special volume consists of both invited chapters and open-solicited
chapters written by leading researchers in the field. All papers were peer
reviewed by at least two recognised reviewers. This book contains 10 chapters
divided into five sections:
vii
Section 1 is concerned with the processing of music images, or Optical Music
Recognition (OMR). A focus is made upon recognising printed typeset music
from a scanned image of music score. Section 2 extends the recognition of
music notation to handwritten rather than printed typeset music, and also
moves into the on-line context with a consideration of dynamic pen-based
input. Section 3 focuses upon lyric recognition and the identification and
representation of conventional lyric text combined with the symbols of music
notation. Section 4 considers the importance of music description languages
with emerging applications, including the context of Web-based multi-media
and concert planning systems sensitive to the emotional content of music.
Finally, Section 5 considers the difficulty of evaluating automatic perceptive
systems, discussing the issues and providing some benchmark test data.
Section 1: Off-Line Music Processing
• Chapter 1: Staff Detection and Removal, Ichiro Fujinaga
• Chapter 2: An Off-line Optical Music Sheet Recognition,
Pierfrancesco Bellini, Ivan Bruno, Paolo Nesi
• Chapter 3: Wavelets for Dealing with Super-Imposed Objects in
Recognition of Music Notation, Susan E. George
Section 2: Handwritten Music Recognition
• Chapter 4: Optical Music Analysis for Printed Music Score and
Handwritten Music Manuscript, Kia Ng
• Chapter 5: Pen-Based Input for On-Line Handwritten Music Nota-
tion, Susan E. George
Section 3: Lyric Recognition
• Chapter 6: Multilingual Lyric Modeling and Management,
Pierfrancesco Bellini, Ivan Bruno, Paolo Nesi
• Chapter 7: Lyric Recognition and Christian Music, Susan E. George
Section 4: Music Description and its Applications
• Chapter 8: Towards Constructing Emotional Landscapes with
Music, Dave Billinge, Tom Addis
• Chapter 9: Modeling Music Notation in the Internet Multimedia
Age, Pierfrancesco Bellini, Paolo Nesi
viii
Section 5: Evaluation
• Chapter 10: Evaluation in the Visual Perception of Music, Susan E.
George
Description of Each Chapter
In Chapter 1, Dr. Ichiro Fujinaga describes the issues involved in the detection
and removal of stafflines of musical scores. This removal process is an important
step for many optical music recognition systems and facilitates the segmenta-
tion and recognition of musical symbols. The process is complicated by the fact
that most music symbols are placed on top of stafflines and these lines are
often neither straight nor parallel to each other. The challenge here is to
remove as much of the stafflines as possible while preserving the shapes of the
musical symbols, which are superimposed on stafflines. Various problematic
examples are illustrated and a detailed explanation of an algorithm is
presented. Image processing techniques used in the algorithm include: run-
length coding, connected-component analysis, and projections.
In Chapter 2, Professor Pierfrancesco Bellini, Mr. Ivan Bruno and Professor
Paolo Nesi compare OMR with OCR and discuss the O3MR system. An overview
of the main issues and a survey of the main related works are discussed. The
O3MR system (Object Oriented Optical Music Recognition) system is also
described. The used approach in such system is based on the adoption of
projections for the extraction of basic symbols that constitute graphic
elements of the music notation. Algorithms and a set of examples are also
included to better focus concepts and adopted solutions.
In Chapter 3, Dr. Susan E. George investigates a problem that arises in OMR
when notes and other music notation symbols are super-imposed upon
stavelines in the music image. A general purpose knowledge-free method of
image filtering using two-dimensional wavelets is investigated to separate the
super-imposed objects. The filtering provides a unified theory of staveline
removal/symbol segmentation, and practically is a useful pre-processing
method for OMR.
In Chapter 4, Dr. Kia Ng examines a method of recognising printed music —
both handwritten and typeset. The chapter presents a brief background of the
field, discusses the main obstacles, and presents the processes involved for
printed music scores processing; using a divide-and-conquer approach to sub-
segment compound musical symbols (e.g., chords) and inter-connected groups
(e.g., beamed quavers) into lower-level graphical primitives such as lines and
ellipses before recognition and reconstruction. This is followed by discussions
on the developments of a handwritten manuscripts prototype with a segmen-
tation approach to separate handwritten musical primitives. Issues and
ix
approaches for recognition, reconstruction and revalidation using basic music
syntax and high-level domain knowledge, and data representation are also
presented.
In Chapter 5, Dr. Susan E. George concentrates upon the recognition of
handwritten music entered in a dynamic editing context with use of pen-based
input. The chapter makes a survey of the current scope of on-line (or dynamic)
handwritten input of music notation, presenting the outstanding problems in
recognition. A solution using the multi-layer perception artificial neural
network is presented, explaining experiments in music symbol recognition
from a study involving notation writing from some 25 people using a pressure-
sensitive digitiser for input. Results suggest that a voting system among
networks trained to recognize individual symbols produces the best recogni-
tion rate.
In Chapter 6, Professor Pierfrancesco Bellini, Mr. Ivan Bruno and Professor
Paolo Nesi present an object-oriented language capable of modelling music
notation and lyrics. This new model makes it possible to “plug” on the symbolic
score different lyrics depending on the language. This is done by keeping
separate the music notation model and the lyrics model. An object-oriented
model of music notation and lyrics are presented with many examples. These
models have been implemented in the music editor produced within the
WEDELMUSIC IST project. A specific language has been developed to associate
the lyrics with the score. The most important music notation formats are
reviewed focusing on their representation of multilingual lyrics.
In Chapter 7, Dr. Susan E. George presents a consideration of lyric recognition
in OMR in the context of Christian music. Lyrics are obviously found in other
music contexts, but they are of primary importance in Christian music — where
the words are as integral as the notation. This chapter (i) identifies the
inseparability of notation and word in Christian music, (ii) isolates the
challenges of lyric recognition in OMR providing some examples of lyric
recognition achieved by current OMR software and (iii) considers some
solutions outlining page segmentation and character/word recognition ap-
proaches, particularly focusing upon the target of recognition, as a high level
representation language, that integrates the music with lyrics.
In Chapter 8, Dr. Dave Billinge and Professor Tom Addis investigate language
to describe the emotional and perceptual content of music in linguistic terms.
They aim for a new paradigm in human-computer interaction that they call
tropic mediation and describe the origins of the research in a wish to provide
a concert planner with an expert system. Some consideration is given to how
music might have arisen within human culture and in particular why it presents
unique problems of verbal description. An initial investigation into a discrete,
stable lexicon of musical effect is summarised and the authors explain how and