Music Representation and Transformation in Software. Structure and Algorithms in Python

341 Pages·2022·17.518 MB·English

by  Donald P. Pazel

Donald P. Pazel
Music
Representation
and Transformation
in Software
Structure and Algorithms
in Python
Music Representation and Transformation in Software
Donald P. Pazel
Music Representation
and Transformation in Software
Structure and Algorithms in Python
Donald P. Pazel
Fishkill, NY, USA
ISBN 978-3-030-97471-8 ISBN 978-3-030-97472-5 (eBook)
https://doi.org/10.1007/978-3-030-97472-5

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2022
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting,
reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic
adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even
in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore
free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true
and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied,
with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains
neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface

Introduction

Suppose you have an interest in programming computer music and need a starting point. Perhaps you
have tried various music applications or tools, such as mixers, production systems, or notation systems,
but feel a need to dig deeper, to be more hands-on and work with music fundamentals. Perhaps you
have familiarity with concepts of musical notes, keys, chords, scores, and other music elements, but
you don’t know where to begin in designing and programming about them, let alone translating them
to MIDI®1. You want to be able to put all those elements to work through programming, but you need
direction to get to that next killer music app that you have in mind. This book guides you to that sweet
spot of developing and using your programming skills to design and build systems about music.
However, this book does more than provide that kind of instruction. This book takes the reader on a
journey through music concepts in an organized approach that develops music essentials, from the
concepts of tone, pitch, and time, through notes, intervals, chords, and scores while at the same time
interpreting these elements as software artifacts. As these many concepts are covered, we do so with
close attention to the organization of and relationships amongst these concepts and reflect those as
Python classes and objects, and so learn about music from a software design viewpoint. By this process,
we approach this design in a logical and structured way.
The objective for doing this is to make music theory elements programmatically accessible. The result-
ing software should allow you to be able to code in the language of music, letting you to think primarily
in musical concepts about your programming objectives, while leaving various music calculations,
such as key or chord construction, transparent. That is the point behind the software discussion on
music representation in the first part of this book.
The second part of the book is devoted to music transformation algorithms. These transforms comprise
an interesting application of music representation for modifying music through key change, melodic
reflection, and much more. This part of the book is more academic and has more of a research feel.
However, the connection to programming prominently remains with discussions grounded in

1 MIDI is a registered trademark of the Association of Manufacturers of Electronic Instruments
(AMEI).

v
Preface
implementation details and software structure considerations. We hope the reader finds inspiration in
this section for novel applications or advanced interests in the wide area of computer music research.
The Origin of this Book
The origin of this book has its roots in the late 90's. At that time, I had the honor of being a member of
a computer music research team at IBM®2 TJ Watson Research Center in New York. This department
should not be confused with being a club of musicians that jammed after work. Rather, the Computer
Music Center (CMC) focused on MIDI-based composition software and real-time music modification
as full-time work in the Mathematics Department of this esteemed IBM research center. The depart-
ment was technically and principally led by computer music veterans Danny Oppenheim and Jim
Wright. During my tenure there we constructed MusicSketcher [1], a prototype application that demon-
strated a wide range of music transformational capability, including dynamic chord replacement, and
harmony preserving dynamic note shifts, and other capabilities. These transformations went by the
name Smart Harmony [2]. My work experiences in this department served as an introduction to com-
puter music, MIDI, and various music tools. The novelty and creativity behind the work there was very
impressive, and I found the talks, works, and efforts there stimulating in ideas.
Many years and many different careers later, along with finally having freedom of time, I decided to
take a fresh look at a few ideas in computer music research, but now in the context of some 20 to 30
years of technological change. The prior work with which I was familiar was preserved in papers and
memory. This forced me to think through from scratch the foundations for music representations and
transformational capabilities. Additionally, instead of using C/C++ as we had years earlier, I now have
access to Java or Python, computer languages with which I have become more adept through my pro-
fessional industrial experiences. Computing hardware also improved considerable over time. Further,
VST®3 digital audio provided advanced capability in MIDI rendering and provided access to rich
sound samples which are currently plentiful in the market.
I considered building on an idea for music collaboration that I thought about while I was in the CMC
long ago [3]. The idea was to allow multiple people to log into a group-based music session, choose
instruments, insert riffs or melody lines, and dynamically modify them in real time collaboratively. As
I moved forward on this idea, the effort took on many technical challenges. Firstly, and most im-
portantly, I needed a rich flexible music representation with functionality that translates into MIDI and
music performance dynamics. While I thought this was my main concern, I quickly learned that there
are many other critical features outside of music to address. As this project was designed to have a
web-based front end and significantly, real-time collaborative interaction, it required a coordinating
server as well. After much effort, I achieved a prototypical result that was very close to what I wanted.
Multiple players could engage in music performance through a web interface as a shared experience,
much in the way I envisioned. A tune is composed on a shared (web-based) screen using selectable
multi-measure music fragments stitched together. The players take on different instrument roles, say
one on drums, another guitar, etc. Once the playback started, each player controls their instrument’s
pitch (while automatically maintaining harmony) and volume levels. The tempo and harmony (chords)
are also adjusted in real time through a special “conductor” role. A server takes care of all the interac-
tivity and music dynamics, and ensures the players are relatively synchronized to a unified experience.
As these features became demonstrable, the key question became "Where to go from here?" A question
that remains unsettled still. Even as I contemplated the work’s uncertain future, there remained a nag-
ging consideration that I had reluctantly been putting off and needed to consider.

2 IBM is a registered trademark of International Business Machines.
3 VST is a trademark of Steinberg Media Technologies, GmbH.
vi
Prerequisites and Approach
The problem which with I struggled was that this project was simply too massive for one person to
contain within reasonable time. Although initially I focused on building a proper music representation,
I quickly became absorbed with server architecture details, distributed playback algorithms, client syn-
chronization, web interfaces (JavaScript), and general maintenance. Quickly, the core music represen-
tation became brittle and sketchy with the load of work. What started as a music project, became more
of a “soup to nuts” web application project, a terribly interesting one, but one that left scarce time and
energy for addressing the increasingly insufficient core music representative and transformative issues
on which I wanted and needed to spend much more time and energy.
I took a break from the project for reflection, to reexamine the work that had been done, determine how
to improve upon it, and see to where that circumspection leads. As I realized early on, music represen-
tation is a deep and interesting software modeling exercise in itself. The intuitive concepts of music
require rigorous definition. The closer I looked, it seemed that although we are generally adept with
many music concepts like intervals, scales, chords, and the like, our understanding of these are some-
what superficial and at times not so well structured, if not incomplete, and rough around the edges. It’s
like the many skills we adopt from rote learning. We work with certain skills daily but don’t think
deeply about them nor even understand their foundations with much clarity. Looking at the music
representation I had developed, I found needless complexity that seemed to come more from lack of
foundation (from admittedly a rush to get that component finished as an expense for other needs) than
say sloppy code.
Yet another important aspect surfaced, that music representation comes with a set of algorithms that
are in fact, genuinely interesting. The computation of scales, computing interval end notes, and similar,
are interesting topics, that in a thought-provoking way made me question how as a musician I had done
those things, and if there are better ways to think about those topics, especially, when designing soft-
ware to manipulate them. As an example, the notion of time in music is interesting, that along with the
varied temporal representations comprises an interesting study, including the conversions of one tem-
poral representation to another. It is interesting to think about the properties of a musical note, to ques-
tion why we limit to specific durations, and pitches. These are just a few examples of what came to
light. It speaks to the larger issue of questioning our understanding of music, what restrictions might
be too rigid, and if anything can be done to lessen those restrictions.
I never returned to that initial distributed application. Instead, I decided to focus on music representa-
tion, the related foundational algorithms, and eventually to music transformational issues, from the
perspective of a software engineer and to some degree, a mathematician. This turn of focus resulted in
a sizeable body of study and code that is detailed in this book. The result is a far more fulfilling adven-
ture than I could have imagined.
This experience is what led me to writing this book. After years of working solely on music represen-
tation, and years looking at how to build transformational algorithms with that representation, I thought
I would share that experience here in hopes of motivating others to carry on and perfect that effort and
inspire people towards efforts in computer music.
Prerequisites and Approach
Readers are expected to have basic programming skills, and some familiarity with Python. This in-
cludes familiarity with computational statements, logical structures such as if and loop (for/while)
statements, and function or procedure definitions with calling protocols, including parameter passing.
Also, a familiarity with the concepts of list and dictionary (or maps), embodied as Python lists and
maps or dicts, is also assumed. Unlike many topical introductory books with claims to programming
or a specific programming language training, this book assumes basic programming skills in Python.
If needed, there are many excellent books for learning Python such as [4] [5] or online references such
as [6].
v ii
Preface
The reader should have some loosely defined notions of musical elements. That is, the reader should
have notions about what notes, intervals and chords are. However, the book is built around introducing
all these concepts in a methodical structured way. So, unlike the programming prerequisites, the music
prerequisites are less strict.
Object-Oriented design skills are not required but are a critical part of the book’s teachings. This book
places a heavy emphasis on learning object-oriented (O-O or OO) design and thinking. Designing in
an object-oriented manner forces one to step back from a problem space, and pull apart the concepts
and their dependencies, and make judgments on how to approach a programming task. It forces devel-
opers to:
• code a software model that clearly maps to the problem domain’s conceptual model, and in a sense,
“talks in its terms”.
• reflect relationships and dependencies directly in the software model that are relevant in the do-
main’s conceptual model.
• keep in mind efficiencies and inefficiencies, or limitations of programming artifacts used in the
software model.
• stretch for generality where needed, but within reasonable bounds.
What the reader should realize is that like with so many software endeavors, the software design pro-
cess never ends. This is particularly so with the music domain, with, for example, the introduction of
different instruments, different styles of music, and so forth, each introducing their own conceptual
needs. We do not claim to present in this book the best possible music representation but do believe
this model goes a long way on generality and will change over time to be more inclusive, and to intro-
duce the reader to software design in the object-oriented style.
Since music many times involves calculations, especially involving dynamics over time, some mathe-
matics is involved. The reader should be able to understand simple equations for calculations and sim-
ilar and be capable of translating these into code.
What this Book is Not About
As important as describing what this book is about, is mentioning what it is not about. Importantly, this
is not a book about music notation, nor how to build music notation software. Music notation for sure
is important and related to the topic of this book. Notation concerns the visual and textual communi-
cation of music content and instruction for performance. Consequently, notation incorporates a map-
ping of music concepts to text/symbol and/or visa-versa. This book’s concern is with embodying music
concepts as software model objects, something more abstract that could be useful in building a music
notation system but is not one in and of itself. By the way, there are numerous exemplar music notation
systems on the market: Dorico®4, Sibelius®5, Notion®6, and Finale®7 to name a few. As a warning to
more ambitious readers, building a music notation editor is a herculean effort. Entire teams of devel-
opers and testers are actively devoted to such endeavors with extensive time and resource commitments
by a few companies.
Secondly, this is not a book about building MIDI rendering software. MIDI and some of its important
aspects and translation of music models to MIDI are described in detail here and provide a useful

4 Dorico is a registered trademark of Steinberg Media Technologies GMbH.
5 Sibelius is a registered trademark of Avid Technology Europe LTD.
6 Notion is a registered trademark of Presonus Expansion, L.L.C.
7 Finale is a registered trademark of MakeMusic, Inc.
viii
Intended Audience
educational introduction for readers interested in learning about MIDI. However, construction of MIDI
rendering software is a topic unto itself and is outside the scope of this book.
Intended Audience
A wide variety of readership is envisioned for this book. Primarily it is for students to accelerate their
software development capabilities in designing and building music software. With that, we offer the
following thoughts for the varied reader constituency:

Students
Students should take seriously the technical prerequisites. It is important to be able to program proce-
durally in some programming language. If you already know Python, fine. If not and you pick up
programming languages easily, fine. However, knowing the basics of Python is the expectation of this
book. The book also delivers an introduction to object-oriented principles and how to program to them
in Python. The book applies object-oriented principles to varied music elements and accelerates in
depth rapidly. The algorithms, even in the representation section at times can be complex and intricate,
and in some cases are expressed in mathematical formulae which the user should be able to understand
and translate to code.
In a few instances, advanced computer science concepts are discussed, for example, red-black trees
and constraint engines. Students need not understand the implementations of these concepts in detail,
but should concentrate in understanding the features they provide, how to use them, their interfaces,
and chiefly their strengths and limitations. The more intrigued student may want to invest more time
on theory and implementation details in these areas.
While the primary focus for entry-level students should be on the representation part of the book, the
transformation part could be of great interest and well-worth reading. It comprises a combination of
practical applications of music representation and an introduction to constraint management. Students,
if not guided by an instructor, should read through the transformations and at least understand what
they contribute musically, and understand them technically as best can. Reading through the many
examples should help in providing this understanding.
For representation chapters, the end of chapter questions are mostly for provoking thought about the
range in complexity of what is presented. The student is encouraged to read through the questions and
select a few (if not assigned by the instructor) to think through as best can. For the transformation
chapters, each chapter ends with a “final thoughts” section which highlights both positive and less than
positive aspects of the chapter’s topic. It is meant to be thought provoking, and fuel for further efforts.

Instructors
Instructors can consider using this book for a one semester course on computer music programming.
The representation part could easily occupy a semester. For classes comprised of students with lower-
level programming skills, the instructor could augment the course with a robust Python review. That
said, it is recommended to spend a good deal of time on object-oriented concepts and how to code
Python to an object-oriented methodology. After that, the book can be followed chapter to chapter with
the idea of introducing new music concepts in succession, along with representation and coding prin-
ciples.
If the semester has more time to offer, one can consider introducing topics from the transformation
section, with at least a guide through the transformations and examples. The constraint engine itself
can be technically demanding for students, and the instructor can defer on its implementation details,
and instead highlight its interface and usage. The constraint engine is an interesting topic, and advanced
ix
Preface
students may want to consider augmenting or working with it as an advanced project in this or other
domains.
Practitioners
Practitioners may come to this book for a variety of reasons, and in a variety of ways. If you want to
learn how to approach object-oriented programming with computer music as a domain, then a straight-
forward read is recommended. Otherwise, treating it as a reference text, picking and choosing chapters
in the representation section is useful.
As for the transformation section, there are many topics from which to choose. A few of the transforms
make use of the constraint engine, but many don’t. It is a matter of looking through the transform types
to determine which topics might be of interest.
Experienced Researchers
Experienced researchers may generally find the representation section less interesting than the trans-
formation section, except perhaps for the later chapters of the representation section concerning varia-
ble dynamics and harmony representation. We would expect this audience to be more interested in the
transformation section of the book, on which there are a wide range of topics to study and expand upon.
Regarding the Musical Examples
The reader will find an abundance of musical examples. While music written on a page is not as satis-
fying as listening to it, there are several ways to listen to them.
• Use the score to midi conversion code discussed in this book to generate MIDI files and play
the examples on a MIDI player of choice.
• Although not covered in this book, there is a primitive VST host referenced in the code base
that can be used with Python interface code in the code base.
• The author found copying generated music to a notation application with playback is very
practical and useful, and allows for quickly changing an example’s dynamics, tempo, etc.
• Hands-on playing examples on a musical instrument.
The author is aware that many books of this type go through great effort to provide support for MIDI
playback. The decision was made to invest effort on the main ideas of this book on music representation
and transformation, even at the expense of not providing what would at best be a poorly functional
MIDI playback software package that would only detract from the learning experience.

About the Software

As a bonus this book is associated with an accessible software package that reflects the work of this
text. This software can be found at:
https://github.com/dpazel/music_rep
This software package reflects the class hierarchy and details described throughout this text. It is cov-
ered by MIT license with no guarantees nor warrantees.
x