Table Of ContentNATO ASI Series
Advanced Science Institutes Series
A series presenting the results of activities sponsored by the NATO Science Committee,
which aims at the dissemination of advanced scientific and technological knowledge,
with a view to strengthening links between scientific communities.
The Series is published by an international board of publishers in conjunction with the
NATO Scientific Affairs Division.
A Life Sciences Plenum Publishing Corporation
B Physics London and New York
C Mathematical and Physical Sciences Kluwer Academic Publishers
D Behavioural and Social Sciences Dordrecht, Boston and London
E Applied Sciences
F Computer and Systems Sciences Springer-Verlag
G Ecological Sciences Berlin Heidelberg New York Barcelona
H Cell Biology Budapest Hong Kong London Milan
I Global Environmental Change Paris Santa Clara Singapore Tokyo
Partnership Sub-Series
1. Disarmament Technologies Kluwer Academic Publishers
2. Environment Springer-Verlag
3. High Technology Kluwer Academic Publishers
4. Science and Technology Policy Kluwer Academic Publishers
5. Computer Networking Kluwer Academic Publishers
The Partnership Sub-Series incorporates activities undertaken in collaboration with
NATO's Cooperation Partners, the countries of the CIS and Central and Eastern
Europe, in Priority Areas of concern to those countries.
NATO-PCO Database
The electronic index to the NATO ASI Series provides full bibliographical references
(with keywords and/or abstracts) to about 50 000 contributions from international
scientists published in all sections of the NATO ASI Series. Access to the NATO-PCO
Database compiled by the NATO Publication Coordination Office is possible in two
ways:
-via online FILE 128 (NATO-PCO DATABASE) hosted by ESRIN, Via Galileo Galilei,
I -00044 Frascati, Italy.
-via CD-ROM "NATO Science & Technology Disk" with user-friendly retrieval
software in English, French and German(© WTV GmbH and DATA WARE
Technologies Inc. 1992).
The CD-ROM can be ordered through any member of the Board of Publishers or
through NATO-PCO, B-3090 Overijse, Belgium.
Series F: Computer and Systems Sciences, Vol. 150
Springer-Verlag
Berlin Heidelberg GmbH
Speechreading by
Humans and Machines
Models, Systems, and Applications
Edited by
David G. Stork
Marcus E. Hennecke
Ricoh California Research Center
2882 Sand Hill Road #115
Menlo Park, CA 94025-7022, USA
and
Department of Electrical Engineering
Stanford University
Stanford, CA 94305, USA
Springer
Proceedings of the NATO Advanced Study Institute on Speechreading
by Man and Machine, held in Castera-Verzudan, France, August 28 -
September 8,1995
Library of Congress Cata logtng-tn-Publ icatton Data
Speechreading by humans and machInes: models, systems. and
applicatIons I edited by David G. Stork, Marcus E. Hennecke.
p. cm. -- (NATO ASI series. Series F, Computer and systems
sCiences; vol. 150)
Papers presented at a workshop held at the Chateau de Bonas,
Castera-Verduzan. from Aug. 28 to Sept. 8, 1995.
Includes bibliographical references and indexes.
ISBN 978-3-642-08252-8 ISBN 978-3-662-13015-5 (eBook)
DOI 10.1007/978-3-662-13015-5
1. Human informatIon processlng--Congresses. 2. Llpreadlng
-PsychologIcal aspects--Congresses. 3. Visual perception-
-Congresses. 4. Speech perception--Congresses. 5. Automatic speech
recognition--Congresses. I. Stork. DavId G. II. Hennecke, Marcus
E. III. Series: NATO ASI series. Series F, Computer and systems
scIences; no. 150.
BF444.S63 1996
006.4'54--dc20 96-23719
CIP
CR Subject Classification (1991): I.2.7, I.2.10, I.2.6
ISBN 978-3-642-08252-8
This work is subject to copyright. All rights are reserved, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data
banks. Duplication of this publication or parts thereof is permitted only under the provisions
of the German Copyright Law of September 9, 1965, in its current version, and permission for
use must always be obtained from Springer-Verlag Berlin Heidelberg GmbH.
Violations are liable for prosecution under the German Copyright Law.
© Springer-Verlag Berlin Heidelberg 1996
Originally published by Springer-Verlag Berlin Heidelberg New York in 1996
Softcover reprint of the hardcover 1st edition 1996
Typesetting: Camera-ready by editor
Printed on acid-free paper
SPIN: 10486151 45/3142 - 5 432 1 0
Preface
This book is one outcome of the NATO Advanced Studies Institute (ASI)
Workshop, "Speechreading by Man and Machine," held at the Chateau de
Bonas, Castera-Verduzan (near Auch, France) from August 28 to Septem
ber 8, 1995 - the first interdisciplinary meeting devoted the subject of
speechreading ("lipreading"). The forty-five attendees from twelve countries
covered the gamut of speechreading research, from brain scans of humans
processing bi-modal stimuli, to psychophysical experiments and illusions, to
statistics of comprehension by the normal and deaf communities, to models of
human perception, to computer vision and learning algorithms and hardware
for automated speechreading machines.
The first week focussed on speechreading by humans, the second week by
machines, a general organization that is preserved in this volume. After the in
evitable difficulties in clarifying language and terminology across disciplines
as diverse as human neurophysiology, audiology, psychology, electrical en
gineering, mathematics, and computer science, the participants engaged in
lively discussion and debate. We think it is fair to say that there was an
atmosphere of excitement and optimism for a field that is both fascinating
and potentially lucrative.
Of the many general results that can be taken from the workshop, two of
the key ones are these:
• The ways in which humans employ visual image for speech recogni
tion are manifold and complex, and depend upon the talker-perceiver
pair, severity and age of onset of any hearing loss, whether the topic
of conversation is known or unknown, the level of noise, and so forth.
For simple stimuli, we seem to be tantalizingly close to a definitive
model of how information is integrated in the human nervous system,
at least for bi-modal speech. However, at this time the complexity of
language, speech and vision precludes a clear and predictive model of
full speechreading competence.
• A number of laboratories, using a variety of techniques, all conclude
that incorporating visual image of a talker improves automatic speech
recognition, even above that of state-of-the-art acoustic recognizers in
noise-free environments. The most dramatic benefits are, however, in
real-world conditions of noise or multiple distracting talkers. Thus the
greatest impact of speechreading may be to broaden the range of ap
plicability of speech recognition systems.
VI
We have taken our editorial duties very seriously and in ways that we hope
will make the volume useful for many years to come:
Editorial supervision All authors were encouraged to have their drafts
read by at least one, and preferably more, attendees to help insure
clarity, correctness, and completeness (e.g., of the reference lists). This
sharing of papers also facilitated cross-referencing among the papers
presented here.
Bibliography The bibliography is the most complete source of papers and
books on speechreading ever compiled, and includes references beyond
those cited by the papers in this volume. As such, this bibliog~aphy
should be a useful reference for established researchers as well as a
gateway for those seeking to enter the field.
Patents Technologists building automated speechreading systems may be
concerned with patents, not only for intellectual property issues, but
also for inspiration and ideas - after all, some of the best ideas never
appear in academic journals or books. For this reason, we have included
the most complete list of US patents on speechreading issued since 1976.
Panel discussions Much of the value of attending a conference or workshop
arises from the informal discussions and debates over dinner or during
a stroll in the woods; insights and ideas are discussed that are, alas,
rarely disseminated in print until much later. Of course it is impossible
to record all such informal discussions, but informed opinion was evi
denced in the five lively and interesting panel discussions, summarized
here. In some cases priorities for future research were set and questions
clarified. In particular, one of the recommendations was that we should
meet again in 1999, a date chosen to give researchers adequate time to
explore the results of the 1995 workshop.
Terminology Part of the difficulty in bringing together researchers from
diverse fields is the finding a common terminology. Although we cannot
guarantee that all papers use terminology in precisely the same way, at
least during the Workshop an effort was made to this end.
Uniform style We managed to keep the style fairly uniform despite the
use of a number of different word processing programs in order to help
readers · Goncentrate on the content, and may make the book easier for
use in graduate courses in linguistics, audiology, computer engineering,
and so on. The extensive author and subject indexes and keywords are
all meant to increase the usefulness of the book.
VII
Many people contributed to the workshop and this volume. We would like
to thank all participants, not only for their presentations and papers, but
also for lively discussions, camaraderie and spirited petanque competition.
Warm thanks go to co-directors Christian Benoit and N. Michael Brooke, who
chaired sessions and helped in numerous organizational matters. Christian,
in particular, was invaluable in making local arrangements and helping us
overcome the problems of finances in a European community not yet fully
converted to a single currency. The members of the panel discussions did
an admirable job in guiding and clarifying those discussions, and especially
in recording them accurately. Madam Simon and her very able staff at the
Chateau de Bonas helped to set a relaxed yet invigorating atmosphere, amidst
the lovely grounds, with excellent food and local Armanac. Everyone will
remember for many years the intimate candlelight concert of French Baroque
music performed on traverse, tMorbe and viola da gamba.
We would also like to thank Stanford graduate student Vicky Lu, who
compiled the author and subject indexes (an excellent way to learn about the
literature!), as well as Rowan Fairgrove at the California Research Center,
who performed the computer search for speechreading patents and J. Andrew
Ross and Dr. Hans Wossner of Springer-Verlag for production assistance.
We mentioned above that this volume is just one of the outcomes of the
workshop. The personal meetings, new collaborations, and recommendations
for research are yet others that will help to keep our interdisciplinary field
healthy and thriving.
May 1996
W kshop Director
C ief Scientist
Ricoh CRC
Consulting Assoc. Prof.
Stanford U.
Marcus E. Hennecke
Research Programmer
Ricoh CRC
Graduate Assistant
Stanford U.
osi Adjoudani M. Brooke Auer, Jr. Saldaii.a Robert Sokol
. CA. N.E. H. s,
P6 12 18 24 30 are
n
i
L
ke z-
an z ein nec gue
eskow MovellHiki CoianiBernstE. Hen Rodri
J. B5 J. 11 17S. T. 23 L. 29 M.35 eandro
L
n n,
o a
es aj
0. Garcia 4 C. Benoit 10 Green 16 K. J. Robert-Ribes 22 M.-A. Cathiard 28 E. Vatikiotis-Bat34 Si Wei Lu, Eric Pet
o,
a tt
m e
meele Massaro Luettin Cohen Silsbee Hiraya o Lavag
P. S3 D. 9 J. 15 21M. P. 27 33M. , Fabi
a
n
er Gra
l
K. Pichora-Ful2 D.G. Stork 8 Vogt 14M. P. Bertelson 20 D. Burnham 26 R. Campbell 32 nuejols, Manuel
r
o
C
n
a e
s al Dur tin
w m r
- a
P. Kricos 1 7M. Sams MattheI. 13 C. Abry 19 Pique25M. A. Fuster31 ot shown: M
N
Acknowledgements
The generous support of the following institutions, which made possible this
ASI and the research presented by the below participants, is gratefully ac
knowledged:
• The North Atlantic Treaty Organization (NATO)- Sponsor
• The Ricoh California Research Center - Sponsor
• National Institute on Deafness and Other Communication Disorders,
National Institutes of Health (NIH): Lynne E. Bernstein, Michael
Cohen, Kerry Green, Dom Massaro, and Helena Saldana
• ESPRIT, France Telecom, and the French Cognitive Science Pro
gramme: Christian Abry, Christian Benoit, and Marie-Agnes Cathiard
• Centre Nationale de la Recherche Scientifique (CNRS): Christian Benoit
• The Engineering and Physical Sciences Research Council of the United
Kingdom: N. Michael Brooke
• Economic and Social Science Research Council of Great Britain, Lever
hulme Trust of Great Britain: Ruth Campbell
• The University of British Columbia and The National Science and
Engineering Research Council of Canada: Kathy Pichora-Fuller
• National Science Foundation (NSF): Peter Silsbee
• European Office of Aerospace Research and Development (EOARD):
Robert Kaucic
• Ministere de !'Education de la Communaute Franc;aise: Paul Bertelson
• The Australian Research Council (ARC): Denis Burnham
• The Ministry of Science and Art Baden-Wlirttemberg: Michael Vogt
