Facial animation using muscles to control the mimic articulation of phonemes

97 Pages·2012·22.44 MB·English

by  Anna Mempel

anna mempel
FACIAL ANIMATION USING MUSCLES TO
CONTROL THE MIMIC ARTICULATION OF
PHONEMES

FACIAL ANIMATION USING MUSCLES TO
CONTROL THE MIMIC ARTICULATION OF
PHONEMES
MASTER THESIS
anna mempel
Professor Dr. Helmut Weberpals
Professor Dr. -Ing. Rolf-Rainer Grigat
Institute of Computer Technology
Hamburg University of Technology
6 2012
December
Anna Mempel: Facialanimationusingmusclestocontrolthemimic
6
articulationofphonemes,MasterofScience(M.Sc.),© December
2012
o my om and ad
T M D
ACKNOWLEDGEMENTS
In the first place I would like to thank my advisors Professor
Helmut Weberpals and Professor Rolf-Rainer Grigat for their
inspiration, continuous support, and advice during the course
of this thesis.
I am grateful to Dr. Thomas Punz for his constructive sup-
port during the preparation of this thesis.
Last but not least I owe my loving thanks to my family and
friends, who supported me during this thesis and my studies.
v
ABSTRACT
This master thesis describes an approach to facial animation
using the muscle feature of Autodesk Maya. The skeletal, mus-
cular and skin anatomy of the human head is abstracted from
nature. Themuscularstructureismodelledtoactasahighlyre-
usable interface between any skull and skin model. The muscle
actions are based on the Facial Action Coding System and are
implemented using weights painted on the skin mesh. By com-
bining several muscle actions it is possible to animate facial ex-
pressions, e.g. emotions or phonemes. By animating the mimic
articulation of several visemes, the accuracy, usability and per-
formance of the developed approach is determined.
ZUSAMMENFASSUNG
Diese Master Thesis stellt einen Ansatz zur Gesichtsanimation
mit Autodesk Mayas Muskel-Funktion vor. Die Anatomie der
menschlichen Gesichtsknochen und -muskeln, sowie der Haut
wird von der Realität abstrahiert. Das Muskel-Modell wird als
wiederverwendbare Schnittstelle zwischen beliebigen Schädel-
und Haut-Modellen implementiert. Basierend auf dem Facial
Action Coding System werden die Muskelaktionen durch Ge-
wichtung des Haut-Modells modelliert. Das kombinierte Ak-
tivieren einzelner Muskeln ermöglicht die Animation von Ge-
sichtsausdrücken, wie Emotionen oder Phoneme. Die Genauig-
keit, Nutzbarkeit und Leistungsfähigkeit des Ansatzes werden
nachgewiesen, indem verschiedene Viseme mimisch artikuliert
werden.
vi
CONTENTS
1 introduction 1
2 facial animation state of the art 5
—
21 5
. Controlling the Animation . . . . . . . . . . . . . .
22 7
. Transitions between facial expressions . . . . . . .
23 7
. Anatomical Simulation . . . . . . . . . . . . . . . .
3 application of computer facial animation 9
31 9
. Animation Industry . . . . . . . . . . . . . . . . . .
32 9
. Video Game Industry . . . . . . . . . . . . . . . . .
33 10
. Medicine and Therapy . . . . . . . . . . . . . . . .
34 10
. Lip-Reading and Speech Animation . . . . . . . .
35 11
. Video Teleconferencing and Photography . . . . .
36 11
. Avatars and Social Agents . . . . . . . . . . . . . .
4 phonetics and phonology 13
41 15
. Phonemes versus Visemes . . . . . . . . . . . . . .
42 16
. Mapping Phonemes to Visemes . . . . . . . . . . .
5 anatomy of the human face and skull 21
51 21
. Skeletal anatomy of the human face . . . . . . . .
52 23
. Muscular anatomy of the human face . . . . . . .
53 35
. Skin structure of the human face . . . . . . . . . .
6 solution overview 37
61 37
. Selecting Phonemes and Visemes . . . . . . . . . .
62 38
. Abstracting Anatomy . . . . . . . . . . . . . . . . .
63 39
. The Skin and the Skull Model . . . . . . . . . . . .
64 42
. Autodesk Maya . . . . . . . . . . . . . . . . . . . .
641 42
. . Deformers of the Maya Muscle System . .
642 43
. . Muscle Types . . . . . . . . . . . . . . . . .
643 44
. . Renderers . . . . . . . . . . . . . . . . . . .
65 45
. Prospective Application Areas . . . . . . . . . . . .
7 creating a model of facial muscles 47
71 47
. Linear Muscles . . . . . . . . . . . . . . . . . . . . .
711 47
. . Mathematical Description . . . . . . . . . .
712 48
. . Modelling Linear Muscles . . . . . . . . . .
vii
contents
viii
72 50
. Sphincter Muscles . . . . . . . . . . . . . . . . . . .
721 50
. . Mathematical Description . . . . . . . . . .
722 51
. . Modelling Sphincter Muscles . . . . . . . .
73 53
. Mirroring Muscles . . . . . . . . . . . . . . . . . .
74 53
. Finalising the Muscle Model . . . . . . . . . . . . .
75 54
. Connecting Muscles . . . . . . . . . . . . . . . . . .
76 54
. Reusing the Muscle Model . . . . . . . . . . . . . .
8 implementing muscle behavior 57
81 57
. Modelling Linear Muscle Action . . . . . . . . . .
82 62
. Modelling Sphincter Muscle Action . . . . . . . .
83 64
. Resulting Muscle Actions . . . . . . . . . . . . . .
84 64
. Reusing the Muscle Behavior . . . . . . . . . . . .
9 animating phonemes and visemes 67
91 67
. Applying Mouth Shapes for selected Phonemes .
911 67
. . Bilabial Consonants /m/, /b/, /p/ . . . .
912 69
. . Labiodental Consonants /f/, /v/ . . . . .
913 70
. . Front Vowels /a/, /i/, and /e/ . . . . . . .
914 71
. . Back Vowel /o/ . . . . . . . . . . . . . . . .
92 72
. Capturing Mouth Shapes . . . . . . . . . . . . . . .
10 conclusion and future work 73
bibliography 75
index 81
LIST OF TABLES
1 20
Table Phoneme-to-Viseme Maps . . . . . . . . . .
2
Table Overview of the Muscles of the Model,
their Category and the abstracted Origin
40
and Insertion. . . . . . . . . . . . . . . . . .
LIST OF FIGURES
11 1972
Figure . Rendering and Animation System in
72 2
[Par ] . . . . . . . . . . . . . . . . . . . . .
12 12 3
Figure . Facial Animation Details on Shrek [Shr b]
13 08 4
Figure . Believability Flip [PW ] . . . . . . . . . . .
21
Figure . (a) Muscular Anatomy and (b) Muscular
ActionsthatunderlieActionUnitsrespons-
ibleforAppearanceChangesoftheLower
Face. Numbers on (a) refer to the Action
Unit that makes Use of the Muscle. Num-
bers on (b) indicate the Origin and the
End of each Line indicates the Insertion
02 6
of the Muscle [EFH ] . . . . . . . . . . . .
31
Figure . GameIndustryImprovementsoverthelast
YearsusingtheComputerGameGothicas
12 10
an Example [got ] . . . . . . . . . . . . . .
41 12 13
Figure . Consonants of the IPA [IPA ] . . . . . . .
42
Figure . Places of Articulation in the Vocal Tract
10 99 14
[Net , Ass ] . . . . . . . . . . . . . . . . .
43 12 15
Figure . Vowels of the IPA [IPA ] . . . . . . . . . .
44
Figure . PrestonBlairPhoneme-to-VisemeMapping
12 17
[Pre ] . . . . . . . . . . . . . . . . . . . . .
44
Figure . PrestonBlairPhoneme-to-VisemeMapping
12 18
[Pre ] . . . . . . . . . . . . . . . . . . . . .
51
Figure . Important Areas and Sections of the Face
02 21
[EFH ] . . . . . . . . . . . . . . . . . . . .
52 10 22
Figure . Human Skull, Front View [Net ] . . . . .
53 10 22
Figure . Human Skull, Side View [Net ] . . . . . .
54 12 23
Figure . Muscle Structure [mus ] . . . . . . . . . .
55 10 24
Figure . Muscles of the Face and the Head [Net ]
ix
list of figures
x
56
Figure . Muscles of (a) the Mouth and (b) the Lips
12 24
[ana ] . . . . . . . . . . . . . . . . . . . . .
57 12 25
Figure . Origins of Muscles on the Skull [ana ] . .
58
Figure . Origins of Muscles on the Maxillary Bone
12 26
and Part of the Zygomatic Bone [ana ] . .
59 9 02 27
Figure . AU (Nose Wrinkler) [EFH ] . . . . . . .
510 10 27
Figure . Bones that form the Orbit [Net ] . . . . .
511 10 02 28
Figure . AU (Upper Lip Raiser) [EFH ] . . . . .
512 12 02 29
Figure . AU (Lip Corner Puller) [EFH ] . . . . .
513 13 02 30
Figure . AU (Sharp Lip Puller) [EFH ] . . . . .
514 14 02 31
Figure . AU (Dimpler) [EFH ] . . . . . . . . . .
515 15 02 32
Figure . AU (Lip Corner Depressor) [EFH ] . .
516 16 02 32
Figure . AU (Lower Lip Depressor) [EFH ] . . .
517 17 02 33
Figure . AU (Chin Raiser) [EFH ] . . . . . . . .
518 20 02 34
Figure . AU (Lip Stretcher) [EFH ] . . . . . . .
519 22 02 35
Figure . AU (Lip Funneler) [EFH ] . . . . . . .
520 12 35
Figure . Layers of the Skin [ski a] . . . . . . . . . .
61 40
Figure . Sketch of the Muscle Position . . . . . . . .
62
Figure . Quads (highlighted) and Triangles of the
41
Skin Mesh . . . . . . . . . . . . . . . . . . .
63 08 42
Figure . Maya Muscle, Bone and Capsule [DTM ]
64
Figure . Directional Light illuminating the Mater-
44
ial and Skin Shader of the Skin Mesh . . .
71
Figure . Linear Muscle Description: The displace-
ment of P to P(cid:48) is calculated . . . . . . . . . 48
72
Figure . ModellingZygomaticusMajorasPolygonal
49
Muscle . . . . . . . . . . . . . . . . . . . . .
73
Figure . SphincterMuscleDescription: Thedisplace-
ment of P to P(cid:48) is calculated . . . . . . . . . 50
74
Figure . Modelling Orbicularis Oris (a) as upper
and lower Part and (b) as left and right
52
Part . . . . . . . . . . . . . . . . . . . . . . .
75 53
Figure . Final Muscle Model . . . . . . . . . . . . . .
76
Figure . ReferenceSkullModelrequiresRearrange-
55
ment of Muscles . . . . . . . . . . . . . . . .
81
Figure . (a) Geometry and (b) initial Sticky Weight
58
of the Upper Skull . . . . . . . . . . . . . .
82
Figure . Pivot Point of the Mandible seen from (a)
59
Front and (b) Side. . . . . . . . . . . . . . .
83
Figure . (a)StickyWeightoftheMandible. (b)Trans-
lation, Rotation and (c) Combinations of
59
them enable Mouth Opening . . . . . . . .