Automatic Generation of Computer Animation: Using AI for Movie Animation

385 Pages·2002·3.78 MB·English

by  Ruqian Lu

Lecture Notes in Artificial Intelligence 2160
SubseriesofLectureNotesinComputerScience
EditedbyJ.G.CarbonellandJ.Siekmann
Lecture Notes in Computer Science
EditedbyG.Goos,J.Hartmanis,andJ.vanLeeuwen
3
Berlin
Heidelberg
NewYork
Barcelona
HongKong
London
Milan
Paris
Tokyo
Ruqian Lu Songmao Zhang
Automatic Generation
of ComputerAnimation
Using AI for Movie Animation
1 3
SeriesEditors
JaimeG.Carbonell,CarnegieMellonUniversity,Pittsburgh,PA,USA
Jo¨rgSiekmann,UniversityofSaarland,Saarbru¨cken,Germany
Authors
RuqianLu
SongmaoZhang
AcademiaSinica,InstituteofMathematics
Beijing100080,P.R.China
E-mail:{rqlu/smzhang}@math08.math.ac.cn
Cataloging-in-PublicationDataappliedfor
DieDeutscheBibliothek-CIP-Einheitsaufnahme
Lu,Ruqian:
Automaticgenerationofcomputeranimation:usingAIformovieanimation/
RuqianLu;SongmaoZhang.-Berlin;Heidelberg;NewYork;Barcelona;
HongKong;London;Milan;Paris;Tokyo:Springer,2002
(Lecturenotesincomputerscience;2160:Lecturenotesinartificial
intelligence)
ISBN3-540-43114-4
CRSubjectClassification(1998):I.2,I.4,I.3,J.5
ISSN0302-9743
ISBN3-540-43114-4Springer-VerlagBerlinHeidelbergNewYork
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,re-useofillustrations,recitation,broadcasting,
reproductiononmicrofilmsorinanyotherway,andstorageindatabanks.Duplicationofthispublication
orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,1965,
initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer-Verlag.Violationsare
liableforprosecutionundertheGermanCopyrightLaw.
Springer-VerlagBerlinHeidelbergNewYork
amemberofBertelsmannSpringerScience+BusinessMediaGmbH
http://www.springer.de
©Springer-VerlagBerlinHeidelberg2002
PrintedinGermany
Typesetting:Camera-readybyauthor,dataconversionbyBollerMediendesign
Printedonacid-freepaper SPIN:10840290 06/3142 543210
Preface
We are both fans of watching animated stories. Every evening, before or after din-
ner, we always sit in front of the television and watch the animation program,
which is originally produced and shown for children. We find ourselves becoming
younger while immerged in the interesting plot of the animation: how the princess
is first killed and then rescued, how the little rat defeats the big cat, etc. But what
we have found in those animation programs are not only interesting plots, but also
a big chance for the application of computer science and artificial intelligence
techniques. As is well known, the cost of producing animated movies is very high,
even with the use of computer graphics techniques. Turning a story in text form
into an animated movie is a long and complicated procedure. We came to the con-
clusion that many parts of this process could be automated by using artificial in-
telligence techniques. It is actually a challenge and test for machine intelligence.
So we decided to explore the possibility of a full life cycle automation of com-
puter animation generation. By full life cycle we mean the generation process of
computer animation from a children(cid:146)s story in natural language text form to the
final animated movie. It is of course a task of immense difficulty. However, we
decided to try our best and to see how far we could go.
After nearly ten years of effort, we have been successful in developing a rough
prototype of this technique which resulted in a software package called SWAN. A
cartoon produced by SWAN, called (cid:147)the three brothers(cid:148), was shown on CCTV
(Chinese Central Television Station) in October 1995. We are aware of the fact
that this prototype is rough, because our technique is still far from being mature
enough to produce delicate animated movies of commercial value. Nevertheless,
we have been encouraged by the conclusion of an appraisal committee chaired by
Academician, Prof. Yunmei Dong: the industrialization of the SWAN technique
would hold great potential for economical and social benefits.
It is appropriate to make a note here about the contents of this book. Following
the completion of our SWAN project, we carried out a review and assessment of
the techniques developed in it. Based on the review, we have improved and re-
designed many parts of its methodology, partly in the interest of scientific re-
search, partly for the sake of preparing the implementation of the second version
of SWAN. In this sense, the contents of this book conforms to the new design, not
quite to the originally implemented one.
The SWAN project has received a grant from a special foundation of the Presi-
dent of the Chinese Academy of Sciences. We owe special thanks to the former
President of the Chinese Academy of Sciences, Current President of the Chinese
Association of Science and Technology, Academician, Prof. Guangzhao Zhou; the
Preface VI
former Vice-President of the Chinese Academy of Sciences, Current Vice-
President of the Chinese Association of Science and Technology, Academician,
Prof. Qiheng Hu; and the former head of department of the Bureau for High-Tech,
current Vice-Director of the Institute for Software, Mr. Wenzeng Yang. We are
grateful to the Institute of Mathematics, Institute of Computing Technology, Lab
for Intelligent Information Processing, Lab for Management, Decision, and Infor-
mation Sciences, Center for Brain and Mind Science of the Chinese Academy of
Sciences for their financial support. SWAN has also been supported by the Chi-
nese Natural Science Foundation (especially project 69733020), by Key Projects
of the Ministry of Science and Technology and by 863 High-Tech Projects.
More than 30 graduate students have participated in the SWAN project and
made their contributions. Following is a list of some of their names: Qing Geng,
Wenhong Zhu, Xiaobin Li, Lixin Yu, Zhenghao Jin, Dejie Yang, Yue Xu, Haihu
Shi, Ronglin Wan, Zhaobin Chen, Ying Zhao, Yuan Li, Yi Luo, Bo Gao, Yuejiao
Zhou, Yinghao Ma, Yizhen Li, Yujie Wang, Haixia Du, Lu Li, Ping Yang, Lu
Fan, Guangfeng Ji, Xuesheng Zhang, Lengning Liu, Fan Yang, Xiaolong Jin,
Chengmin Sun, Xiangliang Meng, Jingwei Cai.
We are also very thankful to Prof. Zichu Wei, who has made an important con-
tribution to the SWAN project during discussions about the design and imple-
mentation of the SWAN software, and through his involvement in the manage-
ment of the project.
We thank Prof. Christian Freksa of the University of Hamburg, Prof Bernd
Krieg-Brueckner and Prof. Manfred Wischnewski of the University of Bremen for
their invitation to give a series of seminar talks about SWAN at these universities.
We also thank Prof. Manfred Broy of the Technical University of Munich and
Prof. J(cid:246)rg Siekmann of the University of the Saarland for their invitation to intro-
duce SWAN at these universities. We owe special thanks to Prof. J(cid:246)rg Siekmann.
Without his encouragement and support the publishing of this manuscript would
have been impossible. While preparing the talks, the first author of this book had
the opportunity to further improve the manuscript.
Last, but not least, we would like to thank Ms. Wenyan Zhang and Dr. Cungen
Cao for their valuable help in the final editing of the manuscript. We also thank
the editors of the LNAI series and Springer-Verlag, and Ms. Karin Henzold for
their kind support and help in the editing and publishing of this manuscript.
June, 2001 Ruqian Lu
Songmao Zhang
Contents
1 Overview of Research on Computer Animation and Related Topics............1
1.1 Key Frame Based Computer Animation......................................................1
1.1.1 Basic Ideas and Numerical Techniques................................................1
1.1.2 Example of a Desktop System for Animation Generation: The Life
Forms.............................................................................................................2
1.2 Articulate Planning: Kinematics and Dynamics..........................................3
1.3 Path Planning...............................................................................................5
1.4 Camera Planning..........................................................................................7
1.5 Music Planning............................................................................................9
1.6 Natural Language Understanding and Story Analysis...............................12
1.6.1 Early Approach of Story Grammar....................................................13
1.6.2 Scripts and Frames.............................................................................15
1.6.3 The Plan Understanding Approach.....................................................16
1.6.4 The Story Structure............................................................................19
1.7 Plot Planning and Generation....................................................................21
1.8 Generate Computer Animation from Natural Language Instructions........23
1.9 Generate Animation from Natural Language Stories.................................26
2 SWAN: Full Life Cycle Automation of Computer Animation.....................29
2.1 An Overview of SWAN.............................................................................29
2.1.1 How SWAN Was Born......................................................................29
2.1.2 The FLICA Methodology...................................................................29
2.1.3 Five Levels of Representation............................................................30
2.2 Moon Light: The Natural Language Level Representation........................31
2.2.1 The Limited Chinese Natural Language.............................................31
2.2.2 Check the Validity of the Story Based on Commonsense
Knowledge..................................................................................................31
2.3 Semantic Analysis and Its Representation.................................................32
2.3.1 A Representation Based on Case Frames...........................................32
2.3.2 A Short Introduction to the GF1 Syntax.............................................33
2.3.3 The Roles...........................................................................................33
2.3.4 The Objects........................................................................................35
2.3.5 The Environments..............................................................................36
2.3.6 The Actions........................................................................................37
2.3.7 An Example........................................................................................37
2.4 Pragmatic Analysis and Qualitative Script Planning.................................39
2.4.1 Pragmatic Analysis.............................................................................39
VIII Contents
2.4.2 The Qualitative Planning....................................................................40
2.4.3 The Rainbow Language......................................................................42
2.4.4 Camera Planning and Light Planning.................................................45
2.4.5 Light and Color Planning...................................................................46
2.4.6 An Example........................................................................................47
2.5 Quantitative Planning and Representation.................................................49
2.5.1 The Evergreen Language (cid:151) Global Definition..................................49
2.5.2 The Evergreen Language (cid:151) Detailed Definition...............................51
2.6 Geometric Planning...................................................................................54
2.6.1 Layout Planning.................................................................................54
2.6.2 Path Planning: The Tiger Planner.......................................................55
2.6.3 Path and Move Planning of Cameras.................................................60
2.6.4 Modification of Objects.....................................................................60
2.7 Summary, Acknowledgements, and Prospects..........................................60
2.8 Notation Used in This Book......................................................................62
3 Understanding the Limited Chinese Natural Language...............................65
3.1 The Chinese Language...............................................................................65
3.2 The Moon Light Language........................................................................67
3.2.1 Characters and Words........................................................................67
3.2.2 Syntactic Construction.......................................................................69
3.2.3 The Syntactic Rules............................................................................73
3.3 Commonsense Knowledge Check and Unification Grammar...................76
3.4 Semantic Disambiguation Based on Commonsense Knowledge...............84
4 Story Understanding: The Theory..................................................................93
4.1 Story Representation and Parsing..............................................................94
4.1.1 Case Frame Forest..............................................................................94
4.1.2 An Example of Story and Its Representation.....................................96
4.1.3 Story Parsing Process.........................................................................97
4.2 Forest Grammar.........................................................................................98
4.2.1 A General Introduction to Forest Grammar.......................................98
4.2.2 Definiton of Forest Grammar.............................................................98
4.2.3 A Classification of Forest Grammars...............................................101
4.3 Story Parsing Grammar...........................................................................102
4.3.1 Definition of Story Parsing Grammar..............................................102
4.3.2 An Example of Story Parsing Grammar...........................................104
4.4 Weak Precedence Story Parsing Grammar (WPSPG).............................105
4.4.1 Definition of WPSPG.......................................................................105
4.4.2 Syntactic Parsing Algorithm of WPSPG..........................................110
4.4.3 An Example of Story Parsing...........................................................117
4.4.4 Data about Runtime Efficiency of the Syntactic Parsing of
WPSPG.....................................................................................................118
4.5 Story Parsing Grammar with Attributes...................................................119
4.6 Conclusions..............................................................................................120
Contents IX
4.6.1 Story Parsing Grammar and Rumelhart’s Story Grammar................120
4.6.2 From SPG to SPG with Attributes....................................................121
4.6.3 Significance......................................................................................122
5 Story Understanding: The Practice..............................................................123
5.1 Goal of Story Understanding...................................................................123
5.2 Story Graph: The Shallow Approach.......................................................126
5.3 The Temporal Grammar Approach..........................................................136
5.3.1 Deficiency of the Traditional Approaches........................................136
5.3.2 Introduction to Temporal Models.....................................................140
5.3.3 The Heuristic Temporal Forest Grammar.........................................144
5.4 Abstract Representation: The Deep Approach.........................................150
5.4.1 The Main Idea..................................................................................150
5.4.2 Silver Beach: The Abstract Representation Language.....................152
5.4.3 Determine Story Patterns..................................................................155
5.4.4 Use Story Patterns to Determine Global Information.......................158
5.4.5 Construct New Patterns....................................................................161
5.5 Semantic Disambiguation Based on Deep Commonsense Reasoning.....164
5.5.1 The Anaphora Problem Revisited.....................................................164
5.5.2 A Solution Based on Deep Commonsense Knowledge....................166
6 Plot Planning and Act Planning....................................................................173
6.1 A Knowledge Base of Dramas and Plots.................................................174
6.1.1 A Hierarchy of Drama Agents and Plot Agents...............................175
6.1.2 Generalized Case Frames and Basic Plot Representation.................176
6.2 Skeletal Plot Planning..............................................................................191
6.2.1 Plot Planning through Inheritance and Instantiation.........................191
6.2.2 Plot Planning through Crossover and Mutation................................200
6.2.3 Plot Planning through Abstraction and Partial Presentation.............204
6.3 Qualitative Temporal Planning................................................................207
6.3.1 A Notation for Time Representation in Plots...................................207
6.3.2 Plot Processing at Qualitative Temporal Level................................209
6.4 Dividing the Story in a Series of Acts......................................................212
6.5 Solving the Quantification Problem.........................................................213
6.5.1 Introduction......................................................................................213
6.5.2 Commonsense Problems Relating to the Quantity...........................215
6.5.3 Principles of Quantity-Oriented Commonsense Reasoning.............218
6.5.4 An Algorithm of Quantity Estimation..............................................224
7 Director(cid:146)s Planning........................................................................................229
7.1 The Mission of an Animation Director in SWAN...................................229
7.2 DPRS: A Language for Director Planning...............................................231
7.2.1 Basic Scenarios of the Episodes.......................................................232
7.2.2 Additional Ingredients......................................................................235
7.2.3 Natural Environment........................................................................236
7.2.4 Background......................................................................................236
X Contents
7.2.5 Roles(cid:146) Characteristics.......................................................................237
7.2.6 Relation and Atmosphere among Roles...........................................238
7.2.7 Director(cid:146)s Intention and Attitude......................................................239
7.3 The Generation of Global DPRS.............................................................240
7.3.1 Determine the General Style of a Movie..........................................241
7.3.2 Determine the Basic Rhythms..........................................................243
7.4. The Generation of Local GPRS..............................................................244
7.4.1 Divide the Story into Scenes............................................................244
7.4.2 Determine Principal Roles in Each Scene........................................248
7.4.3 Determine the Basic Scenarios.........................................................248
7.5 Determine and Implement Abstract Ingredients......................................250
7.5.1 The Director(cid:146)s Attitude....................................................................250
7.5.2 Implement Abstract Ingredients.......................................................251
8 Camera Planning............................................................................................255
8.1 The Camera Planning Language Morning Glow.....................................255
8.2 The Work of Edit.....................................................................................256
8.3 The Routine Camera Statements..............................................................259
8.3.1 Classification of Routine Camera Statements..................................260
8.3.2 From DPRS to RCS (1): A Rough Generator of RCS......................264
8.3.3 From DPRS to RCS (2): Ingredient Forest.......................................266
8.3.4 From DPRS to RCS (3): The Case Based RCS Planner...................269
8.3.5 The Virtual Ingredient Forest...........................................................272
8.4 High Level Camera Primitives.................................................................274
8.4.1 Overview..........................................................................................274
8.4.2 Syntax of HLCP...............................................................................276
8.4.3 From RCS to HLCP.........................................................................281
8.5 The Basic Camera Primitives...................................................................283
9 Light, Color, and Role Planning...................................................................287
9.1 Light Planning.........................................................................................287
9.1.1 Glorious: A Language for Light Planning........................................287
9.1.2 Sublanguage Dragon: General Light Effect Specification...............288
9.1.3 Sublanguage Poplar: Particular Light Effect Specification..............292
9.1.4 Generation of Dragon and Poplar Specifications.............................298
9.1.5 Sublanguage BLP: Light Source Specification................................300
9.1.6 A Wide Spectrum Language............................................................302
9.2 Color Planning.........................................................................................304
9.2.1 The Importance of Color Use...........................................................304
9.2.2 Jade: A Language for Color Requirement Planning.........................305
9.2.3 Generation of Jade Statements.........................................................311
9.3 Role Portraying........................................................................................312
9.3.1 Behavior Planning............................................................................312
9.3.2 Outlook Planning..............................................................................313