Speech synthesis method and apparatus, program, recording medium and robot apparatus

26 Pages·2013·1.81 MB·English

US007062438B2
(12) United States Patent (10) Patent N0.: US 7,062,438 B2
Kobayashi et a]. (45) Date of Patent: Jun. 13, 2006
(54) SPEECH SYNTHESIS METHOD AND 6,810,378 B1 * 10/2004 Kochanski et a1. ....... .. 704/258
APPARATUS, PROGRAM, RECORDING 6,823,309 B1 * 11/2004 Kato et a1. ................ .. 704/267
MEDIUM AND ROBOT APPARATUS OTHER PUBLICATIONS
(75) Inventors: Kenichiro Kobayashi, KanagaWa (JP); Macon, M.W.; Jensen-Link, L.; Oliverio, J .; Clements,
Nobuhide Yamazaki, KanagaWa (JP); M.A.; George, EB. “A singing Voice synthesis system based
Makoto Akabane, Tokyo (JP) on sinusoidal modeling” Acoustics, Speech, and Signal
Processing, 1997. lCASSP-97., 1997 IEEE, International
(73) Assignee: Sony Corporation, Tokyo (JP) Conference on, V01; 1 , 21-24,*
( * ) Notice: Subject to any disclaimer, the term of this (Continued)
patent is extended or adjusted under 35 Primary Examineriwayne Young
U'S'C' 154(1)) by 302 days: Assistant ExamineriHuyen X. Vo
_ (74) Attorney, Agent, or F irmiFrommer Lawrence & Haug
(21) Appl' NO" 10/388’107 LLP; William S. Frommer; Thomas F. Presson
(22) Filed: Mar. 13, 2003 (57) ABSTRACT
(65) Prior Publication Data A sentence or a singing is to be synthesized With a natural
Us 2004/0019485 A1 Jan 29, 2004 speech close to the human Voice. To this end, singing
metrical data are formed in a tag processing unit 211 in a
(30) Foreign Application Priority Data singing synthesis unit 212 in a speech synthesis apparatus
200 based on singing data and an analyzed text portion. A
Mar. 15, 2002 (JP) ........................... .. 2002-073385 language analysis unit 213 performs language processing on
text portions other than the singing data. As for a text portion
(51) Int. Cl. regl. stered 1. n a natural metr1. cal d1. ct1. onary, as determ. med by
G10L 13/08 (2006.01) _ _ thi. s language process.m g, correspond. mg natural metr1. cal
(52) US. Cl. ....... ...... 704/260, 704/270, 704/258 data is Selected and its parameters are adjusted in a metrical
(58) Field of Classi?cation Search .............. .. 704/ 270, data adjustment unit 222 based on phonemic Segment data of
_ _ 704060, 258> 267> 205; 84/ 6097610 a phonemic segment storage unit 223 in the metrical data
See aPPhCaUOn ?le for Complete Search hlstory- adjustment unit 222. As for a text portion not registered in
_ the natural metrical dictionary, a phonemic symbol string is
(56) References Clted generated in a natural metrical dictionary storage unit 214,
US PATENT DOCUMENTS after Which metrical data are generated in a metrical gener
* ating unit 221. AWaVeform generating unit 224 concatenates
5,642,470 A 6/1997 Yamamoto et a1. ....... .. 704/270 necessary phonemic Segment data’ based on the natural
5’890’l17 A * 3/1999 sllverman """"""" " 704/260 metrical data metrical data and the sin in metrical data to
6,226,614 B1 * 5/2001 Mizuno et a1. . 704/260 t ’ h f dat g g
6,304,846 B1* 10/2001 George et a1. .. 704/270 generae Speec wave on“ 2"
6,424,944 B1 * 7/2002 Hikawa . . . . . . . . . . . . . .. 704/260
6,446,040 B1 * 9/2002 Socher et a1. ............. .. 704/260 24 Claims, 12 Drawing Sheets
00
FEELING STATE/ 2
210 CHARACTER INFORMATION 220
SPEECH SYNTHESIS
LANGUAGE PROCESSING 215 MBOL 52‘ UNIT
SPEECH SY
UNIT SPEECH SYMBOL SEQUENCE slél?gg?lq-G METRICAL DATA
214 GENERATING UNIT smemc METRIGAL UNIT
213 DATA
NATURAL METRICAL
DlCTlONARY ANALYSIS UNIT UNIT
STORAGE Z23
UNIT 211 212 2%4
TAG SINGING
PROCESSING —> SYNTHESIS WAVEFORM
UNIT UNIT STORAGE GENERATING UNIT
UNIT
TEXT SPEECH
US 7,062,438 B2
Page 2
OTHER PUBLICATIONS Patent Abstracts of Japan, publication No. 02-027397 dated
. . Jan. 30, 1990.
Patent Abstracts of Japan’ pubhcanon NO’ 09244869 dated Patent Abstracts of Japan, publication No. 07-146695 dated
Sep. 19, 1997. J 6 1995
Patent Abstracts of Japan, publication No. 11-184490 dated 1111' ’ '
Jul. 9, 1999. * cited by examiner
U.S. Patent Jun. 13, 2006 Sheet 1 0f 12 US 7,062,438 B2
cow
U.S. Patent
Jun. 13, 2006 Sheet 2 0f 12 US 7,062,438 B2
Input text N 31
t
Analyze tag S2
0
Form singing metrical data ,2, S3
0
Language analysis __, S4
0
Generate metrical data/ N 85
natural metncal data
t
Adjust parameter M 36
0
Speech synthesis N 87
FIG.2
U.S. Patent Jun. 13, 2006 Sheet 3 0f 12 US 7,062,438 B2
QUE
EHj2Iw2mmOm93Pm.mP022?<E0m25<Qum;
|>QQQ QQQQQQ< m
w>QQQwQQQQQQmz<< j w<o m>QQQmQQQQQQmZ<E n wZI
m>QQQQQQQQwQ<U Z w< w>QQQwQQQQQQm<z o <mw .>QQQQEQQQQQO<m E wOO

U.S. Patent Jun. 13, 2006 Sheet 5 0f 12 US 7,062,438 B2
120
119

U.S. Patent Jun. 13, 2006 Sheet 7 0f 12 US 7,062,438 B2
$3252 F826
58m $22
/s /3
sE\
2$58253/%¢
522\218a
6.55322355m02%51ma:;%z 8 m8w
$2A0559¢513 ,‘ ~$@555255 5
aE562|5 %a
22
I
mud/‘252
.5260
%
U.S. Patent Jun. 13, 2006 Sheet 8 0f 12 US 7,062,438 B2
i
m681.231;
"i1f
{“||||||||||||||||||||||||||||||||| |||||||||||||||||||||||||||| |||||||||||||||||||||||||||||||| |L|I_
n__
NI20525 5
__ n__
l......_,.........................-...................................... ....... ............. L,__ r
___
In
A22.,-.w-.-.‘ zm/E_5 2 "52 5I02uH: lmlti1al1imeww-................J.... ............................... . ............ .............
mTmw5Ew5moMoEpE&pmzMnz>sSs3zZ?;?Q8wQm
||||||||||||||||I|||||||N|||||||F| }-iiiIllllll-i1}-Illllilllllllilil
_<o|n_
mmm4,“ m 320"52Q92“:$$mm0m5252w8 ::558a8825:z wmz223“09Eg55.285@o2625220é: 5x2m
nmcsasaasm22
is"_22ZZF5Z2:20QO:022E5E2_ 0Z552Em_Q32O55S.m5E%0E0 E
m1zsaaaN5
_
\/_~n~~~\\_"
_\\\\
.i.... 2i22z088c5E288m8m3%%c“