Manifold-Kernels Comparison in MKPLS for Visual Speech Recognition

0.33 MB·

by  Amr Bakry

Manifold-Kernels Comparison in MKPLS for Visual Speech Recognition
AmrBakry,AhmedElgammal
RutgersUnivesity
[email protected], [email protected]
Abstract videoattractstheresearchers.Graphicalmodelshavebeenused
Speech recognition is a challenging problem. Therefore, extensivelyinVSRandAVSR.Onetechniquethatcanbeused
consideringthevisualinformationbesidestheacousticstream, toextractthedynamicsinthevideoisHiddenMarkovModel
isessentialforimprovingtherecognitionaccuracyinreal-life (HMM).HMMencodesthestochastictemporalrelationshipbe-
un-constraint situations. MKPLS is shown to be successful tween sequence of observations [6]. In [7], HMM was used
framework for solving visual speech recognition. However, forencodingthevisualdynamicsofspeechusingActiveShape
6 MKPLSlacksin-deepanalysistoleverageitspower.Thiswork Model(ASM)andActiveAppearanceModel(AAM).Amore
1 isintendedtostudytwocorebuildingblocksinthepipelineof general graphical models, Dynamic Bayesian Network(DBN)
0 MKPLS:manifoldparameterizationandmanifoldkernel. For model, has been used in [8] with different visual articulation
2 manifoldparameterization,westudytheeffectofchangingthe units called articulatory features. Graph embedding has been
numberofcentersandtheregularizationfactorinordertofind usedin[9]tomodelthetemporalrelationshipbetweenframes,
n thebestparametersofthisstep. Quantifyingthesimilaritybe- thenthegraphisusedlater,in[10],forestimatingacurvethat
a
tweenvisualunitsischallengingstepforcomputingthekernel. representthedynamicsinvideo. Graphicalbasedmethodstry
J
Therefore, we explore three kernel categories: matrix-based, tocapturethesmoothtemporalchangesbetweentheusedvisual
2 curve-basedandsubspace-basedkernels. Intuitionbehindeach units,buttheymayloosesomevisualinformationthatmaybe
2 kernel choice is provided and quantitative comparison among crucialfordiscriminatingsmallspeechchunkslikesingleletter

them is conducted. In other words, this study is intended to utterance.
]
V reasonaboutthekernelchoiceforVSR.Wecomparedifferent Ontheotherhand,theworkin[11]isbasedonextracting
kernelsusingMKPLSframework.Weusetwopublicdatasets: asinglespatio-temporalfeaturevectorforrepresentingthevi-
C
OuluVsandAvLettersdatabases. sualandtemporalinformationforthewholespeechvideo. In
. [12] optical flow was used for extracting the whole word fea-
s
c 1. Introduction tures. These two approaches outperform in the case of small
[ sizevideosbutitmightbesensitivetoframeoutliers.Manifold-

Audiovisualspeechrecognition(AVSR)hasbeeninvestigated KPLS(MKPLS)frameworkisproposedin[13]. Itfindsacon-
1
intensivelyinthelastfewdecades[1]. Speciallyafterbimodal ciselow-dimensionalembeddingforeachvisualunit. MKPLS
v
fusionofaudioandvisualstimuliinperceivingspeechhasbeen has three phases: manifold parameterization, manifold latent
1
demonstrated by the McGurk effect [2]. For example, when space embedding and manifold classification. It uses Kernel-
6
the spoken sound /ga/ is seen as /ba/, most people perceive based supervised dimensionality reduction technique, namely,
8
the sound as /da/ [2]. More specifically, with the advances in Kernel-PLS(KPLS)[14].
5
computervision, visualspeechrecognition(VSR),alsocalled ChoiceoftheusedkernelinKPLSisverycriticaltoachieve
0
lipreading,haveattractedresearchattention[3]. VSRsystems thebestperformance.Thisparthasnotbeendiscussedindetails
.
1 gainimportancewiththeneedforcontrollingmachinesverbally in[13]. Inthiswork,weexploreanddiscussseveralsimilarity
0 innoisyacousticenvironment.Anexampleofthatisthecaren- measure for visual units. In MKPLS, visual units are repre-
6 vironment,wherethenoise(e.g. frommotorandradio)makes sentedbytheirmanifoldparameterizations.Eventhoughkernel
1 itverydifficultforaudiospeechrecognition.Anotherpotential discussionisdonefromthepointofviewofMKPLS.Ourfind-
:
v example is to control robots in outer space where there is no ingscanbegeneralizedtobeappliedforotherframeworks.We
i mediaforaudiotransmission. applyMKPLSforAVLetters[7]andOuluVs[11],andtheex-
X Mapping between phonemes and visemes 1 tends to be perimental results contrast the performance variation between
r many-to-one,i.e, thesamevisemecanappearformanydiffer- thekernels.
a entphonemes. Thisshowsthatvisualinformationsolelymight Afterthisintroduction,theproblemstatement,mathemati-
notbeenoughforachievingthespeechrecognitiontask. Asa calmodelingandtheMKPLSframeworkarepresentedinSec-
result, VSR is challenging problem, specially, when using in- tion 2.2. Then, the used kernels are presented in-details in
formationonlyfromplanmarker-lessandreal-lifeimages. Section 3. Finally, the experimental results and the applied
Ontheotherhand,speakeridentificationistightlycoupled databasesarelistedinSection4.
problemwithspeechrecognition[4,5,3]. Speakeridentifica-
tion is defined as the ability to identify the speaker within a 2. Manifold-KPLS
groupofusersfromsolelyspeechrelatedfeatures,likevoiceor
mouthmotion. 2.1. ProblemDefinition
Theappearanceisnottheidealfeaturestobeusedforsolv-
Given set of visual units, we need to recognize new test unit,
ingthelipreadingproblem,whilethedynamicsintheutterance
andinfertheidentityofthespeaker. Thevisualunitcouldbe
1Visemeisthevisualphoneme.Itisdefinedasthesmallestdiscrim- viseme,wordoracompletesentence. Foreachtrainingvisual
inativeunitforvisualspeech unit is assigned to specific speech class and specific speaker.
Both the training and testing visual units are represented by anymanifoldM ,representedbyitsparameterizationC ,the
ν ν
sequenceofimages(frames)extractedfromthespeechvideo. correspondingembeddedpointcanbecomputedby
Eachframeexposesonlythemouthareaofthespeaker.
t =v R. (3)
ν ν
2.2. Frameworkdescription
Where the projection matrix R is learned from KPLS algo-
Manifold Kernel Partial Least Squares (MKPLS) framework rithm, and v = K(C ,.) ∈ RN. K measures the simi-
ν ν
is proposed in [13]. For convenience, we briefly describe the
laritybetweenC andalltrainingmanifoldparameterizations
ν
mathematicalmodelandtheframeworkpipelineshere.
{C ,k = 1···N}. The choice of kernel K and its compu-
Letusdenotethek-thsequencebyS = {xk ∈ RD,i = k
k i tation is discussed in detailed in Section3. Because we solve
1···n }, where the image xk ∈ RD. Let y represents
k i k twoproblems,speechrecognitionandspeakeridentification,we
the class labels for the k-th sequence. For the particular
learnoneRforeachtask.WelearnR basedonspeechlabels,
c
case of speech recognition and speaker identification, y ∈
k andwelearnR forspeakeridentificationwithsubjectlabels.
p
{c ,··· ,c }×{p ,···p }.Herec isthespeechlabel,andp
1 K 1 L i j As a result, for each v , we get two embedding in two latent
istheperformeridentity. LetMk ⊂ RD isalow-dimensional spaces:tc =v R intνhespeechlatentspaceandtp =v R
manifold connects the images of sequence S . The basic as- ν ν c ν ν p
k inthespeakerlatentspace.
sumptionisthatallthesemanifolds(M ∀k)aretopologically
k Finallyinmanifoldclassification,givenalatentspaceem-
equivalent,howevereachofthemhasdifferentgeometryinRD.
beddingt ,MKPLSusesseveraltechniquestoclassifyitsuch
ν
Thisassumptionisstatedclearlyin[13].
asRegressionforclassification(RfC)[13],Supportvectorma-
In principal, MKPLS pipeline has three phases: individ-
chines (SVM) and K-nearest neighbor (KNN). In the latent
ualmanifoldparameterization, latentspaceembeddingandfi-
space of speech, we want to infer the speech label (c) while
nallyinference/classification. ConsiderthemanifoldM con-
k inthespeakerspace,weneedtoinferthesubjectlabel(p).
nects the n frames of specific Visual Unit (VU). Assuming,
k
that we have a unified manifold U. The result of individual
3. Manifold-to-manifoldKernels
manifoldparameterizationisasinglerepresentationforVU
whichholdsthetopologicaldeformationofMkwithrespectto The parameterization, extracted out of the first phase of MK-
U.Anyfurtherprocessingisdonebasedontheseparameteriza- PLS,holdsthedynamicsineachvideowhichencodesspeech-
tions. relatedinformationalongwithspeaker-relatedinformation.Be-
ThemanifoldMkisrepresentedbyaparameterizationCk cause MKPLS uses kernel-based approach for dimensionality
withrespecttoasetofbasis{ψ1,ψ2,··· ,ψn}.Thesebasisare reduction, the kernel choice is critical for achieving the best
anonlinearfunctionofpointsonU. WeuseGaussianRadial performance. In this section, we investigate several types of
Basis Function (Gaussian-RBF): ψi(z) = exp(σ(cid:107)z−wi(cid:107)) kernels.
where wi;i = 1···n are fixed points on U. The goal is to MKPLS claims that, to define manifold-to-manifold ker-
findaregressionfunctionγ(t)=C(cid:62)kΨ(t)whichminimizethe nel, it suffices to define it in the parameterization space,i.e,
objectivefunction K (M ,M ) =. K(C ,C ). Therefore, we need to
manifold i j i j
definekernelsoverthespaceofparameterizations,whichcon-
(cid:88)nk (cid:13)(cid:13)xk−γk(zk)(cid:13)(cid:13)2+λΩ[γk], (1) sequently, measure the similarity between manifolds in terms
(cid:13) i i (cid:13) oftheirgeometricdeformationfromthecommonmanifoldrep-
i
resentation. MKPLS gives us the ability to plugin any valid
where (cid:107)·(cid:107) is the Euclidean norm, Ψ(t) = kernel.Inthissection,weinvestigateseveralchoiceofkernels:
[ψ1(zt),ψ2(zt),··· ,ψn(zt)](cid:62),Ωisaregularizationfunction matrix-basedkernels,curve-basedkernelsandsubspace-based
thatenforcesthesmoothnessinthelearnedfunction, andλis kernels.
the regularizer. Representer theory helps to find closed form
forCas 3.1. Matrix-basedkernels
C(cid:62) =(A(cid:62)A +λG)−1A(cid:62)X(cid:62), (2)
k k k k k Sincethedimensionalityofallparameterizationsisunique,we
whereA isann ×nmatrixwithA =exp(σ(cid:107)z −w (cid:107))
k k (ij) i j canmeasurethesimilaritybetweenthembymeasuringthesim-
andGisann×nmatrixwithG = exp(σ(cid:107)w −w (cid:107)).
(ij) i j ilaritybetweenthecorrespondingcolumn. Thisistheideabe-
X isthen ×DdatamatrixforUV .Thedetailsoflearning
k k k hindthematrix-basedkernels.
themanifoldparameterizationcanbefoundin[13].
The choice of λ and n is crucial for better performance.
3.1.1. Cosine-similaritykernel(Cosine)
Figure 1 shows the trade-off between value of λ and n. This
choicedependsupontheapplication. Inthiswork,weneedto We can measure the similarity between columns using cosine
capturethesmoothdynamicsinthevisualunits. Therefore,we theanglebetweenthem. Asaresult,theoverallsimilaritybe-
chooseλ = 50. Itisclearthatn = 16exposemorevariations tween two parameterizations is the sum over all colmn-wise
thanwithn=8.Moreinformationcanbeusefulinsomecases similarities. Therefore, thecosine-manifoldkernelcanbede-
andcanbemoreconfusinginothers. Threrefore,inSection4, finedas
weshowbothconfigurations. K (C ,C )= tr(CiC(cid:62)j )2 , (4)
In Latent space embedding, kernel partial least squares cos i j ||C || ||C ||
i F j F
(KPLS) [14] is adopted for embedding the parameterizations
where(cid:107)·(cid:107) ismatrixFrobeniusnorm.
{C ,k = 1···N} into a low-dimensional latent space Rm, F
k
as{t ∈ Rm,k = 1···N}. KPLSissupervisedmethod,so
k 3.1.2. Euclidean-distancekernel(Eculid)
itusesthesetoflabels{y ,k = 1···N}fortheembedding.
k
Supervisedembeddingguaranteestoacheivethemostconcise Inthiskernel,wemeasuretheEuclideandistancebetweenthe
andinformativelow-dimensionallatentspaceembedding. For i-thcolumninparameterizationC (u )anditscorresponding
1 1i
Figure1:TheparameterizationCisD×nmatrix.EachplothasDlines,andeachlineisaplotforvaluesprogressionofarowinC.
Atlargevaluesofλtheparameterizationissmoothenoughtocapturelargedynamicsinthevisualunit.
Table1:SubjectSemi-dependentspeechrecognitiononOuluVsdatabase
n=8and LBPu2 n=16and LBPu2
1×2 1−8×8 1×1 1−8×8
m= 10 30 50 80 100 130 200 10 30 50 80 100 130 200
Cosine 62.19 78.13 81.72 81.41 81.72 82.03 81.09 58.28 77.19 79.22 79.53 79.22 79.84 79.53
Euclid 61.25 79.06 79.38 79.53 79.84 80.16 78.91 56.72 75.16 75.00 75.63 75.94 75.31 74.69
EditDist 62.50 75.63 66.72 43.44 22.81 21.41 22.34 59.53 70.16 61.25 41.25 35.00 24.69 15.62
Frechet 29.53 27.81 25.47 17.34 15.97 14.53 11.88
Grassm 28.91 37.34 41.87 42.19 39.69 37.66 31.88 24.38 26.41 29.53 28.44 26.25 25.63 25.63
GrassmCC 28.91 37.34 41.87 42.19 39.84 37.34 27.81 24.53 26.41 29.53 28.44 25.94 25.31 21.72
GrassmDiff 28.13 35.00 37.81 39.17 37.29 31.67 25.63
columninparameterizationC (u ).Hence,theoverallmatrix 3.2.2. Edit-distancekernel(EditDist)
2 2i
kernelδ=(cid:80)n ||u −v ||2,andthematrixsimilarityis
i=1 i i 2 The idea of this metric is similar to minimum edit distance
between strings. Two main difference between EditDist and
K(C ,C )=exp(−ωδ) (5)
1 2 Frechetalgorithms: inEditDisttheoveralldistanceisthesum
ofdistancebetweenallmatcheswhileFrechettakesthemaxi-
whereωisanormalizationfactor.ForK(.,.)tobevalidkernel,
mumofallmatches,andEditDistconsidersunmatchedpoints
it needs to be symmetric positive definite matrix (SPD). The
asbeingmatchedwiththeoriginwhileFrechetignorestheun-
exponentialfunctiontakescareofthepositivedefinitenesspart.
matched points. Emperically, we found that the best column-
Forthesymmetry,theuseddistancemeasureshouldbemetric,
wise distance, in both EditDist and Frechet, is the Euclidean
whichissatisfiedforEuclideandistancecase.
distance.
3.2. Curve-basedKernels
3.3. Subspace-basedKernel
Inthiscategory,weconsiderthecolumnsoftheparameteriza-
Each parameterization C represents n-dimensional subspace
tionmatrixaspointsinRD,andthematrixdefinesacurvecon- inRD.Therefore,wecankusesubspace-to-spacemetrictomea-
necting those points. The matching between columns should
sure the similarity in parameterization space. This gives the
obeytheordering.Thismeansthatiftwocolumnsi,jfromthe
mostgeneralcomparisonbetweenmatrices. Becauseitconsid-
firstmatrixmatchthecolumnsu,v fromothermatrixrespec-
ersthesubspacespannedbythecolumnsofeachparameteriza-
tively,theu≤viffi<j. Foreachofthefollowingdistances,
tionwithoutencodinganyordering.
theparameterizationkerneliscomputedusingEqn5.
3.3.1. Grassmanniankernel
3.2.1. Fre´chet-distanceKernel(Frechet)
EverycoefficientmatrixC isD×d.SinceD(cid:29)d,henceC
k i
Fre´chetdistanceisaknownmetrictomeasurethedistancebe- representddimensionalsubspaceinRD.Therefore,thematrix
tweentwocurves, thattakesintoaccountthelocationandor- CbelongstoGrassmannianmanifoldG . Formoredetails
D,d
dering of the points along the curves. Here, we use discrete aboutGrassmannianmanifolds,thereaderisreferredto[15].
Fre´chetdistance,alsoknownascouplingdistance,inwhichwe There are several approaches for measuring the similarity
assumethatthecurvesarepiece-wiselinear.Thebasicideathat, onGrassmannianmanifold,weusetheonedefinedin[16].
eachpointinonecurveismatchedwithitsclosestpointonthe
othercurve,andthedistancedwillbethemaximumEuclidean K =a Kcc+a Kproj (6)
ij 1 ij 2 ij
distancebetweeneachtwomatchedpoints. Attheend,notall
pointsarematchedbetweenthetwocurves. Where Kproj, Kcc are the projection kernel and the canon-
ij ij
Table2:SpeakerIndepenent-speechrecognitonAccuracyonOuluVsdatabase
n=8and LBPu2 n=16and LBPu2
1×2 1−8×8 1×1 1−8×8
m= 10 30 50 80 100 130 200 10 30 50 80 100 130 200
Cosine 49.22 50.00 51.41 49.84 50.00 49.84 50.00 50 51.88 49.06 49.06 50.31 50.00 50.63
Euclid 48.59 55.16 55.78 55.00 55.47 55.47 55.00 48.44 61.25 58.44 57.50 57.81 58.75 57.19
EditDist 47.97 50.94 41.25 26.09 21.72 18.13 16.41 48.44 48.44 43.75 37.81 31.56 21.87 16.25
Frechet 11.25 13.28 12.50 12.34 13.75 13.44 13.75
Grassm 29.84 30.31 32.34 31.09 28.59 24.84 31.56 22.19 23.75 25.00 19.69 21.25 19.06 15.31
GrassmCC 29.69 30.31 32.34 31.09 28.75 23.91 23.28 22.19 23.75 25.00 19.69 21.25 19.37 17.19
ical correlation kernel respectively, and a ,a are weighting selfvsLBPoftheimagesconcatenatedwithparameterization
1 2
constants. The projection kernel is defined by Kproj = ofthediscretedifferencebetweenthoseimages.
ij
(cid:13)(cid:13)Π(cid:62)i Πj(cid:13)(cid:13)2F whereΠk istheorthogonalversionofthecoeffi-
cientmatrixC ,computedbyGram-Schmidtorthogonalization 4.2.1. Visualspeechrecognition
k
algorithm.Thecanonicalcorrelationkernelisdefinedby
Two test protocols has been adopted for visual speech recog-
Kcc = max max a(cid:62)b (7) nition:SpeakerIndependent(SI)andSpeakerSemi-Dependent
ij p q
ap∈span{Πi}bq∈span{Πj} (SSD)asdefinedin [13].
Subjecttoa(cid:62)a = b(cid:62)b = 1ifp = q,and0otherwise. We SpeakerSemi-DependentVSR(SSD):Herewetestonone
p q p q repeatoftheavailablevideosandtrainbasedontheremaining
use two Grassmannian kernels: Grassm defined by Eq 6 and
repeatsforthesamesubjects. Inthisconfigurationallsubjects
GrassmCCdefinedbyEq7.
andphrasesarepresentedinthetrainingset. Table1showthe
SinceGrassmanniandistancedoesnotconsidertheorder-
SSDspeechrecognitionaccuracyforOuluVsdatabasewiththe
ingoftheparameterizationcolumns,wecanencodesometem-
twofeatureconfigurations. Table4showstheresultofmatrix-
poralinformationbyusingtheparameterizationofdifferenceof
basedkernelsandcurve-basedkernelsappliedavletters.
theinputfeatures.WedenotethisexperimentbyGrassmDiff.
4. Experiments
Table4:SSDspeechrecognitiononAvLetters
4.1. Databases n=8and LBPu2
3×4 1:4×8
WeapplyMKPLSforOuluVsdatabase[11]. OuluVshasten m= 10 30 50 80 100 130 200
differenteverydayphrases. Eachphraseisutteredby20sub- Cosine 50.77 56.67 60.77 62.82 63.85 64.49 63.85
jectsuptofivetimes. Weusethesametestprotocoalusedin Euclid 51.41 56.41 60.38 64.49 65.13 64.52 64.74
[13]. The frame rate was set to 25 fps. The dataset contains EditDist 51.41 56.54 60.51 64.36 65.13 65.00 64.74
sequenceofimagesformouthareawithaverageresolutionof Frechet 23.59 34.62 36.28 34.49 35.64 34.49 29.23
120×60pixels.LBP[17]visualfeaturesisextactedfromim-
ages. Twofeatureconfigurationshavebeenused: thefirstcon-
Speaker Independent VSR (SI): the challenge here is to
figurationisLBP withn = 16(16basisforGaussian-
1:8×8 showthescalabilityofthemodel,meanhowfarthemodelcan
RBF) and the second one is LBPu2 with n = 8. We
1×2 1:8×8 recognizethespokenphrasebasedonthedynamicsevenifthe
also apply MKPLS for AVLetters database [7] which has ten
speakerisnotseenbeforeinthetrainingset.Inthisexperiment,
subjects. Each speaker repeats every English letter (A···Z)
weuseone-speaker-out.Table2showtheSIspeechrecognition
exactlythreetimes, withatotalof780videosequences. The
accuracyforOuluVsforthetwoconfigurations.
speakerwasrequestedtostartandendutteranceofeveryletter
inaneutralstate(mouthclosed). Weapplysinglefeaturecon-
4.2.2. SpeakerIdentification(SpId):
figuration: 3×4 cell-grid with four-resolutions LBP features
( LBPu2 )withn=8.
3×4 1:4×8 The goal in this experiment is to find the speaker within the
Inallexperiments,therecognitionrateismeasuredasthe
registersetofusers. Thechallengeistofindthespeakerfrom
ratiobetweenthecorrectlyrecognizedclipsandthetotalnum-
the limited available information in the mouth area. We take
berofclips.
onerepetitionoutfortesting,andtrainoverallotherrepetitions.
Table3showsthespeakeridentificationaccuracywhenapplied
4.2. ExperimentalResults
toOuluVsforthetwotestconfigurations.
In this section, we present the empirical results of MKPLS
whenpluginedwitheachoneofthekernelsdescribedinSec- 4.3. Discussion
tion3. Togiverealcomparisonbetweentheproposedkernels,
we need to explore different parameters of MKPLS pipeline: From the numbers, we can clearly see the superiority of both
Thenumber(n)ofGaussain-RBFbasisψthatweusetolearn techniques of Matrix-based kernels (Cosine and Euclid) in all
the individual manifold parameterization, we show results for test and features configurations. For m = 10, EditDist gives
n=8,16.Thedimensionlaityofthemanifoldlatentspace,we thebestresultsforSSD-speechrecognitoinandSpeakerIden-
usem=10,30,50,80,100,130,200,whichcoverwiderange tification. GrassDiff gives slightly better resutls than Grassm
ofthepossiblevalues. abd GrassCC, since it encodes temporal information. Frechet
EspeciallyforGrassmann-basedkernels3,weshowtheaf- kernelprovesfailureinthisapplication.ForAvletters,EditDist
fectofusingtheparameterizationoftheLBPoftheimagesit- andEuclidgivethebestrecognitionrate.
Table3:SpeakerIdentificationAccuracyonOuluVsdatabase
n=8and LBPu2 n=16and LBPu2
1×2 1−8×8 1×1 1−8×8
m= 10 30 50 80 100 130 200 10 30 50 80 100 130 200
Cosine 93.91 99.69 99.69 99.69 99.69 99.69 99.69 92.66 99.69 99.69 99.69 99.69 99.69 99.69
Euclid 93.75 99.53 99.53 99.53 99.53 99.53 99.53 93.91 99.53 99.53 99.53 99.69 99.53 99.53
EditDist 94.22 99.53 99.53 99.37 99.06 71.88 69.69 92.81 99.69 99.53 99.53 99.53 99.69 99.53
Frechet 83.91 95.16 89.06 75.94 64.84 50.63 17.97 88.13 96.09 87.81 62.81 27.66 27.66 27.66
Grassm 84.69 99.38 99.37 99.53 99.06 98.75 97.81 92.19 99.22 99.06 98.91 98.44 98.33 97.56
GrassmCC 84.69 99.38 99.37 99.53 99.06 98.75 97.81 92.19 99.22 99.06 98.91 98.44 98.33 95.94
5. Conclusion [13] A. Bakry and A. Elgammal, “MKPLS: Manifold Kernel
PartialLeastSquaresforLipreadingandSpeakerIdentifi-
WeinvestigatedthekernelchoiceforthemiddlephaseofMK-
cation,”ComputerVisionandPatternRecognition,2013.
PLS framework. We explored three kind of manifold ker-
nels: matrix-based kernel, curve-based kernel and subspace- [14] R.RosipalandL.Trejo,“Kernelpartialleastsquaresre-
based kernel. We compare the kernels based on different pa- gressioninreproducingkernelhilbertspace,”TheJournal
rameterconfigurationforMKPLS.Theexperimentsshowsthat ofMachineLearningResearch,pp.97–123.
usingparameterization-to-parameterizationkernelcandelegate [15] A. Edelman, T. a. Arias, and S. T. Smith, “The Geome-
manifold-to-manifold kernel. The results shows the superior- tryofAlgorithmswithOrthogonalityConstraints,”SIAM
ityofthematrix-basedkernelforvisualspeechrecogniton.For Journal on Matrix Analysis and Applications, no. 2, pp.
speakeridentificationtasks,allkernelsgivesperfectresults. 303–353,Jan.
[16] M. Harandi and C. Sanderson, “Graph embedding dis-
6. References criminant analysis on Grassmannian manifolds for im-
proved image set matching,” IEEE Conference on Com-
[1] G. Potamianos and C. Neti, “Audio-visual automatic puter Vision and Pattern Recognition, pp. 2705–2712,
speech recognition: An overview,” Issues in Visual and
Jun.
Audio-VisualSpeechProcessing.
[17] T. Ojala, “Multiresolution gray-scale and rotation in-
[2] H.McGurkandJ.MacDonald,“Hearinglipsandseeing
variant texture classification with local binary patterns,”
voices,”Nature,vol.264,no.23December,pp.746–748,
PAMI,no.7,pp.971–987.
1976.
[3] D. Shiell and L. Terry, “Audio-Visual and Visual-Only
Speech and Speaker Recognition: Issues about The-
ory,SystemDesign,andImplementation,”Visualspeech
recognition:lipsegmentationandmapping.
[4] J. Luettin, N. Thacker, and S. Beet, “Speaker identifica-
tion by lipreading,” International Conference on Spoken
LanguageProcessing,pp.1–4.
[5] C.SandersonandK.Paliwal,“Identityverificationusing
speechandfaceinformation,”DigitalSignalProcessing,
no.5,pp.449–480,Sep.
[6] L.Rabiner,“AtutorialonhiddenMarkovmodelsandse-
lectedapplicationsinspeechrecognition,”Proceedingsof
theIEEE,no.2,pp.257–286.
[7] I.MatthewsandT.Cootes,“Extractionofvisualfeatures
forlipreading,”PAMI,vol.24,no.2,pp.198–213,2002.
[8] K. Saenko and K. Livescu, “Visual speech recognition
with loosely synchronized feature streams,” IEEE Inter-
nationalConferenceonComputerVision.
[9] Z.ZhouandG.Zhao,“Lipreading: AGraphEmbedding
Approach,”ICPR,pp.523–526,2010.
[10] Z.Zhou, G.Zhao, andM.Pietikainen, “Towardsaprac-
tical lipreading system,” Computer Vision and Pattern
Recognition,2011.
[11] G. Zhao, “Lipreading with local spatiotemporal descrip-
tors,”IEEETransactionsonMultimedia,pp.1–11,2009.
[12] A. Shaikh, D. Kumar, and W. Yau, “Lip Reading using
OpticalFlowandSupportVectorMachines,”IEEEInter-
nationalCongressonImageandSignalProcessing,vol.1,
pp.327–330,Oct.2010.