3D Face Recognition Performance under Adversarial Conditions

16 Pages·2007·1.56 MB·English

ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
3DFACERECOGNITIONPERFORMANCEUNDERADVERSARIALCONDITIONS
ArmanSavran1,OyaC¸eliktutan1,AydınAkyol2,JanaTrojanova´ 3,HamdiDibekliog˘lu4,
SemihEsenlik1,NesliBozkurt5,CemDemirkır1,ErdemAkagu¨ndu¨z5,KeremC¸alıs¸kan6,Nes¸e
Alyu¨z4,Bu¨lentSankur1,I˙lkayUlusoy5,LaleAkarun4,TevfikMetinSezgin7
1 Bog˘azic¸iUniversity,DepartmentofElectricalandElectronicEngineering,˙Istanbul,Turkey
2 ˙IstanbulTechnicalUniversity,DepartmentofComputerEngineering,Turkey
3 DepartmentofCybernetics,UniversityofWestBohemiainPilsen,CzechRepublic
4 Bog˘azic¸iUniversity,DepartmentofComputerEngineering,Turkey
5 MiddleEasternTechnicalUniversity,Dept. ofElectricalandElectronicsEngineering,Turkey
6 MiddleEasternTechnicalUniversity,InformaticsInstitute,Turkey
7 UniversityofCambridge,ComputerLaboratory,UK
ABSTRACT 1.1. 3DRecognitionofNon-cooperativePersons
Weaddressthequestionof3Dfacerecognitionandexpression Mostoftheexistingmethodsforfacialfeaturedetectionandper-
understandingunderadverseconditionslikeillumination,pose, sonrecognitionassumefrontalandneutralviewsonly. Inthese
and accessories. We therefore conduct a campaign to build a studies data are therefore collected from cooperative subjects,
3D face database including systematic variation of poses, dif- whoexposetheirfacesinastillpositioninfrontofthescanner,
ferent types of occlusions, and a rich set of expressions. The frontalposes,andavoidextremeexpressionsandanyoccluding
expressions consist of a judiciously selected subset of Action material. This may be uncomfortable to subjects. In fact, the
Units as well as the six basic emotions. The database is de- second generation techniques vie for improved quality of life
signed to enable various research paths from face recognition viabiometry,henceenrollmentandupdatingshouldproceedin
tofaciallandmarkingandtoexpressionestimation.Preliminary waysthatdonotencumberthesubject. Ontheotherextreme,a
resultsarepresentedontheoutcomeofthreedifferentlandmark- subject, awareofpersonidentificationcameras, maytrytoes-
ing methods as well as one registration method. As expected, chewbeingrecognizedbyposingawkwardlyandworsestill,by
observednon-neutralandnon-frontalfacesdemandnewrobust resortingtoocclusionsviadanglinghair,eyeglasses,facialhair
algorithmstoachieveanacceptableperformance. etc.Also,the3Ddatamayhavedifferenttranslation,rotationor
scalingduetothecontrolledenvironmentalparameterssuchas
KEYWORDS
theacquisitionsetupanddeviceproperties.Boththenatural,un-
3D face database – Facial expressions – Facial landmarking – controlledbehaviourofsubjectsandthemimicsandacrobatics
Facerecognition oftheeschewerseriouslydamagetheperformanceofboth2D
and 3D recognition algorithms. Furthermore, we believe that
3Dcaptureoffacedatacanmitigatesignificantlymostofthese
1. INTRODUCTION
effects. Usingadatabasebuiltspecificallyforthispurpose,we
testtheperformanceof3Dfaceidentificationalgorithmsaswell
Thehistoryofautomatedfacerecognitiondatesbacktotheearly
asthoseofautomaticlandmarkingandregistrationundernon-
1960s.Theoverwhelmingmajorityofthesetechniquesarebased
frontalposes,inpresenceoffacialexpression,gesturesandoc-
on2Dfacedata. Theearlyapproacheswerefocusedonthege-
clusions.
ometryofkeypoints(eyes, mouthandnose)andrelationsbe-
tweenthem(length,angles).Inthe1990s,theprincipalcompo-
nentanalysis(PCA)basedeigenfacealgorithmwasintroduced 1.2. Facialexpressionunderstanding
and it became a standard reference in face recognition. De-
spitetheplethoraof2Dalgorithms,thereisnotyetafullyreli- Understandingoffacialexpressionshaswideimplicationsrang-
ableidentificationandverificationmethodthatcanoperateun- ingfrompsychologicalanalysistoaffectiveman-machineinter-
deradverseconditionsofillumination,pose,accessoriesandex- faces.Oncetheexpressionisrecognized,itcanalsobeneutral-
pression. Twosolutionstothisimpasseconsistini)Exploring izedforimprovedpersonrecognition.Automaticrecognitionof
new modalities, such as 3D facial data; ii) The use of multi- facialexpressionsischallengingsincethegeometryoftheface
biometrics,thatis,theemploymentofmorethanonebiometric canchangerapidlyasaresultoffacialmusclecontractions.Ex-
modalityandtheirjudiciousfusion. pressions are often accompanied by purposeful or involuntary
Recentlyfacerecognizersusing3Dfacialdatahavegained pose variations of the subject’s face. 3D representation of the
popularity due to their lighting and viewpoint independence. humanfacecontainsmoreinformationthanits2Dappearance.
Thishasalsobeenenabledbythewideravailabilityof3Drange Weconjecturethatnotallshapechangesonthefacialsurfaces
scanners. The 3D face processing can be envisioned as a sin- duetoexpressionsarereflectedinthe2Dimages. Certainsur-
glemodalitybiometricapproachinlieuofthe2Dversionorin facesdeformations,bulgesandcreases,especiallyinthesmooth
acomplementarymodeinamulti-biometricscheme. Another areasoftheface,arehardtotrackin2Dwhiletheyareapparent
goalapplicationof3Dfacialdataistheunderstandingoffacial on3Ddata.
expressionsinanaffectivehuman-computerinterface. Among the facial signal processing problems, we want to
Basedontheideasabove,twomaingoalsareaddressedin addresstotheabovestatedgoalsbyprovidingsolutionstothe
thisproject: followingproblems:
87
ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
1.3. 3Dfaceregistration 2. DATABASECONTENT
3Dshapesneedtobealignedtoeachotherandshouldbebrought In this project, on the one hand we model attempts to inval-
intoacommoncoordinateframebeforeanycomparativeanaly- idate 3D face recognition and any other effort to mislead the
siscanbemade.The3Dfaceregistrationistheprocessofdefin- systemortoinduceafakecharacter. Tothiseffect,wecapture
ingatransformationthatwillcloselyaligntwofaces.Firstofall, 3Dfacedataimitatingdifficultsurveillanceconditionsandnon-
thepermissibletransformationsshouldbeset.Rigidtransforma- cooperatingsubjects,tryingvariousrealisticbuteffectiveocclu-
tionsallowonlyfortranslation, rotationorscaling. Non-rigid sionsandposes. Ontheotherhand, wecollect3Dfacialdata
transformationsgoonestepfurtherandallowpatchwisedefor- correspondingtovariousactionunitsandtovariousemotional
mationoffacialsurfaceswithintheconstraintsofestimatedlan- expressions.
ndmark points. Secondly, a similarity measure should be de- The database consists of 81 subjects in various poses, ex-
termined that favors successful recognition. Different similar- pressions and occlusion conditions. The images are captured
itymeasuresarepossibleforregistrationtechniquessuchasthe using Inspeck Mega Capturor II [6], and where each scan is
point-to-pointorpoint-to-surfacedistances. manually labeled for facial landmark points such as nose tip,
innereyecorner,etc.Ourdatabasehastwoversions:
• Database1isdesignedforbothexpressionunderstanding
1.4. 3Dfacelandmarking andfacerecognition. Herethereare47peoplewith53
different face scans per subject. Each scan is intended
Facial landmarks are essential for such tasks as face registra- tocoveroneposeand/oroneexpressiontype,andmost
tion, expression understanding and any related processing. In ofthesubjectshaveonlyoneneutralface,thoughsome
general,registrationprocessisguidedbyasetoffiducialpoints of them have two. Totally there are 34 expressions, 13
called landmarks [1] that are used to define the transform be- poses, fourocclusionsandoneortwoneutralfaces. In
tweentwosurfaces. Theregistrationtechniquesthatareexam- addition,Database1alsoincorporates30professionalac-
inedinthescopeofthisprojectmakeuseofasetoflandmark tors/actresses out of 47, which hopefully provide more
pointsthatarelabeledforeachfacesurfacetobealigned. Pre- realisticoratleastmorepronouncedexpressions.
liminary studies of landmarking of uncontrolled facial expres-
• Database2includes34subjectswithonly10expressions,
sions[2]indicatethatitremainsstillasanopenproblem.
13poses,fourocclusionsandfourneutralfacesandfour
neutralfaces,thusresultinginatotalof31scanspersub-
ject.
1.5. 3Dfacialexpressionandfacerecognitiondatabasecon-
Themajorityofthesubjectsareagedbetween25and35.There
struction
are51menand30womenintotal,andmostofthesubjectsare
Caucasian.
Thereareveryfewpubliclyavailabledatabasesofannotated3D
Thisdatabasecanbeusedforautomaticfaciallandmarkde-
facial expressions. Wang et. al [3] studied recognition of ex-
tection,facedetection,faceposeestimation,registration,recog-
pressionsbyextractingprimitivesurfacefeaturesthatdescribe
nition and facial expression recognition purposes. Hence we
theshape.TheyusetheirdatabasededicatedtoexpressionsBU-
alsolabeledfacialfiducialpointsmanually,andprovidethisin-
3DFE [4]. It includes only emotional expressions (happiness,
formationwiththedatabase.
surprise,fear,sadness,anger,disgust)withfourintensitylevels
In the following subsections, the collected facial expres-
from100people.
sions,headposesandocclusionsareexplained.
Inthisproject,weaimtostudyamorecomprehensivesetof
expressionsbycoveringmanyexpressionsbasedonFacialAc-
2.1. FacialExpressions
tionCodingSystem(FACS)[5]inadditiontothesixbasicemo-
tionalexpressions. Thereare44ActionUnits(AUs)definedin We have considered two types of expressions. In the first set,
FACS.AUsareassumedtobebuildingblocksofexpressions, the expressions are based on AU of the FACS [5]. However,
andthustheycangivebroadbasisforfacialexpressions. Also, within this project a subset of them, which are more common
since each of them is related with activation of distinct set of andeasiertoenact,isconsidered. Theselectedactionunitsare
muscles,theycanbeassessedquiteobjectively.Wealsoinclude groupedinto19lowerfaceAUs,fiveupperfaceAUsandthree
variousposesandocclusionconditions. Wehavecollecteddata AUcombinations.Itisimportanttonotethat,forsomesubjects
from 81 subjects, preprocessed and manually landmarked the properlyproducingsomeAUsmaynotbepossible,becausethey
data. We have defined the metadata standard. We have also arenotabletoactivaterelatedmusclesortheydonotknowhow
performedbasicregistrationandclassificationtests.Thismulti- to control them. Therefore, in the database some expressions
expression and multi-pose database, which will eventually be arenotavailableforsomesubjects.Also,unlessalltheacquired
madepublic, willbeinstrumentalinadvancingtheexpression AUsarevalidatedbytrainedAUcoders,thecapturedAUscan
analysisresearchand developalgorithmsfor3Dfacerecogni- notbeverified.
tionunderadverseconditions. Inthesecond set, facialexpressionscorresponding tocer-
The rest of the report is as follows: Section 2 describes tain emotional expressions are collected. We have considered
the database content, and the database structure is detailed in thefollowingemotions:happiness,surprise,fear,sadness,anger
Section3. Section4givesthenecessarydetailsofthedataac- and disgust. It is stated that these expressions are universal
quisition setup. Landmarking issues are dealt in two separate amonghumanraces[7].
sections: Section 5 and 6, respectively, presents manual land- Duringacquisitionofeachactionunit,subjectsweregiven
marking and automatic landmarking techniques. The 3D face explicationsabouttheseexpressionsandtheyweregivenfeed-
registrationmethodappliedtoourdatabaseisgiveninSection backiftheydidnotenactcorrectly.Alsotofacilitatetheinstruc-
7. Finally,Section8showstheperformanceresultsofregistra- tions,avideoclipshowingthecorrectfacialmotionforthecor-
tion, automaticlandmarkingandrecognition. Conclusionsare respondingactionunitisdisplayedonthemonitor[8,9]. How-
drawninSection9. ever,inthecaseofemotionalexpressions,therewerenovideo
88
ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
1. LowerFaceActionUnits
• LowerLipDepressor-AU16
• LipsPart-AU25
• JawDrop-AU26
• MouthStretch-AU27(*)
• LipCornerPuller-AU12(*)
• LeftLipCornerPuller-AU12L
• RightLipCornerPuller-AU12R
• LowIntensityLipCornerPuller-AU12LW
• Dimpler-AU14
• LipStretcher-AU20
• LipCornerDepressor-AU15
• ChinRaiser-AU17
Figure1:LowerFaceActionUnits:lowerlipdepressor(a),lips
• LipFunneler-AU22
part(b), jawdrop(c), mouthstretch(d), lipcornerpuller(e),
• LipPuckerer-AU18 low intensity lower lip depressor (f), left lip corner puller (g),
• LipTightener-AU23 rightlipcornerpuller(h),dimpler(i),lipstretcher(j),lipcorner
depressor(k),chinraiser(l),lipfunneler(m),lippuckerer(n),
• LipPresser-AU24
liptightener(o),lippresser(p),lipsuck(q),upperlipraiser(r),
• LipSuck-AU28(*) nosewrinkler(s),cheekpuff(t).
• UpperLipRaiser-AU10
• NoseWrinkler-AU9(*)
• CheekPuff-AU34(*)
2. UpperFaceActionUnits
• OuterBrowRaiser-AU2(*)
• BrowLowerer-AU4(*)
Figure2:UpperFaceActionUnits:outerbrowraiser(a),brow
• InnerBrowRaiser-AU1
lowerer(b),innerbrowraiser(c),squint(d),eyesclosed(e).
• Squint-AU44
• EyesClosed-AU43(*)
3. SomeActionUnitCombinations
• Jaw Drop (26) + Low Intensity Lip Corner
Puller
• LipFunneler(22)+LipsPart(25)(*)
• LipCornerPuller(12)+LipCornerDepressor
Figure3: SomeActionUnitCombinations: lipcornerpuller+
(15)
lip corner depressor (a), jaw drop + lip corner puller (b), lip
4. Emotions funneler+lipspart(c).
• Happiness(*)
• Surprise
• Fear
• Sadness
• Anger
• Disgust
Figure 4: Emotional expressions: happiness (a), surprise (b),
Theexpressionsmarkedwith(*)areavailableinbothDatabase1and
fear(c),sadness(d),angry(e),disgust(f).
Database2,buttheothersareonlyfoundinDatabase1.
Table1:Lowerfaceactionunits.
Thetotalof35expressionsconsistof19AUsbelongingto
lower part of the face (Fig. 1), five AUs from the upper face
orphotoguidelinessothatsubjectstriedtoimprovise. Onlyif (Fig. 2), three AU combinations (Fig. 3) and six emotional
theywereabletoenact,theyweretoldtomimictheexpression expressions (Fig. 4) as listed in Table 1. In these tables, the
inarecordedvideo.Moreover,amirrorisplacedinfrontofthe expressionsmarkedwith(*)areavailableinbothDatabase1and
subjectsinordertoletthemcheckthemselves. Database2,buttheothersareonlytobefoundinDatabase1.
89
ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
Figure6: Occlusions: eye occlusion(a), mouthocclusion (b),
eyeglasses(c),hair(d).
• Occlusionofeyewithhand-asnaturalaspossible
• Occlusionofmouthwithhand-asnaturalaspossible
Figure 5: Poses: neutral pose (a); yaw rotations of +10◦ (b), • Eyeglasses(notsunglasses,normaleyeglasses)
+20◦(c),+30◦(d),+45◦(e),+90◦(f),-45◦(g)and-90◦(h)re-
• Hair
spectively;pitchrotationsofstrongupwards(i),slightupwards
(j), slight downwards (k), strong downwards (l); bottom-right
Table3:Occlusions
(m)andupperright(n).
2.2. HeadPoses theireyesortheyweresurprisedbyputtingtheirhandsovertheir
mouth. Second,fortheeyeglassesocclusion,wemadesothat
Various poses of head are acquired for each subject (Fig. 5). subjectsuseddifferenteyeglassesfromapool. Finally,ifsub-
There are three types of head which are seven angles of yaw, jects’hairswerelongenough,wehavealsoscannedtheirfaces
fouranglesofpitch,andtwocrossrotationswhichincorporate withhairpartlyoccludingtheirface.
bothyawandpitch. Fortheyawrotations,subjectsalignthem-
selvesbyrotatingthechaironwhichtheysittoalignwithstraps
placedonthefloorcorrespondingtovariousangles. Forpitch 3. THESTRUCTUREOFTHEDATABASE
andcrossrotations,werequestedthesubjectstolookatmarks
placedonthewallsbyturningtheirheadsonly(i.e.,noeyero- In many image retrieval database applications two approaches
tation). Thus,wecanobtainacoarseapproximationofrotation arewidelyused. Oneisstoringanyimage-likedataontohard
angles.TheheadposesarelistedTable2. diskandtherestofthedatatoanRDBMS(RelationalDatabase
ManagementSystem)whilethesecondapproachisstoringim-
ageanditsmetadatadirectlyontheharddisk. Whilethefirst
1. YawRotations
one provides easy querying, it has the disadvantage of using
• +10◦ and managing third-party RDBMS software. Also not every
• +20◦ RDBMSmayworkproperlyincross-platformsystems.Inaddi-
tionitnecessitatesknowledgeofusingSQL(StructuredQuery
• +30◦ Language). Wesuspectthatnotallresearchersarewellversed
• +45◦ inSQL.Forthesereasonswedecidedtousethesecondalterna-
tive, thatis, storageofimagesandtheirmetadataonthehard-
• +90◦
disk. Forthispurpose,datastorageandmappingstructuresare
• -45◦ designedandatraditionaldatabasedesignusedinRDBMSsis
• -90◦ considered.Inthisdesignwemakeuseofprimarykeyandweak
referencing. Aprimitivereferentialintegrityisusedinsidethe
2. PitchRotations software for preventing redundancy and making relations be-
tweentree-likestructures.Thefirstversionofthedesignisgiven
• Strongupwards
inAppendix12.1.
• Slightupwards One canmap the entiredatabase and thestorage structure
• Slightdownwards basedontheabovedesign. WechoosetouseXMLstructurein
datastoragebecauseinadditiontohavingahumanreadablefor-
• Strongupwards
matXMLisalsocommonlyusedinnonRDBMSapplications
3. CrossRotations intherecentyears. Arecordisarowinatable, eachcolumn
inarecordshowsusanattributeoftherecord. Everytablede-
• Yawandpitch1(approximately20◦ pitchand
finesitsattributesandlistsitsdataafterdeclarationofattributes.
45◦yaw)
ContentofthesubjecttableexportedfromSubject.xmlisgiven
• Yawandpitch2(approximately-20◦pitchand asanexampleinAppendix12.2.
45◦yaw) Populatingthedatabasemanuallyisalwaysveryhardand
thus a problematic application. One can easily commit errors
Table2:Headposes duringacquisitionandprocessingstages. Inordertoovercome
thisproblemwedecidedtowriteaprogramusingourdatabase
structurewhichhelpstheacquisitionapplication.Classdiagram
oftheprogramshowsthatitisageneraltime-savingstratagem.
2.3. Occlusions
Initial class designs involve inheritance of the form structures
Thisdatabasealsocontainsseveraltypesofocclusionsthatare foreasyusageandalsoinvolveusageofbasicanti-patternslike
listedinTable3andshowninFig.6. Singleton for assuring that only single instances of Business
Fortheocclusionofeyesandmouth,subjectschooseanat- Classesexistinsidetheprogramformanagingdatasessions. A
uralposeofthemselves; forexample, asiftheywerecleaning schemaoftheclassdiagramisfoundinFig.7.
90
ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
Figure 7: A schema of the class diagram. Business classes Figure8:Usercaninsertnewsubjectstothedatabasefromthis
areusedforconstructingtheobjectslikeSubject, Sessionand screen;editingordeletingpreviouslysavedsubjectsisalsodone
Records.Sincetheseobjectsareusedasobjectconstructionar- fromhere.
chitecture,eachclasshasitspersistencystructure,lettingitto
bewrittentodatabaseinanychange. Thisisastandardmech-
anismcalled“Reflection”in“ObjectRelational”
Inordertowriteaquickcodeinlimitedtimewedecidedto
use .net 2.0 framework and C# as the programming language.
Wecouldn’ttrythecrossplatformcompatibilityofthecodeus-
ing Mono in Linux, but we’ve implemented the code in cross
platformmanner. TheLandMarkqueryingpartofthesoftware
is finished for the use of participants during Manual - Auto-
maticLandmarkingcomparisons. Forlandmarkingusershave
no chance but add the destination directory names as input to
their software. After using our software user will be able to
querythesystembygender,occlusionoranystructuretheywant
andsavetheresultstoadocumentwhichcanbeinputtotheir
implementations.Afterhavingsuchaninputdocumenttheycan
easilytesttheiralgorithmsonthedatatheyexactlywant.While
queryingandfindingthedatadirectorypathsisworkingprop-
Figure 9: User can insert the action units used during posing
erly,exportingthisstructuretoaninputfilewillbeimplemented
from this screen. Each action unit, its code and explanation
withtheacknowledgementscomingfromtheuserswhowantto
has to be inserted. This also lets user to relate his/her action
testtheirlandmarkingalgorithmsonthedatabase. Somescreen
units to be used in their landmarking study to general struc-
shotsofthesoftwareisshowninFig.8,9and10.
ture. This part is vital for researchers for choosing their own
As a conclusion, the proposed database structure does not
databasestructuretobeusedintheirlandmarkingalgorithms.
need any third-party component or querying structure. Since
it is XML based, it is human readable, and can also be used
in many acquisition database structures owing to its easy and
color. Moreover,duringacquisition,eachsubjectwearsaband
genericimplementation. Wehopethestructurewe’veproposed
tokeephis/herhairabovetheforehead. Thisalsosimplifiesthe
willevolveandbecomeadatabasestructureforsimilarresearch
segmentationoffacebefore3Dreconstruction.
applications.
Theon-boardsoftwareinthescannerisusedforacquisition
and 3D model reconstruction. Inspeck’s 3D digitizer is based
4. DATAACQUISITION onanopticalprinciple, whichcombinesActiveOpticalTrian-
gulationandPhaseShiftedMoir(Fig. 12). Thesurfaceofthe
Facial data are acquired using Inspeck Mega Capturor II 3D, objectmustreflectlightbacktothe3Ddigitizer. Hencetrans-
which is a commercial structured-light based 3D digitizer de- parentsurfaces(likeglass)cannotbescanned. Certainfringe
vice[6]. Itisabletocaptureafaceinlessthanasecond. Our patterns, thatareshiftedversionsofeachothers, areprojected
acquisitionsetupisdisplayedinFig. 11. Subjectsaremadeto onthe3Dsurfaceofanobject,andasaresultthesepatternsare
sitatadistanceofabout1.5metersawayfromthe3Ddigitizer. deformedaccordingtoobjectsurface.IntheInspeck’ssystem,a
Toobtainahomogenouslightingdirectedonthefaceweusea fringepatternisintheformofstraightverticallines.Allthe3D
1000W halogen lamp without any other ambient illumination. informationisextractedfromthesedeformedpatterns. During
However, due to the strong lighting of this lamp and the de- acquisitionweemployedusuallyfourfringepatterns.However,
vice’sprojector, usuallyspecularreflectionsoccurontheface. sometimes we preferred to use three to diminish the effect of
Thisdoesnotonlyaffectthetextureimageofthefacebutcan motionforsomesubjects.
alsocausenoiseinthe3Ddata. Topreventit,weapplyaspe- Oncetherawdataareacquired,threestagesofprocessing
cialpowdertothesubject’sfacewhichdoesnotchangetheskin (calledPreProcessing,ProcessingandPostProcessinginthede-
91
ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
Figure10:Queryingoftheimagedataofmanuallandmarksand Figure11: Acquisitionsetup. Attheleft3DDigitizer,whichis
sortingduetosubjectorpose. Usercanquerythedatapaths mountedonatripod,isseen. Amirror,anLCDdisplayanda
related to subject and their poses to be used in landmarking. 1000Whalogenlampareplacedbelow,aboveandbehindofit
Softwareusercanselectthesubjectsandtheactionunitshe/she respectively. A seat is positioned about 1.5 meters away from
wantstorelateduringlandmarking.Thisqueryreturnstheuser the3DDigitizer.Onthefloor,justbelowtheseat,yellowstraps
pathsofthedatatheywanttouse. areplacedaccordingtorotationangles.
vice’sjargon)arecarriedouttoobtainthe3Dinformation(Fig.
13). InthePreProcessingstage,firstthephasefunctioniscal-
culated (top right image in Fig. 13). Consequently the depth
informationismeasuredbyusingsomereferencepoints,called
parallaxpoints(topleftimageinFig. 13). Mostofthetimewe
allowedtheInspecksoftwaretodetecttheseparallaxpointsau-
tomatically;however,werevertedtomanualmarkingincaseof
doubt.Next,anInterestAreaisselectedforthefaceimage.We
setthiszonemanuallybycreatingoneormorepolygonsthatde- Figure12:3Dmodelreconstruction.Ontheleftaretherawdata
fineit(topmiddleimageinFig. 13). AclearlydefinedInterest containingtextureand4frameswithshiftedfringepattern.The
Areaisveryhelpfultoremovethebackgroundclutter.Thuswe lastpictureshowsparallaxpoints. AfterprocessingbyInspeck
cancanceloutextrastreaks,whichcomefromthedigitizingof Fapssoftwareweobtainthe3Dmodel.
thebackgroundarea.
After segmenting the face, phase unwrapping operation is
performedtoobtain3DdataintheProcessingstage. However, adjustedtocapturefourpicturesthatlaterconstitutethe
duringphaseunwrappingsomediscontinuityerrors,whichare rawdatafor3Dfaceimagesandonemorepictureoftex-
erroneous depth levels associated with discontinuous surfaces, tureimage.Duringacquisition,thesubjectmuststaystill
mayoccur. Theverifieddiscontinuitiesarefixedbysequential (betterholdthebreath),sothatthepicturesareconsistent
pushorpulloperations. with each other. Otherwise the wavelike defects occur
globallyaroundtheface. Inthiscasetheacquisitionis
Finally in the Post-Processing stage, the so-far processed
repeated.
data, currently in machine units, is converted to a data set in
geometric parameters (millimeters). To convert the data into • Hairocclusion:Fringesprojectedonhairbecomesunde-
milimeters we determine a reference from a set of candidate tectable,hencethecorrespondingpartsonthe3Dimage
reference(parallax)points. Generally,thepointwhichhasthe isnoisy.Hairoccludedpartsareincludedinthefaceseg-
highestconfidencevalueisselectedandthegeometricparame- mentation,sinceavoidingthemisnotpossible.However,
tersoftheotherpointsintheimagearedeterminedaccordingly. withsmoothingoperationwecanobtainlessnoisydata.
Also, in this stage we sometimes perform simple filtering op- • Hand occlusion and open mouth: In the case of hand
erationstocorrectslightopticaldistortions. Wheneverserious occlusionandopenmouth, depthlevelchangessharply
distortionsoccurredwepreferredtore-capturethedata. overtheedges,causingadiscontinuousfringepattern.In
Theresultisexportedintheformof“.net”file,whichisa the3Dimage,handandtongueareconstructedatwrong
specificfileformatforthedata-processingsoftwareofInspeck depthlevels.
Inc.[6].Theoutputfilestoresthe3Dcoordinatesofthevertices
• Facial hair and eyes: Beards, eyebrows and eyelashes
togetherwithcolorinformationasarelatedseparatetextureout-
causesmallfluctuations. Alsoeyescannotbekeptcon-
putfilein“.bmp”format.
stantbecauseofhighlightintensity,resultinginaspiky
Sincetheremightbeproblemsforsomeofthescans,FAPS
surfaceontheeyes.Theseproblemsareignored,notad-
preprocessing has been done simultaneously with data acqui-
ditionalfilteringprocessesareapplied.
sition. If a problem is observed on the data then that scan is
repeatedonthespot. Somecommonlyoccurringproblemsare • Eyeglasses:Fringepatterncannotbereflectedbackfrom
asfollows(Fig.14): transparentsurfaces.Eyeglassesseemtodeformthearea
they cover on the face. The resulting noisy surface is
• Movementduringacquisition: TheInspeckhardwareis includedinthefacesegmentation.
92
ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
Figure14: Commonlyoccurringproblemsduringimageacqui-
sitionandfacereconstruction. Attopofthefigure,noisedueto
hair,movement,andfacialhairisseenonthefacescans.Atthe
bottomleft,amistakeinthedepthlevelofthetongue,andatthe
right,itscorrectionisdisplayed.
Figure13: ReconstructionbyFAPS.Pre-processing(toprow),
processing (middle row) and post-processing (bottom row)
stages. Toprow: Parallaxpoints,polygontodefinetheregion
of interest, phase image. Middle row: Depth image, parallax
points, 3D shape, 3D texture. Bottom row: A scanned phase
fromdifferentviewpointsafterpost-processingsuchasholefill-
ing.
Figure15:Manuallymarkedlandmarkpoints.
5. MANUALDATALANDMARKING
Allofourfaceimagesarelandmarkedmanuallyinordertobe In the end of manual landmarking, we obtain 2D and 3D
used in various training and testing methods as ground truth. locationsof24facialfiducialpointsforeveryscanofeverysub-
Oneachfacescan,24pointsaremarkedprovidedthattheyare ject.ThelandmarkdatastructureisstoredinaMATLAB“mat”
visibleinthegivenscan.ThelandmarkpointsarelistedinTable file,oneforeachsubject. Thesamedataisalsosavedin“text”
4andshowninFig.15. fileformatinordernottoforcetheuserstoMATLAB.
1.Outerlefteyebrow 2.Middleofthelefteyebrow
6. AUTOMATICLANDMARKING
3.Innerlefteyebrow 4.Innerrighteyebrow
5.Middleoftherighteyebrow 6.Outerrighteyebrow
Facedetection,normalizationorregistrationshouldbeperformed
7.Outerlefteyecorner 8.Innerlefteyecorner
before any face recognition algorithm. ICP (Iterative Closest
9.Innerrighteyecorner 10.Outerrighteyecorner
Point)[10]andTPS(ThinPlateSpline)[11]arethemostlyused
11.Nosesaddleleft 12.Nosesaddleright 3D to 3D registration methods. However, they perform well
13.Leftnosepeak 14.Nosetip only when the models being registered are very close to each
15.Rightnosepeak 16.Leftmouthcorner other since these methods are sensitive to initialization condi-
17.Upperlipoutermiddle 18.Rightmouthcorner tions.Thus,aroughregistrationshouldbeperformedbeforethe
19.Upperlipinnermiddle 20.Lowerlipinnermiddle fineregistrationbyICPorTPSmethods.
21.Lowerlipoutermiddle 22.Chinmiddle Colbry, Stockman and Jain [12] proposed a rough regis-
23.Leftearlobe 24.Rightearlobe tration method based on 3D anchor points detected from the
face data before application of ICP. They detected facial fea-
Table4:Landmarkpointsmanuallylabeledoneachscan. turesbasedontheirshapeindexes,intensityvaluesandspatial
relationswheretheyusedboth3Dshapeand2Dimageoutputs
LandmarkingsoftwareisimplementedasaMATLABGUI. ofthescannerinordertolocateanchorpoints.Similarly,Luand
This software is designed so that all landmark points can be Jain[13],Changet. al. [14]andBoehnenandRuss[15]used
markedguidedbyareferencetemplateimageforthelandmarks 3Dand2Dinformationtogetherinordertoextractfacialanchor
in Fig. 15. This scheme helps us avoid errors which can oc- points.Inmostofthesemethods,detectionusuallystartsfroma
cur when marking many landmark points over lots of images. singlepointandthenfollowedbyanorderofotherpoints. For
The3Dcoordinatescorrespondingtothe2Dmarkedpointson examplein[13],firstnosewasneededtobedetectedsothatthe
thetextureimageisextractedautomaticallyusingthissoftware. eye and mouth corners could be detected next. Nose tip was
Here we make the assumption that 2D image and 3D data are detectedasfollows: First,posewasquantizedintosomenum-
registeredalthoughtheyarenot100%registered. ber of angles. Then, the point with the maximum projection
93
ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
value along the corresponding pose direction was found. Fi-
nally,basedonthenoseprofile,thebestcandidateforanosetip
wasselected.Afterlocatingthenosetip,otherpointsarelocated
byconsideringthespatialrelationshipsamongthepoints,shape
indexesforthesepointsobtainedfrom3Ddataandcornerness
forthesepointsdeterminedfromtheintensityimage. In[14],
eye cavities were detected first by investigating the mean and
Gaussiancurvaturevaluesandbyusingsomeheuristics. Then,
nosetipwasdetectedasbeingapeakwhichhasaspatialrela-
tionship with the eye pits. In [12], facial points are detected
based on their mean and Gaussian curvature values indepen-
dentlyand,again,toresolvebetweenmultipledetections,spatial
relationsbetweentheanchorpointswereconsidered. Thiswas
donebyarelaxationalgorithmwheresomeruleswereusedfor
anchorpointsandrelationsbetweentheminordertoeliminate
incorrectsetsofanchorpoints. Figure 17: A total of 22 feature points is landmarked (b) by
In this project we applied three different methods for au- usingautomaticallylandmarked7featurepoints(a).
tomaticlandmarking. Thereare24goallandmarkpointseach
ofwhichismarkedmanuallyontheENTERFACE07database.
Althoughsevenofthelandmarkpointsareconsideredtobethe
6.2. Robust2DLandmarking(RL)
mostfiducialones,eachmethoddealswithitsownsetofpoints.
Thethreecompetitormethodsarebrieflydescribedbelowand Thismethodisquitesimilartothemethoddescribedinprevi-
theirtestresultsarepresentedinSection8(TestingandEvalua- ous subsection, in that it is also a two-tier method which cap-
tionSection). tures first the seven fiducial landmarks. It also proceeds from
aninitialcoarseevaluationon80x60imagestoarefinedversion
of 640x480 resolution images. The fiducial ones are the four
6.1. StatisticalAutomaticLandmarking(SAL)
eyecorners,thetipofthenose,andthetwomouthcorners[2].
Thismethodexploitsthefacedepthmapsobtainedby3Dscans. Thismethoddiffersfromtheoneinprevioussubsectionasfol-
Inthetrainingstage,anincrementalalgorithmisusedtomodel lows:weusetemplatesforeachlandmarkconsistingofasubset
patches around facial features as mixture models. These mix- ofDCTcoefficientsandweobtainscores(likelihoods)employ-
turemodelsare,inturn,usedtoderivelikelihoodsforlandmarks ingSVMclassifiers.Inthecoarselocalization,8x8DCTblocks
duringtesting[16]. Thelikelihoodtestsyieldthesevenfiducial are extracted and trained separately for each facial landmark.
featurepoints(innerandoutereyecorners,nosetipandmouth Wepermitinitiallyfourcandidatescorrespondingtothehighest
corners)andtheremaining15landmarksareestimatedbasedon peaksinthematchingscoremapofeachlandmark. Thisisin
theinitialsevenfiducialonesbyback-projection. Forenhanced order not to miss any feature points. The 4x7 = 28 landmark
reliability, the seven landmarks are first estimated on a coarse candidatesarereducedtoafinalsevenbasedonanexhaustive
scale,andthenrefinedusingalocalsearch[17]. Furthermore, graphsearchtechnique.Twomethodsareproposedforthistask:
incorrectlocalizationsaredetectedandfixedwiththeGOLLUM ProbabilisticGraphModel-IandProbabilisticGraphModel-II.
Algorithm[18]. GOLLUMAlgorithmusessmallernumberof InPGM-I,wefirstfindasubsetofthemostreliablelandmarks
points to check all point localizations. The assumption is that basedontheanthropomorphicdatalearnedduringatrainingses-
these points contain statistically discriminatory local informa- sionandthenestimatethepositionofthemissinglandmarks.On
tion,andcanbeindependentlylocalized. Anexampleresultis theotherhand,PGM-IIisdesignedforsimplerscenarios,with-
showninFig.16. outvarietyinilluminationandposes.Itsystematicallysearches
InasecondtierapplicationoftheGOLLUMalgorithm,the forthesevenfeaturelandmarkamongthefourpeaksofthesim-
locations of 15 additional feature points are estimated. These ilarityscoremap. Facialfeaturelandmarksaredeterminedac-
pointsarebasedonthecoordinatesofthesevenfiducialones. cording to companion features, called the support set for that
The initial seven points contain richer statistically discrimina- feature. Once the coarse-stage landmarks are located we pro-
torylocalinformationandcanthusbeindependentlylocalized, ceed with the refinement stage on the original high-resolution
whereastheremaining15pointshavemuchweakerimageevi- image. Thesearchproceedswitha21x21windowfromaround
dencearoundthem,hencetheyneedthestructuralinformation thecoarselocalizationpoints.Eachconfigurationformedbyre-
oftheformerones. InFig. 17alloftheselandmarkpointsare liablelandmarksissettoorigin,scaledtoafixedlengthandro-
shown. tated.Sincethisproceduremakestheconfigurationindependent
oftheposeandscaleoftheface, thisalgorithmachieveshigh
accuracyevenundervariousposesandforthisreasoniscalled
robust 2D landmarking. In our future work, we aim to apply
thismethodto3Dscansandextendouralgorithmtocoverthe
remaining15featuresasthemethodintheprevioussubsection.
6.3. TransformationInvariant3DFeatureDetection(TIFD)
Intransformandscaleinvariantfeaturedetectionalgorithm,3D
fundamental elements such as peaks, pits and saddles are ex-
tractedwiththeirscaleinformationfromthe3Dscandata.With
Figure16:Depthmapofthefaceimage(a),statisticallylocated
apredefinedgraphstructurebasedonagroupoffundamentalel-
featurepoints(b)andcorrectedfeaturepointsontheface(c).
ementsandtheirspecialrelationships,wecanconstructatopol-
ogy of these elements which could then be used to define the
94
ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
object. And this graph structure can be used for object detec- erativeClosestPoint)[10]andTPS(ThinPlateSpline)[11]are
tion,registrationandrecognitionpurposes. themostfrequentlyused3D-to-3Dregistrationmethods. How-
Themethodfindsthemean(H)andGaussian(K)curvatures ever,thesemethodsperformsatisfactorilyonlywhenthemodels
[10,13]onthesurface.AfterwardsusingGaussianpyramiding, toberegisteredareveryclosetoeachothersincethesemethods
theHandKvaluesarecomputedforthehigherscales.Inallof areinitializationdependent.Thus,acoarseregistrationisamust
thescalelevels,fundamentalelements(peaks,pitsandsaddles) beforeanyfurtherfineregistrationviaICPorTPSmethods.And
arefoundusingHandKvaluesasin[10,13]. InFig. 18,the this coarse registration is usually obtained by automatic land-
higher levels of these fundamental elements are depicted with markingdefinedintheprevioussection.
color(blue: peak, cyan: pit, red: saddleridge, yellow: saddle Even though Iterative Closest Point algorithm has a high
valley)andareshownonthe3Dsurface.Wenamethis3Dvol- computationalcost,itseaseofimplementationandaccurateend-
umea“UVSspace”where“u”and“v”aresurfacecoordinates resultsmakeitapreferredmethodintheregistrationof3Ddata.
and“s”isforscale.Byextractingtheconnectedcomponentsin- The ICP method finds a dense correspondence between a test
sidethisUVSvolume,weobtainthefundamentalelementswith surfaceandagallerysurfacebytransformingthetestsurfaceto
theirpositionandscaleinformation.Thedetailsofthealgorithm fitthegallerywhileminimizingthemeansquareerror[10].The
aregivenin[19]and[20]. transformationisrigid,consistingofatranslationandarotation.
FortheENTERFACE’07project,wehavecreateda3Dtopol- Becausethistransformwithsixdegreesoffreedomisnonlinear,
ogyforthehumanfaceasformedbyonepeak,twopitsandone the error minimization problem has to be solved by means of
saddlewhichrepresentthenose,theeyepitsandthenosesad- iterativemethods.
dle, respectively. A four node graphical model is constructed The most accurate results obtained so far in the literature,
whereeachfundamentalelementisanodeinthegraphandspa- make use of a one-to-all registration in which a test shape is
tialrelationshipsbetweenthefundamentalelementsarecarried alignedseparatelytoeachofthefacesinthegalleryandthetest
bytheedgesbetweenthenodes. Thespatialrelationsbetween faceiscomparedtoeachgalleryfaceseparately. Toovercome
a node couples are the position difference between the nodes thecomputationalcostofone-to-allregistration,in[21]itwas
normalizedbyscale,normaldifferencebetweenthenodesand proposedtouseanaveragefacemodel(AFM).Intheregistra-
scaledifference.Thesevaluesaremodeledstatisticallyfromthe tionwithAFM,asingleregistrationofatestfaceisadequateto
training scans. Then during testing, a graph search is accom- alignitwithallthepre-registeredfacesinthegallery.Motivated
plished on the topology obtained from the test scans. As the bythisworkonregistrationviaAFM,in[22]itwasproposed
topologyforahumanfaceisformedashavingtwoeyepits,one tousemultipleAFMsfordifferentfacialcategoriestoincrease
nosesaddleandonenosepeak,amongthe1peak-2pits-1saddle the accuracy of registration. Multiple AFMs can be achieved
combinations,themostprobablecombinationisselectedasthe eitherbymanuallygroupingthefacesintodifferentcategories
correctmodel. or by clustering them into classes in an unsupervised manner.
Thus, usingthisrestrictedtopology, wecandetecttwoin- The latter approach is a more accurate one, in which the dis-
nereyecorners,middleofnosesaddleandtopofthenos.These criminatingfacialpropertiesexistinginthedatasetareacquired
differfromthelabeledlandmarkpointsbutthelattercanbeex- automatically. AlthoughthismethodwasnotusedduringeN-
trapolatedeasily. Sincethelocationoffaceisfoundinthe3D TERFACE’07,itisparticularlysuitedtothedatasetbecauseof
scandataaswellaslandmarkpoints,poseestimationandregis- itsmanyinherentvariations.Wewillapplyitinthefuture.
trationcanbeperformedeasilyafterwards.
8. TESTINGANDEVALUATION
Thedatabaseof3Dfacescanshavebeenprocessedwithcondi-
tioningoperations. Inaddition,theycontainground-truthland-
marks;hencetheycanpotentiallybeusedfortestinginvarious
taskssuchasfacedetection,automaticfacialfeaturedetection,
registration, face recognition and facial expression estimation.
DuringtheeNTEFACE’07projectwehavelimitedourselvesto
testing automatic landmarking and registration algorithms due
to time limitations. Work on other tasks will be completed in
thenearfuture.
8.1. LandmarkingTests
Forthecomparativeevaluationoflandmarklocalizers,wehave
Figure18: Left: 3Dsurfacestructurecenterscoloredbytheir
usedanormalizeddistance. Wedividedthelocalizationerror,
types(blue: peak,cyan: pit,red: saddleridge,yellow: saddle
measuredasEuclideandistancebetweenthemanuallymarked
valley)andthegraphrepresentingtheface,Right:UVSvolume
and automatically estimated landmark positions by the inter-
shown over the 3D data. Each level represents a layer in the
ocular distance. A landmark is considered correctly detected
scalespace. Connectedcomponentsarefoundinsidethisvol-
if its deviation from the true landmark position is less than a
umeinordertoextractthefundamentalelements,namelypeaks
given threshold, called the acceptance threshold. This thresh-
(blue), pits (cyan), saddle ridges(red) and saddle valleys (yel-
olditselfistypicallyapercentageoftheinter-oculardistance.
low).
Curvesobtainedinthismannerareusedtopresentthesuccess
ofthealgorithms(Fig.19,21and22).
Forautomaticlandmarkingtests,wechose66subjectsthat
7. AFMBASEDICPREGISTRATION possessedmultiplefrontalandneutralposes.Thesamesubjects
wereusedforbothtrainingandtesting. Morespecifically, we
Inany3Dfaceprocessing,beitfacerecognitionorexpression trainedouralgorithmsusingoneneutralfrontalposefromeach
understanding,theregistrationoffacesisacrucialstep.ICP(It- subject,andthentesteditusingalternateneutralfrontalposes,
95
ProceedingsoftheeNTERFACE’07WorkshoponMultimodalInterfaces,I˙stanbul,Turkey,July16-August10,2007
whichwerenotusedfortraining.Inaddition,wealsotestedthe • 28subjectsforeyeormouthocclusion.
landmarkingonthreedifficultcategories:onefromactionunits
InFig.20,outputofthealgorithmisshownforsomerepresent-
(open mouth, AU27), one from rotations (+20◦ yaw) and one
ingposes. Asexpected,thebestresultsareobtainedforneutral
fromocclusionscans(mouthocclusion).
poses. InFig. 21, performanceofthefeaturelocalizerversus
acceptancethresholdisgivenforthreetestingset.Accordingto
8.1.1. StatisticalAutomaticLandmarking(SAL) results,theproposedmethodismorerobusttofacialexpressions
andocclusionratherthanrotation.
InthetrainingstepofStatisticalAutomaticLandmarking,neu-
tralposesof40subjects,inour3DFace&ExpressionDatabase,
havebeenused. ToevaluatethesuccessoftheStatisticalAuto-
maticLandmarkingalgorithm,atestsethasbeenprepared.The
testingsetconsistsof:
• 20subjectsforneutralpose
• 20subjectsforneutralposewithglasses,
• 20subjectsforneuralposewitheyeormouthocclusion.
Accordingtotestresults,itisseenthatresultswiththeneutral
Figure20: Landmarkingoutcomesoftheautomaticfeaturelo-
posewithocclusionisbetter(Fig.19).Becauseofthestatistical
calizer for neutral pose, yaw, rotation, mouth stretch and eye
nature of the algorithm, initial landmarking step is processed
occlusion,respectivelyfromlefttoright.
similarly for all poses. In other words, occlusion in the pose
does not affect statistical initial landmarking. After statistical
initial landmarking, fine localization process corrects the mis-
placedfeaturepointsbyusingthedepthmap.Inthisstep,apose
withocclusioncanbecorrectedmoreefficientlythananeutral
pose.Becausethereareseverallocalminimasaroundthemouth
intheneutralposeandthisdecreasestheefficiencyofthefine
localizationalgorithm.Namely,mouthocclusion(byhand)pro-
vides to get rid off local minimas around mouth. As a result,
wecansaythatiftheposeandtheposeanglesarestable,this
approachgivesgoodresultsfortheposeswithocclusions.
Figure 21: Performance of 2D Landmarking. Red: pose with
expression,Blue:mouthoreyeocclusion,Green:rotation.
8.1.3. TransformationInvariant3DFeatureDetection(TIFD)
InordertotesttheperformanceoftheTIFDmethodontheeN-
TERFACE’07 scans we have made some of experiments. Be-
forehand, we have trained the system using 66 neutral scans
Figure19: StatisticalAutomaticLandmarkingtestresults. Re-
taken from 66 different people. By training, we mean extrac-
sultsforneutralpose,withmouthocclusionandwitheyeglasses
tion of a statistical model for the relations between the nodes,
occlusionisshowninasgreen,redandbluecurvesrespectively.
namelythenosepeak,thenosesaddleandtheeyepitsbytrain-
ingaGaussianMixtureModel[20].Duringtesting,themethod
searchesforaquartedofapeak,asaddleandtwopitswhichare
8.1.2. Robust2DLandmarking(RL)
statisticallysimilarintermsofrelativescaleandrelativeorien-
tationtothetrainedmodelofneutralscans.
Inthissection,wepresentedresultsofourautomaticlandmark-
Fortesting, wehaveusedthreegroupsofscans. Thefirst
ingalgorithmtestedoneNTERFACEDatabase. Wehaveused
group included the scans with the expression of mouth open-
68 samples with neutral poses as training set. For each sam-
ing. Sinceopeningthemouthdoesnotchangethelocationsof
ple,DCTcoefficientsofmanuallylandmarkedpointswerecom-
thefournodesordoesnotoccludethem,thesuccessrateswere
putedandthen,wetrainedanSVMclassifierwiththesecoeffi-
veryhigh. Wehaveused25modelsofdifferentpeoplehaving
cients. Inordertoverifytherobustnessofproposedalgorithm,
thisexpression. InFig. 22, redcurvedepictsthesuccessrate
we tested our classifier on samples with different poses. The
forthisexperiment. Bysuccessrate,wemeantheaveragesuc-
testingsetconsistsof:
cessamongallfeaturepoints.Ifthedifferencebetweenthefour
• 12subjectsforrotation(+10◦yawrotation),
points of the detected quartet and their four originally marked
• 28subjectsforfacialactionunit(mouthstretch), pointsarebelowathreshold,thesuccessis“one”forthatmodel;
96