Table Of ContentSpringer Tracts in Advanced Robotics 149
Martin Pfanne
In-Hand Object
Localization and
Control: Enabling
Dexterous
Manipulation
with Robotic Hands
Springer Tracts in Advanced Robotics
Volume 149
SeriesEditors
BrunoSiciliano,DipartimentodiIngegneriaElettricaeTecnologie
dell’Informazione,UniversitàdegliStudidiNapoliFedericoII,Napoli,Italy
OussamaKhatib,ArtificialIntelligenceLaboratory,DepartmentofComputer
Science,StanfordUniversity,Stanford,CA,USA
AdvisoryEditors
NancyAmato,ComputerScience&Engineering,TexasA&MUniversity,College
Station,TX,USA
OliverBrock,FakultätIV,TUBerlin,Berlin,Germany
HermanBruyninckx,KULeuven,Heverlee,Belgium
WolframBurgard,InstituteofComputerScience,UniversityofFreiburg,Freiburg,
Baden-Württemberg,Germany
RajaChatila,ISIR,Pariscedex05,France
FrancoisChaumette,IRISA/INRIA,Rennes,Ardennes,France
WanKyunChung,RoboticsLaboratory,MechanicalEngineering,POSTECH,
Pohang,Korea(Republicof)
PeterCorke,QueenslandUniversityofTechnology,Brisbane,QLD,Australia
PaoloDario,LEM,ScuolaSuperioreSant’Anna,Pisa,Italy
AlessandroDeLuca,DIAGAR,SapienzaUniversitàdiRoma,Roma,Italy
RüdigerDillmann,HumanoidsandIntelligenceSystemsLab,KIT-Karlsruher
InstitutfürTechnologie,Karlsruhe,Germany
KenGoldberg,UniversityofCalifornia,Berkeley,CA,USA
JohnHollerbach,SchoolofComputing,UniversityofUtah,SaltLake,UT,USA
LydiaE.Kavraki,DepartmentofComputerScience,RiceUniversity,Houston,TX,
USA
VijayKumar,SchoolofEngineeringandAppliedMechanics,Universityof
Pennsylvania,Philadelphia,PA,USA
BradleyJ.Nelson,InstituteofRoboticsandIntelligentSystems,ETHZurich,
Zürich,Switzerland
FrankChongwooPark,MechanicalEngineeringDepartment,SeoulNational
University,Seoul,Korea(Republicof)
S.E.Salcudean,TheUniversityofBritishColumbia,Vancouver,BC,Canada
RolandSiegwart,LEEJ205,ETHZürich,InstituteofRobotics&Autonomous
SystemsLab,Zürich,Switzerland
GauravS.Sukhatme,DepartmentofComputerScience,UniversityofSouthern
California,LosAngeles,CA,USA
TheSpringerTractsinAdvancedRobotics(STAR)publishnewdevelopmentsand
advances in the fields of robotics research, rapidly and informally but with a high
quality. The intent is to cover all the technical contents, applications, and multi-
disciplinaryaspectsofrobotics,embeddedinthefieldsofMechanicalEngineering,
Computer Science, Electrical Engineering, Mechatronics, Control, and Life
Sciences,aswellasthemethodologiesbehindthem.Withinthescopeoftheseries
aremonographs,lecturenotes,selectedcontributionsfromspecializedconferences
andworkshops,aswellasselectedPhDtheses.
Specialoffer:Forallclientswithaprintstandingorderweofferfreeaccesstothe
electronicvolumesoftheSeriespublishedinthecurrentyear.
IndexedbySCOPUS,DBLP,EICompendex,zbMATH,SCImago.
AllbookspublishedintheseriesaresubmittedforconsiderationinWebofScience.
Martin Pfanne
In-Hand Object Localization
and Control: Enabling
Dexterous Manipulation
with Robotic Hands
MartinPfanne
InstitutfürRobotikundMechatronik
DeutschesZentrumfürLuft-und
Raumfahrt
Oberpfaffenhofen-Weßling,Germany
ISSN 1610-7438 ISSN 1610-742X (electronic)
SpringerTractsinAdvancedRobotics
ISBN 978-3-031-06966-6 ISBN 978-3-031-06967-3 (eBook)
https://doi.org/10.1007/978-3-031-06967-3
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SwitzerlandAG2022
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Series Editor’s Foreword
Atthedawnofthecentury’sthirddecade,roboticsisreachinganelevatedlevelof
maturityandcontinuestobenefitfromtheadvancesandinnovationsinitsenabling
technologies.Theseallarecontributingtoanunprecedentedefforttobringingrobots
to the human environment in hospitals and homes, factories, and schools; in the
fieldrobotsfightingfires,makinggoodsandproducts,pickingfruits,andwatering
thefarmland,andsavingtimeandlives.Robotstodayholdthepromiseofmaking
a considerable impact in a wide range of real-world applications from industrial
manufacturingtohealthcare,transportation,andexplorationofthedeepspaceand
sea.Tomorrow,robotswillbecomepervasiveandtouchuponmanyaspectsofmodern
life.
TheSpringerTractsinAdvancedRobotics(STAR)isdevotedtobringingtothe
research community the latest advances in the robotics field on the basis of their
significanceandquality.Throughwideandtimelydisseminationofcriticalresearch
developmentsinrobotics,ourobjectivewiththisseriesistopromotemoreexchanges
andcollaborationsamongtheresearchersinthecommunityandcontributetofurther
advancementsinthisrapidlygrowingfield.
The monograph by Martin Pfanne is based on the author’s doctoral thesis. The
contentsarefocusedondexterousmanipulationandcontrolofobjectswithrobotic
hands and are organized into six chapters. The challenge of localization and in-
hand manipulation of various objects is tackled by resorting to a novel grasp state
estimationmethodthatintegratesinformationfromtactilesensing,proprioception,
andvisionintoacommonformulation.
RichofexamplesdevelopedbymeansofextensiveexperimentationontheDLR
humanoid robot David in a range of grasping scenarios, this volume is a very fine
additiontotheSTARseries!
Naples,Italy BrunoSiciliano
April2022 STAREditor
v
Preface
Thefindingsthatarepresentedinthismanuscriptaretheresultofmyresearchatthe
Institute of Robotics and Mechatronics at the German Aerospace Center (DLR). I
conductedthisworkasagraduatestudentbetween2015and2021incooperationwith
theCenterforCognitiveInteractionTechnology(CITEC)atBielefeldUniversity.
As two of the leading figures at these institutions and the principal supervisors
of my doctoral studies, I would like to express my deepest gratitude to Prof. Alin
Albu-SchäfferandProf.HelgeRitterforprovidingmethisopportunityandfortheir
continuedsupport.IwouldalsoliketothankDr.FreekStulpforhisguidanceduring
theearlyyearsofmyresearch.ThankyoutoJensReineckeforinitiallyopeningthe
doorstoDLRformeasadiplomastudentin2012andforhisfriendshipinthemany
yearssincethen.MydeepgratitudegoestoDr.MaximeChalonforhisinvaluable
supportofmyresearchandthemanyfruitfuldiscussionsonallaspectsofdexterous
manipulation. Many thanks to Manuel Stoiber, my former master student, whose
workgreatlycontributedtothealgorithmicframeworkthatispresentedinthisbook.
TheDLRhumanoidrobotDavidwastheprimaryresearchplatformforthework
duringmystudies.Therefore,IwouldliketothanktheentireDavidteamforenabling
metoconductthisresearchinthefirstplace.
ThankyoutoDr.BastianDeutschmann,Dr.MaximeChalon,Dr.DanielLeidner
and Dr. Freek Stulp for their helpful advice and comments during the writing of
mydissertation,onwhichthisbookisbased.Moreover,IamverythankfultoProf.
AntonioBicchiforagreeingtobepartoftheevaluationcommitteeofmydoctoral
examination.
Finally,Iwouldliketoverymuchthankmyfamily.Inparticular,Iameternally
gratefultomywife,Carolina,withoutwhomthisworkwouldnothavebeenpossible.
Andaveryspecialthankyougoestomydaughter,EvaMarie,forbeingmyinspiration
duringthefinalyearsofthecreationofthisbook.
LandsbergamLech,Germany MartinPfanne
July2022
vii
Contents
1 Introduction ................................................... 1
1.1 DexterousManipulation .................................... 4
1.2 Contribution .............................................. 9
1.3 OrganizationofthisWork .................................. 12
References ..................................................... 14
2 RelatedWork .................................................. 15
2.1 DexterousRoboticHands ................................... 15
2.1.1 ABriefHistoryofRoboticHands ..................... 15
2.1.2 DLRDavid ........................................ 18
2.2 DexterousManipulation .................................... 20
2.2.1 Overview .......................................... 20
2.2.2 GraspStateEstimation .............................. 22
2.2.3 Impedance-BasedObjectControl ..................... 25
2.2.4 Learning-BasedMethods ............................. 26
References ..................................................... 27
3 GraspModeling ................................................ 33
3.1 Definitions ............................................... 33
3.2 Kinematics ............................................... 37
3.2.1 ForwardKinematics ................................. 37
3.2.2 GraspMatrix ....................................... 39
3.2.3 HandJacobian ...................................... 41
3.2.4 ContactModel ...................................... 43
3.3 Dynamics ................................................ 46
3.3.1 RigidBodyDynamics ............................... 46
3.3.2 GraspDynamics .................................... 48
3.3.3 ContactDynamics .................................. 49
3.4 GraspSubspaces .......................................... 50
3.5 TypesofGrasps ........................................... 52
References ..................................................... 55
ix
x Contents
4 GraspStateEstimation ......................................... 57
4.1 Introduction .............................................. 57
4.1.1 Concept ........................................... 58
4.1.2 ProblemStatement .................................. 61
4.2 ProbabilisticGraspStateEstimation ......................... 63
4.2.1 Fundamentals ...................................... 63
4.2.2 ParticleFilter ....................................... 64
4.2.3 ExtendedKalmanFilter .............................. 67
4.2.4 FilterSelection ..................................... 69
4.3 ContactDetectionandLocalization .......................... 70
4.3.1 CollisionDetection .................................. 70
4.3.2 JointTorqueMeasurements .......................... 71
4.3.3 ContactPointLocalization ........................... 72
4.4 StateEstimationfromFingerPositionMeasurements ........... 75
4.4.1 GraspStateDefinition ............................... 76
4.4.2 MotionModel ...................................... 76
4.4.3 MeasurementModel ................................ 79
4.4.4 Extensions ......................................... 81
4.5 DataFusionwithFiducialMarkers ........................... 86
4.5.1 AprilTag ........................................... 86
4.5.2 MeasurementModel ................................ 88
4.5.3 CameraLocalization ................................ 89
4.5.4 TargetTracking ..................................... 91
4.6 DataFusionwithContourFeatures .......................... 92
4.6.1 FeatureExtraction .................................. 93
4.6.2 MeasurementModel ................................ 96
4.7 DataFusionwithVisualObjectTracking ..................... 97
4.7.1 Multi-ModalityVisualObjectTracking ................ 98
4.7.2 MeasurementModel ................................ 99
4.8 DataFusionUnderMeasurementDelays ...................... 100
4.9 ExperimentalValidation .................................... 105
4.9.1 GraspAcquisition ................................... 105
4.9.2 Pick-and-Place ..................................... 113
4.9.3 In-HandManipulation ............................... 116
4.10 Summary ................................................ 120
References ..................................................... 122
5 Impedance-BasedObjectControl ................................ 125
5.1 Introduction .............................................. 125
5.1.1 Concept ........................................... 126
5.1.2 ProblemStatement .................................. 128
5.2 ControllerDesign ......................................... 131
5.2.1 ObjectImpedance ................................... 131
5.2.2 ForceDistribution ................................... 134
5.2.3 ArchitectureOverview ............................... 135
Contents xi
5.3 ObjectImpedanceControl .................................. 136
5.3.1 ObjectPositioning .................................. 136
5.3.2 MaintainingtheGraspConfiguration .................. 138
5.4 InternalForces ............................................ 139
5.4.1 ForceDistribution ................................... 140
5.4.2 QuadraticOptimization .............................. 144
5.4.3 Extensions ......................................... 145
5.5 TorqueMapping .......................................... 147
5.5.1 ForceMapping ..................................... 147
5.5.2 NullspaceControl ................................... 148
5.6 GraspReconfiguration ..................................... 149
5.6.1 AddingandRemovingContacts ....................... 150
5.6.2 GraspAcquisition ................................... 151
5.7 EnablingIn-HandManipulation ............................. 153
5.7.1 FingerGaitingInterface .............................. 153
5.7.2 ContactPointRelocation ............................. 155
5.8 ExperimentalValidation .................................... 157
5.8.1 TrackingPerformance ............................... 158
5.8.2 StabilizingtheGraspAcquisition ...................... 160
5.8.3 FingerGaiting ...................................... 167
5.9 Summary ................................................ 169
References ..................................................... 170
6 Conclusion .................................................... 173
6.1 SummaryandDiscussion ................................... 173
6.2 Outlook .................................................. 177
References ..................................................... 180
