Table Of ContentSpringer Tracts in Advanced Robotics 85
Editors
Prof.BrunoSiciliano Prof.OussamaKhatib
DipartimentodiInformatica ArtificialIntelligenceLaboratory
eSistemistica DepartmentofComputerScience
UniversitàdiNapoliFedericoII StanfordUniversity
ViaClaudio21,80125Napoli Stanford,CA94305-9010
Italy USA
E-mail:[email protected] E-mail:[email protected]
Forfurthervolumes:
http://www.springer.com/series/5208
EditorialAdvisoryBoard
OliverBrock,TUBerlin,Germany
HermanBruyninckx,KULeuven,Belgium
RajaChatila,ISIR-UPMC&CNRS,France
HenrikChristensen,GeorgiaTech,USA
PeterCorke,QueenslandUniv.Technology,Australia
PaoloDario,ScuolaS.AnnaPisa,Italy
RüdigerDillmann,Univ.Karlsruhe,Germany
KenGoldberg,UCBerkeley,USA
JohnHollerbach,Univ.Utah,USA
MakotoKaneko,OsakaUniv.,Japan
LydiaKavraki,RiceUniv.,USA
VijayKumar,Univ.Pennsylvania,USA
SukhanLee,SungkyunkwanUniv.,Korea
FrankPark,SeoulNationalUniv.,Korea
TimSalcudean,Univ.BritishColumbia,Canada
RolandSiegwart,ETHZurich,Switzerland
GauravSukhatme,Univ.SouthernCalifornia,USA
SebastianThrun,StanfordUniv.,USA
YangshengXu,ChineseUniv.HongKong,PRC
Shin’ichiYuta,TsukubaUniv.,Japan
N
SdeTrAtRhe(aSupsrpinicgeesrTofraEcUtsRinOANd(vEaunrcoepdeRanobRootbicost)ichsasRbeeseeanrpcrhoNmeottwedorukn)- RERuersoOepaeracBnhROO************TICS
NetworkU
E
Radu Bogdan Rusu
Semantic 3D Object Maps
for Everyday Robot
Manipulation
ABC
Author
RaduBogdanRusu
OpenPerception,Inc.
SanFrancisco,CA
USA
ISSN1610-7438 e-ISSN1610-742X
ISBN978-3-642-35478-6 e-ISBN978-3-642-35479-3
DOI10.1007/978-3-642-35479-3
SpringerHeidelbergNewYorkDordrechtLondon
LibraryofCongressControlNumber:2012954153
(cid:2)c Springer-VerlagBerlinHeidelberg2013
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.
PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpub-
lication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforany
errorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,withrespect
tothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Foreword
Robotics is undergoing a major transformation in scope and dimension. From a
largely dominant industrial focus, robotics is rapidly expanding into human envi-
ronmentsandvigorouslyengagedinitsnewchallenges.Interactingwith,assisting,
serving, and exploring with humans, the emerging robots will increasingly touch
peopleandtheirlives.
Beyondits impacton physicalrobots,the bodyof knowledgeroboticshaspro-
duced is revealing a much wider range of applications reaching across diverse
research areas and scientific disciplines, such as: biomechanics, haptics, neuro-
sciences,virtualsimulation,animation,surgery,andsensornetworksamongothers.
Inreturn,thechallengesofthenewemergingareasareprovinganabundantsource
ofstimulationandinsightsforthefieldofrobotics.Itisindeedattheintersectionof
disciplinesthatthemoststrikingadvanceshappen.
TheSpringerTractsin AdvancedRobotics(STAR)isdevotedtobringingtothe
research community the latest advances in the robotics field on the basis of their
significance and quality. Through a wide and timely dissemination of critical re-
searchdevelopmentsin robotics,ourobjectivewith this seriesis to promotemore
exchanges and collaborations among the researchers in the community and con-
tributetofurtheradvancementsinthisrapidlygrowingfield.
The monograph by Radu Bogdan Rusu is a contribution aimed at developing
novelrepresentationsofrobots’operatingenvironments.Thepromisingtechnology
ofpointcloud-basedperceptionisintroducedtoderivesemantic3Dobjectmodels
enabling complex mobile manipulation scenarios. Results are validated in a rich
setofexperimentsondifferentrobotsperformingavarietyoftasksindynamically
changing indoor environments, revealing a promising outlook toward the goal of
employingautonomousrobotsineverydayhabitats.
Remarkably,themonographisbasedontheauthor’sdoctoralthesis,whichwas
a finalistforthe TenthEditionofthe EURONGeorgesGiraltPhDAward devoted
tothebestPhDthesisinRoboticsinEurope.AveryfineadditiontoSTAR!
Naples,Italy BrunoSiciliano
October2012 STAREditor
Abstract
Environmentmodelsserveasimportantresourcesforanautonomousrobotbypro-
viding it with the necessary task-relevant information about its habitat. Their use
enablesrobotstoperformtheirtasksmorereliably,flexibly,andefficiently.Asau-
tonomousroboticplatformsgetmoresophisticatedmanipulationcapabilities,they
also needmoreexpressiveandcomprehensiveenvironmentmodels:formanipula-
tionpurposestheirmodelshavetoincludetheobjectspresentintheworld,together
with their position, form, and other aspects, as well as an interpretation of these
objectswithrespecttotherobottasks.
This book proposes Semantic 3D Object Models as a novel representation of
therobot’soperatingenvironmentthatsatisfiestheserequirementsandshowshow
these models can be automatically acquired from dense 3D range data. The book
contributesin two importantways to the researcharea acquisitionof environment
models.
ThefirstcontributionisanovelframeworkforSemantic3DObjectModelacqui-
sition from PointCloudData.The functionalityofthis frameworkincludesrobust
alignmentandintegrationmechanismsforpartialdataviews,fastsegmentationinto
regionsbasedonlocalsurfacecharacteristics,andreliableobjectdetection,catego-
rization, and reconstruction.The computedmodelsare semantic in that they infer
structuresinthedatathataremeaningfulwithrespecttotherobottask.Examplesof
suchobjectsaredoorsandhandles,supportingplanes,cupboards,walls,ormovable
smallerobjects.
The second key contribution is point cloud representations based on 3D point
feature histograms (3D-PFHs), which model the local surface geometry for each
point.3D-PFHsdistinguishthemselvesfromalternative3Dfeaturerepresentations
inthattheyareveryfasttocompute,robustagainstvariationsinposeandsampling
density,andcopewellwithnoisysensordata.Theirusesubstantiallyimprovesthe
qualityoftheSemantic3DObjectModelsacquired,aswellasthespeedwithwhich
they are computed. 3D-PFHs come with specific software tools that allow for the
learningofsurfacecharacteristicsbasedontheirunderlyinggeometry,theassembly
ofmostdistinctive3Dpointsfromagivencloud,aswellaslimitedview-invariant
correspondencesearchfor3Dregistration.
The contributionspresentedin this bookhave been fullyimplementedand em-
pirically evaluated on different robots performing different tasks in different en-
vironments. The first demonstration relates to the problem of cleaning tables by
disposing the objects on them into a garbagebin with a personalroboticassistant
VIII Abstract
in the presence of humans in its working space. The frameworkfor Semantic 3D
ObjectModelacquisitionisdemonstratedandusedtoconstructdynamic3Dcolli-
sionmaps,annotatethesurroundingworldwithsemanticlabels,andextractobject
clusters supported by tables in real-time performance. The second demonstration
presentsanon-the-flymodelacquisitionsystem for doorand handleidentification
from noisy 3D point cloud maps. Experimental results show good robustness in
the presence of large variations in the data, without suffering from the classical
under or over-fitting problemsusually associated with similar initiatives based on
machinelearningclassifiers. Thethirdapplicationexampletacklestheproblemof
real-timesemanticmappingofindoorenvironmentswithdifferentkindsofterrain
classes, suchas walkwaysandstairs, forthe navigationofa six-leggedrobotwith
terrain-specificwalkingmodes.
Contents
Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V
Abstract. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . VII
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IX
ListofFigures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XIII
ListofTables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XIX
ListofAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XXI
ListofSymbolsandNotations . . . . . . . . . . . . . . . . . . . . . . XXIII
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Why“3D”SemanticPerception? . . . . . . . . . . . . . . . . . 2
1.2 ComputationalProblems . . . . . . . . . . . . . . . . . . . . . 3
1.3 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 BookOutlineandContributions . . . . . . . . . . . . . . . . . 9
PartI Semantic 3DObject Mapping Kernel
2 3DMapRepresentations . . . . . . . . . . . . . . . . . . . . . . . 15
2.1 DataAcquisition . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 DataRepresentation. . . . . . . . . . . . . . . . . . . . . . . . 20
2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3 MappingSystemArchitectures . . . . . . . . . . . . . . . . . . . . 27
4 3DPointFeatureRepresentations . . . . . . . . . . . . . . . . . . 33
4.1 The“Neighborhood”Concept . . . . . . . . . . . . . . . . . . 35
4.2 FilteringOutliers . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.3 SurfaceNormalsandCurvatureEstimates . . . . . . . . . . . . 40
4.4 PointFeatureHistograms(PFH) . . . . . . . . . . . . . . . . . 44
4.5 FastPointFeatureHistograms(FPFH) . . . . . . . . . . . . . . 51
4.6 FeaturePersistence . . . . . . . . . . . . . . . . . . . . . . . . 54
X Contents
4.7 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5 FromPartialtoCompleteModels . . . . . . . . . . . . . . . . . . 61
5.1 PointCloudRegistration . . . . . . . . . . . . . . . . . . . . . 61
5.2 DataResampling . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.3 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6 ClusteringandSegmentation . . . . . . . . . . . . . . . . . . . . . 75
6.1 FittingSimplifiedGeometricModels . . . . . . . . . . . . . . . 76
6.2 BasicClusteringTechniques . . . . . . . . . . . . . . . . . . . 78
6.3 FindingEdgesin3DData. . . . . . . . . . . . . . . . . . . . . 80
6.4 SegmentationviaRegionGrowing . . . . . . . . . . . . . . . . 81
6.5 ApplicationSpecificModelFitting . . . . . . . . . . . . . . . . 83
6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
PartII Mapping ofIndoorEnvironments
7 StaticSceneInterpretation . . . . . . . . . . . . . . . . . . . . . . 89
7.1 HeuristicRule-BasedFunctionalReasoning . . . . . . . . . . . 91
7.2 LearningtheSceneStructure . . . . . . . . . . . . . . . . . . . 100
7.3 ExportingandUsingtheModels . . . . . . . . . . . . . . . . . 105
7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
8 SurfaceandObjectClassLearning. . . . . . . . . . . . . . . . . . 109
8.1 LearningLocalSurfaceClasses. . . . . . . . . . . . . . . . . . 111
8.1.1 GeneratingTrainingData. . . . . . . . . . . . . . . . . 113
8.1.2 MostDiscriminativeFeatureSelection . . . . . . . . . . 115
8.1.3 SupervisedClassLearningUsingSupportVector
Machines . . . . . . . . . . . . . . . . . . . . . . . . . 118
8.2 FastGeometricPointLabeling . . . . . . . . . . . . . . . . . . 123
8.3 GlobalFastPointFeatureHistogramsforObject
Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
9 ParametricShapeModelFitting . . . . . . . . . . . . . . . . . . . 137
9.1 ObjectSegmentation . . . . . . . . . . . . . . . . . . . . . . . 138
9.2 HybridShape-SurfaceObjectModels . . . . . . . . . . . . . . 141
9.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
Contents XI
PartIII Applications
10 TableCleaninginDynamicEnvironments . . . . . . . . . . . . . . 149
10.1 Real-TimeCollisionMapsforMotionRe-planning . . . . . . . 152
10.2 SemanticInterpretationof3DPointCloudMaps . . . . . . . . 154
10.3 SystemEvaluation . . . . . . . . . . . . . . . . . . . . . . . . 156
10.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
11 IdentifyingandOpeningDoors . . . . . . . . . . . . . . . . . . . . 161
11.1 DetectingDoors . . . . . . . . . . . . . . . . . . . . . . . . . . 165
11.2 DetectingHandles. . . . . . . . . . . . . . . . . . . . . . . . . 167
11.3 SystemEvaluation . . . . . . . . . . . . . . . . . . . . . . . . 171
11.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
12 Real-TimeSemanticMapsfromStereo . . . . . . . . . . . . . . . 177
12.1 LeavingFlatlandMappingArchitecture . . . . . . . . . . . . . 179
12.1.1 VisualOdometer . . . . . . . . . . . . . . . . . . . . . 181
12.1.2 SpatialDecomposition . . . . . . . . . . . . . . . . . . 184
12.1.3 PolygonalModeling . . . . . . . . . . . . . . . . . . . 186
12.1.4 MergingandRefinement . . . . . . . . . . . . . . . . . 187
12.1.5 SemanticLabeling . . . . . . . . . . . . . . . . . . . . 189
12.1.6 3DMappingPerformance . . . . . . . . . . . . . . . . 191
12.2 SemanticMapUsageandApplications . . . . . . . . . . . . . . 193
12.2.1 HybridModelVisualizations . . . . . . . . . . . . . . . 193
12.2.2 MotionPlanningforNavigation . . . . . . . . . . . . . 194
12.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
13 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
A 3DGeometryPrimer . . . . . . . . . . . . . . . . . . . . . . . . . 201
A.1 EuclideanGeometryandCoordinateSystems . . . . . . . . . . 201
A.2 DistanceMetrics . . . . . . . . . . . . . . . . . . . . . . . . . 202
A.3 GeometricShapes . . . . . . . . . . . . . . . . . . . . . . . . . 203
B SampleConsensus . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
C MachineLearning . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
C.1 SupportVectorMachines . . . . . . . . . . . . . . . . . . . . . 207
C.2 ConditionalRandomFields . . . . . . . . . . . . . . . . . . . . 210
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
