Semantic Mapping with Mobile Robots

259 Pages·2011·11.07 MB·English

Semantic Mapping with Mobile Robots
ANDRZEJ PRONOBIS
Doctoral Thesis in Computer Vision and Robotics
Stockholm, Sweden 2011
TRITA-CSC-A 2011:10 Computer Vision and Active Perception
ISSN-1653-5723 School of Computer Science and Communication
ISRN-KTH/CSC/A--11/10--SE KTH Royal Institute of Technology
ISBN 978-91-7501-039-7 SE-100 44 Stockholm, Sweden
Copyright © 2011 by Andrzej Pronobis except where otherwise stated.
Tryck: AJ E-print AB
iii
Abstract
After decades of unrealistic predictions and expectations, robots have fi-
nallyescapedfromindustrialworkplacesandmadetheirwayintoourhomes,
offices, museums and other public spaces. These service robots are increas-
ingly present in our environments and many believe that it is in the area of
service and domestic robotics that we will see the largest growth within the
next few years. In order to realize the dream of robot assistants perform-
inghuman-liketaskstogetherwithhumansinaseamlessfashion,weneedto
providethemwiththefundamentalcapabilityofunderstandingcomplex,dy-
namic and unstructured environments. More importantly, we need to enable
them the sharing of our understanding of space to permit natural cooper-
ation. To this end, this thesis addresses the problem of building internal
representations of space for artificial mobile agents populated with human
spatial semantics as well as means for inferring that semantics from sensory
information. More specifically, an extensible approach to place classification
is introduced and used for mobile robot localization as well as categorization
andextractionofspatialsemanticconceptsfromgeneralplaceappearanceand
geometry. Themodelscanbeincrementallyadaptedtothedynamicchanges
in the environment and employ efficient ways for cue integration, sensor fu-
sion and confidence estimation. In addition, a system and representational
approachtosemanticmapping ispresented. Thesystemincorporatesandin-
tegratessemanticknowledgefrommultiplesourcessuchasthegeometryand
generalappearanceofplaces,presenceofobjects,topologyoftheenvironment
aswellashumaninput. Aconceptual map isdesignedandusedformodeling
and reasoning about spatial concepts and their relations to spatial entities
and their semantic properties. Finally, the semantic mapping algorithm is
built into an integrated robotic system and shown to substantially enhance
theperformanceoftherobotonthecomplextaskofactiveobjectsearch. The
presentedevaluationsshowtheeffectivenessofthesystemanditsunderlying
componentsanddemonstrateapplicabilitytoreal-worldproblemsinrealistic
human settings.
Keywords: spatialunderstanding,semanticmapping,placerecognition,
place categorization, mobile robotics.
iv
Acknowledgments
Therearemanypeoplewhohavesupported,encouragedandinspiredmethrough-
outthisthesis. Ihaveworkedwithandwassurroundedbyanumberofgreatpeople
who contributed in many ways to this research and deserve special mention.
First and foremost, I would like to express my gratitude to Patric Jensfelt, my
main advisor, for his supervision, great engagement and guidance. Thank you for
supporting my ideas and all aspects of my work, for rich and inspiring discussions
and great collaboration, and last but certainly not least for your friendship, un-
derstanding and great sense of humor. I owe many thanks to Barbara Caputo for
givingmetheopportunitytoexploretheexcitingfieldofcomputervision,introduc-
ing me to research in general, as well as hosting me in her group at IDIAP during
my internship. Thank you for your good advices, endless enthusiasm, supervision
and great help. I am also very grateful to Henrik I. Christensen for inspiring me
to pursue my research in the direction of robotics as well as for his support and
sharing with me his vision of AI. Many thanks go to Stefan Carlsson, Jan-Olof
Eklundh and Danica Kragic for their valuable advices and encouragement during
the years spent at KTH.
This thesis has been made possible with the help and cooperation of Alper
Aydemir and Kristoffer Sjöö as well as Adrian Bishop, Moritz Göbelbecker, Marc
Hanheide,LuoJie,ÓscarMartínezMozos,MuhammadMuneebUllah,LiXingand
Hendrik Zender, and all the great people involved in the CoSy and CogX projects.
Kristoffer and Alper, I will never forget our coding nights and the Turkisk peppar!
Furthermore, my thanks go to our administrative staff: Jeanna Ayoubi and Friné
Portal. You were always very kind, helpful and patient.
It was a pleasure to share doctoral studies and life with wonderful people like
Babak, Oscar, Niklas, Mattias and Pedro among others who become close friends.
Special thanks go to my dearest friend Babak for being a part of my family and a
person I can always count on. And thanks for insightful discussions and excellent
proof reading! Babak, Marianna, Mattias, Niklas, Oscar and Pedro thanks for
sharingmygreatpassionforclimbingandinspiringmetotrynewthingsandreach
newgoals,especiallytofinishSvenskKlassiker. Thatwasabsolutelyfantastictime
together! Babak, Njupeskär was just the beginning!
The time at CVAP would not have been the same without: Ahmad, Alessan-
dro, Alireza, Arnaud, Carl-Henrik, Christian, Dan, Elin, Gareth, Gert, Hedvig,
Heydar, Javier, Jeannette, Jorge, Josephine, Kai, Magnus, Miroslav, Mårten, Re-
naud, Omid, Sagar, Simone, Yasemin, Vahid. There are many more, former and
current members of CVAP that have influenced my life in positive ways. The time
spent in Switzerland would not have been as amazing without my climbing and
skiing buddies: Ganga, Ferran, Petr and Stano and other great people: Bogdan,
Chris,Constantin,Danil,Elisa,Francesco,Ghita,Guillermo,Hamed,Kate,Niklas,
Radu, Roger, Tatiana, Tristan and Vincent. Thank you all!
v
Special thanks go to my Polish friends: Anna J. & Marcin, Anna P. (Dzi-
ubasek 1), Grześ (Dziubasek 2), Krzyś & Gabrysia & VaNTa Junior :-), Maciej &
Fredrik, Pit & Dominika and Zygi for being always next to me and deeply in my
heart. When we meet it feels like we never split! Krzyś, Grześ, Zygi, you’re the
best climbing mates!
Lastly, and most importantly, I want to thank my family. My parents for
constant love, support and always being for me. Kocham Was drodzy Rodzice!
Finally, my Marianka who gave me strength to always move forward, through all
the ups and downs of the PhD time. Thank you for giving it all meaning to me
and making my world so rich, beautiful and complete.
Andrzej Pronobis
Stockholm, May 2011
ThisworkwassupportedbytheVRprojectCOMPLEX,theEUFP7ProjectCogX
and the SSF through its Centre for Autonomous Systems.
vi
List of Papers
The thesis is based on the following papers:
[A] AndrzejPronobis,BarbaraCaputo,PatricJensfelt,andHenrikI.Chris-
tensen. A realistic benchmark for visual indoor place recognition.
Robotics and Autonomous Systems (RAS), 58(1):81–96, January 2010.
[B] Andrzej Pronobis, Jie Luo, and Barbara Caputo. The more you learn,
thelessyoustore: Memory-controlledincrementalSVMforvisualplace
recognition. Image and Vision Computing (IMAVIS), Special Issue
on Online Pattern Recognition and Machine Learning Techniques for
Computer-Vision: TheoryandApplications,28(7):1080–1097,July2010.
[C] Andrzej Pronobis and Barbara Caputo. Confidence-based cue integra-
tion for visual place recognition. In Proceedings of the 2007 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS’07),
San Diego, CA, USA, October 2007.
[D] Andrzej Pronobis, Oscar M. Mozos, Barbara Caputo, and Patric Jens-
felt. Multi-modal semantic place classification. The International
Journal of Robotics Research (IJRR), Special Issue on Robotic Vision,
29(2-3):298–320, February 2010.
[E] Andrzej Pronobis, Kristoffer Sjöö, Alper Aydemir, Adrian N. Bishop,
andPatricJensfelt. Representingspatialknowledgeinmobilecognitive
systems. In Proceedings of the 11th International Conference on Intel-
ligent Autonomous Systems (IAS-11), Ottawa, Canada, August 2010.
[F] Andrzej Pronobis and Patric Jensfelt. Understanding the real world:
combining objects, appearance, geometry and topology for semantic
mapping. Technical Report TRITA-CSC-CV 2011:1 CVAP319, Kung-
liga Tekniska Högskolan, CVAP/CAS, May 2011. The paper contains
contributions described in the following submissions:
• Andrzej Pronobis and Patric Jensfelt. Understanding the real
world: combining objects, appearance, geometry and topology for
semantic mapping. In ICRA 2011 Workshop on Semantic Percep-
tion, Mapping and Exploration, Shanghai, China, May 2011.
• Andrzej Pronobis and Patric Jensfelt. Hierarchical multi-modal
place categorization. In European Conference on Mobile Robots
(ECMR’11), Örebro, Sweden, September 2011. (Submitted).
vii
[G] MarcHanheide,CharlesGretton,RichardW.Dearden,NickA.Hawes,
Jeremy L. Wyatt, Andrzej Pronobis, Alper Aydemir, Moritz Göbel-
becker, and Hendrik Zender. Exploiting probabilistic knowledge un-
der uncertain sensing for efficient robot behaviour. In Proceedings of
the 22nd International Joint Conference on Artificial Intelligence (IJ-
CAI’11), Barcelona, Spain, July 2011.
viii
In addition to papers [A]-[G], the following papers have also been produced in
part by the author of the thesis:
[1] AndrzejPronobis,BarbaraCaputo,PatricJensfelt,andHenrikI.Chris-
tensen. A discriminative approach to robust visual place recognition.
In Proceedings of the 2006 IEEE/RSJ International Conference on In-
telligent Robots and Systems (IROS’06), Beijing, China, October 2006.
[2] Jie Luo, Andrzej Pronobis, and Barbara Caputo. SVM-based transfer
of visual knowledge across robotic platforms. In Proceedings of the
5th International Conference on Computer Vision Systems (ICVS’07),
Bielefeld, Germany, March 2007.
[3] Jie Luo, Andrzej Pronobis, Barbara Caputo, and Patric Jensfelt. In-
cremental learning for place recognition in dynamic environments. In
Proceedings of the 2007 IEEE/RSJ International Conference on Intel-
ligent Robots and Systems (IROS’07), San Diego, CA, USA, October
2007.
[4] Andrzej Pronobis, Oscar M. Mozos, and Barbara Caputo. SVM-based
discriminative accumulation scheme for place recognition. In Proceed-
ings of the 2008 IEEE International Conference on Robotics and Au-
tomation (ICRA’08), Pasadena, CA, USA, May 2008.
[5] Muhammad Muneeb Ullah, Andrzej Pronobis, Barbara Caputo, Jie
Luo, Patric Jensfelt, and Henrik I. Christensen. Towards robust place
recognition for robot localization. In Proceedings of the 2008 IEEE In-
ternationalConferenceonRoboticsandAutomation(ICRA’08),Pasadena,
CA, USA, May 2008.
[6] Heydar Maboudi Afkham, Alireza Tavakoli Targhi, Jan-Olof Eklundh,
andAndrzejPronobis. Jointvisualvocabularyforanimalclassification.
In Proceedings of the 2008 19th International Conference on Pattern
Recognition (ICPR’08), Tampa, FL, USA, December 2008.
[7] Andrzej Pronobis and Barbara Caputo. COLD: COsy Localization
Database. The International Journal of Robotics Research (IJRR), 28
(5):588–594, May 2009.
[8] Andrzej Pronobis, Kristoffer Sjöö, Alper Aydemir, Adrian N. Bishop,
and Patric Jensfelt. A framework for robust cognitive spatial map-
ping. In Proceedings of the 14th International Conference on Advanced
Robotics (ICAR’09), Munich, Germany, June 2009.
ix
[9] AndrzejPronobis,LiXing,andBarbaraCaputo. OverviewoftheCLEF
2009 robot vision track. In Carol Peters, Barbara Caputo, Julio Gon-
zalo, Gareth J. F. Jones, Jayashree Kalpathy-Cramer, Henning Müller,
and Theodora Tsikrika, editors, Multilingual Information Access Eval-
uation Vol. II Multimedia Experiments, volume 6242 of Lecture Notes
in Computer Science, pages 110–119. Springer, 2010.
[10] AndrzejPronobis, HenrikChristensen, andBarbaraCaputo. Overview
of the ImageCLEF@ICPR 2010 Robot Vision track. In Devrim Ünay,
ZehraÇataltepe,andSelimAksoy,editors,Recognizing Patterns in Sig-
nals, Speech, Images and Videos,volume6388ofLecture Notes in Com-
puter Science, pages 171–179. Springer Berlin / Heidelberg, 2010.
[11] Andrzej Pronobis, Marco Fornoni, Henrik I. Christensen, and Barbara
Caputo. The Robot Vision track at ImageCLEF 2010. In Working
Notes for the CLEF 2010 Workshop, Padua, Italy, 2010. ISBN 978-88-
904810-0-0.
[12] Andrzej Pronobis and Barbara Caputo. The robot vision task. In
HenningMüller,PaulClough,ThomasDeselaers,andBarbaraCaputo,
editors, ImageCLEF, volume 32 of The Information Retrieval Series,
pages 185–198. Springer Berlin Heidelberg, 2010. ISBN 978-3-642-
15181-1.
[13] Li Xing and Andrzej Pronobis. Multi-cue discriminative place recog-
nition. In Carol Peters, Barbara Caputo, Julio Gonzalo, Gareth J. F.
Jones,JayashreeKalpathy-Cramer,HenningMüller,andTheodoraTsikrika,
editors, Multilingual Information Access Evaluation Vol. II Multimedia
Experiments,volume6242ofLecture Notes in Computer Science,pages
315–323. Springer, 2010.
[14] AndrzejPronobis,PatricJensfelt,KristofferSjöö,HendrikZender,Geert-
Jan M. Kruijff, Oscar M. Mozos, and Wolfram Burgard. Semantic
modelling of space. In Henrik I. Christensen, Geert-Jan M. Kruijff,
and Jeremy L. Wyatt, editors, Cognitive Systems, volume 8 of Cogni-
tive Systems Monographs, pages 165–221. Springer Berlin Heidelberg,
2010. ISBN 978-3-642-11694-0.
[15] Kristoffer Sjöö, Hendrik Zender, Patric Jensfelt, Geert-Jan M. Krui-
jff, Andrzej Pronobis, Nick Hawes, and Michael Brenner. The ex-
plorer system. In Henrik I. Christensen, Geert-Jan M. Kruijff, and
Jeremy L. Wyatt, editors, Cognitive Systems, volume 8 of Cognitive
Systems Monographs, pages 395–421. Springer Berlin Heidelberg, 2010.
ISBN 978-3-642-11694-0.
x
[16] Jeremy L. Wyatt, Alper Aydemir, Michael Brenner, Marc Hanheide,
NickHawes,PatricJensfelt,MatejKristan,Geert-JanM.Kruijff,Pierre
Lison, Andrzej Pronobis, Kristoffer Sjöö, Alen Vrečko, Hendrik Zen-
der, Michael Zillich, and Danijel Skočaj. Self-understanding & self-
extension:a systems and representational approach. IEEE Transac-
tions on Autonomous Mental Development (TAMD), Special Issue on
Representations and Architectures for Cognitive Systems, 2(4):282–303,
December 2010.
[17] Alper Aydemir, Kristoffer Sjöö, John Folkesson, Andrzej Pronobis, and
Patric Jensfelt. Search in the real world: Active visual object search
based on spatial relations. In Proceedings of the 2011 IEEE Interna-
tional Conference on Robotics and Automation (ICRA’11), Shanghai,
China, May 2011.
[18] KristofferSjöö,AndrzejPronobis,andPatricJensfelt. Functionaltopo-
logical relations for qualitative spatial representation. In Proceedings
of the 15th International Conference on Advanced Robotics (ICAR’11),
Tallinn, Estonia, June 2011.