Mixed and augmented reality in medicine

305 Pages·2019·29.2 MB·English

by  Peters

Mixed and Augmented
Reality in Medicine
Series in Medical Physics and Biomedical Engineering
Series Editors: John G. Webster, E. Russell Ritenour, Slavik Tabakov,
and Kwan Hoong Ng
Recent books in the series:
Targeted Muscle Reinnervation: A Neural Interface for Artificial Limbs
Todd A. Kuiken, Aimee E. Schultz Feuser, Ann K. Barlow (Eds)
Advanced MR Neuroimaging: From Theory to Clinical Practice
Ioannis Tsougos
Handbook of X-ray Imaging: Physics and Technology
Paolo Russo (Ed)
Quantitative MRI of the Brain: Principles of Physical Measurement, Second Edition
Mara Cercignani, Nicholas G. Dowell, Paul S. Tofts (Eds)
Graphics Processing Unit-Based High Performance Computing in Radiation Therapy
Xun Jia, Steve B. Jiang (Eds)
A Guide to Outcome Modeling in Radiotherapy and Oncology: Listening to the Data
Issam El Naqa (Ed)
Radiotherapy and Clinical Radiobiology of Head and Neck Cancer
Loredana G. Marcu, Iuliana Toma-Dasu, Alexandru Dasu, Claes Mercke
Advances in Particle Therapy: A Multidisciplinary Approach
Manjit Dosanjh, Jacques Bernier (Eds)
Problems and Solutions in Medical Physics: Diagnostic Imaging Physics
Kwan Hoong Ng, Jeannie Hsiu Ding Wong, Geoffrey D. Clarke
Advances and Emerging Technologies in Radiation Oncology Physics
Siyong Kim, John Wong
Clinical Radiotherapy Physics with MATLAB: A Problem-Solving Approach
Pavel Dvorak
Proton Therapy Physics, Second Edition
Harald Paganetti (Ed)
For more information about this series, please visit: https://www.crcpress.com/Series-in-Medical-
Physics-and-Biomedical-Engineering/book-series/CHMEPHBIOENG
Series in Medical Physics and Biomedical Engineering
Series Editors: John G. Webster, E. Russell Ritenour, Slavik Tabakov,
and Kwan Hoong Ng Mixed and Augmented
Recent books in the series: Reality in Medicine
Targeted Muscle Reinnervation: A Neural Interface for Artificial Limbs
Todd A. Kuiken, Aimee E. Schultz Feuser, Ann K. Barlow (Eds)
Advanced MR Neuroimaging: From Theory to Clinical Practice
Ioannis Tsougos
Handbook of X-ray Imaging: Physics and Technology
Paolo Russo (Ed)
Quantitative MRI of the Brain: Principles of Physical Measurement, Second Edition
Mara Cercignani, Nicholas G. Dowell, Paul S. Tofts (Eds)
Graphics Processing Unit-Based High Performance Computing in Radiation Therapy
Xun Jia, Steve B. Jiang (Eds)
Edited by
A Guide to Outcome Modeling in Radiotherapy and Oncology: Listening to the Data Terry M. Peters
Issam El Naqa (Ed)
Robarts Research Institute, Western University, Canada
Radiotherapy and Clinical Radiobiology of Head and Neck Cancer
Cristian A. Linte
Loredana G. Marcu, Iuliana Toma-Dasu, Alexandru Dasu, Claes Mercke
Department of Biomedical Engineering and
Advances in Particle Therapy: A Multidisciplinary Approach
Chester F Carlson Center for Imaging Science
Manjit Dosanjh, Jacques Bernier (Eds)
Rochester Institute of Technology, USA
Problems and Solutions in Medical Physics: Diagnostic Imaging Physics
Ziv Yaniv
Kwan Hoong Ng, Jeannie Hsiu Ding Wong, Geoffrey D. Clarke
Office of High Performance Computing and
Advances and Emerging Technologies in Radiation Oncology Physics
Communications, National Library of Medicine
Siyong Kim, John Wong
National Institutes of Health, and TAJ Technologies Inc, USA
Clinical Radiotherapy Physics with MATLAB: A Problem-Solving Approach
Pavel Dvorak Jacqueline Williams
Robarts Research Institute, Western University, Canada
Proton Therapy Physics, Second Edition
Harald Paganetti (Ed)
For more information about this series, please visit: https://www.crcpress.com/Series-in-Medical-
Physics-and-Biomedical-Engineering/book-series/CHMEPHBIOENG
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2019 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed on acid-free paper
International Standard Book Number-13: 978-1-138-06863-6 (Hardback)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts
have been made to publish reliable data and information, but the author and publisher cannot assume
responsibility for the validity of all materials or the consequences of their use. The authors and publishers
have attempted to trace the copyright holders of all material reproduced in this publication and apologize
to copyright holders if permission to publish in this form has not been obtained. If any copyright material
has not been acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information storage
or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.
com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood
Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and
registration for a variety of users. For organizations that have been granted a photocopy license by the
CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
About the Series
The Series in Medical Physics and Biomedical Engineering describes the applica-
tions of physical sciences, engineering, and mathematics in medicine and clinical
research.
The series seeks (but is not restricted to) publications in the following topics:
• Artificial organs
• Assistive technology
• Bioinformatics
• Bioinstrumentation
• Biomaterials
• Biomechanics
• Biomedical engineering
• Clinical engineering
• Imaging
CRC Press • Implants
Taylor & Francis Group • Medical computing and mathematics
6000 Broken Sound Parkway NW, Suite 300
• Medical/surgical devices
Boca Raton, FL 33487-2742
• Patient monitoring
© 2019 by Taylor & Francis Group, LLC
• Physiological measurement
CRC Press is an imprint of Taylor & Francis Group, an Informa business
• Prosthetics
No claim to original U.S. Government works • Radiation protection, health physics, and dosimetry
Printed on acid-free paper • Regulatory issues
• Rehabilitation engineering
International Standard Book Number-13: 978-1-138-06863-6 (Hardback)
• Sports medicine
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts • Systems physiology
have been made to publish reliable data and information, but the author and publisher cannot assume • Telemedicine
responsibility for the validity of all materials or the consequences of their use. The authors and publishers
• Tissue engineering
have attempted to trace the copyright holders of all material reproduced in this publication and apologize
to copyright holders if permission to publish in this form has not been obtained. If any copyright material • Treatment
has not been acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, The Series in Medical Physics and Biomedical Engineering is an international series
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or that meets the need for up-to-date texts in this rapidly developing field. Books in the
hereafter invented, including photocopying, microfilming, and recording, or in any information storage
series range in level from introductory graduate textbooks and practical handbooks
or retrieval system, without written permission from the publishers.
to more advanced expositions of current research. The Series in Medical Physics and
For permission to photocopy or use material electronically from this work, please access www.copyright. Biomedical Engineering is the official book series of the International Organization
com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood
for Medical Physics.
Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and
registration for a variety of users. For organizations that have been granted a photocopy license by the
CCC, a separate system of payment has been arranged.
THE INTERNATIONAL ORGANIZATION FOR MEDICAL PHYSICS
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe. The International Organization for Medical Physics (IOMP) represents over 18,000
Visit the Taylor & Francis Web site at medical physicists worldwide and has a membership of 80 national and 6 regional
http://www.taylorandfrancis.com organizations, together with a number of corporate members. Individual medical
and the CRC Press Web site at physicists of all national member organisations are also automatically members.
http://www.crcpress.com
v
vi About the Series
The mission of IOMP is to advance medical physics practice worldwide by
disseminating scientific and technical information, fostering the educational and
professional development of medical physics, and promoting the highest quality
medical physics services for patients.
A World Congress on Medical Physics and Biomedical Engineering is held every
three years in cooperation with International Federation for Medical and Biological
Engineering (IFMBE) and International Union for Physics and Engineering
Sciences in Medicine (IUPESM). A regionally based international conference, the
International Congress of Medical Physics (ICMP) is held between world congresses.
IOMP also sponsors international conferences, workshops, and courses.
The IOMP has several programmes to assist medical physicists in develop-
ing countries. The joint IOMP Library Programme supports 75 active libraries in
43 developing countries, and the Used Equipment Programme coordinates equip-
ment donations. The Travel Assistance Programme provides a limited number of
grants to enable physicists to attend the world congresses.
IOMP co-sponsors the Journal of Applied Clinical Medical Physics. The IOMP
publishes, twice a year, an electronic bulletin, Medical Physics World. IOMP also
publishes e-Zine, an electronic news letter about six times a year. IOMP has an
agreement with Taylor & Francis for the publication of the Medical Physics and
Biomedical Engineering series of textbooks. IOMP members receive a discount.
IOMP collaborates with international organizations, such as the World Health
Organizations (WHO), the International Atomic Energy Agency (IAEA), and other
international professional bodies such as the International Radiation Protection
Association (IRPA) and the International Commission on Radiological Protection
(ICRP), to promote the development of medical physics and the safe use of radiation
and medical devices.
Guidance on education, training, and professional development of medical physi-
cists is issued by IOMP, which is collaborating with other professional organizations
in development of a professional certification system for medical physicists that can
be implemented on a global basis.
The IOMP website (www.iomp.org) contains information on all the activities of
the IOMP, policy statements 1 and 2, and the ‘IOMP: Review and Way Forward,’
which outlines all the activities of IOMP and plans for the future.
Contents
Preface.......................................................................................................................ix
Editors .......................................................................................................................xi
Contributors ...........................................................................................................xiii
Chapter 1 Overview of Mixed and Augmented Reality in Medicine ...................1
Terry M. Peters
Chapter 2 Tracking and Calibration ....................................................................15
Elvis C. S. Chen and Burton Ma
Chapter 3 Registration ........................................................................................29
Lena Maier-Hein, Stefanie Speidel, Esther Stenau,
Elvis C. S. Chen, and Burton Ma
Chapter 4 Display Technologies .........................................................................47
Ulrich Eck and Tobias Sielhorst
Chapter 5 Active Sensorimotor Augmentation in Robotics-Assisted
Surgical Systems ................................................................................61
Seyed Farokh Atashzar, Michael Naish, and Rajni V. Patel
Chapter 6 Visual Perception and Human–Computer Interaction in
Surgical Augmented and Virtual Reality Environments ...................83
Roy Eagleson and Sandrine de Ribaupierre
Chapter 7 Interaction in Augmented Reality Image-Guided Surgery ................99
Simon Drouin, D. L. Collins, and M. Kersten-Oertel
Chapter 8 Cognitive Oriented Design and Assessment of Augmented
Reality in Medicine ..........................................................................115
Pierre Jannin and Thierry Morineau
Chapter 9 Augmented Reality in Aging and Medical Education .....................127
Pascal Fallavollita and Nassir Navab
vii
viii Contents
Chapter 10 Cost Effective Simulation: A Mixed Reality Approach ...................135
Ziv Yaniv, Özgür Güler, and Ren Hui Gong
Chapter 11 Augmented Reality-Based Visualization for Echocardiographic
Applications ......................................................................................155
Gabriel Hanssen Kiss, Cameron Lowell Palmer, Ole Christian
Mjølstad, Håvard Dalen, Bjørn Olav Haugen, and Hans Torp
Chapter 12 Ultrasound Augmented Laparoscopy: Technology and Human
Factors ..............................................................................................171
Uditha L. Jayarathne
Chapter 13 Toward Clinically Viable Ultrasound-Augmented Laparoscopic
Visualization ....................................................................................189
Xinyang Liu and Raj Shekhar
Chapter 14 Augmented Reality in Image-Guided Robotic Surgery ...................203
Wen Pei Liu and Russell H. Taylor
Chapter 15 Augmented Reality for Reducing Intraoperative Radiation
Exposure to Patients and Clinicians during X-Ray Guided
Procedures ........................................................................................217
Nicolas Loy Rodas and Nicolas Padoy
Chapter 16 Augmented and Virtual Visualization for Image-Guided
Cardiac Therapeutics ........................................................................231
Cristian A. Linte, Terry M. Peters, and Michael S. Sacks
Chapter 17 3D Augmented Reality-Based Surgical Navigation and
Intervention ......................................................................................251
Zhencheng Fan, Cong Ma, Xinran Zhang, and Hongen Liao
Chapter 18 Augmenting Haptic Perception in Surgical Tools ............................265
Randy Lee, Roberta L. Klatzky, and George D. Stetten
Index ......................................................................................................................283
Preface
The origins of this book date back to the MICCAI Workshop on Augmented
Environments for Computer-assisted Interventions (AE-CAI) held October 2015
in Munich, Germany. The AE-CAI workshop, which this year (2018) celebrates its
12th edition, continues to serve as a leading platform for dissemination of cutting-
edge work in augmented, virtual, and mixed reality visualization for image-guided
interventions, simulation, and training.
After being approached by Taylor & Francis to consider editing a book on the
topic of the workshop, we discussed the idea and agreed that such a collection would
fit an under-represented sector in the literature. It also would serve as a timely, ten-
year update and augmentation of the 2008 publication, Image-guided Interventions:
Technology and Applications, edited by Peters and Cleary. However, to do a new
book justice, it would need to be focused more specifically on mixed reality in
the context of interventions, simulation, and training. During the 2015 edition of
AE-CAI in Munich, we received overwhelming enthusiasm from authors, present-
ers, and attendees to contribute to this book.
Jacqueline Williams generously agreed to take on the role of Executive Editor,
and during the past 18 months, we received contributions from the authors repre-
sented in the 18 chapters in this book. The title, Mixed and Augmented Reality in
Medicine was selected to represent the widest range of visualization environments
across the reality-virtuality continuum, ranging from purely real to purely virtual.
Although all such environments fall under mixed reality, depending on the extent of
their real to virtual content, augmented reality has evolved as the widely agreed upon
terminology to describe even mixed realities.
This book overviews a variety of mixed reality environments and their applica-
tion in medicine. It begins with a historical overview chapter on medical augmented
realities. Subsequent chapters focus on the fundamental technologies for building
augmented reality environments, including surgical tracking, registration, display
technologies, and haptics. Additional chapters focus on applications in education
and training, visualization for medical education and rehabilitation, cost-effective
simulations, augmented reality visualization for an ultrasound and ultrasound-
enhanced laparoscopic applications, X-ray guided procedures, minimally invasive
cardiac diagnosis and therapy, surgical navigation and intervention, as well as aug-
mented haptic perception in surgery. Finally, the collection would be incomplete
without discussing perceptual capacities and constraints, interaction paradigms and
cognitive-oriented design, and assessment of medical augmented realities. These
topics are covered in three chapters dedicated to perception and cognition in
augmented reality.
ix