Table Of ContentVISUAL TRACKING IN
CONVENTIONAL MINIMALLY
INVASIVE SURGERY
VISUAL TRACKING IN
CONVENTIONAL MINIMALLY
INVASIVE SURGERY
Shahram Payandeh
Simon Fraser University
Burnaby, Canada
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2017 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed on acid-free paper
Version Date: 20160714
International Standard Book Number-13: 978-1-4987-6395-0 (Hardback)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts
have been made to publish reliable data and information, but the author and publisher cannot assume
responsibility for the validity of all materials or the consequences of their use. The authors and publishers
have attempted to trace the copyright holders of all material reproduced in this publication and apologize to
copyright holders if permission to publish in this form has not been obtained. If any copyright material has
not been acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit-
ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying, microfilming, and recording, or in any information storage or retrieval system,
without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.
com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood
Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and
registration for a variety of users. For organizations that have been granted a photocopy license by the CCC,
a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used
only for identification and explanation without intent to infringe.
Library of Congress Cataloging‑in‑Publication Data
Names: Payandeh, Shahram, 1957- author.
Title: Visual tracking in conventional minimally invasive surgery / Shahram
Payandeh.
Description: Boca Raton : Taylor & Francis, 2017. | Includes bibliographical
references and index.
Identifiers: LCCN 2016022118 | ISBN 9781498763950
Subjects: LCSH: Surgical instruments and apparatus--Design and construction.
Classification: LCC RD71 .P39 2017 | DDC 617.9/178--dc23
LC record available at https://lccn.loc.gov/2016022118
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents
Summary, ix
Chapter 1 ▪ Overview Of MiniMally invasive surgery 1
OVERVIEW OF MINIMALLY INVASIVE SURGERY TRAINING
ENVIRONMENTS 5
CONVENTIONAL VERSUS ROBOT-ASSISTED MINIMALLY
INVASIVE SURGERY 7
MOTIVATIONS AND OBJECTIVES OF THIS BOOK 8
LITERATURE OVERVIEW 10
LAYOUT OF THE BOOK 15
SUMMARY 16
Chapter 2 ▪ endOsCOpe setup and CalibratiOn 17
OPTICAL SYSTEM 17
Light Source 17
Endoscope and Cables 18
Video Camera 19
Surgical Tools 19
IMAGE PROCESSING 21
Endoscopic Camera Calibration 22
Lens Distortion and Correction Methods 26
Radial Lens Distortion Model 28
Estimation of Coefficients 30
EXPERIMENTAL STUDIES 32
Locating the Real Image Center 32
v
vi ◾ Contents
Obtaining Image Coordinates 33
Computation of the Intrinsic Parameters 33
Calibration Parameters 35
Distortion Correction 36
SUMMARY 37
Chapter 3 ▪ Marker-based traCking 39
DESIGN OF MARKER 39
UNIT CONVERSION 42
Pixel-to-Millimeter Conversion 42
Results and Analysis 44
GESTURE RECOGNITION OF SURGICAL TOOLS 49
Target Tool Segmentation 52
Feature Extraction 56
Gesture Pattern Recognition Using Artificial Neural Networks 61
Results and Analysis 65
DESIGN OF SURGEON–COMPUTER INTERFACES 74
SUMMARY 79
Chapter 4 ▪ Markerless traCking: gaussian apprOaCh 81
OVERVIEW OF MOVEMENT TRACKING USING MOTION
HISTORY IMAGES 82
FEATURE-BASED TRACKING 85
Overview of the Feature-Based Tracking Algorithm 88
Two-Dimensional Tracking of Surgical Tools 89
Kalman Filter 89
Initialization and Preprocessing 92
Surgical Tool Tracking Using Standard Kalman Filtering 95
Modified Kalman Filter 98
Extended Kalman Filter 100
Adaptive Gaussian Mixture Model (AGGM) 103
SURGICAL TOOL TIP EXTRACTION 105
Contents ◾ vii
Two-Dimensional Tracking of Multiple Tools 109
SURGICAL TOOL TRACKING IN 3-D 110
Three-Dimensional Pose Estimation Using Surgical Tool Tip
and Edges 110
EXPERIMENTAL STUDIES 112
Single-Tool Tracking 112
Multiple-Tool Tracking 116
THREE-DIMENSIONAL INTERACTIVE SURGICAL MOUSE 118
Design of an Augmented Surgical Environment 121
Menu Design 122
Interaction Design 122
Implementation of a 3-D Mouse 126
Aligning the Endoscopic and Virtual Images 126
Three-Dimensional Manipulation of Augmented Virtual
Images 130
Two-Handed Manipulation of a Virtual Suture 132
SUMMARY 133
Chapter 5 ▪ Markerless traCking: nOn-gaussian apprOaCh 137
REVIEW OF PARTICLE FILTER 138
Example Implementations of Particle Filter 140
MODIFICATION TO THE RESAMPLING STRATEGY 145
HYBRID APPROACH: INTEGRATING THE PF AND AGMM 146
EXPERIMENTAL STUDIES 149
SUMMARY 153
Chapter 6 ▪ regiOn-based traCking 155
FEATURE DESCRIPTION AND EXTRACTION 156
SIFT Description and Extraction 156
SURF Description and Extraction 160
ORB Description and Extraction 162
MEASURING FEATURE MATCHING 164
viii ◾ Contents
ESTIMATION OF PLANAR HOMOGRAPHY 167
EXPERIMENTAL STUDIES 171
Testing Results of General Image Data 172
SUMMARY 183
REFERENCES, 191
APPENDIX I, 201
APPENDIX II, 205
APPENDIX III, 217
APPENDIX IV, 221
INDEX, 227
Summary
Conventional minimally invasive surgery, which includes lapa-
roscopic surgery, has become commonplace in repairing or replacing
tissues and organs within the cavity of the body. This is done by making
small incisions, through which an endoscope and long surgical tools are
fed in order to reach the inside of the body cavity. By viewing images of the
surgical scene on a monitor, the surgeon is able to perform various surgi-
cal procedures. The basis of this conventional surgical procedure, which
has remained unchanged, is being practiced all over the globe, from the
richest nations to the less privileged ones. The procedure is generally per-
formed by one main surgeon and an assistant surgeon, who is responsible
for holding and navigating the endoscope. To master this type of surgical
procedure one needs to have a certain degree of hand–eye coordination
and dexterity. This level of expertise can usually be gained through vari-
ous virtual or physical training environments. Over the past few years,
various degrees of automation have also been introduced in order to assist
the surgeon in performing such operations. These can range from a full
telerobotic setup to the more conventional designs of surgeon–computer
interfaces. The key aspects of developing automation can range from the
capability of visually tracking the surgical tools during the operation or
training period to tracking a specific landmark located on the surgical
scene. These tracking capabilities are particularly important for conven-
tional surgical cases in which economic factors prohibit health provid-
ers from deploying telerobotic surgical systems. As such, any automation
during the surgical procedure must be done through the image processing
of the surgical scene obtained through an endoscope, which is the theme
of this book.
The purpose of this book is to introduce the various tools and meth-
odologies that can be utilized to enhance a conventional surgical setup
with some degree of automation. The main focus of this book is on the
ix
