The IDA Pro book: The unofficial guide to the world's most popular disassembler

676 Pages·2011·6.14 MB·English

by  Chris Eagle

E 2
D N
I
T D
I
J
IIDDAA PPRROO EBP MP TT HH EE ON
DDEE--OOBBFFUUSSCCAATTEEDD SUB
2ND EDITION
II DD AA PP RR OO
J
E M TT
BP P
HH
B
U
S
No source code? No problem. With IDA Pro, the inter- • Use code graphing to quickly make sense of cross- EE BB OO OO KK
active disassembler, you live in a source code–optional references and function calls
world. IDA can automatically analyze the millions of
opcodes that make up an executable and present you • Extend IDA to support new processors and filetypes II
using the SDK
with a disassembly. But at that point, your work is just DD T H E U N O F F I C I A L G U I D E T O T H E
beginning. With The IDA Pro Book, you’ll learn how • Explore popular plug-ins that make writing IDA scripts
W O R L D ’ S M O S T P O P U L A R D I S A S S E M B L E R
to turn that mountain of mnemonics into something you easier, allow collaborative reverse engineering, and AA
can actually use. much more
Hailed by the creator of IDA Pro as “profound, compre- • Use IDA’s built-in debugger to tackle hostile and
C H R I S E A G L E
PP
hensive, and accurate,” the second edition of The IDA obfuscated code
Pro Book covers everything from the very first steps to
advanced automation techniques. You’ll find complete Whether you’re analyzing malware, conducting vulnerabil- RR
ity research, or reverse engineering software, a mastery “I wholeheartedly recommend The
coverage of IDA’s new Qt-based user interface, as
of IDA Pro is crucial to your success. Take your skills to the IDA Pro Book to all IDA Pro users.”
well as increased coverage of the IDA debugger, the OO
next level with this 2nd edition of The IDA Pro Book. —Ilfak Guilfanov,
Bochs debugger, and IDA scripting (especially using
creator of IDA Pro
IDAPython). But because humans are still smarter than ABOUT THE AUTHOR
computers, you’ll even learn how to use IDA’s latest BB
Chris Eagle is a Senior Lecturer of Computer Science
interactive and scriptable interfaces to your advantage.
at the Naval Postgraduate School in Monterey, CA.
Save time and effort as you learn to:
He is the author of many IDA plug-ins and co-author of OO
• Navigate, comment, and modify disassembly Gray Hat Hacking (McGraw-Hill), and he has spoken
• Identify known library routines, so you can focus your at numerous security conferences, including Blackhat, OO
Defcon, Toorcon, and Shmoocon.
analysis on other areas of the code
KK
J
M
THE FINEST IN GEEK ENTERTAINMENT™
www.nostarch.com E
E
A B
G P
$69.95($79.95CDN) L P
This b o o k u“sIe sL aI Ela yF-LflAaTt .b”inding that won’t snap shut. SOFTWARE DEVELOPMPROGRAMMING/SHELVE IN: E B
ENT U
S

PRAISE FOR THE FIRST EDITION OF THE IDA PRO BOOK
“I wholeheartedly recommend The IDA Pro Book to all IDA Pro users.”
—ILFAK GUILFANOV, CREATOR OF IDA PRO
“A very concise, well laid out book. . . . The step by step examples, and much
needed detail of all aspects of IDA alone make this book a good choice.”
—CODY PIERCE, TIPPINGPOINT DVLABS
“Chris Eagle is clearly an excellent educator, as he makes the sometimes very
dense and technically involved material easy to read and understand and also
chooses his examples well.”
—DINO DAI ZOVI, TRAIL OF BITS BLOG
“Provides a significantly better understanding not of just IDA Pro itself, but
of the entire RE process.”
—RYAN LINN, THE ETHICAL HACKER NETWORK
“This book has no fluff or filler, it’s solid information!”
—ERIC HULSE, CARNAL0WNAGE BLOG
“The densest, most accurate, and, by far, the best IDA Pro book ever
released.”
—PIERRE VANDEVENNE, OWNER AND CEO OF DATARESCUE SA
“I highly recommend this book to anyone, from the person looking to begin
using IDA Pro to the seasoned veteran.”
—DUSTIN D. TRAMMELL, SECURITY RESEARCHER
“This book does definitely get a strong buy recommendation from me. It’s
well written and it covers IDA Pro more comprehensively than any other
written document I am aware of (including the actual IDA Pro Manual).”
—SEBASTIAN PORST, SENIOR SOFTWARE SECURITY ENGINEER, MICROSOFT
“Whether you need to solve a tough runtime defect or examine your
application security from the inside out, IDA Pro is a great tool and this book
is THE guide for coming up to speed.”
—JOE STAGNER, PROGRAM MANAGER, MICROSOFT

THE IDA PRO BOOK
2 N D E D I T I O N
The Unofficial Guide to the
World’s Most Popular
Disassembler
by Chris Eagle
San Francisco
THE IDA PRO BOOK, 2ND EDITION. Copyright © 2011 by Chris Eagle.
All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or
mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior
written permission of the copyright owner and the publisher.
Printed in Canada
15 14 13 12 11 1 2 3 4 5 6 7 8 9
ISBN-10: 1-59327-289-8
ISBN-13: 978-1-59327-289-0
Publisher: William Pollock
Production Editor: Alison Law
Cover and Interior Design: Octopod Studios
Developmental Editor: Tyler Ortman
Technical Reviewer: Tim Vidas
Copyeditor: Linda Recktenwald
Compositor: Alison Law
Proofreader: Paula L. Fleming
Indexer: BIM Indexing & Proofreading Services
For information on book distributors or translations, please contact No Starch Press, Inc. directly:
No Starch Press, Inc.
38 Ringold Street, San Francisco, CA 94103
phone: [phone]; fax: [phone]; [email protected]; www.nostarch.com
The Library of Congress has cataloged the first edition as follows:
Eagle, Chris.
The IDA Pro book : the unofficial guide to the world's most popular disassembler / Chris Eagle.
p. cm.
Includes bibliographical references and index.
ISBN-13: 978-1-59327-178-7
ISBN-10: 1-59327-178-6
1. IDA Pro (Electronic resource) 2. Disassemblers (Computer programs) 3. Debugging in computer science. I.
Title.
QA76.76.D57E245 2008
005.1'4--dc22
2008030632
No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and
company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark
symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the
benefit of the trademark owner, with no intention of infringement of the trademark.
The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been
taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any
person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the
information contained in it.
This book is dedicated to my mother.

B R I E F C O N T E N T S
Acknowledgments.........................................................................................................xix
Introduction..................................................................................................................xxi
PART I: INTRODUCTION TO IDA
Chapter 1: Introduction to Disassembly..............................................................................3
Chapter 2: Reversing and DisassemblyTools....................................................................15
Chapter 3: IDA Pro Background......................................................................................31
PART II: BASIC IDA USAGE
Chapter 4: Getting Started with IDA................................................................................43
Chapter 5: IDA Data Displays.........................................................................................59
Chapter 6: DisassemblyNavigation................................................................................79
Chapter 7: DisassemblyManipulation...........................................................................101
Chapter 8: Datatypes and DataStructures......................................................................127
Chapter 9: Cross-References andGraphing....................................................................167
Chapter 10: The Many Faces of IDA.............................................................................189
PART III: ADVANCED IDA USAGE
Chapter 11: Customizing IDA.......................................................................................201
Chapter 12: Library Recognition Using FLIRT Signatures...................................................211
Chapter 13: Extending IDA’s Knowledge.......................................................................227
Chapter 14: Patching Binaries and Other IDA Limitations.................................................237
PART IV: EXTENDING IDA’S CAPABILITIES
Chapter 15: IDA Scripting............................................................................................249
Chapter 16: The IDA Software Development Kit..............................................................285
Chapter 17: The IDA Plug-in Architecture.......................................................................315
Chapter 18: Binary Files and IDA LoaderModules..........................................................347
Chapter 19: IDA Processor Modules..............................................................................377
PART V: REAL-WORLDAPPLICATIONS
Chapter 20: Compiler Personalities...............................................................................415
Chapter 21: Obfuscated Code Analysis.........................................................................433
Chapter 22: Vulnerability Analysis................................................................................475
Chapter 23: Real-World IDA Plug-ins.............................................................................499
PART VI: THE IDA DEBUGGER
Chapter 24: The IDA Debugger....................................................................................513
Chapter 25: Disassembler/Debugger Integration............................................................539
Chapter 26: Additional Debugger Features....................................................................569
Appendix A: Using IDA Freeware 5.0...........................................................................581
Appendix B: IDC/SDK Cross-Reference..........................................................................585
Index.........................................................................................................................609
viii Brief Contents