Transparent System Introspection in Support of Analyzing Stealthy Malware

168 Pages·2016·1.43 MB·English

Transparent System Introspection
in Support of Analyzing Stealthy Malware
A Dissertation
Presented to
the faculty of the School of Engineering and Applied Science
University of Virginia
In partial fulfillment
of the requirements for the degree
Doctor of Philosophy Computer Engineering
by
Kevin Joseph Leach
December 2016

APPROVAL SHEET
The dissertation
is submitted in partial fulfillment of the requirements
for the degree of
Doctor of Philosophy
Kevin Joseph Leach, author
The dissertation has been read and approved by the examining committee:
Dr. Westley Weimer, Advisor
Dr. Joanne Dugan, Committee Chair
Dr. Marty Humphrey
Dr. Ronald D. Williams
Dr. Laura E. Barnes
Dr. Angelos Stavrou
Accepted for the School of Engineering and Applied Science:
Craig H. Benson, Dean, School of Engineering and Applied Science
December 2016

Abstract
The proliferation of malware has increased dramatically and seriously degraded the privacy
of users and the integrity of hosts. Millions of unique malware samples appear every year,
which has driven the development of a vast array of analysis tools. Malware analysis is
often performed with the assistance of virtualization or emulation for rapid deployment.
Malware samples are run in an instrumented virtual machine or analysis tool, and existing
introspection techniques help an analyst determine its behavior. Unfortunately, a growing
body of malware samples has begun employing anti-debugging, anti-virtualization, and anti-
emulation techniques to escape or otherwise subvert these analysis mechanisms.
These anti-analysis techniques often require measuring differences between the analysis en-
vironment and the native environment (e.g., executing more slowly in a debugger). We call
these measurable differences artifacts. Malware samples that use artifacts to exhibit stealthy
behavior have increased the effort required to analyze and understand each stealthy sam-
ple. Additionally, traditional automated techniques fail against such samples because they
produce measurable artifacts. We desire a transparent approach that produces no artifacts,
thereby admitting the analysis of stealthy malware. We refer to this challenge as the debug-
ging transparency problem. Solving this problem is thus concerned with reducing artifacts
or permitting reliable analysis in the presence of artifacts.
We present a system consisting of two approaches to address the debugging transparency
problem and then demonstrate how these components can apply to currently available com-
puter systems. We present two techniques capable of transparently acquiring snapshots of
memory and disk activity that can be used to analyze stealthy malware. First, we dis-
cuss a novel use of a custom Field-Programmable Gate Array that provides snapshots of
memory and disk activity with no measurable timing artifacts. Second, we present a novel
use of System Management Mode on x86 platforms that produces no functional artifacts at
the expense of producing timing artifacts. Finally, we present an approach to evaluating the
i
tradeoffspacethatexistsbetweenanalysistransparencyandthefidelityofintrospectiondata
provided by such an analysis system. Together, these approaches form a cohesive solution
to the debugging transparency problem that admits analyzing stealthy malware.
ii
Dedication
The graduate school experience has been a long endurance test. There are several people to
whom I owe a tremendous amount of gratitude.
First, to Wes Weimer, thank you for serving as my doctoral advisor. Thank you for taking
me as a student in spite of my coming from an entirely different field of research. Thank you
for your tireless support throughout the doctoral program, and thank you for your eternal
patience with me as a time-consuming student. Thank you for advising me even though so
many of my publications did not include you as a co-author. Thank you for your professional
and life advice, and thank you for improving my research acumen and writing skills. Most
of all, thank you for helping me figure out what I want to do with my life. Without your
help, I would not have been able to complete my doctoral degree.
To Angelos Stavrou, thank you for being my advisor while I completed my master’s degree.
Attending GMU was the best professional decision of my life, and you have been like a
coach for becoming a better security researcher and professional. Thank you so much for the
opportunities you have provided to me, and thank you for your timeless advice. I wish you
the best of success in your career, your company, and your family.
To Fengwei Zhang, thank you for the years of productive collaboration. We certainly seem
to be able to come up with research-worthy ideas. I am fortunate to be able to call you a
colleague and friend. I hope there will be years of collaboration to come, and I hope you
find all the success you deserve.
To Chad Spensky, thank you for setting me up with the internship at Lincoln Laboratory.
Thankyouforsecuringtwoyearsoffundingformydoctoralstudies—youhelpedmegainalot
of research freedom during my time at UVa. Thank you for being a friend and collaborator,
and thank you for cooking such good barbecue. I hope we can continue to collaborate in the
future.
iii
To Marty Humphrey, thank you for enduring my sardonic comments in your operating
systems class and in your office. I appreciate all of the general advice you have given me and
the conversations we have had during my time at UVa.
To Joanne Dugan, thank you for your unwavering determination in seeing that the computer
engineering program succeeds and that the computer engineering students are represented.
The computer engineering program has been great to me, both as an undergrad and a
doctoral student.
To Laura Barnes, thank you for tolerating me as a student when I first came to UVa as a
graduate student. Your research group is one of the reasons I was able to stick it through in
spite of a difficult first year.
To Ronald Williams, thank you for serving on my committee. Your question about caching
during my proposal turned out to be a significant concern and led to a refined system. It
certainly made reviewers happy on a subsequent journal submission.
To Natalie Edwards, thank you for your eternal patience and administrative help through-
out the doctoral process. Thank you especially for making sure I got paid! Travel and
reimbursement were made much easier due to your efforts.
To Nick Napoli, thank you for being a friend throughout this Iditarod through Hell they call
graduate school. I hope we can remain friends and work together in the future.
To Lin Gong, thank you for being my friend since my first year at UVa. Thank you for
introducing me to so many people in the department, and thank you for your confidence and
conversations over the years.
To Jon Dorn, Kevin Angstadt, Jamie Floyd, Nora Evans, Erin Griffiths, and the rest of
WRG, thank you for your help and support. Thank you for the laughs in the office, and
thank you for tolerating my loud mechanical keyboard. To Dorn and Kevin A., thank you
both for your help on the quadcopter demo while I was interning at GrammaTech. I hope
iv
you all have amazing success in whatever your future endeavors may be.
To Kate Highnam, my former undergraduate workhorse, thank you for your diligence and
hard work. I wish you the best of luck.
I am extremely fortunate to have loving and supportive family. To my parents, Richard
and Linda, thank you for your love and all your advice through the years. Thank you for
your financial support, and thank you for putting up with me. Most of all, thank you for
encouraging me to complete this doctoral degree. I was hesitant to complete the degree early
on, but I am so glad I have seen it through to the end.
To my brother, Eric, thank you for helping me print large volumes of text. Thank you for
your patience with me, and thank you for including me in the Fantasy Football league.
To Yu Huang, my wonderful fiancée and the love of my life, thank you for your love and
support. Thank you for tolerating me, and thank you for being so patient and good-natured.
Iamluckytohaveyouinmylife. Mostimportantly,thankyouforhelpingmefindsomewhat-
related Chinese quotes to place in my dissertation chapters. I can’t wait for what the future
holds for us. 我爱你。
Most of all, thank you, the reader, for taking the time to read my dissertation. Thank you
for contributing to scientific research to bolster human knowledge. I hope you find this work
informative.
v
vi