The evolution of an x86 virtual machine monitor

16 Pages·2010·0.28 MB·English

The Evolution of an x86 Virtual Machine Monitor
Ole Agesen, Alex Garthwaite, Jeffrey Sheldon, Pratap Subrahmanyam
{agesen, alextg, jeffshel, pratap}@vmware.com
ABSTRACT While trap-and-emulate virtualization on mainframes was
TwelveyearshavepassedsinceVMwareengineersfirstvirtu- wellunderstoodatthetime,itwas–unsurprisingly–nota
alizedthex86architecture. Thistechnologicalbreakthrough design goal for the 4004, whose first application was in fact
kickedoffatransformationofanentireindustry,andvirtu- a Busicom calculator [7]. However, as x86 CPUs expanded
alizationisnow(onceagain)athrivingbusinesswithawide their reach, the case for virtualization gradually emerged.
range of solutions being deployed, developed and proposed. There was just one problem, seemingly fundamental: the
But at the base of it all, the fundamental quest is still the generally accepted wisdom held that x86 virtualization was
same: running virtual machines as well as we possibly can impossible, or at best merely impractical, due to inherited
on top of a virtual machine monitor. architecturalquirksandarequirementtoretaincompatibil-
ity with legacy code.
We review how the x86 architecture was originally virtual-
ized in the days of the Pentium II (1998), and follow the This impasse was broken twelve years ago when VMware
evolution of the virtual machine monitor forward through engineers first virtualized the x86 architecture entirely in
the introduction of virtual SMP, 64 bit (x64), and hard- software. By basing the virtual machine monitor (VMM)
ware support for virtualization to finish with a contempo- on binary translation, the architectural features that pre-
rary challenge, nested virtualization. vented trap-and-emulate virtualization of the x86 architec-
ture could be handled precisely and efficiently [1]. This
CategoriesandSubjectDescriptors technological breakthrough kicked off a transformation of
an entire industry: five years ago, Intel started shipping
C.0 [General]: Hardware/software interface; C.4 [Perfor-
CPUs with hardware support for instruction set virtualiza-
mance of systems]: Performance attributes; D.4.7 [Ope-
tion (VT-x), three years ago AMD started shipping CPUs
rating Systems]: Organizationand design
withhardwaresupportformemoryvirtualization(RVI),and
virtualization is now one of the most important ways to
GeneralTerms
utilize the increasing number of cores per socket delivered
Performance, Design by the continued run of Moore’s law. Today, virtualiza-
tion provides server consolidation, fault tolerance, security
Keywords andresourcemanagement,andfacilitatestest,development
anddeploymentacrossamultitudeofoperatingsystemsand
Virtualization, Virtual Machine Monitor, Dynamic Binary
versions. Moreover, virtualization is the foundation of the
Translation, x86, vSMP, VT, SVM, MMU, TLB, Nested
ongoing shift towards cloud computing.
Paging.
Still, and common to all these use cases, the fundamental
1. INTRODUCTION
questremainsthesame: runningvirtualmachines(VMs)as
In 1970, engineers at Intel designed the world’s first inte- well as we possibly can on top of a VMM.
gratedmicroprocessor,the4bit4004,with2124transistors
runningat100kHz[20]. 40yearslater,overclassicssuchas Previous publications have described specific technical as-
the 8008, 8080, 8088, 8086, 80286, 80386, 80486, and Pen- pectsofVMware’sx86VMM,includingdevicevirtualization
tium, this 4 bit architecture has evolved into a powerful 64 [17], binary translation [2], and a comparison between soft-
bit multicore CISC architecture (“x86”), with implementa- wareandhardwareapproaches[1]. Thepresentpapertakes
tions containing 1+ billion transistors running at 3+ GHz. a historical approach, looking back over the past decade in
Amazingly, after this fantastic evolution, modern x86 pro- approximate chronological order. We first distinguish be-
cessors still retain the ability to run ancient code directly. tween hypervisor and VMM to establish a context for the
remainderofthepaper(Section2). Thenwegobackto1998
As the x86 architecture evolved, it came to support a vast toreviewhowthex86architecturewasvirtualizedusingbi-
diversity of operating systems and applications. Its use ex- narytranslationinthePentiumIIera(Section3). Ournext
panded into almost every segment of the computing indus- stop is 2003, where we introduce virtual SMP (Section 4).
try, from embedded systems and handheld devices to per- By 2004 the new feature is x64, AMD’s 64 bit extension
sonalcomputersandservers,andfrominformationprocess- of the x86 architecture (Section 5). A year later, in 2005,
ing to web services to high performance computing.
3
hardware support for virtualization emerged with further
support arriving in 2007-8 (Section 6). Finally, we review Linux OS/2 Win 7 BSD
acontemporarychallenge,nestedvirtualization(Section7),
...
briefly list other topics that space limitations do not per-
mit us to discuss in detail (Section 8), and then offer our VMM VMM VMM VMM
concludingremarks (Section 9).
2. CONTEXT vmkernel
Before describing details of the x86 VMM, let us differ-
entiate between hypervisor and VMM. Existing literature
mostlytreatsthesetermsassynonyms,butforthepurposes Figure 1: The ESX hypervisor: one vmkernel per host,
ofthispaperitisbeneficialtoascribeanarrowerroletothe and one VMM per virtual machine.
VMM. In our vocabulary, a VMM is an entity specifically
responsible for virtualizing a given architecture, including
attempts to read or modify privileged state, the processor
the instruction set, memory, interrupts, and basic I/O op-
generates a trap that transfers control to the VMM. The
erations. A hypervisor combines an operating system (OS)
VMMthenemulatestheinstructionusinganinterpreterand
withaVMM.TheOSmaybeageneral-purposeone,suchas
resumesdirectexecutionoftheguestatthenextinstruction.
Linux or Windows, or it may be one developed specifically
for the purpose of running VMs.
The x86 architecture was considered non-virtualizable be-
causethe processorcouldnotbe configuredto generatethe
For example, VMware’s vSphere ESX hypervisor is com-
requiredtraps1. Oneexampleisthe popfinstructionwhich
prised of the vmkernel and a VMM. The vmkernel contains
loads a set of flags from the stack into the %eflags regis-
abootloader,anx86hardwareabstractionlayer,I/Ostacks
ter. Whenexecutedinprivilegedmode,popfloadsallflags,
for storage and networking, as well as memory and CPU
whichincludeamixofALUflags(zeroflag,carryflag,etc.)
schedulers. To run a VM, the vmkernel loads the VMM,
and system flags (interrupt flag, I/O privilege level, etc.).
whichencapsulatesthedetailsofvirtualizingthex86archi-
When executed in user mode, system flags remain unmodi-
tecture, including all 16 and 32 bit legacy modes as well
fiedregardlessofthecontentsofthestack. Instructionslike
as 64 bit long mode. The VM executes directly on top of
popfthwartanyattempttobuildatrap-and-emulateVMM
the VMM, touching the hypervisor only through the VMM
for x86. If a deprivileged guest kernel attempts to clear the
surface area.
interruptflagusingpopf,notrapisgeneratedandtheVMM
hasnowayofknowingitshouldnotdeliverinterruptstothe
VMware’s products create a separate instance of the VMM
guest.
for each running VM (“guest”) rather than using a single
VMM to run all VMs on the“host”; see Figure 1. This ap-
One could certainly avoid the reliance on traps by running
proachsimplifiestheoverallsystembyprovidingaclearsep-
the guest on an interpreter, but this introduces too much
aration of concerns: the vmkernel manages host-wide tasks
overhead to allow virtual machines to be used widely. Bi-
suchasscheduling,whileVMMsmanageper-VMtasks. More-
nary translation (BT), however, offers the ability to inter-
over, it leads to a natural separation of mechanism, mostly
cept any guest instruction and modify its execution with
implemented in the VMM, and policy, mostly implemented
justasmuchflexibilityasfullinterpretation,butwithmuch
inthevmkernel,usingknowledgeofalltheVMsrunningon
lower overheads. To further improve performance, BT can
the host. Finally, with multi-tenancy, where separate cus-
be limited to guest kernel mode code. Guest user mode
tomers buy VM execution time on the same host, the use
code can run directly on the CPU in“direct execution,”as
of a separate VMM for each VM greatly simplifies reason-
wouldbedoneinatrap-and-emulateVMM,sinceitalready
ingaboutisolation,althoughitdoesnotenforceisolationas
expects to run with user mode semantics such as ignoring
the VMM is a privileged program. (Resource management,
attemptedchangestotheinterruptflag. CombiningBTand
and quality of service properties still remain to be satisfied
direct execution limits translator overheads to the time the
globally by the vmkernel.)
guest spends running kernel code, which is typically a mi-
nority of total execution time.
While we have described this architecture in the context of
VMware’s ESX hypervisor, the same general idea extends
The following sections provide a more detailed description
to our hosted products such as VMware Workstation and
of how a BT-based VMM runs a guest. Section 3.1 de-
Fusion: here, the VMM runs alongside the host OS, again
scribes a simple, yet highly efficient dynamic translator for
with one VMM instanceper virtualmachine. In fact, other
unrestricted x86 code. Sections 3.2 and 3.3 show how to
than a“shim”platform layer, the same VMM is used in all
usepagingandsegmentationtoestablishanenvironmentin
of VMware’s products.
which the translated code can execute. Finally, Section 3.4
outlines an adaptive feedback-based optimization that cus-
3. VIRTUALIZING32-BIT x86
In 1998, the x86 architecture had no hardware support for 1The x86 architecture distinguishes between faults, which abort
virtualizationandwasgenerallyconsideredtobeunvirtual- executionofaninstructionpriortocommittinganystatechange,
and traps, which happen after an instruction completes. Tech-
izable[15]. Atthattimevirtualmachinesweretypicallyrun
nically, an x86 VMM would need to use faults. However, for
usinganapproachknownastrap-and-emulate[13,14]. Ina
consistency with classical literature, we prefer the term “trap-
trap-and-emulate style VMM, the guest code runs directly and-emulate,”eventhoughtheprecisex86termshouldbe“fault-
on the CPU, but with reduced privilege. When the guest and-emulate.”
4
tomizes translationsto observed guest behavior. jmp doTest
spin: sti ; enable interrupts
3.1 BinaryTranslation pause ; yield hardware thread
cli ; disable interrupts
TobuildaVMMthatcanruneverypossibleguestrequires doTest: mov %ebx,%eax
a binary translator that can handle the entire x86 instruc- lock ; If %eax==(%edx) write
tion set. The translator provides the ability to intercept cmpxchg %eax,%ecx,(%edx); %(edx) = %ecx else
preciselythevirtualizationsensitiveinstructionswithoutre- ; %eax = (%edx)
test %eax,%eax ; Set flags from %eax
quiring trap semantics. Since we cannot patch instructions
jnz spin ; Jump if not zero
in the guest’s own image (such patching would be visible
done: pop %ebx
to the guest), we must translate all guest kernel code and ret
alwaysexecutefromaguest-invisibletranslationcache. For-
tunately,ourinputandoutputarebothx86,sotheinstruc-
tions that do not need interception can be translated with We invoked this code in a virtual machine, logging all code
little or no modification. translated. Theabovecodeisnottheinputtothetranslator;
rather, its binary (“hex”) representation is input:
Our translator has these properties:
53 89 c2 fa b9 01 00 00 00 31 db ...
• Binary. Input is binary x86 code, not source code.
• Dynamic. Translationhappensatruntime,interleaved
The translator reads the guest’s memory at the address in-
with execution of the generated code.
dicatedbytheguestprogramcounter(%eip),classifyingthe
• On demand. Code is translated only when it is about bytes as prefixes, opcodes or operands to produce interme-
toexecute. Thislazinessside-stepstheproblemofdis- diaterepresentation(IR)objects. EachIRobjectrepresents
tinguishingcode from data. one guest instruction.
• System level. The translator makes no assumptions The translator accumulates IR objects into a translation
abouttheguestcode. Rulesaresetbythex86ISA,not unit (TU), stopping at 12 instructions or a terminating in-
byahigher-levelABI.Incontrast,anapplication-level struction (usually control flow). The fixed-size cap allows
translator like Dynamo [4] might assume that“return stack allocation of all data structures without risking over-
addresses are always produced by calls” to generate flow; in practice the cap is rarely reached since control flow
faster code. The VMM does not: it must run a buffer tends to terminate TUs sooner. Thus, in the common case
overflow that clobbers a return address precisely as it a TU is a basic block. The first TU in our example is:
wouldhaverunnatively(producingthesamehexadec-
imal numbers in the resulting error message).
push %ebx
• Subsetting. The translator’s input is the full x86 in-
mov %eax,%edx
structionset,includingallprivilegedinstructions;out- cli
put is a subset (mostly user-mode instructions). mov $1,%ecx
xor %ebx,%ebx
jmp doTest
To explain how the translator operates, we work through a
shortexample,aspinlockfunctionthatacquiresalockand
returns to the caller with interrupts disabled. (Such “irq Translatingfromx86tox86subset,mostcodecanbetrans-
locks”may protect kernel data structures that are accessed lated IDENT (for“identically”). The push, movs, and xor
both from normal and interrupt handling contexts.) all fall in this category.
Sincecliisaprivilegedinstruction,itmustbehandledspe-
void SP_LockIRQ(SPLock *lock) { cially by the VMM. Unlike popf, cli can generate a trap,
DisableInterrupts(); but it is more efficient to avoid the trap by translating it
while (CompareExchange(lock, 0, 1) != 0) { non-identically. The cli translation must clear the virtual
EnableInterrupts();
interrupt flag, which exists as part of an in-memory image
Pause();
DisableInterrupts(); of the guest’s virtual CPU (VCPU), so we use an and in-
} struction:
}
and $0xfd,%gs:vcpu.flags
We compiled the C code into this 32-bit binary:
Sincethetranslatordoesnotpreservecodelayout,allcontrol
push %ebx ; callee saved
flow instructionsmustuse non-IDENTtranslations. Inthis
mov %eax,%edx ; %edx = %eax = lock
examplethejmpbecomesatranslator-invokingcontinuation
cli ; disable interrupts
mov $1,%ecx ; %ecx = 1 (indicated by square brackets). Putting this together, the
xor %ebx,%ebx ; %ebx = 0 resulting translation from the first TU of our example is:
5
push %ebx mov %ecx,%gs:scratchECX
mov %eax,%edx pop %eax
and $0xfd,%gs:vcpu.flags movzx %al,%ecx
mov $1,%ecx jmp %gs:rtc(4*%ecx)
xor %ebx,%ebx
jmp [doTest]
Above, there are four CCFs with the leading instruction
in each one marked with an asterisk. The continuation to
Each translator invocation consumes one TU and produces the spin label remains untranslated as it has not executed
onecompiledcodefragment(CCF).AlthoughweshowCCFs yet. The code that was executed now sits in a straight line
in textual form with labels like vcpu.flags, in reality the without jumping about as the original code did.
translator produces binary code directly.
ThelastCCFaboveterminateswitha“launch”sequencefor
AfterproducingtheaboveCCF,theVMMwillexecutethe areturntranslation,thedetailsofwhichhavebeendescribed
codewhichendswithacalltothetranslatortoproducethe previously [2].
translationfordoTest. ThissecondTUisallIDENTexcept
forthefinalconditionaljnzbranchforwhichthetranslator Forabiggerexamplethanthespinlock,butneverthelessone
emits two continuations(one for each successor): thatrunsinexactlythesamemanner,bootingWindowsXP
Professional and then immediately shutting it down trans-
lates933,44432-bitTUsand28,33916-bitTUs. Whilethis
jnz [spin]
may seem like a lot, translating each unit takes just 3 mi-
jmp [done]
croseconds for a total translation time of about 3 seconds.
Againstabackgroundofaoneminuteboot/halt,andkeep-
To speed up inter-CCF transfers, our translator, like pre- ing in mind that a boot workload has an unusually high
vious ones [9], employs a“chaining”optimization, allowing proportion of cold code, the cost of running the translator
oneCCFtojumpdirectlytoanotherwithoutcallingoutof is acceptable.
the translation cache (TC). These chaining jumps replace
thecontinuationjumps,whichthereforeare“executeonce.” Thetranslatordoesnotattemptto“improve”thetranslated
Moreover, it is often possible to elide chaining jumps and code. We assume that if guest code is performance critical,
fall through from one CCF into the next. the OS developers have optimized it and a simple binary
translator would find few remaining opportunities. Thus,
For conditionalbranches,at most one of the two successors instead of applying deep analysis to support manipulation
can use fall through. The other must remain in the trans- of guest code, we disturb it minimally.
lated code as a conditional branch, initially invoking the
continuation, but, once the translated target is produced, Most virtual registers bind to their physical counterparts
redirected to this target. (Sometimes, to avoid code dupli- during execution of TC code to facilitate IDENT transla-
cation,nosuccessorcanusefall-through,sothefinaltransla- tion. Oneexceptionisthesegmentregister%gs. Itprovides
tionusesajcc/jmppairofinstructionstoconnecttoeachof an escape into VMM-level data structures; see Section 3.3.
the successors.) Since translation and execution interleave, The ret translation above uses %gs overrides to spill %eax
the first of the two continuations to execute is most likely and %ecx into VMM memory so that they can be used as
to receive the beneficial fall-through treatment. If the first workingregistersinthetranslationofret. Later,ofcourse,
and subsequent executions follow similar paths, this tends the guest’s %eax and %ecx values must be reloaded into the
to straighten code for good i-cache performance. In effect, hardware registers.
the translator builds execution traces in the TC, even as it
works through guest code in smaller TU chunks. Aswithregisters,thetranslatorbindsguestALU-flags(CF,
PF, AF, ZF, SF, OF) to their physical counterparts. Since
This interleaving of translation and execution continues for many x86 ALU instructions modify flags, nontrivial trans-
as long as the guest runs kernel code, with a decreasing lationsoftenmustsaveandrestoreguestflagsaroundflags-
proportion of translation as the TC gradually captures the clobbering operations. For example, this applies to the cli
guest’s working set. For the spin lock example the transla- translation,wheretheuseofandclobbersguestflags. How-
tion, after one spin-free acquisition, results in this code in ever, in the above example, the translator avoided flags
the TC: save/restore code by looking ahead to see that the guest
soon will execute an xor, which (re)defines all flags. To en-
surethateventheguest’sinterrupthandlerhasaconsistent
* push %ebx ; IDENT view of flags, the VMM defers virtual interrupts until the
mov %eax,%edx xor that terminates the flags-optimizedregion.
and $0xfd,%gs:vcpu.flags ; PRIV
mov $1,%ecx ; IDENT
3.2 VirtualizedMemory: ShadowPageTables
xor %ebx,%ebx
* mov %ebx,%eax ; IDENT The x86 architecture has supported virtual memory since
lock the80386withanMMUconsistingofaTLBandahardware
cmpxchg %eax,%ecx,(%edx)
pagetablewalker. ThewalkerfillstheTLBbytraversinghi-
test %eax,%eax
erarchicalpagetablesinphysicalmemory. Originally,these
jnz [spin] ; JCC
* pop %ebx ; IDENT pagetablesweretwolevelsdeepbutwereextendedtothree
* mov %eax,%gs:scratchEAX ; RET_LAUNCH and later four levels (see Section 5). The walker may spec-
6
ulateaheadandcachemappingswheneveritpleases,andis has been used to create a shadow page table entry. This
allowed to keep those mappings in the TLB until software is solved with a general purpose mechanism in the VMM
explicitlyflushestheTLBentrywithaninvlpginstruction known as traces, which mediate access to guest memory by
or by switching contexts (assigning to the page table root the guest or by the VMM. Traces provide clients with no-
control register %cr3). Additionally, the TLB is allowed to tification upon access to pages of interest. The MMU uses
evict entries unpredictably,at any time. traces for shadow page table consistency, but it is also par-
tially responsible for implementing traces. A traced access
The VMM’s MMU serves two purposes. It maintains isola- may be detected when it results in a page fault caused by
tion of the VMM by ensuring that the guest cannot access anattemptedaccess,whetherbyguestorVMM.Thepage-
memory belonging to the hypervisor, the host, or another fault handler then notifies the trace clients before resuming
VM. It also ensures that the virtual address space visible execution. Alternatively, traces may be“fired”on memory-
to the guest reflects what the guest expressed via its in- accessesthroughadaptivetechniqueslikethosedescribedin
memory page tables. These concerns can be viewed as two Section 3.4, or through explicit checks in the VMM itself.
separatelayersofindirectionstackedontopofoneanother:
(1) the guest creates page tables to describe how to map Withtraces,itispossibletoavoidthrowingawaytheshadow
guest virtual addresses (gVAs) to guest physical addresses pagetablewhenswitchingtoanewaddressspace. Instead,
(gPAs), and (2) the VMM creates mappings from gPAs to the MMU can selectively invalidate shadow page table en-
host physical addresses (hPAs). trieswhentheyaremodified. Eventhen,thecostofhidden
faults is one of the top virtualizationoverheads.
Toprovidenative-speedmemoryaccessfordirectexecution
and IDENT translations (see Section 3.1), the composite To further reduce the number of hidden faults, the MMU
mapping from gVA to hPA ultimately must reside in the can combine the trace processing and the (re)validation of
hardware TLB. This can be achieved by pointing %cr3 at a the shadow page table into a single step, using a technique
“shadow page table”that stores the composite mappings. called“eager validate.” When the MMU receives a notifi-
cation that the guest has modified a page table, instead of
The cost paid by the first access is, however, considerably invalidating the shadow, the MMU immediately populates
morethanatraditionalTLBmisscost. Tounderstandthis, it with an updated entry. Now the guest can access the
we must examine what happens when the guest accesses memory addresses mapped by the page table, without in-
a particular gVA. First, the memory access causes a page curring a hidden fault. Although physical CPUs generally
fault (several hundred cycles in the circa 2002 processors). would continue using the old mapping until the next TLB
Then,theVMMwalkstheguest’spagetablesinsoftwareto flush,theyareallowedtoevictthemappingandrefetchfrom
determine the gPA backing that gVA (again costing a few page tables any time. So a VCPU using eager validate ap-
hundred cycles). Next, the VMM determines the hPA that pears to have a TLB that always drops and reloads entries
backsthatgPA.Often,thisstepisfast,butuponfirsttouch immediately when the in-memory page table is modified.
it requires the host OS to allocate a backing page. Finally,
the VMM allocates a shadow page table for the mapping Eager validate has its own costs since it requires receiving
andwiresitintotheshadowpagetabletree. Thepagefault a notification whenever a page changes. Page table entries
andthesubsequentshadowpagetableupdateareanalogous might be modified, but never used, or a piece of memory
to a normal TLB fill in that they are invisible to the guest, thatwasonceapagetablemightbeusedforotherpurposes.
sotheyhavebeencalled“hiddenpagefaults.”Hiddenfaults Either way, there is a pressure on the MMU to remove the
can have a 1000-fold increase in cost over a TLB fill, but trace if it is not helping to reduce page faults.
tendtobelessfrequentduetohighervirtualTLBcapacity
(i.e., higher shadow page table capacity). This leads to a tension between three paths that all MMU
tuning revolves around:
Oncetheguesthasestablisheditsworkingsetintheshadow
page table, memory accesses run at native speed until the
guest switches to a different address space. TLB semantics • The hidden page fault that occurs when the guest ac-
on x86 require that context switches flush the TLB, so a cesses a page not mapped by the shadow page table.
naive MMU must throw away the shadow page table and
start over. We say such an MMU is“noncaching.” • Switchingfromoneaddressspacetoanother(onbehalf
of the guest) and making sure the new shadow page
Theonlywaytoavoidthecostofrebuildingtheshadowpage table space is in sync with the guest page table.
tableaftereachaddressspaceswitchistokeeparoundcopies
• Guest writes to page table entries that the VMM is
of multiple shadow page tables, one for each address space.
monitoring with a trace.
Here, the x86 semantics of implicit TLB flushes on context
switches gets in the way. If a guest changes a mapping in
its page tables after a shadow entry has been created, then
The overheads of these three operations all trade against
switches to a new address space, and then switches back
one another. For instance, it is possible to eliminate all
to the first address space, the guest can expect the new
costswhentheguestmodifiesapagetablebynottracingthe
mapping to be visible because of the implicit TLB flush.
page,butthisincursmorehiddenpagefaults. Alternatively,
hiddenfaultscanbeeliminatedbyeagerlyshadowingentire
To solve this problem, the MMU needs a way of detecting
address spaces, but this increases costs to switch address
when the guest modifies a primary page table entry which
spaces. Finally, it is easy to make context switches quick,
7
but only when the shadow page table is in sync with the Segments
guest’s page table. %ds (cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)
base
%fs (cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)(cid:0)(cid:0)(cid:2)(cid:2)
Findingtherightbalancebetweenthethreeisverydifficult (cid:0)(cid:2)(cid:0)(cid:2)(cid:0)(cid:2)(cid:0)(cid:2)(cid:0)(cid:2)(cid:0)(cid:2)
base
and varies from workload to workload.
guest vmm
Inadditiontothisthree-waycycletrade-off,theMMUmust
0x00000000 0xffc00000 0xffffffff
factorinshadowpagetablememoryusage. Someworkloads,
like Windows Remote Desktop Services, have working sets Figure 2: Segment-truncation protecting the upper
solargethattheyoverwhelmanyreasonablefixedallocation 4 MB containing the VMM address-space.
forshadowpagetables,causingshadowpagetableeviction,
i.e.,thrashing. Tocounterthisproblem,theMMUmustop-
timize shadow eviction, applying an LRU-like reclamation
into 32 bit processors, although modern operating systems
strategy. Moreover, the MMU must dynamically size the
have largely discontinued reliance on segmentation, prefer-
shadow page table cache according to measured needs, bal-
ring the additional flexibility offered by paging. Segmenta-
ancingthememoryconsumptionagainsttheutilityofother
tion gives us almost exactly what we need to enforce the
possible uses.
boundarybetweenguestandVMMwithinasharedaddress
space: a fast bounds check on memory accesses.
3.3 Segmentation
The guest expects to use the full feature set of the x86 ar- In more detail, the x86 architecture provides six segment
chitecture, including the entire virtual address space. This registers: one is used by instruction fetches (%cs), one for
presents a challenge to the VMM, which must simultane- stack accesses (%ss), and the remaining four for other data
ously satisfy the guest’s expectation as well as provide for accesses (%ds, %es, %fs, %gs). Each segment register con-
its own needs. In particular, the translator must keep the tains a base and a limit and will generate a fault if used
TC and auxiliary data structures mapped to permit execu- with an address greater than the limit. By configuring the
tion of translated code. For example, when running a 32 data segments to allow access only to the guest portion of
bitguestona32bitCPU,theguestandVMMmustsome- theaddressspace,butallowingthecodesegment%cstoac-
howsharethe4GBaddressspaceinamannerthatisboth cess the VMM space, we have achieved the code/data split
efficient and invisible to the guest. that a BT-based VMM needs to hide the TC. The VMM
candifferentiatebetweenguestdataaccessesandtranslator
Acommonwayofcontrollingaccessestomemoryisthrough accesses to runtime data the same way by restricting %ds,
the use of page permissions. Abstractly, the VMM may set %es and %fs to the guest range while reserving %gs for the
upmappingsintheshadowpagetablestodisallowguestac- VMM range; see Figure 2. When the translator emits in-
cesstotheVMMportionoftheaddressspace. Forexample, structions that should read or write VMM data, it includes
theVMMcouldusesupervisor-onlymappingsfortheVMM a %gs prefix. When translating a guest memory access, the
addressrangeandusermappingsforthe(usermodepartof translator makes sure to not use %gs. If a guest instruction
the) guest address range. This approach works well with a uses %gs, the translator must strip the prefix and replace
trap-and-emulatestyleVMM;itisalsotheapproachweuse it with another segment prefix, say %fs (setting up %fs ac-
when running guest user mode code in direct execution. cording to what the guest expected to have in %gs). Alter-
natively,thetranslatorcanemitcodetoperformthevirtual
However,page-basedseparationrequiresthattheguestand %gs-segmentation“insoftware”withoutrelyingonaphysical
the VMM run at different privilege levels so it does not segment register.
work well for running translated guest kernel code. When
running translated code, the TC itself must be accessible Using segmentation to isolate guest memory accesses from
(for execution), yet must remain invisible to guest data ac- theVMMsolvesoneproblem,buttheVMMmustalsoem-
cesses. Whileexecute-onlymappingscouldaccomplishthis, ulatetheguest’suseofthesamesegmentationfunctionality.
x86pagetableshavenosupportforexecute-onlymappings. Mostmodernoperatingsystemsmakelittleornouseofseg-
In addition, an execute-only approach would not apply to mentation. They generally define their segments to have
VMM-definedauxiliarydatastructuresthatmustbeacces- a base of zero and a maximal limit, a case we refer to as a
sible to translated code yet inaccessible when referencing “flat”segment. Sinceflatsegmentsarecommon,weplacethe
memory on behalf of the guest. VMM at the top of the address space so that flat segments
can be precisely“truncated”to prevent access to the VMM
Having ruled out paging as a way to protect the VMM whileallowingaccesstoallremainingvirtualaddresses. His-
fromguestmemoryreferencesintranslatedcode,whichap- torically,VMwarechosetoallocatethetop4MBofthevir-
proaches remain? One possibility is to use the BT infras- tual address space to the VMM. A non-flat guest segment
tructureitselftoinsertexplicitboundschecksintothetrans- maywraparoundthetopofthe4GBaddressspace. Trun-
lation of guest instructions. However, using bounds checks cationthen must cut away not just the part of the segment
on every guest memory access causes significantoverhead. that overlaps the VMM but also the wrap-around portion,
preventingdirectguestaccesstoaddressesthatarenotused
Fortunately, the x86 architecture has one more mechanism by the VMM.
that can be applied: segmentation. The segmentation sup-
portgoesbacktothe16bit80286processor,predatingeven Theuseofsegmentationtoseparateguestmemoryaccesses
paging. Over the years, segmentation was carried forward fromthoseintroducedbythetranslatorisacriticalcompo-
8
nentofVMware’sabilitytoimplementalowoverheadx86to instruction interpreter, avoiding the page fault and thereby
x86 translator. Guest memory accesses that do not overlap eliminating a large portion of the overhead.
withtheVMMincurnooverheadrelativetothatsamemem-
ory access running“natively”outside of a VM. The major It is possible to save more than just the page fault. Rather
difficultywiththisdesignhasbeentheperformanceclifffor than using a general-purpose interpreter that must decode
instructions that do overlap with the VMM. Such instruc- the guest instruction, the translator can pre-decode the in-
tionsincurafaultandmustbeemulatedwithoutthebenefit struction into an easy-to-interpret form and call an opti-
of the shadow page tables for getting to the corresponding mized interpreter.
guest memory. Fortunately, the VMM can often avoid such
faults by predicting if an instruction may generate memory Reducing page table modification costs has other indirect
accessesintheVMMrange(seeSection3.4),buteventhen benefits: we reduce the pressure on the MMU to drop (and
the instruction will typically execute one or two orders of reinstall) traces, in turn avoiding hidden page faults.
magnitude slower than a native memory access.
Optimized translations for instructions that touch special
This cliff leads to a strong pressure to minimize the fre- memoryareusefulbeyondpagetables: theycanavoidpage
quency that guest memory overlaps with the VMM’s ad- faultsforaccessestomemory-mappeddevicesbyidentifying
dress space. Because only the limit of the segment can be instructions likely to touch such devices. For certain device
adjusted,theVMMisconstrainedtothetopoftheaddress accesses,suchasthosetotheAPIC,theVMMgoesonestep
space. In practice, it has also been limited to use at most further.
4 MB of address space. Making it any larger is undesirable
as the VMM would then overlap with hot data structures TheAPICisamemorymappeddevicebuiltintoeverymod-
formanyWindowsguests. Theresultingscarcityofaddress ern x86 CPU to control the delivery of interrupts. One of
spacehasinfluencedthe designof almostallcomponentsof the device registers in the APIC memory range is the Task
the VMM. Small data structures were squeezed as tightly PriorityRegister(TPR)whichspecifiestheminimalpriority
as possible and large data structures had to be paged to foraninterrupttobedelivered. ManyversionsofWindows,
leave space for a sufficiently large TC to hold the guest’s startingwithNTanduptobutnotincluding2003SP2,ac-
code working set, even as guests grew over the years from cesstheTPRtensofthousandsoftimespersecond. Worse
Windows 95/98, NT, 2000, XP, and Vista to Windows 7. still, certain antivirus software can increase the access rate
by an order of magnitude. For example, running a com-
While the address space pressure at times was a source of pilation job on Windows XP with 2 VCPUs and antivirus
grief, over the years it created an environment that favored software generated more than 800,000 accesses to the TPR
minimality and cleanliness as overriding design principles. per second.
We believe that this, in turn, has given us a more reliable
and more efficient VMM. IncurringapagefaulteachtimeWindowsaccessestheTPR
results in unacceptable performance. Even replacing the
3.4 AdaptiveBT page fault with a call to the interpreter can be too costly,
giventhisaccessfrequency. However,byusingadaptiveBT,
The need to cheaply interpose on certain non-virtualizable
we can emit custom translations for instructions that con-
instructions motivated the use of BT for guest kernel code.
sistently access the APIC TPR. In ideal circumstances, the
But as the VMM matured, BT became increasingly useful
translatorcanprovethataninstructionwillalwaysaccessa
as a means of reducing additionalvirtualizationoverheads.
fixedvirtualaddressthatmapstotheAPIC.Bycooperating
withtheMMUtoreceivenotificationifthataddresswereto
Maintaining coherency of the shadow page tables requires
beremapped(theAPIClivesinthephysicaladdressspace),
trappingaccessestoguestpagetables,whichisaccomplished
thetranslatorcanemithighlyoptimizedtranslations. Inthe
by marking those pages as read-only in the shadow page
case of a guest write to the TPR, the best case translation
tables. Each time the guest writes to a page table natively
contains about half a dozen load/store and ALU instruc-
it is a simple write to memory, but when run in a VM it
tions to verify the pending interrupt priority and store the
can turn into a page fault, followed by emulation of the
newtaskpriorityvalue(inthecommoncase,nointerruptis
instruction,andfinallyanupdatetotheshadowpagetable.
raised). On a modern superscalar CPU, we measured this
translation at just 3 cycles, exceeding even the native per-
Fortunately operating systems typically modify page tables
formance of an APIC TPR write.
fromwithinblocksofcodewiththeexpresspurposeofmod-
ifying page tables. Thus, instructions that modify page ta-
Not all guest APIC accesses can be statically proven to ac-
blesalways(orusually)modifypagetablesandinstructions
cessafixedlocationinmemory;intheremainingcases,the
thatdootherthings(likeupdatingschedulerrecords)never
translator inserts checks to verify that the accessed address
modifypagetables. ThispermitstheVMMtotrackinstruc-
matches the predicted one. If the prediction fails (which is
tionsthatmodifypagetablesandfeedthatinformationback
extremely infrequent), we take a slow recovery path. If it
intothetranslator,allowingcreationofcustomtranslations
failsrepeatedly,wecanretranslate,usingthisnewinforma-
for instructionsthat are likely to modify page tables.
tion.
Sincethesetranslationsassumethatthepagetheyarewrit-
InadditiontothosefortheAPICandtheMMU,thetrans-
ing to is a page table (i.e., write-protected by the VMM),
lator also customizes accesses to segment descriptor pages,
there is no need to attempt to write to the page only to
to certain virtual network adapters,and, as mentioned pre-
receive a page fault. Instead they can jump directly to the
9
viously, to the VMM’s address range. quirementsmaychange. Intheremainderofthissection,we
discuss synchronizationmechanisms and the vSMP MMU.
4. SUPPORTINGMULTIPROCESSORVMs
In early 2002, VMware needed to grow beyond its base in 4.1 ThevSMPProgrammingModel
the desktop and developer focused workstation market. To Since the physical CPUs in an SMP system execute largely
capture server workloads, we considered virtualizing a rec- independently of each other, it is natural to model VCPUs
ognizedserverarchitecturesuchasItaniumorSPARC.Ulti- as independent threads of execution. These threads must
mately,wecontinuedtobetonx86andsetouttoimplement coordinatewhenaccessingshareddevicestate,whenupdat-
virtual SMP to have a story for larger workloads. inggPA-to-hPAmappings,andwhenhandlingtraceevents.
Thecoordinationmustpresentaconsistentviewtotheguest
This project required changes and innovations throughout while being fast and scalable.
the hypervisor: the vmkernel scheduler was challenged; the
virtual BIOS had to change; we needed to come up with a Initially,weusedPosix-likesynchronizationmechanisms,in-
threadingmodel,implementsynchronizationprimitivesthat cludingmutexes,semaphores,andconditionvariables. How-
wouldworkacrossmultiplecontexts,findanewdevicepro- ever,wesoonfoundthatinordertocontrolwhenandwhere
gramming model, generalize traces, understand cross-CPU VCPUshandledcertainevents,weneededadditionalhigher-
code modification, etc. It was all new, unknown and un- level mechanisms: crosscalls, actions, and stop.
certain; in short, it was a“bet the company”effort for our
start-up. In cases where this coordination requires synchronicity, the
VMM employs crosscalls. Crosscalls allow one VCPU to
Amazingly, by the end of that summer, we had managed request some service or action from other VCPUs. The ini-
to boot a 4-VCPU Windows 2000 VM, albeit over several tiating VCPU blocks until the request has been handledby
hours and making good use of checkpoints along the way all the callees.
to recover from crashes. The VM ran, but barely: perfmon
running in the guest showed four CPUs, but it was so slow To understand the motivation for crosscalls, first consider
thattheCPUswerepeggedatnearly100%loadsimplyser- tracesinavSMPVMM.Thesetofpagesthataretracedis
vicingtimerinterrupts! Inspiteofthisdismalperformance, dynamic. Adding a page to this set (installing a trace) re-
we were buoyed by the success, and confident that opti- quiresallVCPUstorespectthepresenceofthetracebefore
mizationwouldaddressthe performanceandscalingissues. theinstallercandependuponit. Anotheruseofcrosscallsis
With little hesitation, we sent a screenshot to everyone in the invalidation of gPA-to-hPA mappings in order to allow
the company trumpeting success. memorytobeshared,unshared,swappedorremapped[19].
From that early milestone it took a significant amount of To reason about crosscalls and their interaction with mu-
work to get anywhere near production quality. We had to texes, we statically rank crosscalls and mutexes based on
restructure much of the VMM. Most importantly, we reim- the kinds of resources they manipulate or guard. VCPUs
plemented traces in a vSMP-aware manner as a foundation can progress from one ranked operation to another only in
foranewcachingMMU.Wealsousedthenewtraceimple- order of increasing rank. Additionally, any VCPU block-
mentation to provide coherency of translated code, and to ing on a mutex or invoking a crosscall must serve same- or
shadow the descriptor tables so that direct execution could higher-rankedincomingcrosscalls. Thesesimpleconstraints,
be reenabled. Much of the performance tuning work in- checkablebyassertions,ensurethatdeadlocksdonotoccur.
volved wrestling with locks introduced into the system; not
surprisingly,oneofthemorechallenginglockswasourequiv- In cases where coordination between VCPUs only requires
alent of the address-space lock for the VMM. This perfor- some eventual processing on the part of other VCPUs (i.e.,
mance work continues to this day as physical systems are constituteanasynchronousrequest),theVMMemploysac-
deployed with ever greater numbers of cores and as these tions. Actions, once delivered to a VCPU, are guaranteed
systemscopewithscalethroughtheuseofNUMAarchitec- to be processed before the execution of the next guest in-
tures. struction,butthecallercannotmakeanyassumptionsabout
how long delivery will take. For example, we use actions to
Mere performance problems were only one of the kinds of remove traces. While we could have used a synchronous
challenges we met. We also had to learn how to address crosscall, an asynchronous action permits looser coupling
races in guests exacerbated by the timing characteristics of and therefore better scalability.
VMs. Examples were quickly found in a variety of early
Windows releases, Netware, Solaris, and some Linux dis- Ourfinalhigher-levelcoordinationmechanismisstop: bring
tributions. Our techniques included controlling the guest’s all VCPUs to a stand-still at base rank and at a virtual in-
perceptionoftime,andmorebizarretechniquessuchas“lock structionboundary. Having“stoppedtheworld,”theVMM
step”execution in which VCPUs go round-robin, executing can perform operations that are safe only in this controlled
one guest instruction at a time, to make it safely through environment. For example, we use a form of stop to check-
race-prone regions of guest code: after you, my friend! point a virtual machine’s state.
Howonestructuresanysystemtosupportconcurrencyhasa Over the years, we found these higher-level coordination
profoundimpactnotjustonperformanceandscalabilitybut mechanisms, and especially ranks, to serve our needs well.
alsoonhowtractablethesystemistoreasonaboutandwork However, they were no silver bullet: we still had to pro-
with, and how it evolves over long time periods where re- gramcarefullytoattainscalability,includingsometimesus-
10
ing lock-free algorithms. Finally,toreducethenumberofhiddenfaultsresultingfrom
remotelyproducedtraces,weaddedeagervalidationintothe
4.2 vSMPMMUVirtualization patch set processing.
Inx86SMPsystems,eachcorehasitsownpagewalkerand
The combined effect of these optimization steps gave us a
TLB.However,mostoperatingsystemsuseasharedpoolof
softwareMMUwithacceptableperformance,albeitonethat
memory for page tables. To bridge from shared page tables
we would continue to refine and tune over the years.
toper-coreTLBs,apreciselycoordinated“TLBshootdown”
approachmust be used to invalidate mappings.
5. VIRTUALIZING64 BITx86
Given the dichotomy between shared page tables and per- AMDstartedshippingx86compatible64bitOpteronCPUs
core TLBs, how should the virtual MMU be designed? On in 2003. Their architecture—called AMD64, x86-64, and
the one hand, shadow page tables can be seen as coun- later simply x64—addeda new 64 bit executionmode, long
terparts to primary page tables, suggesting that we share mode,whileretainingexisting16and32bitexecutionmodes
shadow page tables between VCPUs. On the other hand, for complete x86 backwards compatibility.
shadowpagetableshavesimilaritieswithTLBsinthatthey
cache mappings, suggesting use of per-VCPU shadow page Duringthoseyears,serverworkloadswereincreasinglypush-
table caches. ing against memory limitations. PAE mode extended the
physicaladdressspacefrom4GBto64GB,butdidnothing
For simplicity and to generally decouple VCPUs, we went to relieve virtual address space pressure in the increasingly
withper-VCPUshadowpagetablecaches. Reasoningabout “cramped”32bitenvironment: anyindividualprocesscould
nonsharedcachesseemedeasier,althoughweknewwewould stilladdressatmost4GBofvirtualmemoryatatime. Long
have to watch out for potential drawbacks including: (1) mode, however, provided 64 bit addressing (though imple-
increased memory usage when multiple VCPUs would all mentationscouldcapat48usablebits),therebyeliminating
shadow the same primary page table, and (2) more hidden addressspacepressurecompletely. Inlongmode,4-levelhi-
pagefaultsthanifVCPUscooperatetovalidatetheworking erarchical page tables map 48 bits of virtual addresses to,
set into shared shadows, especially when guest schedulers typically, 48 bits of physical addresses.
migrate processes from VCPU to VCPU.
Long mode also brought instruction set improvements, ex-
OurinitialvSMPMMUwasnoncaching. Wesimplyranin- tending the eight legacy x86 registers to 64 bits in much
dependentinstancesofthefamiliarshadowingalgorithmon the same way as the 80386 processor had widened the reg-
each VCPU. Each VCPU would, independently, take hid- isters from 16 to 32 bits almost two decades earlier. For
den faults, build up private shadow page tables, and fully example, the 16 bit %ax register had grown into the 32 bit
clear these shadow page tables on context switches. The %eaxin1985,andin2003itgrewtobecomethe64bit%rax
noncaching MMU allowed us to run and examine the first register. Moreover, long mode added an additional eight
set of SMP workloads. general-purpose registers to alleviate one of the most often
cited architecturalbottlenecks of the x86 architecture.
Forhigherperformance,weneededcachingtoallowshadow
page tables to survive context switches. And, as in the AMDalsotooktheopportunitytoremoveorsimplifysome
uniprocessorcase,wewouldneedtracestokeeptheshadow legacy features in long mode. Unfortunately, two of these
page tables in sync with the primary page tables. Our first “improvements” directly affected our ability to use BT to
caching vSMP MMU used the trace system directly. Each virtualize x64:
time a VCPU processed a hidden fault, it installed a write
traceonthepaththatthepagewalkertraversedinthepri-
• All segment registers but %fs and %gs were flattened
marypagetablestosatisfythehiddenfault. Thetracesub-
(basezero, maximallimit). For %fsand%gs,onlythe
system tracked which VCPUs had a trace on which pages,
base address functionality remained, i.e., limit checks
allowingforprecisenotificationtotheVCPUsthatwereaf-
were removed. The motivation was, probably, that
fectedbypagetablewrites. Asexpected,thecachingMMU
the base address functionality on two segment regis-
loweredthenumberofhiddenfaultsandgaveusasignificant
terssufficedforprovidingper-threadstoragewithseg-
performance boost.
ment registers. Unfortunately, without limit checks, a
BT-based VMM could no longer use segmentation to
Trace firing was the next optimization target. To fire an
protect itself from the guest.
MMU trace, we crosscalled all affected VCPUs. This syn-
chronicity was both expensive and unnecessary: TLB con- • The lahf and sahf instructions were removed from
sistency rules only require that page table updates are re- longmode. TheseinstructionsmoveALU-flagstoand
flected at the next TLB flush. Thus, instead of carrying from the 8 bit %ah, respectively. For a binary transla-
MMU trace events with synchronous crosscalls, we imple- tor,theyprovideafasterwaytosaveandrestoreflags
mented an inbound“patch set”queue for each VCPU. To thanthealternativeinstructions,pushfandpopf,that
notifyanotherVCPUof a traceevent, the event couldsim- move flags to and from the stack.
ply be added to its patch set. Each VCPU would process
thequeuedeventsinitspatchsetatthenextTLBflush(or
soonerifthefixed-sizedatastructurefilledup). Thisdecou- Lacking lahf/sahf would be an inconvenience, but lacking
pling of trace producer and consumer improved our MMU segmentlimitchecksmadebuildingahighperformanceBT-
performance substantially. based 64 bit VMM much harder. Fortunately, AMD saw
11
enough potential value in having a 64 bit virtualization so- into the vmm32 binary and once into the vmm64 binary.
lutionthattheyaddedsupportforsegmentlimitchecksand Initially, 100% of our source code was 32 bit, and 0% was
lahf/sahfin“Revision D”of the processor. common and 64 bit. By the time we shipped our first 64
bit capable product, VMware Workstation 5.5 in 2005, the
5.1 ThePeerModel breakdown was approximately 15% 32 bit specific source,
15% 64 bit specific source, and 70% common source. We
WiththenecessaryarchitecturalsupportsecuredfromAMD,
considerthisbreakdownasuccessfuloutcome: whilewehad
wesetouttovirtualizelongmodeinmuchthesamemanner
two binaries, each just a few hundred KBs, there was very
as we had previously virtualized the 16/32 bit modes. But,
little source code duplication. (It is not, however, the end-
along the way, we had to make a few intermediate stops.
stage; more on that later.)
We got an Opteron prototype system (in a large box with
As with the existing vmm32, the new vmm64 would use
rollerskate wheels). First, we installed a 32 bit host OS on
BT for privileged code, and direct execution for user code.
it,andverifiedthatourexisting32bitVMMcouldrunVMs
When running translated code, the VMM would use the
onit. Asexpected,thisstepwaseasy: thex64architecture
“reintroduced”(long mode) segment limit checks to protect
was indeed backwards compatible with 32 bit x86. Let us
denote this configuration 32. the VMM’s address space.
32
The first step was to write a 64 bit peer that could merely
Next, we installed a 64 bit host OS, and began pondering
reflect interrupts back to the host, while not even trying to
howtowritethe“worldswitchcode”thatwouldgetfroma
runtheguest. Thisrequiredustousetheworldswitchcode
64bithosttoa32bitVMrunningonour32bitVMM,i.e.,
32. We approached this challenge with some trepidation: from our earlier 3624 milestone if the host OS was 32 bit, or
64 a 64-to-64 bit version for a 64 bit host OS.
back in the 80286 days, CPUs could switch from old-style
8086realmodetonew-style80286protectedmode,butthey
To bring up the 64 bit peer, we repurposed our 32 switch
had no reliable way to get back! Might there be a similar 64
technology one more time, now wrapping it into a minimal
problemlurkinginthetransitionbetween32and64bitpro-
toy OS that would start in real mode, reach 32 bit pro-
tected mode? Fortunately, while this mode switch was del-
tectedmode,thenswitchtolongmodeandrunahandfulof
icate, involving taking the CPU all the way back to 16 bit
instructions before going back to 32 bit mode and shutting
real mode and running instructions on an identity-mapped
down. VirtualizingthisminimalOSkeptusbusyforweeks,
page (i.e., one with the same virtual and physical address),
forcingustodealwithlongmodepaging,descriptortables,
this all worked out. Soon, we had a 32 bit VM running on
control registers, and, of course, 64 bit BT in general.
our VMM with the host in 64 bit mode. The CPU happily
transitionedbetweenlongmodeandlegacymodethousands
Drawinguponexperiencefromour32bitVMM,weknewwe
of times per second.
wantedtohaveafull64bitinterpreterasourguaranteed-to-
be-complete execution mechanism. It would deal with the
Feelingconfidentfromthisexperience,wesetoutinearnest
to virtualize long mode, i.e., implement the 64 and 64 con- most contrived corner cases like a page fault on an access
32 64 straddling a page boundary in the middle of a task switch.
figurations. Since we already had a 16- and 32-bit capable
In other words: CISC fun. The interpreter would be our
VMM, we decided to write a new long-modecapable VMM
complete mechanism, giving us the confidence to write a
thatcoulddojustthat: virtualizex64longmode. Wewould
speed-focused binary translator. We would have one very,
usethis“vmm64”whentheguestwasrunninginlongmode
very complete path and one very fast path.
anduseourexistingVMM,nowrenamed“vmm32”whenthe
guest was running 16 or 32 bit code. This decision allowed
Dividing ourselves into two subteams, one team set out to
ustoreuseourexistingVMMlargelyunchangedwhilekeep-
writethetranslator,theothertowritetheinterpreter. The
inglongmodefunctionalityinanewbinary. Wehopedthis
next day (!), the translator team reported back: we have a
would improve performance and keep the task ahead of us
completetranslator. Indeed,itwascomplete;everytransla-
simpler. Soonwestartedusingtheterm“peers”todescribe
tion called into the interpreter with a decoded instruction:
the(vmm32,vmm64)pairthatwouldcooperatetoruna64
“please run this one.” Now pressure was on the interpreter
bit guest.
team. Soon, they got to the point where the minimal test
couldrun. Meanwhile,thetranslatorteamgraduallyshifted
Inadditiontopromisingcodereuse,thepeermodelassured
theburdenawayfromtheinterpreterbyadding“real”trans-
usthat32bitcodewouldexecuteinexactlythesameman-
lationsforIDENTinstructions,branches,calls,returns,etc.
ner, whether the guest was potentially using long mode on
(in rough order of dynamic frequency).
anx64CPUorusinglegacymodeona32bitlegacyCPU.In
a commercial setting where paying customers commit mis-
Having run the long mode equivalent of“hello world,”the
sioncriticalworkloadstoourplatformandrequireextremely
next challenge was clear: 64 bit Linux and Windows. The
highreliability,avoidingduplicatedfunctionalityisnotjust
detailsofgettingfromtheminimaltestOStoarealOSwere
attractive, but almost a requirement.
overall unsurprising. One particularly memorable moment
was the day we got our first 64 bit blue-screen of death
While we would ship two separate binaries, one 32 bit and
(BSOD) full of amazing 64 bit hex numbers. We happily
one 64 bit, we still wanted as much source code shared be-
printeditout,postedthepictureonourdoor,andsentmail
tweenthetwoaspossible. Westartedthinkingofoursource
with a screen shot to“all,”declaring victory.
code as having three flavors: 32 bit, 64 bit, and common
code. The common code would be compiled twice: once
12