/Documentation/kvm/mmu.txt

https://bitbucket.org/sammyz/iscream_thunderc-2.6.35-rebase · Plain Text · 304 lines · 253 code · 51 blank · 0 comment · 0 complexity · 9116b7f2af63daac129d784e08baa9b7 MD5 · raw file

  1. The x86 kvm shadow mmu
  2. ======================
  3. The mmu (in arch/x86/kvm, files mmu.[ch] and paging_tmpl.h) is responsible
  4. for presenting a standard x86 mmu to the guest, while translating guest
  5. physical addresses to host physical addresses.
  6. The mmu code attempts to satisfy the following requirements:
  7. - correctness: the guest should not be able to determine that it is running
  8. on an emulated mmu except for timing (we attempt to comply
  9. with the specification, not emulate the characteristics of
  10. a particular implementation such as tlb size)
  11. - security: the guest must not be able to touch host memory not assigned
  12. to it
  13. - performance: minimize the performance penalty imposed by the mmu
  14. - scaling: need to scale to large memory and large vcpu guests
  15. - hardware: support the full range of x86 virtualization hardware
  16. - integration: Linux memory management code must be in control of guest memory
  17. so that swapping, page migration, page merging, transparent
  18. hugepages, and similar features work without change
  19. - dirty tracking: report writes to guest memory to enable live migration
  20. and framebuffer-based displays
  21. - footprint: keep the amount of pinned kernel memory low (most memory
  22. should be shrinkable)
  23. - reliablity: avoid multipage or GFP_ATOMIC allocations
  24. Acronyms
  25. ========
  26. pfn host page frame number
  27. hpa host physical address
  28. hva host virtual address
  29. gfn guest frame number
  30. gpa guest physical address
  31. gva guest virtual address
  32. ngpa nested guest physical address
  33. ngva nested guest virtual address
  34. pte page table entry (used also to refer generically to paging structure
  35. entries)
  36. gpte guest pte (referring to gfns)
  37. spte shadow pte (referring to pfns)
  38. tdp two dimensional paging (vendor neutral term for NPT and EPT)
  39. Virtual and real hardware supported
  40. ===================================
  41. The mmu supports first-generation mmu hardware, which allows an atomic switch
  42. of the current paging mode and cr3 during guest entry, as well as
  43. two-dimensional paging (AMD's NPT and Intel's EPT). The emulated hardware
  44. it exposes is the traditional 2/3/4 level x86 mmu, with support for global
  45. pages, pae, pse, pse36, cr0.wp, and 1GB pages. Work is in progress to support
  46. exposing NPT capable hardware on NPT capable hosts.
  47. Translation
  48. ===========
  49. The primary job of the mmu is to program the processor's mmu to translate
  50. addresses for the guest. Different translations are required at different
  51. times:
  52. - when guest paging is disabled, we translate guest physical addresses to
  53. host physical addresses (gpa->hpa)
  54. - when guest paging is enabled, we translate guest virtual addresses, to
  55. guest physical addresses, to host physical addresses (gva->gpa->hpa)
  56. - when the guest launches a guest of its own, we translate nested guest
  57. virtual addresses, to nested guest physical addresses, to guest physical
  58. addresses, to host physical addresses (ngva->ngpa->gpa->hpa)
  59. The primary challenge is to encode between 1 and 3 translations into hardware
  60. that support only 1 (traditional) and 2 (tdp) translations. When the
  61. number of required translations matches the hardware, the mmu operates in
  62. direct mode; otherwise it operates in shadow mode (see below).
  63. Memory
  64. ======
  65. Guest memory (gpa) is part of the user address space of the process that is
  66. using kvm. Userspace defines the translation between guest addresses and user
  67. addresses (gpa->hva); note that two gpas may alias to the same gva, but not
  68. vice versa.
  69. These gvas may be backed using any method available to the host: anonymous
  70. memory, file backed memory, and device memory. Memory might be paged by the
  71. host at any time.
  72. Events
  73. ======
  74. The mmu is driven by events, some from the guest, some from the host.
  75. Guest generated events:
  76. - writes to control registers (especially cr3)
  77. - invlpg/invlpga instruction execution
  78. - access to missing or protected translations
  79. Host generated events:
  80. - changes in the gpa->hpa translation (either through gpa->hva changes or
  81. through hva->hpa changes)
  82. - memory pressure (the shrinker)
  83. Shadow pages
  84. ============
  85. The principal data structure is the shadow page, 'struct kvm_mmu_page'. A
  86. shadow page contains 512 sptes, which can be either leaf or nonleaf sptes. A
  87. shadow page may contain a mix of leaf and nonleaf sptes.
  88. A nonleaf spte allows the hardware mmu to reach the leaf pages and
  89. is not related to a translation directly. It points to other shadow pages.
  90. A leaf spte corresponds to either one or two translations encoded into
  91. one paging structure entry. These are always the lowest level of the
  92. translation stack, with optional higher level translations left to NPT/EPT.
  93. Leaf ptes point at guest pages.
  94. The following table shows translations encoded by leaf ptes, with higher-level
  95. translations in parentheses:
  96. Non-nested guests:
  97. nonpaging: gpa->hpa
  98. paging: gva->gpa->hpa
  99. paging, tdp: (gva->)gpa->hpa
  100. Nested guests:
  101. non-tdp: ngva->gpa->hpa (*)
  102. tdp: (ngva->)ngpa->gpa->hpa
  103. (*) the guest hypervisor will encode the ngva->gpa translation into its page
  104. tables if npt is not present
  105. Shadow pages contain the following information:
  106. role.level:
  107. The level in the shadow paging hierarchy that this shadow page belongs to.
  108. 1=4k sptes, 2=2M sptes, 3=1G sptes, etc.
  109. role.direct:
  110. If set, leaf sptes reachable from this page are for a linear range.
  111. Examples include real mode translation, large guest pages backed by small
  112. host pages, and gpa->hpa translations when NPT or EPT is active.
  113. The linear range starts at (gfn << PAGE_SHIFT) and its size is determined
  114. by role.level (2MB for first level, 1GB for second level, 0.5TB for third
  115. level, 256TB for fourth level)
  116. If clear, this page corresponds to a guest page table denoted by the gfn
  117. field.
  118. role.quadrant:
  119. When role.cr4_pae=0, the guest uses 32-bit gptes while the host uses 64-bit
  120. sptes. That means a guest page table contains more ptes than the host,
  121. so multiple shadow pages are needed to shadow one guest page.
  122. For first-level shadow pages, role.quadrant can be 0 or 1 and denotes the
  123. first or second 512-gpte block in the guest page table. For second-level
  124. page tables, each 32-bit gpte is converted to two 64-bit sptes
  125. (since each first-level guest page is shadowed by two first-level
  126. shadow pages) so role.quadrant takes values in the range 0..3. Each
  127. quadrant maps 1GB virtual address space.
  128. role.access:
  129. Inherited guest access permissions in the form uwx. Note execute
  130. permission is positive, not negative.
  131. role.invalid:
  132. The page is invalid and should not be used. It is a root page that is
  133. currently pinned (by a cpu hardware register pointing to it); once it is
  134. unpinned it will be destroyed.
  135. role.cr4_pae:
  136. Contains the value of cr4.pae for which the page is valid (e.g. whether
  137. 32-bit or 64-bit gptes are in use).
  138. role.cr4_nxe:
  139. Contains the value of efer.nxe for which the page is valid.
  140. role.cr0_wp:
  141. Contains the value of cr0.wp for which the page is valid.
  142. gfn:
  143. Either the guest page table containing the translations shadowed by this
  144. page, or the base page frame for linear translations. See role.direct.
  145. spt:
  146. A pageful of 64-bit sptes containing the translations for this page.
  147. Accessed by both kvm and hardware.
  148. The page pointed to by spt will have its page->private pointing back
  149. at the shadow page structure.
  150. sptes in spt point either at guest pages, or at lower-level shadow pages.
  151. Specifically, if sp1 and sp2 are shadow pages, then sp1->spt[n] may point
  152. at __pa(sp2->spt). sp2 will point back at sp1 through parent_pte.
  153. The spt array forms a DAG structure with the shadow page as a node, and
  154. guest pages as leaves.
  155. gfns:
  156. An array of 512 guest frame numbers, one for each present pte. Used to
  157. perform a reverse map from a pte to a gfn.
  158. slot_bitmap:
  159. A bitmap containing one bit per memory slot. If the page contains a pte
  160. mapping a page from memory slot n, then bit n of slot_bitmap will be set
  161. (if a page is aliased among several slots, then it is not guaranteed that
  162. all slots will be marked).
  163. Used during dirty logging to avoid scanning a shadow page if none if its
  164. pages need tracking.
  165. root_count:
  166. A counter keeping track of how many hardware registers (guest cr3 or
  167. pdptrs) are now pointing at the page. While this counter is nonzero, the
  168. page cannot be destroyed. See role.invalid.
  169. multimapped:
  170. Whether there exist multiple sptes pointing at this page.
  171. parent_pte/parent_ptes:
  172. If multimapped is zero, parent_pte points at the single spte that points at
  173. this page's spt. Otherwise, parent_ptes points at a data structure
  174. with a list of parent_ptes.
  175. unsync:
  176. If true, then the translations in this page may not match the guest's
  177. translation. This is equivalent to the state of the tlb when a pte is
  178. changed but before the tlb entry is flushed. Accordingly, unsync ptes
  179. are synchronized when the guest executes invlpg or flushes its tlb by
  180. other means. Valid for leaf pages.
  181. unsync_children:
  182. How many sptes in the page point at pages that are unsync (or have
  183. unsynchronized children).
  184. unsync_child_bitmap:
  185. A bitmap indicating which sptes in spt point (directly or indirectly) at
  186. pages that may be unsynchronized. Used to quickly locate all unsychronized
  187. pages reachable from a given page.
  188. Reverse map
  189. ===========
  190. The mmu maintains a reverse mapping whereby all ptes mapping a page can be
  191. reached given its gfn. This is used, for example, when swapping out a page.
  192. Synchronized and unsynchronized pages
  193. =====================================
  194. The guest uses two events to synchronize its tlb and page tables: tlb flushes
  195. and page invalidations (invlpg).
  196. A tlb flush means that we need to synchronize all sptes reachable from the
  197. guest's cr3. This is expensive, so we keep all guest page tables write
  198. protected, and synchronize sptes to gptes when a gpte is written.
  199. A special case is when a guest page table is reachable from the current
  200. guest cr3. In this case, the guest is obliged to issue an invlpg instruction
  201. before using the translation. We take advantage of that by removing write
  202. protection from the guest page, and allowing the guest to modify it freely.
  203. We synchronize modified gptes when the guest invokes invlpg. This reduces
  204. the amount of emulation we have to do when the guest modifies multiple gptes,
  205. or when the a guest page is no longer used as a page table and is used for
  206. random guest data.
  207. As a side effect we have to resynchronize all reachable unsynchronized shadow
  208. pages on a tlb flush.
  209. Reaction to events
  210. ==================
  211. - guest page fault (or npt page fault, or ept violation)
  212. This is the most complicated event. The cause of a page fault can be:
  213. - a true guest fault (the guest translation won't allow the access) (*)
  214. - access to a missing translation
  215. - access to a protected translation
  216. - when logging dirty pages, memory is write protected
  217. - synchronized shadow pages are write protected (*)
  218. - access to untranslatable memory (mmio)
  219. (*) not applicable in direct mode
  220. Handling a page fault is performed as follows:
  221. - if needed, walk the guest page tables to determine the guest translation
  222. (gva->gpa or ngpa->gpa)
  223. - if permissions are insufficient, reflect the fault back to the guest
  224. - determine the host page
  225. - if this is an mmio request, there is no host page; call the emulator
  226. to emulate the instruction instead
  227. - walk the shadow page table to find the spte for the translation,
  228. instantiating missing intermediate page tables as necessary
  229. - try to unsynchronize the page
  230. - if successful, we can let the guest continue and modify the gpte
  231. - emulate the instruction
  232. - if failed, unshadow the page and let the guest continue
  233. - update any translations that were modified by the instruction
  234. invlpg handling:
  235. - walk the shadow page hierarchy and drop affected translations
  236. - try to reinstantiate the indicated translation in the hope that the
  237. guest will use it in the near future
  238. Guest control register updates:
  239. - mov to cr3
  240. - look up new shadow roots
  241. - synchronize newly reachable shadow pages
  242. - mov to cr0/cr4/efer
  243. - set up mmu context for new paging mode
  244. - look up new shadow roots
  245. - synchronize newly reachable shadow pages
  246. Host translation updates:
  247. - mmu notifier called with updated hva
  248. - look up affected sptes through reverse map
  249. - drop (or update) translations
  250. Further reading
  251. ===============
  252. - NPT presentation from KVM Forum 2008
  253. http://www.linux-kvm.org/wiki/images/c/c8/KvmForum2008%24kdf2008_21.pdf