Star591
677
代码
代码
Issues1850
Pull Requests1448
项目讨论
Wiki
分析
Star591
677
已合并
[OLK-6.6] Enable SEV SNP kernel patches #21196
PrithivishS创建于 3月11日
[OLK-6.6] Enable SEV SNP kernel patches #21196
已合并
PrithivishS创建于 3月11日
从PrithivishS/kernelopeneuler_24.03_sev_snp_rebase合入到openeuler/kernelOLK-6.6
共 134 个文件变更+12042-2455
MDocumentation/admin-guide/kernel-parameters.txt+10-6
@@ -867,12 +867,16 @@
867 the parameter has no effect.867 the parameter has no effect.
868 868 
869 crash_kexec_post_notifiers869 crash_kexec_post_notifiers
870- Run kdump after running panic-notifiers and dumping870+ Only jump to kdump kernel after running the panic
871- kmsg. This only for the users who doubt kdump always871+ notifiers and dumping kmsg. This option increases
872- succeeds in any situation.872+ the risks of a kdump failure, since some panic
873- Note that this also increases risks of kdump failure,873+ notifiers can make the crashed kernel more unstable.
874- because some panic notifiers can make the crashed874+ In configurations where kdump may not be reliable,
875- kernel more unstable.875+ running the panic notifiers could allow collecting
876+ more data on dmesg, like stack traces from other CPUS
877+ or extra data dumped by panic_print. Note that some
878+ configurations enable this option unconditionally,
879+ like Hyper-V, PowerPC (fadump) and AMD SEV-SNP.
876 880 
877 crashkernel=size[KMG][@offset[KMG]]881 crashkernel=size[KMG][@offset[KMG]]
878 [KNL] Using kexec, Linux can switch to a 'crash kernel'882 [KNL] Using kexec, Linux can switch to a 'crash kernel'
MDocumentation/virt/coco/sev-guest.rst+71-0
@@ -67,6 +67,23 @@ counter (e.g. counter overflow), then -EIO will be returned.
67 };67 };
68 };68 };
69 69 
70+The host ioctls are issued to a file descriptor of the /dev/sev device.
71+The ioctl accepts the command ID/input structure documented below.
72+ 
73+::
74+ 
75+ struct sev_issue_cmd {
76+ /* Command ID */
77+ __u32 cmd;
78+ 
79+ /* Command request structure */
80+ __u64 data;
81+ 
82+ /* Firmware error code on failure (see psp-sev.h) */
83+ __u32 error;
84+ };
85+ 
86+ 
702.1 SNP_GET_REPORT872.1 SNP_GET_REPORT
71------------------88------------------
72 89 
@@ -124,6 +141,60 @@ be updated with the expected value.
124 141 
125See GHCB specification for further detail on how to parse the certificate blob.142See GHCB specification for further detail on how to parse the certificate blob.
126 143 
144+2.4 SNP_PLATFORM_STATUS
145+-----------------------
146+:Technology: sev-snp
147+:Type: hypervisor ioctl cmd
148+:Parameters (out): struct sev_user_data_snp_status
149+:Returns (out): 0 on success, -negative on error
150+ 
151+The SNP_PLATFORM_STATUS command is used to query the SNP platform status. The
152+status includes API major, minor version and more. See the SEV-SNP
153+specification for further details.
154+ 
155+2.5 SNP_COMMIT
156+--------------
157+:Technology: sev-snp
158+:Type: hypervisor ioctl cmd
159+:Returns (out): 0 on success, -negative on error
160+ 
161+SNP_COMMIT is used to commit the currently installed firmware using the
162+SEV-SNP firmware SNP_COMMIT command. This prevents roll-back to a previously
163+committed firmware version. This will also update the reported TCB to match
164+that of the currently installed firmware.
165+ 
166+2.6 SNP_SET_CONFIG
167+------------------
168+:Technology: sev-snp
169+:Type: hypervisor ioctl cmd
170+:Parameters (in): struct sev_user_data_snp_config
171+:Returns (out): 0 on success, -negative on error
172+ 
173+SNP_SET_CONFIG is used to set the system-wide configuration such as
174+reported TCB version in the attestation report. The command is similar
175+to SNP_CONFIG command defined in the SEV-SNP spec. The current values of
176+the firmware parameters affected by this command can be queried via
177+SNP_PLATFORM_STATUS.
178+ 
179+2.7 SNP_VLEK_LOAD
180+-----------------
181+:Technology: sev-snp
182+:Type: hypervisor ioctl cmd
183+:Parameters (in): struct sev_user_data_snp_vlek_load
184+:Returns (out): 0 on success, -negative on error
185+ 
186+When requesting an attestation report a guest is able to specify whether
187+it wants SNP firmware to sign the report using either a Versioned Chip
188+Endorsement Key (VCEK), which is derived from chip-unique secrets, or a
189+Versioned Loaded Endorsement Key (VLEK) which is obtained from an AMD
190+Key Derivation Service (KDS) and derived from seeds allocated to
191+enrolled cloud service providers.
192+ 
193+In the case of VLEK keys, the SNP_VLEK_LOAD SNP command is used to load
194+them into the system after obtaining them from the KDS, and corresponds
195+closely to the SNP_VLEK_LOAD firmware command specified in the SEV-SNP
196+spec.
197+ 
1273. SEV-SNP CPUID Enforcement1983. SEV-SNP CPUID Enforcement
128============================199============================
129 200 
MDocumentation/virt/kvm/api.rst+297-9
@@ -147,10 +147,29 @@ described as 'basic' will be available.
147The new VM has no virtual cpus and no memory.147The new VM has no virtual cpus and no memory.
148You probably want to use 0 as machine type.148You probably want to use 0 as machine type.
149 149 
150+X86:
151+^^^^
152+ 
153+Supported X86 VM types can be queried via KVM_CAP_VM_TYPES.
154+ 
155+S390:
156+^^^^^
157+ 
150In order to create user controlled virtual machines on S390, check158In order to create user controlled virtual machines on S390, check
151KVM_CAP_S390_UCONTROL and use the flag KVM_VM_S390_UCONTROL as159KVM_CAP_S390_UCONTROL and use the flag KVM_VM_S390_UCONTROL as
152privileged user (CAP_SYS_ADMIN).160privileged user (CAP_SYS_ADMIN).
153 161 
162+MIPS:
163+^^^^^
164+ 
165+To use hardware assisted virtualization on MIPS (VZ ASE) rather than
166+the default trap & emulate implementation (which changes the virtual
167+memory layout to fit in user mode), check KVM_CAP_MIPS_VZ and use the
168+flag KVM_VM_MIPS_VZ.
169+ 
170+ARM64:
171+^^^^^^
172+ 
154On arm64, the machine type identifier is used to encode a type and the173On arm64, the machine type identifier is used to encode a type and the
155physical address size for the VM. The lower byte (bits[7-0]) encode the174physical address size for the VM. The lower byte (bits[7-0]) encode the
156address size and the upper bits[11-8] encode a machine type. The machine175address size and the upper bits[11-8] encode a machine type. The machine
@@ -559,7 +578,7 @@ ioctl is useful if the in-kernel PIC is not used.
559PPC:578PPC:
560^^^^579^^^^
561 580 
562-Queues an external interrupt to be injected. This ioctl is overleaded581+Queues an external interrupt to be injected. This ioctl is overloaded
563with 3 different irq values:582with 3 different irq values:
564 583 
565a) KVM_INTERRUPT_SET584a) KVM_INTERRUPT_SET
@@ -1010,7 +1029,7 @@ be set in the flags field of this ioctl:
1010The KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL flag requests KVM to generate1029The KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL flag requests KVM to generate
1011the contents of the hypercall page automatically; hypercalls will be1030the contents of the hypercall page automatically; hypercalls will be
1012intercepted and passed to userspace through KVM_EXIT_XEN. In this1031intercepted and passed to userspace through KVM_EXIT_XEN. In this
1013-ase, all of the blob size and address fields must be zero.1032+case, all of the blob size and address fields must be zero.
1014 1033 
1015The KVM_XEN_HVM_CONFIG_EVTCHN_SEND flag indicates to KVM that userspace1034The KVM_XEN_HVM_CONFIG_EVTCHN_SEND flag indicates to KVM that userspace
1016will always use the KVM_XEN_HVM_EVTCHN_SEND ioctl to deliver event1035will always use the KVM_XEN_HVM_EVTCHN_SEND ioctl to deliver event
@@ -1115,7 +1134,7 @@ Other flags returned by ``KVM_GET_CLOCK`` are accepted but ignored.
1115:Extended by: KVM_CAP_INTR_SHADOW1134:Extended by: KVM_CAP_INTR_SHADOW
1116:Architectures: x86, arm641135:Architectures: x86, arm64
1117:Type: vcpu ioctl1136:Type: vcpu ioctl
1118-:Parameters: struct kvm_vcpu_event (out)1137+:Parameters: struct kvm_vcpu_events (out)
1119:Returns: 0 on success, -1 on error1138:Returns: 0 on success, -1 on error
1120 1139 
1121X86:1140X86:
@@ -1238,7 +1257,7 @@ directly to the virtual CPU).
1238:Extended by: KVM_CAP_INTR_SHADOW1257:Extended by: KVM_CAP_INTR_SHADOW
1239:Architectures: x86, arm641258:Architectures: x86, arm64
1240:Type: vcpu ioctl1259:Type: vcpu ioctl
1241-:Parameters: struct kvm_vcpu_event (in)1260+:Parameters: struct kvm_vcpu_events (in)
1242:Returns: 0 on success, -1 on error1261:Returns: 0 on success, -1 on error
1243 1262 
1244X86:1263X86:
@@ -3138,7 +3157,7 @@ as follow::
3138 };3157 };
3139 3158 
3140An entry with a "page_shift" of 0 is unused. Because the array is3159An entry with a "page_shift" of 0 is unused. Because the array is
3141-organized in increasing order, a lookup can stop when encoutering3160+organized in increasing order, a lookup can stop when encountering
3142such an entry.3161such an entry.
3143 3162 
3144The "slb_enc" field provides the encoding to use in the SLB for the3163The "slb_enc" field provides the encoding to use in the SLB for the
@@ -3532,7 +3551,7 @@ Possible features:
3532 - KVM_RUN and KVM_GET_REG_LIST are not available;3551 - KVM_RUN and KVM_GET_REG_LIST are not available;
3533 3552 
3534 - KVM_GET_ONE_REG and KVM_SET_ONE_REG cannot be used to access3553 - KVM_GET_ONE_REG and KVM_SET_ONE_REG cannot be used to access
3535- the scalable archietctural SVE registers3554+ the scalable architectural SVE registers
3536 KVM_REG_ARM64_SVE_ZREG(), KVM_REG_ARM64_SVE_PREG() or3555 KVM_REG_ARM64_SVE_ZREG(), KVM_REG_ARM64_SVE_PREG() or
3537 KVM_REG_ARM64_SVE_FFR;3556 KVM_REG_ARM64_SVE_FFR;
3538 3557 
@@ -4483,7 +4502,7 @@ This will have undefined effects on the guest if it has not already
4483placed itself in a quiescent state where no vcpu will make MMU enabled4502placed itself in a quiescent state where no vcpu will make MMU enabled
4484memory accesses.4503memory accesses.
4485 4504 
4486-On succsful completion, the pending HPT will become the guest's active4505+On successful completion, the pending HPT will become the guest's active
4487HPT and the previous HPT will be discarded.4506HPT and the previous HPT will be discarded.
4488 4507 
4489On failure, the guest will still be operating on its previous HPT.4508On failure, the guest will still be operating on its previous HPT.
@@ -5099,7 +5118,7 @@ before the vcpu is fully usable.
5099 5118 
5100Between KVM_ARM_VCPU_INIT and KVM_ARM_VCPU_FINALIZE, the feature may be5119Between KVM_ARM_VCPU_INIT and KVM_ARM_VCPU_FINALIZE, the feature may be
5101configured by use of ioctls such as KVM_SET_ONE_REG. The exact configuration5120configured by use of ioctls such as KVM_SET_ONE_REG. The exact configuration
5102-that should be performaned and how to do it are feature-dependent.5121+that should be performed and how to do it are feature-dependent.
5103 5122 
5104Other calls that depend on a particular feature being finalized, such as5123Other calls that depend on a particular feature being finalized, such as
5105KVM_RUN, KVM_GET_REG_LIST, KVM_GET_ONE_REG and KVM_SET_ONE_REG, will fail with5124KVM_RUN, KVM_GET_REG_LIST, KVM_GET_ONE_REG and KVM_SET_ONE_REG, will fail with
@@ -5207,6 +5226,24 @@ Valid values for 'action'::
5207 #define KVM_PMU_EVENT_ALLOW 05226 #define KVM_PMU_EVENT_ALLOW 0
5208 #define KVM_PMU_EVENT_DENY 15227 #define KVM_PMU_EVENT_DENY 1
5209 5228 
5229+Via this API, KVM userspace can also control the behavior of the VM's fixed
5230+counters (if any) by configuring the "action" and "fixed_counter_bitmap" fields.
5231+ 
5232+Specifically, KVM follows the following pseudo-code when determining whether to
5233+allow the guest FixCtr[i] to count its pre-defined fixed event::
5234+ 
5235+ FixCtr[i]_is_allowed = (action == ALLOW) && (bitmap & BIT(i)) ||
5236+ (action == DENY) && !(bitmap & BIT(i));
5237+ FixCtr[i]_is_denied = !FixCtr[i]_is_allowed;
5238+ 
5239+KVM always consumes fixed_counter_bitmap, it's userspace's responsibility to
5240+ensure fixed_counter_bitmap is set correctly, e.g. if userspace wants to define
5241+a filter that only affects general purpose counters.
5242+ 
5243+Note, the "events" field also applies to fixed counters' hardcoded event_select
5244+and unit_mask values. "fixed_counter_bitmap" has higher priority than "events"
5245+if there is a contradiction between the two.
5246+ 
52104.121 KVM_PPC_SVM_OFF52474.121 KVM_PPC_SVM_OFF
5211---------------------5248---------------------
5212 5249 
@@ -5558,7 +5595,7 @@ KVM_XEN_ATTR_TYPE_EVTCHN
5558 from the guest. A given sending port number may be directed back to5595 from the guest. A given sending port number may be directed back to
5559 a specified vCPU (by APIC ID) / port / priority on the guest, or to5596 a specified vCPU (by APIC ID) / port / priority on the guest, or to
5560 trigger events on an eventfd. The vCPU and priority can be changed5597 trigger events on an eventfd. The vCPU and priority can be changed
5561- by setting KVM_XEN_EVTCHN_UPDATE in a subsequent call, but but other5598+ by setting KVM_XEN_EVTCHN_UPDATE in a subsequent call, but other
5562 fields cannot change for a given sending port. A port mapping is5599 fields cannot change for a given sending port. A port mapping is
5563 removed by using KVM_XEN_EVTCHN_DEASSIGN in the flags field. Passing5600 removed by using KVM_XEN_EVTCHN_DEASSIGN in the flags field. Passing
5564 KVM_XEN_EVTCHN_RESET in the flags field removes all interception of5601 KVM_XEN_EVTCHN_RESET in the flags field removes all interception of
@@ -6203,6 +6240,191 @@ to know what fields can be changed for the system register described by
6203``op0, op1, crn, crm, op2``. KVM rejects ID register values that describe a6240``op0, op1, crn, crm, op2``. KVM rejects ID register values that describe a
6204superset of the features supported by the system.6241superset of the features supported by the system.
6205 6242 
6243+4.140 KVM_SET_USER_MEMORY_REGION2
6244+---------------------------------
6245+ 
6246+:Capability: KVM_CAP_USER_MEMORY2
6247+:Architectures: all
6248+:Type: vm ioctl
6249+:Parameters: struct kvm_userspace_memory_region2 (in)
6250+:Returns: 0 on success, -1 on error
6251+ 
6252+KVM_SET_USER_MEMORY_REGION2 is an extension to KVM_SET_USER_MEMORY_REGION that
6253+allows mapping guest_memfd memory into a guest. All fields shared with
6254+KVM_SET_USER_MEMORY_REGION identically. Userspace can set KVM_MEM_GUEST_MEMFD
6255+in flags to have KVM bind the memory region to a given guest_memfd range of
6256+[guest_memfd_offset, guest_memfd_offset + memory_size]. The target guest_memfd
6257+must point at a file created via KVM_CREATE_GUEST_MEMFD on the current VM, and
6258+the target range must not be bound to any other memory region. All standard
6259+bounds checks apply (use common sense).
6260+ 
6261+::
6262+ 
6263+ struct kvm_userspace_memory_region2 {
6264+ __u32 slot;
6265+ __u32 flags;
6266+ __u64 guest_phys_addr;
6267+ __u64 memory_size; /* bytes */
6268+ __u64 userspace_addr; /* start of the userspace allocated memory */
6269+ __u64 guest_memfd_offset;
6270+ __u32 guest_memfd;
6271+ __u32 pad1;
6272+ __u64 pad2[14];
6273+ };
6274+ 
6275+A KVM_MEM_GUEST_MEMFD region _must_ have a valid guest_memfd (private memory) and
6276+userspace_addr (shared memory). However, "valid" for userspace_addr simply
6277+means that the address itself must be a legal userspace address. The backing
6278+mapping for userspace_addr is not required to be valid/populated at the time of
6279+KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated
6280+on-demand.
6281+ 
6282+When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes
6283+userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE
6284+state. At VM creation time, all memory is shared, i.e. the PRIVATE attribute
6285+is '0' for all gfns. Userspace can control whether memory is shared/private by
6286+toggling KVM_MEMORY_ATTRIBUTE_PRIVATE via KVM_SET_MEMORY_ATTRIBUTES as needed.
6287+ 
6288+4.141 KVM_SET_MEMORY_ATTRIBUTES
6289+-------------------------------
6290+ 
6291+:Capability: KVM_CAP_MEMORY_ATTRIBUTES
6292+:Architectures: x86
6293+:Type: vm ioctl
6294+:Parameters: struct kvm_memory_attributes (in)
6295+:Returns: 0 on success, <0 on error
6296+ 
6297+KVM_SET_MEMORY_ATTRIBUTES allows userspace to set memory attributes for a range
6298+of guest physical memory.
6299+ 
6300+::
6301+ 
6302+ struct kvm_memory_attributes {
6303+ __u64 address;
6304+ __u64 size;
6305+ __u64 attributes;
6306+ __u64 flags;
6307+ };
6308+ 
6309+ #define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)
6310+ 
6311+The address and size must be page aligned. The supported attributes can be
6312+retrieved via ioctl(KVM_CHECK_EXTENSION) on KVM_CAP_MEMORY_ATTRIBUTES. If
6313+executed on a VM, KVM_CAP_MEMORY_ATTRIBUTES precisely returns the attributes
6314+supported by that VM. If executed at system scope, KVM_CAP_MEMORY_ATTRIBUTES
6315+returns all attributes supported by KVM. The only attribute defined at this
6316+time is KVM_MEMORY_ATTRIBUTE_PRIVATE, which marks the associated gfn as being
6317+guest private memory.
6318+ 
6319+Note, there is no "get" API. Userspace is responsible for explicitly tracking
6320+the state of a gfn/page as needed.
6321+ 
6322+The "flags" field is reserved for future extensions and must be '0'.
6323+ 
6324+4.142 KVM_CREATE_GUEST_MEMFD
6325+----------------------------
6326+ 
6327+:Capability: KVM_CAP_GUEST_MEMFD
6328+:Architectures: none
6329+:Type: vm ioctl
6330+:Parameters: struct kvm_create_guest_memfd(in)
6331+:Returns: A file descriptor on success, <0 on error
6332+ 
6333+KVM_CREATE_GUEST_MEMFD creates an anonymous file and returns a file descriptor
6334+that refers to it. guest_memfd files are roughly analogous to files created
6335+via memfd_create(), e.g. guest_memfd files live in RAM, have volatile storage,
6336+and are automatically released when the last reference is dropped. Unlike
6337+"regular" memfd_create() files, guest_memfd files are bound to their owning
6338+virtual machine (see below), cannot be mapped, read, or written by userspace,
6339+and cannot be resized (guest_memfd files do however support PUNCH_HOLE).
6340+ 
6341+::
6342+ 
6343+ struct kvm_create_guest_memfd {
6344+ __u64 size;
6345+ __u64 flags;
6346+ __u64 reserved[6];
6347+ };
6348+ 
6349+Conceptually, the inode backing a guest_memfd file represents physical memory,
6350+i.e. is coupled to the virtual machine as a thing, not to a "struct kvm". The
6351+file itself, which is bound to a "struct kvm", is that instance's view of the
6352+underlying memory, e.g. effectively provides the translation of guest addresses
6353+to host memory. This allows for use cases where multiple KVM structures are
6354+used to manage a single virtual machine, e.g. when performing intrahost
6355+migration of a virtual machine.
6356+ 
6357+KVM currently only supports mapping guest_memfd via KVM_SET_USER_MEMORY_REGION2,
6358+and more specifically via the guest_memfd and guest_memfd_offset fields in
6359+"struct kvm_userspace_memory_region2", where guest_memfd_offset is the offset
6360+into the guest_memfd instance. For a given guest_memfd file, there can be at
6361+most one mapping per page, i.e. binding multiple memory regions to a single
6362+guest_memfd range is not allowed (any number of memory regions can be bound to
6363+a single guest_memfd file, but the bound ranges must not overlap).
6364+ 
6365+See KVM_SET_USER_MEMORY_REGION2 for additional details.
6366+ 
6367+4.143 KVM_PRE_FAULT_MEMORY
6368+------------------------
6369+ 
6370+:Capability: KVM_CAP_PRE_FAULT_MEMORY
6371+:Architectures: none
6372+:Type: vcpu ioctl
6373+:Parameters: struct kvm_pre_fault_memory (in/out)
6374+:Returns: 0 if at least one page is processed, < 0 on error
6375+ 
6376+Errors:
6377+ 
6378+ ========== ===============================================================
6379+ EINVAL The specified `gpa` and `size` were invalid (e.g. not
6380+ page aligned, causes an overflow, or size is zero).
6381+ ENOENT The specified `gpa` is outside defined memslots.
6382+ EINTR An unmasked signal is pending and no page was processed.
6383+ EFAULT The parameter address was invalid.
6384+ EOPNOTSUPP Mapping memory for a GPA is unsupported by the
6385+ hypervisor, and/or for the current vCPU state/mode.
6386+ EIO unexpected error conditions (also causes a WARN)
6387+ ========== ===============================================================
6388+ 
6389+::
6390+ 
6391+ struct kvm_pre_fault_memory {
6392+ /* in/out */
6393+ __u64 gpa;
6394+ __u64 size;
6395+ /* in */
6396+ __u64 flags;
6397+ __u64 padding[5];
6398+ };
6399+ 
6400+KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory
6401+for the current vCPU state. KVM maps memory as if the vCPU generated a
6402+stage-2 read page fault, e.g. faults in memory as needed, but doesn't break
6403+CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed.
6404+ 
6405+In the case of confidential VM types where there is an initial set up of
6406+private guest memory before the guest is 'finalized'/measured, this ioctl
6407+should only be issued after completing all the necessary setup to put the
6408+guest into a 'finalized' state so that the above semantics can be reliably
6409+ensured.
6410+ 
6411+In some cases, multiple vCPUs might share the page tables. In this
6412+case, the ioctl can be called in parallel.
6413+ 
6414+When the ioctl returns, the input values are updated to point to the
6415+remaining range. If `size` > 0 on return, the caller can just issue
6416+the ioctl again with the same `struct kvm_map_memory` argument.
6417+ 
6418+Shadow page tables cannot support this ioctl because they
6419+are indexed by virtual address or nested guest physical address.
6420+Calling this ioctl when the guest is using shadow page tables (for
6421+example because it is running a nested guest with nested page tables)
6422+will fail with `EOPNOTSUPP` even if `KVM_CHECK_EXTENSION` reports
6423+the capability to be present.
6424+ 
6425+`flags` must currently be zero.
6426+ 
6427+ 
62065. The kvm_run structure64285. The kvm_run structure
6207========================6429========================
6208 6430 
@@ -6859,6 +7081,30 @@ array field represents return values. The userspace should update the return
6859values of SBI call before resuming the VCPU. For more details on RISC-V SBI7081values of SBI call before resuming the VCPU. For more details on RISC-V SBI
6860spec refer, https://github.com/riscv/riscv-sbi-doc.7082spec refer, https://github.com/riscv/riscv-sbi-doc.
6861 7083 
7084+::
7085+ 
7086+ /* KVM_EXIT_MEMORY_FAULT */
7087+ struct {
7088+ #define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 3)
7089+ __u64 flags;
7090+ __u64 gpa;
7091+ __u64 size;
7092+ } memory_fault;
7093+ 
7094+KVM_EXIT_MEMORY_FAULT indicates the vCPU has encountered a memory fault that
7095+could not be resolved by KVM. The 'gpa' and 'size' (in bytes) describe the
7096+guest physical address range [gpa, gpa + size) of the fault. The 'flags' field
7097+describes properties of the faulting access that are likely pertinent:
7098+ 
7099+ - KVM_MEMORY_EXIT_FLAG_PRIVATE - When set, indicates the memory fault occurred
7100+ on a private memory access. When clear, indicates the fault occurred on a
7101+ shared access.
7102+ 
7103+Note! KVM_EXIT_MEMORY_FAULT is unique among all KVM exit reasons in that it
7104+accompanies a return code of '-1', not '0'! errno will always be set to EFAULT
7105+or EHWPOISON when KVM exits with KVM_EXIT_MEMORY_FAULT, userspace should assume
7106+kvm_run.exit_reason is stale/undefined for all other error numbers.
7107+ 
6862::7108::
6863 7109 
6864 /* KVM_EXIT_NOTIFY */7110 /* KVM_EXIT_NOTIFY */
@@ -7893,6 +8139,27 @@ This capability is aimed to mitigate the threat that malicious VMs can
7893cause CPU stuck (due to event windows don't open up) and make the CPU8139cause CPU stuck (due to event windows don't open up) and make the CPU
7894unavailable to host or other VMs.8140unavailable to host or other VMs.
7895 8141 
8142+7.34 KVM_CAP_MEMORY_FAULT_INFO
8143+------------------------------
8144+ 
8145+:Architectures: x86
8146+:Returns: Informational only, -EINVAL on direct KVM_ENABLE_CAP.
8147+ 
8148+The presence of this capability indicates that KVM_RUN will fill
8149+kvm_run.memory_fault if KVM cannot resolve a guest page fault VM-Exit, e.g. if
8150+there is a valid memslot but no backing VMA for the corresponding host virtual
8151+address.
8152+ 
8153+The information in kvm_run.memory_fault is valid if and only if KVM_RUN returns
8154+an error with errno=EFAULT or errno=EHWPOISON *and* kvm_run.exit_reason is set
8155+to KVM_EXIT_MEMORY_FAULT.
8156+ 
8157+Note: Userspaces which attempt to resolve memory faults so that they can retry
8158+KVM_RUN are encouraged to guard against repeatedly receiving the same
8159+error/annotated fault.
8160+ 
8161+See KVM_EXIT_MEMORY_FAULT for more information.
8162+ 
78967.37 KVM_CAP_ARM_WRITABLE_IMP_ID_REGS81637.37 KVM_CAP_ARM_WRITABLE_IMP_ID_REGS
7897-------------------------------------8164-------------------------------------
7898 8165 
@@ -8470,6 +8737,7 @@ PVHVM guests. Valid flags are::
8470 #define KVM_XEN_HVM_CONFIG_EVTCHN_2LEVEL (1 << 4)8737 #define KVM_XEN_HVM_CONFIG_EVTCHN_2LEVEL (1 << 4)
8471 #define KVM_XEN_HVM_CONFIG_EVTCHN_SEND (1 << 5)8738 #define KVM_XEN_HVM_CONFIG_EVTCHN_SEND (1 << 5)
8472 #define KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG (1 << 6)8739 #define KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG (1 << 6)
8740+ #define KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE (1 << 7)
8473 8741 
8474The KVM_XEN_HVM_CONFIG_HYPERCALL_MSR flag indicates that the KVM_XEN_HVM_CONFIG8742The KVM_XEN_HVM_CONFIG_HYPERCALL_MSR flag indicates that the KVM_XEN_HVM_CONFIG
8475ioctl is available, for the guest to set its hypercall page.8743ioctl is available, for the guest to set its hypercall page.
@@ -8513,6 +8781,11 @@ behave more correctly, not using the XEN_RUNSTATE_UPDATE flag until/unless
8513specifically enabled (by the guest making the hypercall, causing the VMM8781specifically enabled (by the guest making the hypercall, causing the VMM
8514to enable the KVM_XEN_ATTR_TYPE_RUNSTATE_UPDATE_FLAG attribute).8782to enable the KVM_XEN_ATTR_TYPE_RUNSTATE_UPDATE_FLAG attribute).
8515 8783 
8784+The KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE flag indicates that KVM supports
8785+clearing the PVCLOCK_TSC_STABLE_BIT flag in Xen pvclock sources. This will be
8786+done when the KVM_CAP_XEN_HVM ioctl sets the
8787+KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE flag.
8788+ 
85168.31 KVM_CAP_PPC_MULTITCE87898.31 KVM_CAP_PPC_MULTITCE
8517-------------------------8790-------------------------
8518 8791 
@@ -8692,6 +8965,21 @@ block sizes is exposed in KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES as a
869264-bit bitmap (each bit describing a block size). The default value is896564-bit bitmap (each bit describing a block size). The default value is
86930, to disable the eager page splitting.89660, to disable the eager page splitting.
8694 8967 
8968+8.41 KVM_CAP_VM_TYPES
8969+---------------------
8970+ 
8971+:Capability: KVM_CAP_MEMORY_ATTRIBUTES
8972+:Architectures: x86
8973+:Type: system ioctl
8974+ 
8975+This capability returns a bitmap of support VM types. The 1-setting of bit @n
8976+means the VM type with value @n is supported. Possible values of @n are::
8977+ 
8978+ #define KVM_X86_DEFAULT_VM 0
8979+ #define KVM_X86_SW_PROTECTED_VM 1
8980+ #define KVM_X86_SEV_VM 2
8981+ #define KVM_X86_SEV_ES_VM 3
8982+ 
86959. Known KVM API problems89839. Known KVM API problems
8696=========================8984=========================
8697 8985 
MDocumentation/virt/kvm/x86/amd-memory-encryption.rst+189-22
@@ -46,21 +46,16 @@ SEV hardware uses ASIDs to associate a memory encryption key with a VM.
46Hence, the ASID for the SEV-enabled guests must be from 1 to a maximum value46Hence, the ASID for the SEV-enabled guests must be from 1 to a maximum value
47defined in the CPUID 0x8000001f[ecx] field.47defined in the CPUID 0x8000001f[ecx] field.
48 48 
49-SEV Key Management49+The KVM_MEMORY_ENCRYPT_OP ioctl
50-==================50+===============================
51 51 
52-The SEV guest key management is handled by a separate processor called the AMD52+The main ioctl to access SEV is KVM_MEMORY_ENCRYPT_OP, which operates on
53-Secure Processor (AMD-SP). Firmware running inside the AMD-SP provides a secure53+the VM file descriptor. If the argument to KVM_MEMORY_ENCRYPT_OP is NULL,
54-key management interface to perform common hypervisor activities such as54+the ioctl returns 0 if SEV is enabled and ``ENOTTY`` if it is disabled
55-encrypting bootstrap code, snapshot, migrating and debugging the guest. For more55+(on some older versions of Linux, the ioctl tries to run normally even
56-information, see the SEV Key Management spec [api-spec]_56+with a NULL argument, and therefore will likely return ``EFAULT`` instead
57- 57+of zero if SEV is enabled). If non-NULL, the argument to
58-The main ioctl to access SEV is KVM_MEMORY_ENCRYPT_OP. If the argument58+KVM_MEMORY_ENCRYPT_OP must be a struct kvm_sev_cmd::
59-to KVM_MEMORY_ENCRYPT_OP is NULL, the ioctl returns 0 if SEV is enabled
60-and ``ENOTTY`` if it is disabled (on some older versions of Linux,
61-the ioctl runs normally even with a NULL argument, and therefore will
62-likely return ``EFAULT``). If non-NULL, the argument to KVM_MEMORY_ENCRYPT_OP
63-must be a struct kvm_sev_cmd::
64 59 
65 struct kvm_sev_cmd {60 struct kvm_sev_cmd {
66 __u32 id;61 __u32 id;
@@ -81,19 +76,56 @@ are defined in ``<linux/psp-dev.h>``.
81KVM implements the following commands to support common lifecycle events of SEV76KVM implements the following commands to support common lifecycle events of SEV
82guests, such as launching, running, snapshotting, migrating and decommissioning.77guests, such as launching, running, snapshotting, migrating and decommissioning.
83 78 
84-1. KVM_SEV_INIT79+1. KVM_SEV_INIT2
85----------------80+----------------
86 81 
87-The KVM_SEV_INIT command is used by the hypervisor to initialize the SEV platform82+The KVM_SEV_INIT2 command is used by the hypervisor to initialize the SEV platform
88context. In a typical workflow, this command should be the first command issued.83context. In a typical workflow, this command should be the first command issued.
89 84 
90-The firmware can be initialized either by using its own non-volatile storage or85+For this command to be accepted, either KVM_X86_SEV_VM or KVM_X86_SEV_ES_VM
91-the OS can manage the NV storage for the firmware using the module parameter86+must have been passed to the KVM_CREATE_VM ioctl. A virtual machine created
92-``init_ex_path``. If the file specified by ``init_ex_path`` does not exist or87+with those machine types in turn cannot be run until KVM_SEV_INIT2 is invoked.
93-is invalid, the OS will create or override the file with output from PSP.88+ 
89+Parameters: struct kvm_sev_init (in)
94 90 
95Returns: 0 on success, -negative on error91Returns: 0 on success, -negative on error
96 92 
93+::
94+ 
95+ struct kvm_sev_init {
96+ __u64 vmsa_features; /* initial value of features field in VMSA */
97+ __u32 flags; /* must be 0 */
98+ __u16 ghcb_version; /* maximum guest GHCB version allowed */
99+ __u16 pad1;
100+ __u32 pad2[8];
101+ };
102+ 
103+It is an error if the hypervisor does not support any of the bits that
104+are set in ``flags`` or ``vmsa_features``. ``vmsa_features`` must be
105+0 for SEV virtual machines, as they do not have a VMSA.
106+ 
107+``ghcb_version`` must be 0 for SEV virtual machines, as they do not issue GHCB
108+requests. If ``ghcb_version`` is 0 for any other guest type, then the maximum
109+allowed guest GHCB protocol will default to version 2.
110+ 
111+This command replaces the deprecated KVM_SEV_INIT and KVM_SEV_ES_INIT commands.
112+The commands did not have any parameters (the ```data``` field was unused) and
113+only work for the KVM_X86_DEFAULT_VM machine type (0).
114+ 
115+They behave as if:
116+ 
117+* the VM type is KVM_X86_SEV_VM for KVM_SEV_INIT, or KVM_X86_SEV_ES_VM for
118+ KVM_SEV_ES_INIT
119+ 
120+* the ``flags`` and ``vmsa_features`` fields of ``struct kvm_sev_init`` are
121+ set to zero, and ``ghcb_version`` is set to 0 for KVM_SEV_INIT and 1 for
122+ KVM_SEV_ES_INIT.
123+ 
124+If the ``KVM_X86_SEV_VMSA_FEATURES`` attribute does not exist, the hypervisor only
125+supports KVM_SEV_INIT and KVM_SEV_ES_INIT. In that case, note that KVM_SEV_ES_INIT
126+might set the debug swap VMSA feature (bit 5) depending on the value of the
127+``debug_swap`` parameter of ``kvm-amd.ko``.
128+ 
972. KVM_SEV_LAUNCH_START1292. KVM_SEV_LAUNCH_START
98-----------------------130-----------------------
99 131 
@@ -434,13 +466,148 @@ issued by the hypervisor to make the guest ready for execution.
434 466 
435Returns: 0 on success, -negative on error467Returns: 0 on success, -negative on error
436 468 
469+18. KVM_SEV_SNP_LAUNCH_START
470+----------------------------
471+ 
472+The KVM_SNP_LAUNCH_START command is used for creating the memory encryption
473+context for the SEV-SNP guest. It must be called prior to issuing
474+KVM_SEV_SNP_LAUNCH_UPDATE or KVM_SEV_SNP_LAUNCH_FINISH;
475+ 
476+Parameters (in): struct kvm_sev_snp_launch_start
477+ 
478+Returns: 0 on success, -negative on error
479+ 
480+::
481+ 
482+ struct kvm_sev_snp_launch_start {
483+ __u64 policy; /* Guest policy to use. */
484+ __u8 gosvw[16]; /* Guest OS visible workarounds. */
485+ __u16 flags; /* Must be zero. */
486+ __u8 pad0[6];
487+ __u64 pad1[4];
488+ };
489+ 
490+See SNP_LAUNCH_START in the SEV-SNP specification [snp-fw-abi]_ for further
491+details on the input parameters in ``struct kvm_sev_snp_launch_start``.
492+ 
493+19. KVM_SEV_SNP_LAUNCH_UPDATE
494+-----------------------------
495+ 
496+The KVM_SEV_SNP_LAUNCH_UPDATE command is used for loading userspace-provided
497+data into a guest GPA range, measuring the contents into the SNP guest context
498+created by KVM_SEV_SNP_LAUNCH_START, and then encrypting/validating that GPA
499+range so that it will be immediately readable using the encryption key
500+associated with the guest context once it is booted, after which point it can
501+attest the measurement associated with its context before unlocking any
502+secrets.
503+ 
504+It is required that the GPA ranges initialized by this command have had the
505+KVM_MEMORY_ATTRIBUTE_PRIVATE attribute set in advance. See the documentation
506+for KVM_SET_MEMORY_ATTRIBUTES for more details on this aspect.
507+ 
508+Upon success, this command is not guaranteed to have processed the entire
509+range requested. Instead, the ``gfn_start``, ``uaddr``, and ``len`` fields of
510+``struct kvm_sev_snp_launch_update`` will be updated to correspond to the
511+remaining range that has yet to be processed. The caller should continue
512+calling this command until those fields indicate the entire range has been
513+processed, e.g. ``len`` is 0, ``gfn_start`` is equal to the last GFN in the
514+range plus 1, and ``uaddr`` is the last byte of the userspace-provided source
515+buffer address plus 1. In the case where ``type`` is KVM_SEV_SNP_PAGE_TYPE_ZERO,
516+``uaddr`` will be ignored completely.
517+ 
518+Parameters (in): struct kvm_sev_snp_launch_update
519+ 
520+Returns: 0 on success, < 0 on error, -EAGAIN if caller should retry
521+ 
522+::
523+ 
524+ struct kvm_sev_snp_launch_update {
525+ __u64 gfn_start; /* Guest page number to load/encrypt data into. */
526+ __u64 uaddr; /* Userspace address of data to be loaded/encrypted. */
527+ __u64 len; /* 4k-aligned length in bytes to copy into guest memory.*/
528+ __u8 type; /* The type of the guest pages being initialized. */
529+ __u8 pad0;
530+ __u16 flags; /* Must be zero. */
531+ __u32 pad1;
532+ __u64 pad2[4];
533+ 
534+ };
535+ 
536+where the allowed values for page_type are #define'd as::
537+ 
538+ KVM_SEV_SNP_PAGE_TYPE_NORMAL
539+ KVM_SEV_SNP_PAGE_TYPE_ZERO
540+ KVM_SEV_SNP_PAGE_TYPE_UNMEASURED
541+ KVM_SEV_SNP_PAGE_TYPE_SECRETS
542+ KVM_SEV_SNP_PAGE_TYPE_CPUID
543+ 
544+See the SEV-SNP spec [snp-fw-abi]_ for further details on how each page type is
545+used/measured.
546+ 
547+20. KVM_SEV_SNP_LAUNCH_FINISH
548+-----------------------------
549+ 
550+After completion of the SNP guest launch flow, the KVM_SEV_SNP_LAUNCH_FINISH
551+command can be issued to make the guest ready for execution.
552+ 
553+Parameters (in): struct kvm_sev_snp_launch_finish
554+ 
555+Returns: 0 on success, -negative on error
556+ 
557+::
558+ 
559+ struct kvm_sev_snp_launch_finish {
560+ __u64 id_block_uaddr;
561+ __u64 id_auth_uaddr;
562+ __u8 id_block_en;
563+ __u8 auth_key_en;
564+ __u8 vcek_disabled;
565+ __u8 host_data[32];
566+ __u8 pad0[3];
567+ __u16 flags; /* Must be zero */
568+ __u64 pad1[4];
569+ };
570+ 
571+ 
572+See SNP_LAUNCH_FINISH in the SEV-SNP specification [snp-fw-abi]_ for further
573+details on the input parameters in ``struct kvm_sev_snp_launch_finish``.
574+ 
575+Device attribute API
576+====================
577+ 
578+Attributes of the SEV implementation can be retrieved through the
579+``KVM_HAS_DEVICE_ATTR`` and ``KVM_GET_DEVICE_ATTR`` ioctls on the ``/dev/kvm``
580+device node, using group ``KVM_X86_GRP_SEV``.
581+ 
582+Currently only one attribute is implemented:
583+ 
584+* ``KVM_X86_SEV_VMSA_FEATURES``: return the set of all bits that
585+ are accepted in the ``vmsa_features`` of ``KVM_SEV_INIT2``.
586+ 
587+Firmware Management
588+===================
589+ 
590+The SEV guest key management is handled by a separate processor called the AMD
591+Secure Processor (AMD-SP). Firmware running inside the AMD-SP provides a secure
592+key management interface to perform common hypervisor activities such as
593+encrypting bootstrap code, snapshot, migrating and debugging the guest. For more
594+information, see the SEV Key Management spec [api-spec]_
595+ 
596+The AMD-SP firmware can be initialized either by using its own non-volatile
597+storage or the OS can manage the NV storage for the firmware using
598+parameter ``init_ex_path`` of the ``ccp`` module. If the file specified
599+by ``init_ex_path`` does not exist or is invalid, the OS will create or
600+override the file with PSP non-volatile storage.
601+ 
437References602References
438==========603==========
439 604 
440 605 
441-See [white-paper]_, [api-spec]_, [amd-apm]_ and [kvm-forum]_ for more info.606+See [white-paper]_, [api-spec]_, [amd-apm]_, [kvm-forum]_, and [snp-fw-abi]_
607+for more info.
442 608 
443.. [white-paper] https://developer.amd.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf609.. [white-paper] https://developer.amd.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf
444.. [api-spec] https://support.amd.com/TechDocs/55766_SEV-KM_API_Specification.pdf610.. [api-spec] https://support.amd.com/TechDocs/55766_SEV-KM_API_Specification.pdf
445.. [amd-apm] https://support.amd.com/TechDocs/24593.pdf (section 15.34)611.. [amd-apm] https://support.amd.com/TechDocs/24593.pdf (section 15.34)
446.. [kvm-forum] https://www.linux-kvm.org/images/7/74/02x08A-Thomas_Lendacky-AMDs_Virtualizatoin_Memory_Encryption_Technology.pdf612.. [kvm-forum] https://www.linux-kvm.org/images/7/74/02x08A-Thomas_Lendacky-AMDs_Virtualizatoin_Memory_Encryption_Technology.pdf
613+.. [snp-fw-abi] https://www.amd.com/system/files/TechDocs/56860.pdf
March/arm64/configs/openeuler_defconfig+1-0
@@ -810,6 +810,7 @@ CONFIG_HAVE_KVM_VCPU_RUN_PID_CHANGE=y
810CONFIG_KVM_XFER_TO_GUEST_WORK=y810CONFIG_KVM_XFER_TO_GUEST_WORK=y
811CONFIG_KVM_GENERIC_HARDWARE_ENABLING=y811CONFIG_KVM_GENERIC_HARDWARE_ENABLING=y
812CONFIG_HAVE_KVM_PINNED_VMID=y812CONFIG_HAVE_KVM_PINNED_VMID=y
813+CONFIG_KVM_GENERIC_MMU_NOTIFIER=y
813CONFIG_KVM_HISI_VIRT=y814CONFIG_KVM_HISI_VIRT=y
814CONFIG_VIRTUALIZATION=y815CONFIG_VIRTUALIZATION=y
815CONFIG_KVM=y816CONFIG_KVM=y
March/arm64/include/asm/kvm_host.h+0-2
@@ -1054,8 +1054,6 @@ int __kvm_arm_vcpu_get_events(struct kvm_vcpu *vcpu,
1054int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,1054int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
1055 struct kvm_vcpu_events *events);1055 struct kvm_vcpu_events *events);
1056 1056 
1057-#define KVM_ARCH_WANT_MMU_NOTIFIER
1058- 
1059void kvm_arm_halt_guest(struct kvm *kvm);1057void kvm_arm_halt_guest(struct kvm *kvm);
1060void kvm_arm_resume_guest(struct kvm *kvm);1058void kvm_arm_resume_guest(struct kvm *kvm);
1061 1059 
March/arm64/kvm/Kconfig+1-1
@@ -23,7 +23,7 @@ menuconfig KVM
23 bool "Kernel-based Virtual Machine (KVM) support"23 bool "Kernel-based Virtual Machine (KVM) support"
24 depends on HAVE_KVM24 depends on HAVE_KVM
25 select KVM_GENERIC_HARDWARE_ENABLING25 select KVM_GENERIC_HARDWARE_ENABLING
26- select MMU_NOTIFIER26+ select KVM_GENERIC_MMU_NOTIFIER
27 select PREEMPT_NOTIFIERS27 select PREEMPT_NOTIFIERS
28 select HAVE_KVM_CPU_RELAX_INTERCEPT28 select HAVE_KVM_CPU_RELAX_INTERCEPT
29 select KVM_MMIO29 select KVM_MMIO
March/arm64/kvm/mmu.c+1-1
@@ -2389,7 +2389,7 @@ int kvm_mmu_mark_touched_log(struct kvm *kvm)
2389 2389 
2390 while (has_next) {2390 while (has_next) {
2391 has_next = false;2391 has_next = false;
2392- for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {2392+ for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
2393 slots = __kvm_memslots(kvm, i);2393 slots = __kvm_memslots(kvm, i);
2394 kvm_for_each_memslot(memslot, bkt, slots) {2394 kvm_for_each_memslot(memslot, bkt, slots) {
2395 gfn_t start, end;2395 gfn_t start, end;
March/loongarch/include/asm/kvm_host.h+0-1
@@ -322,7 +322,6 @@ void kvm_flush_tlb_all(void);
322void kvm_flush_tlb_gpa(struct kvm_vcpu *vcpu, unsigned long gpa);322void kvm_flush_tlb_gpa(struct kvm_vcpu *vcpu, unsigned long gpa);
323int kvm_handle_mm_fault(struct kvm_vcpu *vcpu, unsigned long badv, bool write, int ecode);323int kvm_handle_mm_fault(struct kvm_vcpu *vcpu, unsigned long badv, bool write, int ecode);
324 324 
325-#define KVM_ARCH_WANT_MMU_NOTIFIER
326void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte);325void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte);
327int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end, bool blockable);326int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end, bool blockable);
328int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end);327int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end);
March/loongarch/kvm/Kconfig+1-1
@@ -30,11 +30,11 @@ config KVM
30 select HAVE_KVM_MSI30 select HAVE_KVM_MSI
31 select KVM_GENERIC_DIRTYLOG_READ_PROTECT31 select KVM_GENERIC_DIRTYLOG_READ_PROTECT
32 select KVM_GENERIC_HARDWARE_ENABLING32 select KVM_GENERIC_HARDWARE_ENABLING
33+ select KVM_GENERIC_MMU_NOTIFIER
33 select KVM_MMIO34 select KVM_MMIO
34 select KVM_XFER_TO_GUEST_WORK35 select KVM_XFER_TO_GUEST_WORK
35 select SCHED_INFO36 select SCHED_INFO
36 select GUEST_PERF_EVENTS if PERF_EVENTS37 select GUEST_PERF_EVENTS if PERF_EVENTS
37- select MMU_NOTIFIER
38 select PREEMPT_NOTIFIERS38 select PREEMPT_NOTIFIERS
39 select KVM_VFIO39 select KVM_VFIO
40 help40 help
March/loongarch/kvm/mmu.c+2-2
@@ -698,7 +698,7 @@ static bool fault_supports_huge_mapping(struct kvm_memory_slot *memslot,
698 *698 *
699 * There are several ways to safely use this helper:699 * There are several ways to safely use this helper:
700 *700 *
701- * - Check mmu_invalidate_retry_hva() after grabbing the mapping level, before701+ * - Check mmu_invalidate_retry_gfn() after grabbing the mapping level, before
702 * consuming it. In this case, mmu_lock doesn't need to be held during the702 * consuming it. In this case, mmu_lock doesn't need to be held during the
703 * lookup, but it does need to be held while checking the MMU notifier.703 * lookup, but it does need to be held while checking the MMU notifier.
704 *704 *
@@ -879,7 +879,7 @@ static int kvm_map_page(struct kvm_vcpu *vcpu, unsigned long gpa, bool write)
879 879 
880 /* Check if an invalidation has taken place since we got pfn */880 /* Check if an invalidation has taken place since we got pfn */
881 spin_lock(&kvm->mmu_lock);881 spin_lock(&kvm->mmu_lock);
882- if (mmu_invalidate_retry_hva(kvm, mmu_seq, hva)) {882+ if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn)) {
883 /*883 /*
884 * This can happen when mappings are changed asynchronously, but884 * This can happen when mappings are changed asynchronously, but
885 * also synchronously if a COW is triggered by885 * also synchronously if a COW is triggered by
March/mips/include/asm/kvm_host.h+0-2
@@ -810,8 +810,6 @@ int kvm_mips_mkclean_gpa_pt(struct kvm *kvm, gfn_t start_gfn, gfn_t end_gfn);
810pgd_t *kvm_pgd_alloc(void);810pgd_t *kvm_pgd_alloc(void);
811void kvm_mmu_free_memory_caches(struct kvm_vcpu *vcpu);811void kvm_mmu_free_memory_caches(struct kvm_vcpu *vcpu);
812 812 
813-#define KVM_ARCH_WANT_MMU_NOTIFIER
814- 
815/* Emulation */813/* Emulation */
816enum emulation_result update_pc(struct kvm_vcpu *vcpu, u32 cause);814enum emulation_result update_pc(struct kvm_vcpu *vcpu, u32 cause);
817int kvm_get_badinstr(u32 *opc, struct kvm_vcpu *vcpu, u32 *out);815int kvm_get_badinstr(u32 *opc, struct kvm_vcpu *vcpu, u32 *out);
March/mips/kvm/Kconfig+1-1
@@ -25,7 +25,7 @@ config KVM
25 select HAVE_KVM_EVENTFD25 select HAVE_KVM_EVENTFD
26 select HAVE_KVM_VCPU_ASYNC_IOCTL26 select HAVE_KVM_VCPU_ASYNC_IOCTL
27 select KVM_MMIO27 select KVM_MMIO
28- select MMU_NOTIFIER28+ select KVM_GENERIC_MMU_NOTIFIER
29 select INTERVAL_TREE29 select INTERVAL_TREE
30 select KVM_GENERIC_HARDWARE_ENABLING30 select KVM_GENERIC_HARDWARE_ENABLING
31 help31 help
March/powerpc/include/asm/kvm_host.h+0-2
@@ -62,8 +62,6 @@
62 62 
63#include <linux/mmu_notifier.h>63#include <linux/mmu_notifier.h>
64 64 
65-#define KVM_ARCH_WANT_MMU_NOTIFIER
66- 
67#define HPTEG_CACHE_NUM (1 << 15)65#define HPTEG_CACHE_NUM (1 << 15)
68#define HPTEG_HASH_BITS_PTE 1366#define HPTEG_HASH_BITS_PTE 13
69#define HPTEG_HASH_BITS_PTE_LONG 1267#define HPTEG_HASH_BITS_PTE_LONG 12
March/powerpc/kvm/Kconfig+4-4
@@ -42,7 +42,7 @@ config KVM_BOOK3S_64_HANDLER
42config KVM_BOOK3S_PR_POSSIBLE42config KVM_BOOK3S_PR_POSSIBLE
43 bool43 bool
44 select KVM_MMIO44 select KVM_MMIO
45- select MMU_NOTIFIER45+ select KVM_GENERIC_MMU_NOTIFIER
46 46 
47config KVM_BOOK3S_HV_POSSIBLE47config KVM_BOOK3S_HV_POSSIBLE
48 bool48 bool
@@ -85,7 +85,7 @@ config KVM_BOOK3S_64_HV
85 tristate "KVM for POWER7 and later using hypervisor mode in host"85 tristate "KVM for POWER7 and later using hypervisor mode in host"
86 depends on KVM_BOOK3S_64 && PPC_POWERNV86 depends on KVM_BOOK3S_64 && PPC_POWERNV
87 select KVM_BOOK3S_HV_POSSIBLE87 select KVM_BOOK3S_HV_POSSIBLE
88- select MMU_NOTIFIER88+ select KVM_GENERIC_MMU_NOTIFIER
89 select CMA89 select CMA
90 help90 help
91 Support running unmodified book3s_64 guest kernels in91 Support running unmodified book3s_64 guest kernels in
@@ -194,7 +194,7 @@ config KVM_E500V2
194 depends on !CONTEXT_TRACKING_USER194 depends on !CONTEXT_TRACKING_USER
195 select KVM195 select KVM
196 select KVM_MMIO196 select KVM_MMIO
197- select MMU_NOTIFIER197+ select KVM_GENERIC_MMU_NOTIFIER
198 help198 help
199 Support running unmodified E500 guest kernels in virtual machines on199 Support running unmodified E500 guest kernels in virtual machines on
200 E500v2 host processors.200 E500v2 host processors.
@@ -211,7 +211,7 @@ config KVM_E500MC
211 select KVM211 select KVM
212 select KVM_MMIO212 select KVM_MMIO
213 select KVM_BOOKE_HV213 select KVM_BOOKE_HV
214- select MMU_NOTIFIER214+ select KVM_GENERIC_MMU_NOTIFIER
215 help215 help
216 Support running unmodified E500MC/E5500/E6500 guest kernels in216 Support running unmodified E500MC/E5500/E6500 guest kernels in
217 virtual machines on E500MC/E5500/E6500 host processors.217 virtual machines on E500MC/E5500/E6500 host processors.
March/powerpc/kvm/book3s_hv.c+1-1
@@ -6106,7 +6106,7 @@ static int kvmhv_svm_off(struct kvm *kvm)
6106 }6106 }
6107 6107 
6108 srcu_idx = srcu_read_lock(&kvm->srcu);6108 srcu_idx = srcu_read_lock(&kvm->srcu);
6109- for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {6109+ for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
6110 struct kvm_memory_slot *memslot;6110 struct kvm_memory_slot *memslot;
6111 struct kvm_memslots *slots = __kvm_memslots(kvm, i);6111 struct kvm_memslots *slots = __kvm_memslots(kvm, i);
6112 int bkt;6112 int bkt;
March/powerpc/kvm/powerpc.c+1-6
@@ -632,13 +632,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
632 break;632 break;
633#endif633#endif
634 case KVM_CAP_SYNC_MMU:634 case KVM_CAP_SYNC_MMU:
635-#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE635+ BUILD_BUG_ON(!IS_ENABLED(CONFIG_KVM_GENERIC_MMU_NOTIFIER));
636- r = hv_enabled;
637-#elif defined(KVM_ARCH_WANT_MMU_NOTIFIER)
638 r = 1;636 r = 1;
639-#else
640- r = 0;
641-#endif
642 break;637 break;
643#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE638#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
644 case KVM_CAP_PPC_HTAB_FD:639 case KVM_CAP_PPC_HTAB_FD:
March/riscv/include/asm/kvm_host.h+0-2
@@ -250,8 +250,6 @@ struct kvm_vcpu_arch {
250static inline void kvm_arch_sync_events(struct kvm *kvm) {}250static inline void kvm_arch_sync_events(struct kvm *kvm) {}
251static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}251static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
252 252 
253-#define KVM_ARCH_WANT_MMU_NOTIFIER
254- 
255#define KVM_RISCV_GSTAGE_TLB_MIN_ORDER 12253#define KVM_RISCV_GSTAGE_TLB_MIN_ORDER 12
256 254 
257void kvm_riscv_local_hfence_gvma_vmid_gpa(unsigned long vmid,255void kvm_riscv_local_hfence_gvma_vmid_gpa(unsigned long vmid,
March/riscv/kvm/Kconfig+1-1
@@ -30,7 +30,7 @@ config KVM
30 select KVM_GENERIC_HARDWARE_ENABLING30 select KVM_GENERIC_HARDWARE_ENABLING
31 select KVM_MMIO31 select KVM_MMIO
32 select KVM_XFER_TO_GUEST_WORK32 select KVM_XFER_TO_GUEST_WORK
33- select MMU_NOTIFIER33+ select KVM_GENERIC_MMU_NOTIFIER
34 select PREEMPT_NOTIFIERS34 select PREEMPT_NOTIFIERS
35 help35 help
36 Support hosting virtualized guest machines.36 Support hosting virtualized guest machines.
March/x86/Kbuild+2-0
@@ -28,5 +28,7 @@ obj-y += net/
28 28 
29obj-$(CONFIG_KEXEC_FILE) += purgatory/29obj-$(CONFIG_KEXEC_FILE) += purgatory/
30 30 
31+obj-y += virt/svm/
32+ 
31# for cleaning33# for cleaning
32subdir- += boot tools34subdir- += boot tools
March/x86/boot/compressed/sev.c+96-103
@@ -304,26 +304,6 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
304 sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ);304 sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SEV_ES_GEN_REQ);
305}305}
306 306 
307-static void enforce_vmpl0(void)
308-{
309- u64 attrs;
310- int err;
311- 
312- /*
313- * RMPADJUST modifies RMP permissions of a lesser-privileged (numerically
314- * higher) privilege level. Here, clear the VMPL1 permission mask of the
315- * GHCB page. If the guest is not running at VMPL0, this will fail.
316- *
317- * If the guest is running at VMPL0, it will succeed. Even if that operation
318- * modifies permission bits, it is still ok to do so currently because Linux
319- * SNP guests are supported only on VMPL0 so VMPL1 or higher permission masks
320- * changing is a don't-care.
321- */
322- attrs = 1;
323- if (rmpadjust((unsigned long)&boot_ghcb_page, RMP_PG_SIZE_4K, attrs))
324- sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_NOT_VMPL0);
325-}
326- 
327/*307/*
328 * SNP_FEATURES_IMPL_REQ is the mask of SNP features that will need308 * SNP_FEATURES_IMPL_REQ is the mask of SNP features that will need
329 * guest side implementation for proper functioning of the guest. If any309 * guest side implementation for proper functioning of the guest. If any
@@ -342,7 +322,7 @@ static void enforce_vmpl0(void)
342 MSR_AMD64_SNP_VMPL_SSS | \322 MSR_AMD64_SNP_VMPL_SSS | \
343 MSR_AMD64_SNP_SECURE_TSC | \323 MSR_AMD64_SNP_SECURE_TSC | \
344 MSR_AMD64_SNP_VMGEXIT_PARAM | \324 MSR_AMD64_SNP_VMGEXIT_PARAM | \
345- MSR_AMD64_SNP_VMSA_REG_PROTECTION | \325+ MSR_AMD64_SNP_VMSA_REG_PROT | \
346 MSR_AMD64_SNP_RESERVED_BIT13 | \326 MSR_AMD64_SNP_RESERVED_BIT13 | \
347 MSR_AMD64_SNP_RESERVED_BIT15 | \327 MSR_AMD64_SNP_RESERVED_BIT15 | \
348 MSR_AMD64_SNP_RESERVED_BITS18_22 | \328 MSR_AMD64_SNP_RESERVED_BITS18_22 | \
@@ -383,6 +363,85 @@ void snp_check_features(void)
383 }363 }
384}364}
385 365 
366+/* Search for Confidential Computing blob in the EFI config table. */
367+static struct cc_blob_sev_info *find_cc_blob_efi(struct boot_params *bp)
368+{
369+ unsigned long cfg_table_pa;
370+ unsigned int cfg_table_len;
371+ int ret;
372+ 
373+ ret = efi_get_conf_table(bp, &cfg_table_pa, &cfg_table_len);
374+ if (ret)
375+ return NULL;
376+ 
377+ return (struct cc_blob_sev_info *)efi_find_vendor_table(bp, cfg_table_pa,
378+ cfg_table_len,
379+ EFI_CC_BLOB_GUID);
380+}
381+ 
382+/*
383+ * Initial set up of SNP relies on information provided by the
384+ * Confidential Computing blob, which can be passed to the boot kernel
385+ * by firmware/bootloader in the following ways:
386+ *
387+ * - via an entry in the EFI config table
388+ * - via a setup_data structure, as defined by the Linux Boot Protocol
389+ *
390+ * Scan for the blob in that order.
391+ */
392+static struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp)
393+{
394+ struct cc_blob_sev_info *cc_info;
395+ 
396+ cc_info = find_cc_blob_efi(bp);
397+ if (cc_info)
398+ goto found_cc_info;
399+ 
400+ cc_info = find_cc_blob_setup_data(bp);
401+ if (!cc_info)
402+ return NULL;
403+ 
404+found_cc_info:
405+ if (cc_info->magic != CC_BLOB_SEV_HDR_MAGIC)
406+ sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED);
407+ 
408+ return cc_info;
409+}
410+ 
411+/*
412+ * Indicate SNP based on presence of SNP-specific CC blob. Subsequent checks
413+ * will verify the SNP CPUID/MSR bits.
414+ */
415+static bool early_snp_init(struct boot_params *bp)
416+{
417+ struct cc_blob_sev_info *cc_info;
418+ 
419+ if (!bp)
420+ return false;
421+ 
422+ cc_info = find_cc_blob(bp);
423+ if (!cc_info)
424+ return false;
425+ 
426+ /*
427+ * If a SNP-specific Confidential Computing blob is present, then
428+ * firmware/bootloader have indicated SNP support. Verifying this
429+ * involves CPUID checks which will be more reliable if the SNP
430+ * CPUID table is used. See comments over snp_setup_cpuid_table() for
431+ * more details.
432+ */
433+ setup_cpuid_table(cc_info);
434+ 
435+ /*
436+ * Pass run-time kernel a pointer to CC info via boot_params so EFI
437+ * config table doesn't need to be searched again during early startup
438+ * phase.
439+ */
440+ bp->cc_blob_address = (u32)(unsigned long)cc_info;
441+ 
442+ return true;
443+}
444+ 
386/*445/*
387 * sev_check_cpu_support - Check for SEV support in the CPU capabilities446 * sev_check_cpu_support - Check for SEV support in the CPU capabilities
388 *447 *
@@ -433,7 +492,7 @@ void sev_enable(struct boot_params *bp)
433 bp->cc_blob_address = 0;492 bp->cc_blob_address = 0;
434 493 
435 /*494 /*
436- * Do an initial SEV capability check before snp_init() which495+ * Do an initial SEV capability check before early_snp_init() which
437 * loads the CPUID page and the same checks afterwards are done496 * loads the CPUID page and the same checks afterwards are done
438 * without the hypervisor and are trustworthy.497 * without the hypervisor and are trustworthy.
439 *498 *
@@ -448,7 +507,7 @@ void sev_enable(struct boot_params *bp)
448 * Setup/preliminary detection of SNP. This will be sanity-checked507 * Setup/preliminary detection of SNP. This will be sanity-checked
449 * against CPUID/MSR values later.508 * against CPUID/MSR values later.
450 */509 */
451- snp = snp_init(bp);510+ snp = early_snp_init(bp);
452 511 
453 /* Now repeat the checks with the SNP CPUID table. */512 /* Now repeat the checks with the SNP CPUID table. */
454 513 
@@ -479,7 +538,20 @@ void sev_enable(struct boot_params *bp)
479 if (!(get_hv_features() & GHCB_HV_FT_SNP))538 if (!(get_hv_features() & GHCB_HV_FT_SNP))
480 sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED);539 sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED);
481 540 
482- enforce_vmpl0();541+ /*
542+ * Enforce running at VMPL0.
543+ *
544+ * RMPADJUST modifies RMP permissions of a lesser-privileged (numerically
545+ * higher) privilege level. Here, clear the VMPL1 permission mask of the
546+ * GHCB page. If the guest is not running at VMPL0, this will fail.
547+ *
548+ * If the guest is running at VMPL0, it will succeed. Even if that operation
549+ * modifies permission bits, it is still ok to do so currently because Linux
550+ * SNP guests running at VMPL0 only run at VMPL0, so VMPL1 or higher
551+ * permission mask changes are a don't-care.
552+ */
553+ if (rmpadjust((unsigned long)&boot_ghcb_page, RMP_PG_SIZE_4K, 1))
554+ sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_NOT_VMPL0);
483 }555 }
484 556 
485 if (snp && !(sev_status & MSR_AMD64_SEV_SNP_ENABLED))557 if (snp && !(sev_status & MSR_AMD64_SEV_SNP_ENABLED))
@@ -505,85 +577,6 @@ u64 sev_get_status(void)
505 return m.q;577 return m.q;
506}578}
507 579 
508-/* Search for Confidential Computing blob in the EFI config table. */
509-static struct cc_blob_sev_info *find_cc_blob_efi(struct boot_params *bp)
510-{
511- unsigned long cfg_table_pa;
512- unsigned int cfg_table_len;
513- int ret;
514- 
515- ret = efi_get_conf_table(bp, &cfg_table_pa, &cfg_table_len);
516- if (ret)
517- return NULL;
518- 
519- return (struct cc_blob_sev_info *)efi_find_vendor_table(bp, cfg_table_pa,
520- cfg_table_len,
521- EFI_CC_BLOB_GUID);
522-}
523- 
524-/*
525- * Initial set up of SNP relies on information provided by the
526- * Confidential Computing blob, which can be passed to the boot kernel
527- * by firmware/bootloader in the following ways:
528- *
529- * - via an entry in the EFI config table
530- * - via a setup_data structure, as defined by the Linux Boot Protocol
531- *
532- * Scan for the blob in that order.
533- */
534-static struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp)
535-{
536- struct cc_blob_sev_info *cc_info;
537- 
538- cc_info = find_cc_blob_efi(bp);
539- if (cc_info)
540- goto found_cc_info;
541- 
542- cc_info = find_cc_blob_setup_data(bp);
543- if (!cc_info)
544- return NULL;
545- 
546-found_cc_info:
547- if (cc_info->magic != CC_BLOB_SEV_HDR_MAGIC)
548- sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED);
549- 
550- return cc_info;
551-}
552- 
553-/*
554- * Indicate SNP based on presence of SNP-specific CC blob. Subsequent checks
555- * will verify the SNP CPUID/MSR bits.
556- */
557-bool snp_init(struct boot_params *bp)
558-{
559- struct cc_blob_sev_info *cc_info;
560- 
561- if (!bp)
562- return false;
563- 
564- cc_info = find_cc_blob(bp);
565- if (!cc_info)
566- return false;
567- 
568- /*
569- * If a SNP-specific Confidential Computing blob is present, then
570- * firmware/bootloader have indicated SNP support. Verifying this
571- * involves CPUID checks which will be more reliable if the SNP
572- * CPUID table is used. See comments over snp_setup_cpuid_table() for
573- * more details.
574- */
575- setup_cpuid_table(cc_info);
576- 
577- /*
578- * Pass run-time kernel a pointer to CC info via boot_params so EFI
579- * config table doesn't need to be searched again during early startup
580- * phase.
581- */
582- bp->cc_blob_address = (u32)(unsigned long)cc_info;
583- 
584- return true;
585-}
586- 
587void sev_prep_identity_maps(unsigned long top_level_pgt)580void sev_prep_identity_maps(unsigned long top_level_pgt)
588{581{
589 /*582 /*
March/x86/coco/core.c+52-0
@@ -20,6 +20,11 @@
20enum cc_vendor cc_vendor __ro_after_init = CC_VENDOR_NONE;20enum cc_vendor cc_vendor __ro_after_init = CC_VENDOR_NONE;
21u64 cc_mask __ro_after_init;21u64 cc_mask __ro_after_init;
22 22 
23+static struct cc_attr_flags {
24+ __u64 host_sev_snp : 1,
25+ __resv : 63;
26+} cc_flags;
27+ 
23static bool noinstr intel_cc_platform_has(enum cc_attr attr)28static bool noinstr intel_cc_platform_has(enum cc_attr attr)
24{29{
25 switch (attr) {30 switch (attr) {
@@ -93,6 +98,9 @@ static bool noinstr amd_cc_platform_has(enum cc_attr attr)
93 case CC_ATTR_GUEST_SEV_SNP:98 case CC_ATTR_GUEST_SEV_SNP:
94 return sev_status & MSR_AMD64_SEV_SNP_ENABLED;99 return sev_status & MSR_AMD64_SEV_SNP_ENABLED;
95 100 
101+ case CC_ATTR_HOST_SEV_SNP:
102+ return cc_flags.host_sev_snp;
103+ 
96 default:104 default:
97 return false;105 return false;
98 }106 }
@@ -153,6 +161,50 @@ u64 cc_mkdec(u64 val)
153}161}
154EXPORT_SYMBOL_GPL(cc_mkdec);162EXPORT_SYMBOL_GPL(cc_mkdec);
155 163 
164+static void amd_cc_platform_clear(enum cc_attr attr)
165+{
166+ switch (attr) {
167+ case CC_ATTR_HOST_SEV_SNP:
168+ cc_flags.host_sev_snp = 0;
169+ break;
170+ default:
171+ break;
172+ }
173+}
174+ 
175+void cc_platform_clear(enum cc_attr attr)
176+{
177+ switch (cc_vendor) {
178+ case CC_VENDOR_AMD:
179+ amd_cc_platform_clear(attr);
180+ break;
181+ default:
182+ break;
183+ }
184+}
185+ 
186+static void amd_cc_platform_set(enum cc_attr attr)
187+{
188+ switch (attr) {
189+ case CC_ATTR_HOST_SEV_SNP:
190+ cc_flags.host_sev_snp = 1;
191+ break;
192+ default:
193+ break;
194+ }
195+}
196+ 
197+void cc_platform_set(enum cc_attr attr)
198+{
199+ switch (cc_vendor) {
200+ case CC_VENDOR_AMD:
201+ amd_cc_platform_set(attr);
202+ break;
203+ default:
204+ break;
205+ }
206+}
207+ 
156__init void cc_random_init(void)208__init void cc_random_init(void)
157{209{
158 /*210 /*
March/x86/configs/openeuler_defconfig+8-0
@@ -764,12 +764,19 @@ CONFIG_HAVE_KVM_MSI=y
764CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y764CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
765CONFIG_KVM_VFIO=y765CONFIG_KVM_VFIO=y
766CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y766CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
767+CONFIG_KVM_GENERIC_PRE_FAULT_MEMORY=y
767CONFIG_KVM_COMPAT=y768CONFIG_KVM_COMPAT=y
768CONFIG_HAVE_KVM_IRQ_BYPASS=y769CONFIG_HAVE_KVM_IRQ_BYPASS=y
769CONFIG_HAVE_KVM_NO_POLL=y770CONFIG_HAVE_KVM_NO_POLL=y
770CONFIG_KVM_XFER_TO_GUEST_WORK=y771CONFIG_KVM_XFER_TO_GUEST_WORK=y
771CONFIG_HAVE_KVM_PM_NOTIFIER=y772CONFIG_HAVE_KVM_PM_NOTIFIER=y
772CONFIG_KVM_GENERIC_HARDWARE_ENABLING=y773CONFIG_KVM_GENERIC_HARDWARE_ENABLING=y
774+CONFIG_KVM_GENERIC_MMU_NOTIFIER=y
775+CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES=y
776+CONFIG_KVM_PRIVATE_MEM=y
777+CONFIG_KVM_GENERIC_PRIVATE_MEM=y
778+CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE=y
779+CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE=y
773CONFIG_VIRTUALIZATION=y780CONFIG_VIRTUALIZATION=y
774CONFIG_KVM=m781CONFIG_KVM=m
775CONFIG_KVM_INTEL=m782CONFIG_KVM_INTEL=m
@@ -7231,6 +7238,7 @@ CONFIG_VIRT_DRIVERS=y
7231CONFIG_VMGENID=y7238CONFIG_VMGENID=y
7232# CONFIG_VBOXGUEST is not set7239# CONFIG_VBOXGUEST is not set
7233# CONFIG_NITRO_ENCLAVES is not set7240# CONFIG_NITRO_ENCLAVES is not set
7241+CONFIG_TSM_REPORTS=m
7234CONFIG_EFI_SECRET=m7242CONFIG_EFI_SECRET=m
7235CONFIG_SEV_GUEST=m7243CONFIG_SEV_GUEST=m
7236# CONFIG_TDX_GUEST_DRIVER is not set7244# CONFIG_TDX_GUEST_DRIVER is not set
March/x86/include/asm/cpufeatures.h+4-0
@@ -467,9 +467,13 @@
467#define X86_FEATURE_SEV (19*32+ 1) /* AMD Secure Encrypted Virtualization */467#define X86_FEATURE_SEV (19*32+ 1) /* AMD Secure Encrypted Virtualization */
468#define X86_FEATURE_VM_PAGE_FLUSH (19*32+ 2) /* "" VM Page Flush MSR is supported */468#define X86_FEATURE_VM_PAGE_FLUSH (19*32+ 2) /* "" VM Page Flush MSR is supported */
469#define X86_FEATURE_SEV_ES (19*32+ 3) /* AMD Secure Encrypted Virtualization - Encrypted State */469#define X86_FEATURE_SEV_ES (19*32+ 3) /* AMD Secure Encrypted Virtualization - Encrypted State */
470+#define X86_FEATURE_SEV_SNP (19*32+ 4) /* AMD Secure Encrypted Virtualization - Secure Nested Paging */
470#define X86_FEATURE_V_TSC_AUX (19*32+ 9) /* "" Virtual TSC_AUX */471#define X86_FEATURE_V_TSC_AUX (19*32+ 9) /* "" Virtual TSC_AUX */
471#define X86_FEATURE_SME_COHERENT (19*32+10) /* "" AMD hardware-enforced cache coherency */472#define X86_FEATURE_SME_COHERENT (19*32+10) /* "" AMD hardware-enforced cache coherency */
472#define X86_FEATURE_DEBUG_SWAP (19*32+14) /* AMD SEV-ES full debug state swap support */473#define X86_FEATURE_DEBUG_SWAP (19*32+14) /* AMD SEV-ES full debug state swap support */
474+#define X86_FEATURE_RMPREAD (19*32+21) /* RMPREAD instruction */
475+#define X86_FEATURE_SEGMENTED_RMP (19*32+23) /* Segmented RMP support */
476+#define X86_FEATURE_HV_INUSE_WR_ALLOWED (19*32+30) /* Allow Write to in-use hypervisor-owned pages */
473/* HYGON 3rd CSV */477/* HYGON 3rd CSV */
474#define X86_FEATURE_CSV3 (19*32 + 30) /* HYGON 3rd CSV */478#define X86_FEATURE_CSV3 (19*32 + 30) /* HYGON 3rd CSV */
475 479 
March/x86/include/asm/disabled-features.h+7-1
@@ -117,6 +117,12 @@
117#define DISABLE_IBT (1 << (X86_FEATURE_IBT & 31))117#define DISABLE_IBT (1 << (X86_FEATURE_IBT & 31))
118#endif118#endif
119 119 
120+#ifdef CONFIG_KVM_AMD_SEV
121+#define DISABLE_SEV_SNP 0
122+#else
123+#define DISABLE_SEV_SNP (1 << (X86_FEATURE_SEV_SNP & 31))
124+#endif
125+ 
120/*126/*
121 * Make sure to add features to the correct mask127 * Make sure to add features to the correct mask
122 */128 */
@@ -141,7 +147,7 @@
141 DISABLE_ENQCMD)147 DISABLE_ENQCMD)
142#define DISABLED_MASK17 0148#define DISABLED_MASK17 0
143#define DISABLED_MASK18 (DISABLE_IBT)149#define DISABLED_MASK18 (DISABLE_IBT)
144-#define DISABLED_MASK19 0150+#define DISABLED_MASK19 (DISABLE_SEV_SNP)
145#define DISABLED_MASK20 0151#define DISABLED_MASK20 0
146#define DISABLED_MASK21 0152#define DISABLED_MASK21 0
147#define DISABLED_MASK22 0153#define DISABLED_MASK22 0
March/x86/include/asm/e820/api.h+1-0
@@ -17,6 +17,7 @@ extern bool e820__mapped_all(u64 start, u64 end, enum e820_type type);
17extern void e820__range_add (u64 start, u64 size, enum e820_type type);17extern void e820__range_add (u64 start, u64 size, enum e820_type type);
18extern u64 e820__range_update(u64 start, u64 size, enum e820_type old_type, enum e820_type new_type);18extern u64 e820__range_update(u64 start, u64 size, enum e820_type old_type, enum e820_type new_type);
19extern u64 e820__range_remove(u64 start, u64 size, enum e820_type old_type, bool check_type);19extern u64 e820__range_remove(u64 start, u64 size, enum e820_type old_type, bool check_type);
20+extern u64 e820__range_update_table(struct e820_table *t, u64 start, u64 size, enum e820_type old_type, enum e820_type new_type);
20 21 
21extern void e820__print_table(char *who);22extern void e820__print_table(char *who);
22extern int e820__update_table(struct e820_table *table);23extern int e820__update_table(struct e820_table *table);
March/x86/include/asm/fpu/api.h+3-0
@@ -168,6 +168,9 @@ extern void fpstate_clear_xstate_component(struct fpstate *fps, unsigned int xfe
168 168 
169extern u64 xstate_get_guest_group_perm(void);169extern u64 xstate_get_guest_group_perm(void);
170 170 
171+extern void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr);
172+ 
173+ 
171/* KVM specific functions */174/* KVM specific functions */
172extern bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu);175extern bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu);
173extern void fpu_free_guest_fpstate(struct fpu_guest *gfpu);176extern void fpu_free_guest_fpstate(struct fpu_guest *gfpu);
March/x86/include/asm/iommu.h+1-0
@@ -10,6 +10,7 @@ extern int force_iommu, no_iommu;
10extern int iommu_detected;10extern int iommu_detected;
11extern int iommu_merge;11extern int iommu_merge;
12extern int panic_on_overflow;12extern int panic_on_overflow;
13+extern bool amd_iommu_snp_en;
13 14 
14#ifdef CONFIG_SWIOTLB15#ifdef CONFIG_SWIOTLB
15extern bool x86_swiotlb_enable;16extern bool x86_swiotlb_enable;
March/x86/include/asm/kvm-x86-ops.h+5-0
@@ -120,6 +120,7 @@ KVM_X86_OP(enter_smm)
120KVM_X86_OP(leave_smm)120KVM_X86_OP(leave_smm)
121KVM_X86_OP(enable_smi_window)121KVM_X86_OP(enable_smi_window)
122#endif122#endif
123+KVM_X86_OP_OPTIONAL(dev_get_attr)
123KVM_X86_OP_OPTIONAL(mem_enc_ioctl)124KVM_X86_OP_OPTIONAL(mem_enc_ioctl)
124KVM_X86_OP_OPTIONAL(mem_enc_register_region)125KVM_X86_OP_OPTIONAL(mem_enc_register_region)
125KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)126KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)
@@ -141,6 +142,10 @@ KVM_X86_OP_OPTIONAL(control_pre_system_reset)
141KVM_X86_OP_OPTIONAL(control_post_system_reset)142KVM_X86_OP_OPTIONAL(control_post_system_reset)
142KVM_X86_OP_OPTIONAL(get_hygon_coco_extension)143KVM_X86_OP_OPTIONAL(get_hygon_coco_extension)
143KVM_X86_OP_OPTIONAL(enable_hygon_coco_extension)144KVM_X86_OP_OPTIONAL(enable_hygon_coco_extension)
145+KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
146+KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
147+KVM_X86_OP_OPTIONAL_RET0(private_max_mapping_level)
148+KVM_X86_OP_OPTIONAL(gmem_invalidate)
144 149 
145#undef KVM_X86_OP150#undef KVM_X86_OP
146#undef KVM_X86_OP_OPTIONAL151#undef KVM_X86_OP_OPTIONAL
March/x86/include/asm/kvm_host.h+61-24
@@ -113,6 +113,8 @@
113 KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)113 KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
114#define KVM_REQ_HV_TLB_FLUSH \114#define KVM_REQ_HV_TLB_FLUSH \
115 KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)115 KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
116+#define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \
117+ KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT)
116 118 
117#define CR0_RESERVED_BITS \119#define CR0_RESERVED_BITS \
118 (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \120 (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
@@ -203,6 +205,7 @@ enum exit_fastpath_completion {
203 EXIT_FASTPATH_NONE,205 EXIT_FASTPATH_NONE,
204 EXIT_FASTPATH_REENTER_GUEST,206 EXIT_FASTPATH_REENTER_GUEST,
205 EXIT_FASTPATH_EXIT_HANDLED,207 EXIT_FASTPATH_EXIT_HANDLED,
208+ EXIT_FASTPATH_EXIT_USERSPACE,
206};209};
207typedef enum exit_fastpath_completion fastpath_t;210typedef enum exit_fastpath_completion fastpath_t;
208 211 
@@ -246,28 +249,31 @@ enum x86_intercept_stage;
246 KVM_GUESTDBG_INJECT_DB | \249 KVM_GUESTDBG_INJECT_DB | \
247 KVM_GUESTDBG_BLOCKIRQ)250 KVM_GUESTDBG_BLOCKIRQ)
248 251 
252+#define PFERR_PRESENT_MASK BIT(0)
253+#define PFERR_WRITE_MASK BIT(1)
254+#define PFERR_USER_MASK BIT(2)
255+#define PFERR_RSVD_MASK BIT(3)
256+#define PFERR_FETCH_MASK BIT(4)
257+#define PFERR_PK_MASK BIT(5)
258+#define PFERR_SGX_MASK BIT(15)
259+#define PFERR_GUEST_RMP_MASK BIT_ULL(31)
260+#define PFERR_GUEST_FINAL_MASK BIT_ULL(32)
261+#define PFERR_GUEST_PAGE_MASK BIT_ULL(33)
262+#define PFERR_GUEST_ENC_MASK BIT_ULL(34)
263+#define PFERR_GUEST_SIZEM_MASK BIT_ULL(35)
264+#define PFERR_GUEST_VMPL_MASK BIT_ULL(36)
249 265 
250-#define PFERR_PRESENT_BIT 0266+/*
251-#define PFERR_WRITE_BIT 1267+ * IMPLICIT_ACCESS is a KVM-defined flag used to correctly perform SMAP checks
252-#define PFERR_USER_BIT 2268+ * when emulating instructions that triggers implicit access.
253-#define PFERR_RSVD_BIT 3269+ */
254-#define PFERR_FETCH_BIT 4270+#define PFERR_IMPLICIT_ACCESS BIT_ULL(48)
255-#define PFERR_PK_BIT 5271+/*
256-#define PFERR_SGX_BIT 15272+ * PRIVATE_ACCESS is a KVM-defined flag us to indicate that a fault occurred
257-#define PFERR_GUEST_FINAL_BIT 32273+ * when the guest was accessing private memory.
258-#define PFERR_GUEST_PAGE_BIT 33274+ */
259-#define PFERR_IMPLICIT_ACCESS_BIT 48275+#define PFERR_PRIVATE_ACCESS BIT_ULL(49)
260- 276+#define PFERR_SYNTHETIC_MASK (PFERR_IMPLICIT_ACCESS | PFERR_PRIVATE_ACCESS)
261-#define PFERR_PRESENT_MASK BIT(PFERR_PRESENT_BIT)
262-#define PFERR_WRITE_MASK BIT(PFERR_WRITE_BIT)
263-#define PFERR_USER_MASK BIT(PFERR_USER_BIT)
264-#define PFERR_RSVD_MASK BIT(PFERR_RSVD_BIT)
265-#define PFERR_FETCH_MASK BIT(PFERR_FETCH_BIT)
266-#define PFERR_PK_MASK BIT(PFERR_PK_BIT)
267-#define PFERR_SGX_MASK BIT(PFERR_SGX_BIT)
268-#define PFERR_GUEST_FINAL_MASK BIT_ULL(PFERR_GUEST_FINAL_BIT)
269-#define PFERR_GUEST_PAGE_MASK BIT_ULL(PFERR_GUEST_PAGE_BIT)
270-#define PFERR_IMPLICIT_ACCESS BIT_ULL(PFERR_IMPLICIT_ACCESS_BIT)
271 277 
272#define PFERR_NESTED_GUEST_PAGE (PFERR_GUEST_PAGE_MASK | \278#define PFERR_NESTED_GUEST_PAGE (PFERR_GUEST_PAGE_MASK | \
273 PFERR_WRITE_MASK | \279 PFERR_WRITE_MASK | \
@@ -1249,6 +1255,21 @@ enum kvm_apicv_inhibit {
1249 APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED,1255 APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED,
1250};1256};
1251 1257 
1258+/*
1259+ * Per-VM x86 fields introduced after the OLK-6.6 KABI baseline.
1260+ *
1261+ * To avoid changing the layout of struct kvm_arch (which is embedded in
1262+ * struct kvm and whose offsets are part of the published kABI), new x86
1263+ * per-VM fields are placed here instead and reached through the
1264+ * arch_ext member that is appended at the tail of struct kvm.
1265+ */
1266+struct kvm_arch_ext {
1267+ u8 vm_type;
1268+ bool has_private_mem;
1269+ bool has_protected_state;
1270+ bool pre_fault_allowed;
1271+};
1272+ 
1252struct kvm_arch {1273struct kvm_arch {
1253 unsigned long n_used_mmu_pages;1274 unsigned long n_used_mmu_pages;
1254 unsigned long n_requested_mmu_pages;1275 unsigned long n_requested_mmu_pages;
@@ -1753,6 +1774,7 @@ struct kvm_x86_ops {
1753 void (*enable_smi_window)(struct kvm_vcpu *vcpu);1774 void (*enable_smi_window)(struct kvm_vcpu *vcpu);
1754#endif1775#endif
1755 1776 
1777+ int (*dev_get_attr)(u32 group, u64 attr, u64 *val);
1756 int (*mem_enc_ioctl)(struct kvm *kvm, void __user *argp);1778 int (*mem_enc_ioctl)(struct kvm *kvm, void __user *argp);
1757 int (*mem_enc_register_region)(struct kvm *kvm, struct kvm_enc_region *argp);1779 int (*mem_enc_register_region)(struct kvm *kvm, struct kvm_enc_region *argp);
1758 int (*mem_enc_unregister_region)(struct kvm *kvm, struct kvm_enc_region *argp);1780 int (*mem_enc_unregister_region)(struct kvm *kvm, struct kvm_enc_region *argp);
@@ -1789,6 +1811,10 @@ struct kvm_x86_ops {
1789 int (*control_post_system_reset)(struct kvm *kvm);1811 int (*control_post_system_reset)(struct kvm *kvm);
1790 int (*get_hygon_coco_extension)(struct kvm *kvm);1812 int (*get_hygon_coco_extension)(struct kvm *kvm);
1791 int (*enable_hygon_coco_extension)(struct kvm *kvm, u32 arg);1813 int (*enable_hygon_coco_extension)(struct kvm *kvm, u32 arg);
1814+ void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
1815+ int (*gmem_prepare)(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
1816+ void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
1817+ int (*private_max_mapping_level)(struct kvm *kvm, kvm_pfn_t pfn);
1792};1818};
1793 1819 
1794struct kvm_x86_nested_ops {1820struct kvm_x86_nested_ops {
@@ -1826,6 +1852,7 @@ struct kvm_arch_async_pf {
1826 gfn_t gfn;1852 gfn_t gfn;
1827 unsigned long cr3;1853 unsigned long cr3;
1828 bool direct_map;1854 bool direct_map;
1855+ u64 error_code;
1829};1856};
1830 1857 
1831extern u32 __read_mostly kvm_nr_uret_msrs;1858extern u32 __read_mostly kvm_nr_uret_msrs;
@@ -1897,6 +1924,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
1897 const struct kvm_memory_slot *memslot);1924 const struct kvm_memory_slot *memslot);
1898void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen);1925void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen);
1899void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages);1926void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages);
1927+void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end);
1900 1928 
1901int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);1929int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);
1902 1930 
@@ -2126,6 +2154,15 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd);
2126void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level,2154void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level,
2127 int tdp_max_root_level, int tdp_huge_page_level);2155 int tdp_max_root_level, int tdp_huge_page_level);
2128 2156 
2157+ 
2158+#ifdef CONFIG_KVM_PRIVATE_MEM
2159+#define kvm_arch_has_private_mem(kvm) ((kvm)->arch_ext.has_private_mem)
2160+#else
2161+#define kvm_arch_has_private_mem(kvm) false
2162+#endif
2163+ 
2164+#define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch_ext.has_protected_state)
2165+ 
2129static inline u16 kvm_read_ldt(void)2166static inline u16 kvm_read_ldt(void)
2130{2167{
2131 u16 ldt;2168 u16 ldt;
@@ -2173,15 +2210,15 @@ enum {
2173#define HF_SMM_MASK (1 << 1)2210#define HF_SMM_MASK (1 << 1)
2174#define HF_SMM_INSIDE_NMI_MASK (1 << 2)2211#define HF_SMM_INSIDE_NMI_MASK (1 << 2)
2175 2212 
2176-# define KVM_ADDRESS_SPACE_NUM 22213+# define KVM_MAX_NR_ADDRESS_SPACES 2
2214+/* SMM is currently unsupported for guests with private memory. */
2215+# define kvm_arch_nr_memslot_as_ids(kvm) (kvm_arch_has_private_mem(kvm) ? 1 : 2)
2177# define kvm_arch_vcpu_memslots_id(vcpu) ((vcpu)->arch.hflags & HF_SMM_MASK ? 1 : 0)2216# define kvm_arch_vcpu_memslots_id(vcpu) ((vcpu)->arch.hflags & HF_SMM_MASK ? 1 : 0)
2178# define kvm_memslots_for_spte_role(kvm, role) __kvm_memslots(kvm, (role).smm)2217# define kvm_memslots_for_spte_role(kvm, role) __kvm_memslots(kvm, (role).smm)
2179#else2218#else
2180# define kvm_memslots_for_spte_role(kvm, role) __kvm_memslots(kvm, 0)2219# define kvm_memslots_for_spte_role(kvm, role) __kvm_memslots(kvm, 0)
2181#endif2220#endif
2182 2221 
2183-#define KVM_ARCH_WANT_MMU_NOTIFIER
2184- 
2185int kvm_cpu_has_injectable_intr(struct kvm_vcpu *v);2222int kvm_cpu_has_injectable_intr(struct kvm_vcpu *v);
2186int kvm_cpu_has_interrupt(struct kvm_vcpu *vcpu);2223int kvm_cpu_has_interrupt(struct kvm_vcpu *vcpu);
2187int kvm_cpu_has_extint(struct kvm_vcpu *v);2224int kvm_cpu_has_extint(struct kvm_vcpu *v);
March/x86/include/asm/mem_encrypt.h+2-2
@@ -20,8 +20,10 @@ struct boot_params;
20 20 
21#ifdef CONFIG_X86_MEM_ENCRYPT21#ifdef CONFIG_X86_MEM_ENCRYPT
22void __init mem_encrypt_init(void);22void __init mem_encrypt_init(void);
23+void __init mem_encrypt_setup_arch(void);
23#else24#else
24static inline void mem_encrypt_init(void) { }25static inline void mem_encrypt_init(void) { }
26+static inline void __init mem_encrypt_setup_arch(void) { }
25#endif27#endif
26 28 
27#ifdef CONFIG_AMD_MEM_ENCRYPT29#ifdef CONFIG_AMD_MEM_ENCRYPT
@@ -44,7 +46,6 @@ void __init sme_map_bootdata(char *real_mode_data);
44void __init sme_unmap_bootdata(char *real_mode_data);46void __init sme_unmap_bootdata(char *real_mode_data);
45 47 
46void __init sme_early_init(void);48void __init sme_early_init(void);
47-void __init sev_setup_arch(void);
48 49 
49void sme_encrypt_kernel(struct boot_params *bp);50void sme_encrypt_kernel(struct boot_params *bp);
50void sme_enable(struct boot_params *bp);51void sme_enable(struct boot_params *bp);
@@ -79,7 +80,6 @@ static inline void __init sme_map_bootdata(char *real_mode_data) { }
79static inline void __init sme_unmap_bootdata(char *real_mode_data) { }80static inline void __init sme_unmap_bootdata(char *real_mode_data) { }
80 81 
81static inline void __init sme_early_init(void) { }82static inline void __init sme_early_init(void) { }
82-static inline void __init sev_setup_arch(void) { }
83 83 
84static inline void sme_encrypt_kernel(struct boot_params *bp) { }84static inline void sme_encrypt_kernel(struct boot_params *bp) { }
85static inline void sme_enable(struct boot_params *bp) { }85static inline void sme_enable(struct boot_params *bp) { }
March/x86/include/asm/msr-index.h+46-23
@@ -654,39 +654,55 @@
654#define MSR_AMD64_IBS_REG_COUNT_MAX 8 /* includes MSR_AMD64_IBSBRTARGET */654#define MSR_AMD64_IBS_REG_COUNT_MAX 8 /* includes MSR_AMD64_IBSBRTARGET */
655#define MSR_AMD64_SVM_AVIC_DOORBELL 0xc001011b655#define MSR_AMD64_SVM_AVIC_DOORBELL 0xc001011b
656#define MSR_AMD64_VM_PAGE_FLUSH 0xc001011e656#define MSR_AMD64_VM_PAGE_FLUSH 0xc001011e
657+#define MSR_AMD64_VIRT_SPEC_CTRL 0xc001011f
657#define MSR_AMD64_SEV_ES_GHCB 0xc0010130658#define MSR_AMD64_SEV_ES_GHCB 0xc0010130
658#define MSR_AMD64_SEV 0xc0010131659#define MSR_AMD64_SEV 0xc0010131
659#define MSR_AMD64_SEV_ENABLED_BIT 0660#define MSR_AMD64_SEV_ENABLED_BIT 0
660-#define MSR_AMD64_SEV_ES_ENABLED_BIT 1
661-#define MSR_AMD64_SEV_SNP_ENABLED_BIT 2
662#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)661#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
662+#define MSR_AMD64_SEV_ES_ENABLED_BIT 1
663#define MSR_AMD64_SEV_ES_ENABLED BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)663#define MSR_AMD64_SEV_ES_ENABLED BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)
664+#define MSR_AMD64_SEV_SNP_ENABLED_BIT 2
664#define MSR_AMD64_SEV_SNP_ENABLED BIT_ULL(MSR_AMD64_SEV_SNP_ENABLED_BIT)665#define MSR_AMD64_SEV_SNP_ENABLED BIT_ULL(MSR_AMD64_SEV_SNP_ENABLED_BIT)
665- 666+#define MSR_AMD64_SNP_VTOM_BIT 3
666-/* SNP feature bits enabled by the hypervisor */667+#define MSR_AMD64_SNP_VTOM BIT_ULL(MSR_AMD64_SNP_VTOM_BIT)
667-#define MSR_AMD64_SNP_VTOM BIT_ULL(3)668+#define MSR_AMD64_SNP_REFLECT_VC_BIT 4
668-#define MSR_AMD64_SNP_REFLECT_VC BIT_ULL(4)669+#define MSR_AMD64_SNP_REFLECT_VC BIT_ULL(MSR_AMD64_SNP_REFLECT_VC_BIT)
669-#define MSR_AMD64_SNP_RESTRICTED_INJ BIT_ULL(5)670+#define MSR_AMD64_SNP_RESTRICTED_INJ_BIT 5
670-#define MSR_AMD64_SNP_ALT_INJ BIT_ULL(6)671+#define MSR_AMD64_SNP_RESTRICTED_INJ BIT_ULL(MSR_AMD64_SNP_RESTRICTED_INJ_BIT)
671-#define MSR_AMD64_SNP_DEBUG_SWAP BIT_ULL(7)672+#define MSR_AMD64_SNP_ALT_INJ_BIT 6
672-#define MSR_AMD64_SNP_PREVENT_HOST_IBS BIT_ULL(8)673+#define MSR_AMD64_SNP_ALT_INJ BIT_ULL(MSR_AMD64_SNP_ALT_INJ_BIT)
673-#define MSR_AMD64_SNP_BTB_ISOLATION BIT_ULL(9)674+#define MSR_AMD64_SNP_DEBUG_SWAP_BIT 7
674-#define MSR_AMD64_SNP_VMPL_SSS BIT_ULL(10)675+#define MSR_AMD64_SNP_DEBUG_SWAP BIT_ULL(MSR_AMD64_SNP_DEBUG_SWAP_BIT)
675-#define MSR_AMD64_SNP_SECURE_TSC BIT_ULL(11)676+#define MSR_AMD64_SNP_PREVENT_HOST_IBS_BIT 8
676-#define MSR_AMD64_SNP_VMGEXIT_PARAM BIT_ULL(12)677+#define MSR_AMD64_SNP_PREVENT_HOST_IBS BIT_ULL(MSR_AMD64_SNP_PREVENT_HOST_IBS_BIT)
677-#define MSR_AMD64_SNP_IBS_VIRT BIT_ULL(14)678+#define MSR_AMD64_SNP_BTB_ISOLATION_BIT 9
678-#define MSR_AMD64_SNP_VMSA_REG_PROTECTION BIT_ULL(16)679+#define MSR_AMD64_SNP_BTB_ISOLATION BIT_ULL(MSR_AMD64_SNP_BTB_ISOLATION_BIT)
679-#define MSR_AMD64_SNP_SMT_PROTECTION BIT_ULL(17)680+#define MSR_AMD64_SNP_VMPL_SSS_BIT 10
681+#define MSR_AMD64_SNP_VMPL_SSS BIT_ULL(MSR_AMD64_SNP_VMPL_SSS_BIT)
682+#define MSR_AMD64_SNP_SECURE_TSC_BIT 11
683+#define MSR_AMD64_SNP_SECURE_TSC BIT_ULL(MSR_AMD64_SNP_SECURE_TSC_BIT)
684+#define MSR_AMD64_SNP_VMGEXIT_PARAM_BIT 12
685+#define MSR_AMD64_SNP_VMGEXIT_PARAM BIT_ULL(MSR_AMD64_SNP_VMGEXIT_PARAM_BIT)
686+#define MSR_AMD64_SNP_RESERVED_BIT13 BIT_ULL(13)
687+#define MSR_AMD64_SNP_IBS_VIRT_BIT 14
688+#define MSR_AMD64_SNP_IBS_VIRT BIT_ULL(MSR_AMD64_SNP_IBS_VIRT_BIT)
689+#define MSR_AMD64_SNP_RESERVED_BIT15 BIT_ULL(15)
690+#define MSR_AMD64_SNP_VMSA_REG_PROT_BIT 16
691+#define MSR_AMD64_SNP_VMSA_REG_PROT BIT_ULL(MSR_AMD64_SNP_VMSA_REG_PROT_BIT)
692+#define MSR_AMD64_SNP_SMT_PROT_BIT 17
693+#define MSR_AMD64_SNP_SMT_PROT BIT_ULL(MSR_AMD64_SNP_SMT_PROT_BIT)
694+#define MSR_AMD64_SNP_RESV_BIT 18
680#define MSR_AMD64_SNP_IBPB_ON_ENTRY_BIT 23695#define MSR_AMD64_SNP_IBPB_ON_ENTRY_BIT 23
681#define MSR_AMD64_SNP_IBPB_ON_ENTRY BIT_ULL(MSR_AMD64_SNP_IBPB_ON_ENTRY_BIT)696#define MSR_AMD64_SNP_IBPB_ON_ENTRY BIT_ULL(MSR_AMD64_SNP_IBPB_ON_ENTRY_BIT)
682- 
683-/* SNP feature bits reserved for future use. */
684-#define MSR_AMD64_SNP_RESERVED_BIT13 BIT_ULL(13)
685-#define MSR_AMD64_SNP_RESERVED_BIT15 BIT_ULL(15)
686#define MSR_AMD64_SNP_RESERVED_BITS18_22 GENMASK_ULL(22, 18)697#define MSR_AMD64_SNP_RESERVED_BITS18_22 GENMASK_ULL(22, 18)
687#define MSR_AMD64_SNP_RESERVED_MASK GENMASK_ULL(63, 24)698#define MSR_AMD64_SNP_RESERVED_MASK GENMASK_ULL(63, 24)
688 699 
689-#define MSR_AMD64_VIRT_SPEC_CTRL 0xc001011f700+#define MSR_AMD64_RMP_BASE 0xc0010132
701+#define MSR_AMD64_RMP_END 0xc0010133
702+#define MSR_AMD64_RMP_CFG 0xc0010136
703+#define MSR_AMD64_SEG_RMP_ENABLED_BIT 0
704+#define MSR_AMD64_SEG_RMP_ENABLED BIT_ULL(MSR_AMD64_SEG_RMP_ENABLED_BIT)
705+#define MSR_AMD64_RMP_SEGMENT_SHIFT(x) (((x) & GENMASK_ULL(13, 8)) >> 8)
690 706 
691/* AMD Collaborative Processor Performance Control MSRs */707/* AMD Collaborative Processor Performance Control MSRs */
692#define MSR_AMD_CPPC_CAP1 0xc00102b0708#define MSR_AMD_CPPC_CAP1 0xc00102b0
@@ -775,8 +791,15 @@
775#define MSR_K8_TOP_MEM1 0xc001001a791#define MSR_K8_TOP_MEM1 0xc001001a
776#define MSR_K8_TOP_MEM2 0xc001001d792#define MSR_K8_TOP_MEM2 0xc001001d
777#define MSR_AMD64_SYSCFG 0xc0010010793#define MSR_AMD64_SYSCFG 0xc0010010
778-#define MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT 23794+#define MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT 23
779#define MSR_AMD64_SYSCFG_MEM_ENCRYPT BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)795#define MSR_AMD64_SYSCFG_MEM_ENCRYPT BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
796+#define MSR_AMD64_SYSCFG_SNP_EN_BIT 24
797+#define MSR_AMD64_SYSCFG_SNP_EN BIT_ULL(MSR_AMD64_SYSCFG_SNP_EN_BIT)
798+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT 25
799+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN BIT_ULL(MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT)
800+#define MSR_AMD64_SYSCFG_MFDM_BIT 19
801+#define MSR_AMD64_SYSCFG_MFDM BIT_ULL(MSR_AMD64_SYSCFG_MFDM_BIT)
802+ 
780#define MSR_K8_INT_PENDING_MSG 0xc0010055803#define MSR_K8_INT_PENDING_MSG 0xc0010055
781/* C1E active bits in int pending message */804/* C1E active bits in int pending message */
782#define K8_INTP_C1E_ACTIVE_MASK 0x18000000805#define K8_INTP_C1E_ACTIVE_MASK 0x18000000
March/x86/include/asm/sev-common.h+41-4
@@ -54,8 +54,18 @@
54 (((unsigned long)fn) << 32))54 (((unsigned long)fn) << 32))
55 55 
56/* AP Reset Hold */56/* AP Reset Hold */
57-#define GHCB_MSR_AP_RESET_HOLD_REQ 0x00657+#define GHCB_MSR_AP_RESET_HOLD_REQ 0x006
58-#define GHCB_MSR_AP_RESET_HOLD_RESP 0x00758+#define GHCB_MSR_AP_RESET_HOLD_RESP 0x007
59+#define GHCB_MSR_AP_RESET_HOLD_RESULT_POS 12
60+#define GHCB_MSR_AP_RESET_HOLD_RESULT_MASK GENMASK_ULL(51, 0)
61+ 
62+/* Preferred GHCB GPA Request */
63+#define GHCB_MSR_PREF_GPA_REQ 0x010
64+#define GHCB_MSR_GPA_VALUE_POS 12
65+#define GHCB_MSR_GPA_VALUE_MASK GENMASK_ULL(51, 0)
66+ 
67+#define GHCB_MSR_PREF_GPA_RESP 0x011
68+#define GHCB_MSR_PREF_GPA_NONE 0xfffffffffffff
59 69 
60/* GHCB GPA Register */70/* GHCB GPA Register */
61#define GHCB_MSR_REG_GPA_REQ 0x01271#define GHCB_MSR_REG_GPA_REQ 0x012
@@ -91,14 +101,22 @@ enum psc_op {
91 /* GHCBData[11:0] */ \101 /* GHCBData[11:0] */ \
92 GHCB_MSR_PSC_REQ)102 GHCB_MSR_PSC_REQ)
93 103 
104+#define GHCB_MSR_PSC_REQ_TO_GFN(msr) (((msr) & GENMASK_ULL(51, 12)) >> 12)
105+#define GHCB_MSR_PSC_REQ_TO_OP(msr) (((msr) & GENMASK_ULL(55, 52)) >> 52)
106+ 
94#define GHCB_MSR_PSC_RESP 0x015107#define GHCB_MSR_PSC_RESP 0x015
95#define GHCB_MSR_PSC_RESP_VAL(val) \108#define GHCB_MSR_PSC_RESP_VAL(val) \
96 /* GHCBData[63:32] */ \109 /* GHCBData[63:32] */ \
97 (((u64)(val) & GENMASK_ULL(63, 32)) >> 32)110 (((u64)(val) & GENMASK_ULL(63, 32)) >> 32)
98 111 
112+/* Set highest bit as a generic error response */
113+#define GHCB_MSR_PSC_RESP_ERROR (BIT_ULL(63) | GHCB_MSR_PSC_RESP)
114+ 
99/* GHCB Hypervisor Feature Request/Response */115/* GHCB Hypervisor Feature Request/Response */
100#define GHCB_MSR_HV_FT_REQ 0x080116#define GHCB_MSR_HV_FT_REQ 0x080
101#define GHCB_MSR_HV_FT_RESP 0x081117#define GHCB_MSR_HV_FT_RESP 0x081
118+#define GHCB_MSR_HV_FT_POS 12
119+#define GHCB_MSR_HV_FT_MASK GENMASK_ULL(51, 0)
102#define GHCB_MSR_HV_FT_RESP_VAL(v) \120#define GHCB_MSR_HV_FT_RESP_VAL(v) \
103 /* GHCBData[63:12] */ \121 /* GHCBData[63:12] */ \
104 (((u64)(v) & GENMASK_ULL(63, 12)) >> 12)122 (((u64)(v) & GENMASK_ULL(63, 12)) >> 12)
@@ -111,8 +129,19 @@ enum psc_op {
111 * The VMGEXIT_PSC_MAX_ENTRY determines the size of the PSC structure, which129 * The VMGEXIT_PSC_MAX_ENTRY determines the size of the PSC structure, which
112 * is a local stack variable in set_pages_state(). Do not increase this value130 * is a local stack variable in set_pages_state(). Do not increase this value
113 * without evaluating the impact to stack usage.131 * without evaluating the impact to stack usage.
132+ *
133+ * Use VMGEXIT_PSC_MAX_COUNT in cases where the actual GHCB-defined max value
134+ * is needed, such as when processing GHCB requests on the hypervisor side.
114 */135 */
115#define VMGEXIT_PSC_MAX_ENTRY 64136#define VMGEXIT_PSC_MAX_ENTRY 64
137+#define VMGEXIT_PSC_MAX_COUNT 253
138+ 
139+#define VMGEXIT_PSC_ERROR_GENERIC (0x100UL << 32)
140+#define VMGEXIT_PSC_ERROR_INVALID_HDR ((1UL << 32) | 1)
141+#define VMGEXIT_PSC_ERROR_INVALID_ENTRY ((1UL << 32) | 2)
142+ 
143+#define VMGEXIT_PSC_OP_PRIVATE 1
144+#define VMGEXIT_PSC_OP_SHARED 2
116 145 
117struct psc_hdr {146struct psc_hdr {
118 u16 cur_entry;147 u16 cur_entry;
@@ -163,8 +192,16 @@ struct snp_psc_desc {
163#define GHCB_RESP_CODE(v) ((v) & GHCB_MSR_INFO_MASK)192#define GHCB_RESP_CODE(v) ((v) & GHCB_MSR_INFO_MASK)
164 193 
165/*194/*
166- * Error codes related to GHCB input that can be communicated back to the guest195+ * GHCB-defined return codes that are communicated back to the guest via
167- * by setting the lower 32-bits of the GHCB SW_EXITINFO1 field to 2.196+ * SW_EXITINFO1.
197+ */
198+#define GHCB_HV_RESP_NO_ACTION 0
199+#define GHCB_HV_RESP_ISSUE_EXCEPTION 1
200+#define GHCB_HV_RESP_MALFORMED_INPUT 2
201+ 
202+/*
203+ * GHCB-defined sub-error codes for malformed input (see above) that are
204+ * communicated back to the guest via SW_EXITINFO2[31:0].
168 */205 */
169#define GHCB_ERR_NOT_REGISTERED 1206#define GHCB_ERR_NOT_REGISTERED 1
170#define GHCB_ERR_INVALID_USAGE 2207#define GHCB_ERR_INVALID_USAGE 2
March/x86/include/asm/sev.h+96-1
@@ -87,9 +87,26 @@ extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
87/* Software defined (when rFlags.CF = 1) */87/* Software defined (when rFlags.CF = 1) */
88#define PVALIDATE_FAIL_NOUPDATE 25588#define PVALIDATE_FAIL_NOUPDATE 255
89 89 
90+/* RMUPDATE detected 4K page and 2MB page overlap. */
91+#define RMPUPDATE_FAIL_OVERLAP 4
92+ 
93+/* PSMASH failed due to concurrent access by another CPU */
94+#define PSMASH_FAIL_INUSE 3
95+ 
90/* RMP page size */96/* RMP page size */
91#define RMP_PG_SIZE_4K 097#define RMP_PG_SIZE_4K 0
92#define RMP_PG_SIZE_2M 198#define RMP_PG_SIZE_2M 1
99+#define RMP_TO_PG_LEVEL(level) (((level) == RMP_PG_SIZE_4K) ? PG_LEVEL_4K : PG_LEVEL_2M)
100+#define PG_LEVEL_TO_RMP(level) (((level) == PG_LEVEL_4K) ? RMP_PG_SIZE_4K : RMP_PG_SIZE_2M)
101+ 
102+struct rmp_state {
103+ u64 gpa;
104+ u8 assigned;
105+ u8 pagesize;
106+ u8 immutable;
107+ u8 rsvd;
108+ u32 asid;
109+} __packed;
93 110 
94#define RMPADJUST_VMSA_PAGE_BIT BIT(16)111#define RMPADJUST_VMSA_PAGE_BIT BIT(16)
95 112 
@@ -101,6 +118,54 @@ struct snp_req_data {
101 unsigned int data_npages;118 unsigned int data_npages;
102};119};
103 120 
121+#define MAX_AUTHTAG_LEN 32
122+ 
123+/* See SNP spec SNP_GUEST_REQUEST section for the structure */
124+enum msg_type {
125+ SNP_MSG_TYPE_INVALID = 0,
126+ SNP_MSG_CPUID_REQ,
127+ SNP_MSG_CPUID_RSP,
128+ SNP_MSG_KEY_REQ,
129+ SNP_MSG_KEY_RSP,
130+ SNP_MSG_REPORT_REQ,
131+ SNP_MSG_REPORT_RSP,
132+ SNP_MSG_EXPORT_REQ,
133+ SNP_MSG_EXPORT_RSP,
134+ SNP_MSG_IMPORT_REQ,
135+ SNP_MSG_IMPORT_RSP,
136+ SNP_MSG_ABSORB_REQ,
137+ SNP_MSG_ABSORB_RSP,
138+ SNP_MSG_VMRK_REQ,
139+ SNP_MSG_VMRK_RSP,
140+ 
141+ SNP_MSG_TYPE_MAX
142+};
143+ 
144+enum aead_algo {
145+ SNP_AEAD_INVALID,
146+ SNP_AEAD_AES_256_GCM,
147+};
148+ 
149+struct snp_guest_msg_hdr {
150+ u8 authtag[MAX_AUTHTAG_LEN];
151+ u64 msg_seqno;
152+ u8 rsvd1[8];
153+ u8 algo;
154+ u8 hdr_version;
155+ u16 hdr_sz;
156+ u8 msg_type;
157+ u8 msg_version;
158+ u16 msg_sz;
159+ u32 rsvd2;
160+ u8 msg_vmpck;
161+ u8 rsvd3[35];
162+} __packed;
163+ 
164+struct snp_guest_msg {
165+ struct snp_guest_msg_hdr hdr;
166+ u8 payload[4000];
167+} __packed;
168+ 
104struct sev_guest_platform_data {169struct sev_guest_platform_data {
105 u64 secrets_gpa;170 u64 secrets_gpa;
106};171};
@@ -125,7 +190,7 @@ struct secrets_os_area {
125#define VMPCK_KEY_LEN 32190#define VMPCK_KEY_LEN 32
126 191 
127/* See the SNP spec version 0.9 for secrets page format */192/* See the SNP spec version 0.9 for secrets page format */
128-struct snp_secrets_page_layout {193+struct snp_secrets_page {
129 u32 version;194 u32 version;
130 u32 imien : 1,195 u32 imien : 1,
131 rsvd1 : 31;196 rsvd1 : 31;
@@ -213,6 +278,7 @@ int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, struct sn
213void snp_accept_memory(phys_addr_t start, phys_addr_t end);278void snp_accept_memory(phys_addr_t start, phys_addr_t end);
214u64 snp_get_unsupported_features(u64 status);279u64 snp_get_unsupported_features(u64 status);
215u64 sev_get_status(void);280u64 sev_get_status(void);
281+void sev_show_status(void);
216#else282#else
217static inline void sev_es_ist_enter(struct pt_regs *regs) { }283static inline void sev_es_ist_enter(struct pt_regs *regs) { }
218static inline void sev_es_ist_exit(void) { }284static inline void sev_es_ist_exit(void) { }
@@ -241,6 +307,35 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in
241static inline void snp_accept_memory(phys_addr_t start, phys_addr_t end) { }307static inline void snp_accept_memory(phys_addr_t start, phys_addr_t end) { }
242static inline u64 snp_get_unsupported_features(u64 status) { return 0; }308static inline u64 snp_get_unsupported_features(u64 status) { return 0; }
243static inline u64 sev_get_status(void) { return 0; }309static inline u64 sev_get_status(void) { return 0; }
310+static inline void sev_show_status(void) { }
311+#endif
312+ 
313+#ifdef CONFIG_KVM_AMD_SEV
314+bool snp_probe_rmptable_info(void);
315+int snp_rmptable_init(void);
316+int snp_lookup_rmpentry(u64 pfn, bool *assigned, int *level);
317+void snp_dump_hva_rmpentry(unsigned long address);
318+int psmash(u64 pfn);
319+int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, u32 asid, bool immutable);
320+int rmp_make_shared(u64 pfn, enum pg_level level);
321+void snp_leak_pages(u64 pfn, unsigned int npages);
322+void kdump_sev_callback(void);
323+void snp_fixup_e820_tables(void);
324+#else
325+static inline bool snp_probe_rmptable_info(void) { return false; }
326+static inline int snp_rmptable_init(void) { return -ENOSYS; }
327+static inline int snp_lookup_rmpentry(u64 pfn, bool *assigned, int *level) { return -ENODEV; }
328+static inline void snp_dump_hva_rmpentry(unsigned long address) {}
329+static inline int psmash(u64 pfn) { return -ENODEV; }
330+static inline int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, u32 asid,
331+ bool immutable)
332+{
333+ return -ENODEV;
334+}
335+static inline int rmp_make_shared(u64 pfn, enum pg_level level) { return -ENODEV; }
336+static inline void snp_leak_pages(u64 pfn, unsigned int npages) {}
337+static inline void kdump_sev_callback(void) { }
338+static inline void snp_fixup_e820_tables(void) {}
244#endif339#endif
245 340 
246#endif341#endif
March/x86/include/asm/svm.h+4-1
@@ -286,7 +286,10 @@ static_assert((X2AVIC_MAX_PHYSICAL_ID & AVIC_PHYSICAL_MAX_INDEX_MASK) == X2AVIC_
286 286 
287#define AVIC_HPA_MASK ~((0xFFFULL << 52) | 0xFFF)287#define AVIC_HPA_MASK ~((0xFFFULL << 52) | 0xFFF)
288 288 
289-#define SVM_SEV_FEAT_DEBUG_SWAP BIT(5)289+#define SVM_SEV_FEAT_SNP_ACTIVE BIT(0)
290+#define SVM_SEV_FEAT_RESTRICTED_INJECTION BIT(3)
291+#define SVM_SEV_FEAT_ALTERNATE_INJECTION BIT(4)
292+#define SVM_SEV_FEAT_DEBUG_SWAP BIT(5)
290 293 
291struct vmcb_seg {294struct vmcb_seg {
292 u16 selector;295 u16 selector;
March/x86/include/asm/trap_pf.h+12-8
@@ -2,6 +2,8 @@
2#ifndef _ASM_X86_TRAP_PF_H2#ifndef _ASM_X86_TRAP_PF_H
3#define _ASM_X86_TRAP_PF_H3#define _ASM_X86_TRAP_PF_H
4 4 
5+#include <linux/bits.h>
6+ 
5/*7/*
6 * Page fault error code bits:8 * Page fault error code bits:
7 *9 *
@@ -13,16 +15,18 @@
13 * bit 5 == 1: protection keys block access15 * bit 5 == 1: protection keys block access
14 * bit 6 == 1: shadow stack access fault16 * bit 6 == 1: shadow stack access fault
15 * bit 15 == 1: SGX MMU page-fault17 * bit 15 == 1: SGX MMU page-fault
18+ * bit 31 == 1: fault was due to RMP violation
16 */19 */
17enum x86_pf_error_code {20enum x86_pf_error_code {
18- X86_PF_PROT = 1 << 0,21+ X86_PF_PROT = BIT(0),
19- X86_PF_WRITE = 1 << 1,22+ X86_PF_WRITE = BIT(1),
20- X86_PF_USER = 1 << 2,23+ X86_PF_USER = BIT(2),
21- X86_PF_RSVD = 1 << 3,24+ X86_PF_RSVD = BIT(3),
22- X86_PF_INSTR = 1 << 4,25+ X86_PF_INSTR = BIT(4),
23- X86_PF_PK = 1 << 5,26+ X86_PF_PK = BIT(5),
24- X86_PF_SHSTK = 1 << 6,27+ X86_PF_SHSTK = BIT(6),
25- X86_PF_SGX = 1 << 15,28+ X86_PF_SGX = BIT(15),
29+ X86_PF_RMP = BIT(31),
26};30};
27 31 
28#endif /* _ASM_X86_TRAP_PF_H */32#endif /* _ASM_X86_TRAP_PF_H */
March/x86/include/uapi/asm/kvm.h+356-2
@@ -460,8 +460,13 @@ struct kvm_sync_regs {
460 460 
461#define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE 0x00000001461#define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE 0x00000001
462 462 
463-/* attributes for system fd (group 0) */463+/* vendor-independent attributes for system fd (group 0) */
464-#define KVM_X86_XCOMP_GUEST_SUPP 0464+#define KVM_X86_GRP_SYSTEM 0
465+# define KVM_X86_XCOMP_GUEST_SUPP 0
466+ 
467+/* vendor-specific groups and attributes for system fd */
468+#define KVM_X86_GRP_SEV 1
469+# define KVM_X86_SEV_VMSA_FEATURES 0
465 470 
466struct kvm_vmx_nested_state_data {471struct kvm_vmx_nested_state_data {
467 __u8 vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE];472 __u8 vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE];
@@ -532,6 +537,349 @@ struct kvm_pmu_event_filter {
532#define KVM_PMU_EVENT_FLAG_MASKED_EVENTS BIT(0)537#define KVM_PMU_EVENT_FLAG_MASKED_EVENTS BIT(0)
533#define KVM_PMU_EVENT_FLAGS_VALID_MASK (KVM_PMU_EVENT_FLAG_MASKED_EVENTS)538#define KVM_PMU_EVENT_FLAGS_VALID_MASK (KVM_PMU_EVENT_FLAG_MASKED_EVENTS)
534 539 
540+/* for KVM_CAP_MCE */
541+struct kvm_x86_mce {
542+ __u64 status;
543+ __u64 addr;
544+ __u64 misc;
545+ __u64 mcg_status;
546+ __u8 bank;
547+ __u8 pad1[7];
548+ __u64 pad2[3];
549+};
550+ 
551+/* for KVM_CAP_XEN_HVM */
552+#define KVM_XEN_HVM_CONFIG_HYPERCALL_MSR (1 << 0)
553+#define KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL (1 << 1)
554+#define KVM_XEN_HVM_CONFIG_SHARED_INFO (1 << 2)
555+#define KVM_XEN_HVM_CONFIG_RUNSTATE (1 << 3)
556+#define KVM_XEN_HVM_CONFIG_EVTCHN_2LEVEL (1 << 4)
557+#define KVM_XEN_HVM_CONFIG_EVTCHN_SEND (1 << 5)
558+#define KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG (1 << 6)
559+#define KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE (1 << 7)
560+ 
561+struct kvm_xen_hvm_config {
562+ __u32 flags;
563+ __u32 msr;
564+ __u64 blob_addr_32;
565+ __u64 blob_addr_64;
566+ __u8 blob_size_32;
567+ __u8 blob_size_64;
568+ __u8 pad2[30];
569+};
570+ 
571+struct kvm_xen_hvm_attr {
572+ __u16 type;
573+ __u16 pad[3];
574+ union {
575+ __u8 long_mode;
576+ __u8 vector;
577+ __u8 runstate_update_flag;
578+ struct {
579+ __u64 gfn;
580+#define KVM_XEN_INVALID_GFN ((__u64)-1)
581+ } shared_info;
582+ struct {
583+ __u32 send_port;
584+ __u32 type; /* EVTCHNSTAT_ipi / EVTCHNSTAT_interdomain */
585+ __u32 flags;
586+#define KVM_XEN_EVTCHN_DEASSIGN (1 << 0)
587+#define KVM_XEN_EVTCHN_UPDATE (1 << 1)
588+#define KVM_XEN_EVTCHN_RESET (1 << 2)
589+ /*
590+ * Events sent by the guest are either looped back to
591+ * the guest itself (potentially on a different port#)
592+ * or signalled via an eventfd.
593+ */
594+ union {
595+ struct {
596+ __u32 port;
597+ __u32 vcpu;
598+ __u32 priority;
599+ } port;
600+ struct {
601+ __u32 port; /* Zero for eventfd */
602+ __s32 fd;
603+ } eventfd;
604+ __u32 padding[4];
605+ } deliver;
606+ } evtchn;
607+ __u32 xen_version;
608+ __u64 pad[8];
609+ } u;
610+};
611+ 
612+ 
613+/* Available with KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_SHARED_INFO */
614+#define KVM_XEN_ATTR_TYPE_LONG_MODE 0x0
615+#define KVM_XEN_ATTR_TYPE_SHARED_INFO 0x1
616+#define KVM_XEN_ATTR_TYPE_UPCALL_VECTOR 0x2
617+/* Available with KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_EVTCHN_SEND */
618+#define KVM_XEN_ATTR_TYPE_EVTCHN 0x3
619+#define KVM_XEN_ATTR_TYPE_XEN_VERSION 0x4
620+/* Available with KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG */
621+#define KVM_XEN_ATTR_TYPE_RUNSTATE_UPDATE_FLAG 0x5
622+ 
623+struct kvm_xen_vcpu_attr {
624+ __u16 type;
625+ __u16 pad[3];
626+ union {
627+ __u64 gpa;
628+#define KVM_XEN_INVALID_GPA ((__u64)-1)
629+ __u64 pad[8];
630+ struct {
631+ __u64 state;
632+ __u64 state_entry_time;
633+ __u64 time_running;
634+ __u64 time_runnable;
635+ __u64 time_blocked;
636+ __u64 time_offline;
637+ } runstate;
638+ __u32 vcpu_id;
639+ struct {
640+ __u32 port;
641+ __u32 priority;
642+ __u64 expires_ns;
643+ } timer;
644+ __u8 vector;
645+ } u;
646+};
647+ 
648+/* Available with KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_SHARED_INFO */
649+#define KVM_XEN_VCPU_ATTR_TYPE_VCPU_INFO 0x0
650+#define KVM_XEN_VCPU_ATTR_TYPE_VCPU_TIME_INFO 0x1
651+#define KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADDR 0x2
652+#define KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_CURRENT 0x3
653+#define KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_DATA 0x4
654+#define KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADJUST 0x5
655+/* Available with KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_EVTCHN_SEND */
656+#define KVM_XEN_VCPU_ATTR_TYPE_VCPU_ID 0x6
657+#define KVM_XEN_VCPU_ATTR_TYPE_TIMER 0x7
658+#define KVM_XEN_VCPU_ATTR_TYPE_UPCALL_VECTOR 0x8
659+ 
660+/* Secure Encrypted Virtualization command */
661+enum sev_cmd_id {
662+ /* Guest initialization commands */
663+ KVM_SEV_INIT = 0,
664+ KVM_SEV_ES_INIT,
665+ /* Guest launch commands */
666+ KVM_SEV_LAUNCH_START,
667+ KVM_SEV_LAUNCH_UPDATE_DATA,
668+ KVM_SEV_LAUNCH_UPDATE_VMSA,
669+ KVM_SEV_LAUNCH_SECRET,
670+ KVM_SEV_LAUNCH_MEASURE,
671+ KVM_SEV_LAUNCH_FINISH,
672+ /* Guest migration commands (outgoing) */
673+ KVM_SEV_SEND_START,
674+ KVM_SEV_SEND_UPDATE_DATA,
675+ KVM_SEV_SEND_UPDATE_VMSA,
676+ KVM_SEV_SEND_FINISH,
677+ /* Guest migration commands (incoming) */
678+ KVM_SEV_RECEIVE_START,
679+ KVM_SEV_RECEIVE_UPDATE_DATA,
680+ KVM_SEV_RECEIVE_UPDATE_VMSA,
681+ KVM_SEV_RECEIVE_FINISH,
682+ /* Guest status and debug commands */
683+ KVM_SEV_GUEST_STATUS,
684+ KVM_SEV_DBG_DECRYPT,
685+ KVM_SEV_DBG_ENCRYPT,
686+ /* Guest certificates commands */
687+ KVM_SEV_CERT_EXPORT,
688+ /* Attestation report */
689+ KVM_SEV_GET_ATTESTATION_REPORT,
690+ /* Guest Migration Extension */
691+ KVM_SEV_SEND_CANCEL,
692+ 
693+ /* Second time is the charm; improved versions of the above ioctls. */
694+ KVM_SEV_INIT2,
695+ 
696+ /* SNP-specific commands */
697+ KVM_SEV_SNP_LAUNCH_START = 100,
698+ KVM_SEV_SNP_LAUNCH_UPDATE,
699+ KVM_SEV_SNP_LAUNCH_FINISH,
700+ 
701+ KVM_SEV_NR_MAX,
702+};
703+ 
704+struct kvm_sev_cmd {
705+ __u32 id;
706+ __u32 pad0;
707+ __u64 data;
708+ __u32 error;
709+ __u32 sev_fd;
710+};
711+ 
712+struct kvm_sev_init {
713+ __u64 vmsa_features;
714+ __u32 flags;
715+ __u16 ghcb_version;
716+ __u16 pad1;
717+ __u32 pad2[8];
718+};
719+ 
720+struct kvm_sev_launch_start {
721+ __u32 handle;
722+ __u32 policy;
723+ __u64 dh_uaddr;
724+ __u32 dh_len;
725+ __u32 pad0;
726+ __u64 session_uaddr;
727+ __u32 session_len;
728+ __u32 pad1;
729+};
730+ 
731+struct kvm_sev_launch_update_data {
732+ __u64 uaddr;
733+ __u32 len;
734+ __u32 pad0;
735+};
736+ 
737+ 
738+struct kvm_sev_launch_secret {
739+ __u64 hdr_uaddr;
740+ __u32 hdr_len;
741+ __u32 pad0;
742+ __u64 guest_uaddr;
743+ __u32 guest_len;
744+ __u32 pad1;
745+ __u64 trans_uaddr;
746+ __u32 trans_len;
747+ __u32 pad2;
748+};
749+ 
750+struct kvm_sev_launch_measure {
751+ __u64 uaddr;
752+ __u32 len;
753+ __u32 pad0;
754+};
755+ 
756+struct kvm_sev_guest_status {
757+ __u32 handle;
758+ __u32 policy;
759+ __u32 state;
760+};
761+ 
762+struct kvm_sev_dbg {
763+ __u64 src_uaddr;
764+ __u64 dst_uaddr;
765+ __u32 len;
766+ __u32 pad0;
767+};
768+ 
769+struct kvm_sev_attestation_report {
770+ __u8 mnonce[16];
771+ __u64 uaddr;
772+ __u32 len;
773+ __u32 pad0;
774+};
775+ 
776+struct kvm_sev_send_start {
777+ __u32 policy;
778+ __u32 pad0;
779+ __u64 pdh_cert_uaddr;
780+ __u32 pdh_cert_len;
781+ __u32 pad1;
782+ __u64 plat_certs_uaddr;
783+ __u32 plat_certs_len;
784+ __u32 pad2;
785+ __u64 amd_certs_uaddr;
786+ __u32 amd_certs_len;
787+ __u32 pad3;
788+ __u64 session_uaddr;
789+ __u32 session_len;
790+ __u32 pad4;
791+};
792+ 
793+struct kvm_sev_send_update_data {
794+ __u64 hdr_uaddr;
795+ __u32 hdr_len;
796+ __u32 pad0;
797+ __u64 guest_uaddr;
798+ __u32 guest_len;
799+ __u32 pad1;
800+ __u64 trans_uaddr;
801+ __u32 trans_len;
802+ __u32 pad2;
803+};
804+ 
805+struct kvm_sev_receive_start {
806+ __u32 handle;
807+ __u32 policy;
808+ __u64 pdh_uaddr;
809+ __u32 pdh_len;
810+ __u32 pad0;
811+ __u64 session_uaddr;
812+ __u32 session_len;
813+ __u32 pad1;
814+};
815+ 
816+struct kvm_sev_receive_update_data {
817+ __u64 hdr_uaddr;
818+ __u32 hdr_len;
819+ __u32 pad0;
820+ __u64 guest_uaddr;
821+ __u32 guest_len;
822+ __u32 pad1;
823+ __u64 trans_uaddr;
824+ __u32 trans_len;
825+ __u32 pad2;
826+};
827+ 
828+struct kvm_sev_snp_launch_start {
829+ __u64 policy;
830+ __u8 gosvw[16];
831+ __u16 flags;
832+ __u8 pad0[6];
833+ __u64 pad1[4];
834+};
835+ 
836+/* Kept in sync with firmware values for simplicity. */
837+#define KVM_SEV_SNP_PAGE_TYPE_NORMAL 0x1
838+#define KVM_SEV_SNP_PAGE_TYPE_ZERO 0x3
839+#define KVM_SEV_SNP_PAGE_TYPE_UNMEASURED 0x4
840+#define KVM_SEV_SNP_PAGE_TYPE_SECRETS 0x5
841+#define KVM_SEV_SNP_PAGE_TYPE_CPUID 0x6
842+ 
843+struct kvm_sev_snp_launch_update {
844+ __u64 gfn_start;
845+ __u64 uaddr;
846+ __u64 len;
847+ __u8 type;
848+ __u8 pad0;
849+ __u16 flags;
850+ __u32 pad1;
851+ __u64 pad2[4];
852+};
853+ 
854+#define KVM_SEV_SNP_ID_BLOCK_SIZE 96
855+#define KVM_SEV_SNP_ID_AUTH_SIZE 4096
856+#define KVM_SEV_SNP_FINISH_DATA_SIZE 32
857+ 
858+struct kvm_sev_snp_launch_finish {
859+ __u64 id_block_uaddr;
860+ __u64 id_auth_uaddr;
861+ __u8 id_block_en;
862+ __u8 auth_key_en;
863+ __u8 vcek_disabled;
864+ __u8 host_data[KVM_SEV_SNP_FINISH_DATA_SIZE];
865+ __u8 pad0[3];
866+ __u16 flags;
867+ __u64 pad1[4];
868+};
869+ 
870+#define KVM_X2APIC_API_USE_32BIT_IDS (1ULL << 0)
871+#define KVM_X2APIC_API_DISABLE_BROADCAST_QUIRK (1ULL << 1)
872+ 
873+struct kvm_hyperv_eventfd {
874+ __u32 conn_id;
875+ __s32 fd;
876+ __u32 flags;
877+ __u32 padding[3];
878+};
879+ 
880+#define KVM_HYPERV_CONN_ID_MASK 0x00ffffff
881+#define KVM_HYPERV_EVENTFD_DEASSIGN (1 << 0)
882+ 
535/*883/*
536 * Masked event layout.884 * Masked event layout.
537 * Bits Description885 * Bits Description
@@ -565,4 +913,10 @@ struct kvm_pmu_event_filter {
565/* x86-specific KVM_EXIT_HYPERCALL flags. */913/* x86-specific KVM_EXIT_HYPERCALL flags. */
566#define KVM_EXIT_HYPERCALL_LONG_MODE BIT(0)914#define KVM_EXIT_HYPERCALL_LONG_MODE BIT(0)
567 915 
916+#define KVM_X86_DEFAULT_VM 0
917+#define KVM_X86_SW_PROTECTED_VM 1
918+#define KVM_X86_SEV_VM 2
919+#define KVM_X86_SEV_ES_VM 3
920+#define KVM_X86_SNP_VM 4
921+ 
568#endif /* _ASM_X86_KVM_H */922#endif /* _ASM_X86_KVM_H */
March/x86/kernel/cpu/amd.c+27-3
@@ -21,6 +21,7 @@
21#include <asm/delay.h>21#include <asm/delay.h>
22#include <asm/debugreg.h>22#include <asm/debugreg.h>
23#include <asm/resctrl.h>23#include <asm/resctrl.h>
24+#include <asm/sev.h>
24 25 
25#ifdef CONFIG_X86_6426#ifdef CONFIG_X86_64
26# include <asm/mmconfig.h>27# include <asm/mmconfig.h>
@@ -386,6 +387,28 @@ static void tsa_init(struct cpuinfo_x86 *c)
386 }387 }
387}388}
388 389 
390+static void bsp_determine_snp(struct cpuinfo_x86 *c)
391+{
392+#ifdef CONFIG_ARCH_HAS_CC_PLATFORM
393+ cc_vendor = CC_VENDOR_AMD;
394+ 
395+ if (cpu_has(c, X86_FEATURE_SEV_SNP)) {
396+ /*
397+ * RMP table entry format is not architectural and is defined by the
398+ * per-processor PPR. Restrict SNP support on the known CPU models
399+ * for which the RMP table entry format is currently defined for.
400+ */
401+ if (!cpu_has(c, X86_FEATURE_HYPERVISOR) &&
402+ c->x86 >= 0x19 && snp_probe_rmptable_info()) {
403+ cc_platform_set(CC_ATTR_HOST_SEV_SNP);
404+ } else {
405+ setup_clear_cpu_cap(X86_FEATURE_SEV_SNP);
406+ cc_platform_clear(CC_ATTR_HOST_SEV_SNP);
407+ }
408+ }
409+#endif
410+}
411+ 
389static void bsp_init_amd(struct cpuinfo_x86 *c)412static void bsp_init_amd(struct cpuinfo_x86 *c)
390{413{
391 if (cpu_has(c, X86_FEATURE_CONSTANT_TSC)) {414 if (cpu_has(c, X86_FEATURE_CONSTANT_TSC)) {
@@ -497,9 +520,9 @@ static void bsp_init_amd(struct cpuinfo_x86 *c)
497 break;520 break;
498 }521 }
499 522 
500- 
501 tsa_init(c);523 tsa_init(c);
502 524 
525+ bsp_determine_snp(c);
503 return;526 return;
504 527 
505warn:528warn:
@@ -518,8 +541,8 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)
518 * SME feature (set in scattered.c).541 * SME feature (set in scattered.c).
519 * If the kernel has not enabled SME via any means then542 * If the kernel has not enabled SME via any means then
520 * don't advertise the SME feature.543 * don't advertise the SME feature.
521- * For SEV: If BIOS has not enabled SEV then don't advertise the544+ * For SEV: If BIOS has not enabled SEV then don't advertise SEV and
522- * SEV and SEV_ES feature (set in scattered.c).545+ * any additional functionality based on it.
523 *546 *
524 * In all cases, since support for SME and SEV requires long mode,547 * In all cases, since support for SME and SEV requires long mode,
525 * don't advertise the feature under CONFIG_X86_32.548 * don't advertise the feature under CONFIG_X86_32.
@@ -554,6 +577,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)
554clear_sev:577clear_sev:
555 setup_clear_cpu_cap(X86_FEATURE_SEV);578 setup_clear_cpu_cap(X86_FEATURE_SEV);
556 setup_clear_cpu_cap(X86_FEATURE_SEV_ES);579 setup_clear_cpu_cap(X86_FEATURE_SEV_ES);
580+ setup_clear_cpu_cap(X86_FEATURE_SEV_SNP);
557 }581 }
558}582}
559 583 
March/x86/kernel/cpu/common.c+6-1
@@ -1365,8 +1365,13 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
1365 /*1365 /*
1366 * AMD's AutoIBRS is equivalent to Intel's eIBRS - use the Intel feature1366 * AMD's AutoIBRS is equivalent to Intel's eIBRS - use the Intel feature
1367 * flag and protect from vendor-specific bugs via the whitelist.1367 * flag and protect from vendor-specific bugs via the whitelist.
1368+ *
1369+ * Don't use AutoIBRS when SNP is enabled because it degrades host
1370+ * userspace indirect branch performance.
1368 */1371 */
1369- if ((x86_arch_cap_msr & ARCH_CAP_IBRS_ALL) || cpu_has(c, X86_FEATURE_AUTOIBRS)) {1372+ if ((x86_arch_cap_msr & ARCH_CAP_IBRS_ALL) ||
1373+ (cpu_has(c, X86_FEATURE_AUTOIBRS) &&
1374+ !cpu_feature_enabled(X86_FEATURE_SEV_SNP))) {
1370 setup_force_cpu_cap(X86_FEATURE_IBRS_ENHANCED);1375 setup_force_cpu_cap(X86_FEATURE_IBRS_ENHANCED);
1371 if (!cpu_matches(cpu_vuln_whitelist, NO_EIBRS_PBRSB) &&1376 if (!cpu_matches(cpu_vuln_whitelist, NO_EIBRS_PBRSB) &&
1372 !(x86_arch_cap_msr & ARCH_CAP_PBRSB_NO))1377 !(x86_arch_cap_msr & ARCH_CAP_PBRSB_NO))
March/x86/kernel/cpu/mtrr/generic.c+3-0
@@ -108,6 +108,9 @@ static inline void k8_check_syscfg_dram_mod_en(void)
108 (boot_cpu_data.x86 >= 0x0f)))108 (boot_cpu_data.x86 >= 0x0f)))
109 return;109 return;
110 110 
111+ if (cc_platform_has(CC_ATTR_HOST_SEV_SNP))
112+ return;
113+ 
111 rdmsr(MSR_AMD64_SYSCFG, lo, hi);114 rdmsr(MSR_AMD64_SYSCFG, lo, hi);
112 if (lo & K8_MTRRFIXRANGE_DRAM_MODIFY) {115 if (lo & K8_MTRRFIXRANGE_DRAM_MODIFY) {
113 pr_err(FW_WARN "MTRR: CPU %u: SYSCFG[MtrrFixDramModEn]"116 pr_err(FW_WARN "MTRR: CPU %u: SYSCFG[MtrrFixDramModEn]"
March/x86/kernel/crash.c+3-0
@@ -40,6 +40,7 @@
40#include <asm/intel_pt.h>40#include <asm/intel_pt.h>
41#include <asm/crash.h>41#include <asm/crash.h>
42#include <asm/cmdline.h>42#include <asm/cmdline.h>
43+#include <asm/sev.h>
43 44 
44/* Used while preparing memory map entries for second kernel */45/* Used while preparing memory map entries for second kernel */
45struct crash_memmap_data {46struct crash_memmap_data {
@@ -59,6 +60,8 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
59 */60 */
60 cpu_emergency_stop_pt();61 cpu_emergency_stop_pt();
61 62 
63+ kdump_sev_callback();
64+ 
62 disable_local_APIC();65 disable_local_APIC();
63}66}
64 67 
March/x86/kernel/e820.c+15-29
@@ -28,18 +28,13 @@
28 * the first 128 E820 memory entries in boot_params.e820_table and the remaining28 * the first 128 E820 memory entries in boot_params.e820_table and the remaining
29 * (if any) entries of the SETUP_E820_EXT nodes. We use this to:29 * (if any) entries of the SETUP_E820_EXT nodes. We use this to:
30 *30 *
31- * - inform the user about the firmware's notion of memory layout
32- * via /sys/firmware/memmap
33- *
34 * - the hibernation code uses it to generate a kernel-independent CRC3231 * - the hibernation code uses it to generate a kernel-independent CRC32
35 * checksum of the physical memory layout of a system.32 * checksum of the physical memory layout of a system.
36 *33 *
37 * - 'e820_table_kexec': a slightly modified (by the kernel) firmware version34 * - 'e820_table_kexec': a slightly modified (by the kernel) firmware version
38 * passed to us by the bootloader - the major difference between35 * passed to us by the bootloader - the major difference between
39- * e820_table_firmware[] and this one is that, the latter marks the setup_data36+ * e820_table_firmware[] and this one is that e820_table_kexec[]
40- * list created by the EFI boot stub as reserved, so that kexec can reuse the37+ * might be modified by the kexec itself to fake an mptable.
41- * setup_data information in the second kernel. Besides, e820_table_kexec[]
42- * might also be modified by the kexec itself to fake a mptable.
43 * We use this to:38 * We use this to:
44 *39 *
45 * - kexec, which is a bootloader in disguise, uses the original E82040 * - kexec, which is a bootloader in disguise, uses the original E820
@@ -47,6 +42,11 @@
47 * can have a restricted E820 map while the kexec()-ed kexec-kernel42 * can have a restricted E820 map while the kexec()-ed kexec-kernel
48 * can have access to full memory - etc.43 * can have access to full memory - etc.
49 *44 *
45+ * Export the memory layout via /sys/firmware/memmap. kexec-tools uses
46+ * the entries to create an E820 table for the kexec kernel.
47+ *
48+ * kexec_file_load in-kernel code uses the table for the kexec kernel.
49+ *
50 * - 'e820_table': this is the main E820 table that is massaged by the50 * - 'e820_table': this is the main E820 table that is massaged by the
51 * low level x86 platform code, or modified by boot parameters, before51 * low level x86 platform code, or modified by boot parameters, before
52 * passed on to higher level MM layers.52 * passed on to higher level MM layers.
@@ -532,9 +532,10 @@ u64 __init e820__range_update(u64 start, u64 size, enum e820_type old_type, enum
532 return __e820__range_update(e820_table, start, size, old_type, new_type);532 return __e820__range_update(e820_table, start, size, old_type, new_type);
533}533}
534 534 
535-static u64 __init e820__range_update_kexec(u64 start, u64 size, enum e820_type old_type, enum e820_type new_type)535+u64 __init e820__range_update_table(struct e820_table *t, u64 start, u64 size,
536+ enum e820_type old_type, enum e820_type new_type)
536{537{
537- return __e820__range_update(e820_table_kexec, start, size, old_type, new_type);538+ return __e820__range_update(t, start, size, old_type, new_type);
538}539}
539 540 
540/* Remove a range of memory from the E820 table: */541/* Remove a range of memory from the E820 table: */
@@ -805,7 +806,7 @@ u64 __init e820__memblock_alloc_reserved(u64 size, u64 align)
805 806 
806 addr = memblock_phys_alloc(size, align);807 addr = memblock_phys_alloc(size, align);
807 if (addr) {808 if (addr) {
808- e820__range_update_kexec(addr, size, E820_TYPE_RAM, E820_TYPE_RESERVED);809+ e820__range_update_table(e820_table_kexec, addr, size, E820_TYPE_RAM, E820_TYPE_RESERVED);
809 pr_info("update e820_table_kexec for e820__memblock_alloc_reserved()\n");810 pr_info("update e820_table_kexec for e820__memblock_alloc_reserved()\n");
810 e820__update_table_kexec();811 e820__update_table_kexec();
811 }812 }
@@ -1015,17 +1016,6 @@ void __init e820__reserve_setup_data(void)
1015 1016 
1016 e820__range_update(pa_data, sizeof(*data)+data->len, E820_TYPE_RAM, E820_TYPE_RESERVED_KERN);1017 e820__range_update(pa_data, sizeof(*data)+data->len, E820_TYPE_RAM, E820_TYPE_RESERVED_KERN);
1017 1018 
1018- /*
1019- * SETUP_EFI, SETUP_IMA and SETUP_RNG_SEED are supplied by
1020- * kexec and do not need to be reserved.
1021- */
1022- if (data->type != SETUP_EFI &&
1023- data->type != SETUP_IMA &&
1024- data->type != SETUP_RNG_SEED)
1025- e820__range_update_kexec(pa_data,
1026- sizeof(*data) + data->len,
1027- E820_TYPE_RAM, E820_TYPE_RESERVED_KERN);
1028- 
1029 if (data->type == SETUP_INDIRECT) {1019 if (data->type == SETUP_INDIRECT) {
1030 len += data->len;1020 len += data->len;
1031 early_memunmap(data, sizeof(*data));1021 early_memunmap(data, sizeof(*data));
@@ -1037,12 +1027,9 @@ void __init e820__reserve_setup_data(void)
1037 1027 
1038 indirect = (struct setup_indirect *)data->data;1028 indirect = (struct setup_indirect *)data->data;
1039 1029 
1040- if (indirect->type != SETUP_INDIRECT) {1030+ if (indirect->type != SETUP_INDIRECT)
1041 e820__range_update(indirect->addr, indirect->len,1031 e820__range_update(indirect->addr, indirect->len,
1042 E820_TYPE_RAM, E820_TYPE_RESERVED_KERN);1032 E820_TYPE_RAM, E820_TYPE_RESERVED_KERN);
1043- e820__range_update_kexec(indirect->addr, indirect->len,
1044- E820_TYPE_RAM, E820_TYPE_RESERVED_KERN);
1045- }
1046 }1033 }
1047 1034 
1048 pa_data = pa_next;1035 pa_data = pa_next;
@@ -1050,7 +1037,6 @@ void __init e820__reserve_setup_data(void)
1050 }1037 }
1051 1038 
1052 e820__update_table(e820_table);1039 e820__update_table(e820_table);
1053- e820__update_table(e820_table_kexec);
1054 1040 
1055 pr_info("extended physical RAM map:\n");1041 pr_info("extended physical RAM map:\n");
1056 e820__print_table("reserve setup_data");1042 e820__print_table("reserve setup_data");
@@ -1191,9 +1177,9 @@ void __init e820__reserve_resources(void)
1191 res++;1177 res++;
1192 }1178 }
1193 1179 
1194- /* Expose the bootloader-provided memory layout to the sysfs. */1180+ /* Expose the kexec e820 table to the sysfs. */
1195- for (i = 0; i < e820_table_firmware->nr_entries; i++) {1181+ for (i = 0; i < e820_table_kexec->nr_entries; i++) {
1196- struct e820_entry *entry = e820_table_firmware->entries + i;1182+ struct e820_entry *entry = e820_table_kexec->entries + i;
1197 1183 
1198 firmware_map_add_early(entry->addr, entry->addr + entry->size, e820_type_to_string(entry));1184 firmware_map_add_early(entry->addr, entry->addr + entry->size, e820_type_to_string(entry));
1199 }1185 }
March/x86/kernel/fpu/xstate.c+1-0
@@ -1044,6 +1044,7 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr)
1044 1044 
1045 return __raw_xsave_addr(xsave, xfeature_nr);1045 return __raw_xsave_addr(xsave, xfeature_nr);
1046}1046}
1047+EXPORT_SYMBOL_GPL(get_xsave_addr);
1047 1048 
1048/*1049/*
1049 * Given an xstate feature nr, calculate where in the xsave buffer the state is.1050 * Given an xstate feature nr, calculate where in the xsave buffer the state is.
March/x86/kernel/fpu/xstate.h+0-2
@@ -54,8 +54,6 @@ extern int copy_sigframe_from_user_to_xstate(struct task_struct *tsk, const void
54extern void fpu__init_cpu_xstate(void);54extern void fpu__init_cpu_xstate(void);
55extern void fpu__init_system_xstate(unsigned int legacy_size);55extern void fpu__init_system_xstate(unsigned int legacy_size);
56 56 
57-extern void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr);
58- 
59extern void __user *get_xsave_addr_user(struct xregs_state __user *xsave, int xfeature_nr);57extern void __user *get_xsave_addr_user(struct xregs_state __user *xsave, int xfeature_nr);
60 58 
61static inline u64 xfeatures_mask_supervisor(void)59static inline u64 xfeatures_mask_supervisor(void)
March/x86/kernel/setup.c+1-1
@@ -1123,8 +1123,8 @@ void __init setup_arch(char **cmdline_p)
1123 * Needs to run after memblock setup because it needs the physical1123 * Needs to run after memblock setup because it needs the physical
1124 * memory size.1124 * memory size.
1125 */1125 */
1126- sev_setup_arch();
1127 cc_random_init();1126 cc_random_init();
1127+ mem_encrypt_setup_arch();
1128 1128 
1129 efi_fake_memmap();1129 efi_fake_memmap();
1130 efi_find_mirror();1130 efi_find_mirror();
March/x86/kernel/sev.c+38-3
@@ -62,6 +62,25 @@
62#define AP_INIT_CR0_DEFAULT 0x6000001062#define AP_INIT_CR0_DEFAULT 0x60000010
63#define AP_INIT_MXCSR_DEFAULT 0x1f8063#define AP_INIT_MXCSR_DEFAULT 0x1f80
64 64 
65+static const char * const sev_status_feat_names[] = {
66+ [MSR_AMD64_SEV_ENABLED_BIT] = "SEV",
67+ [MSR_AMD64_SEV_ES_ENABLED_BIT] = "SEV-ES",
68+ [MSR_AMD64_SEV_SNP_ENABLED_BIT] = "SEV-SNP",
69+ [MSR_AMD64_SNP_VTOM_BIT] = "vTom",
70+ [MSR_AMD64_SNP_REFLECT_VC_BIT] = "ReflectVC",
71+ [MSR_AMD64_SNP_RESTRICTED_INJ_BIT] = "RI",
72+ [MSR_AMD64_SNP_ALT_INJ_BIT] = "AI",
73+ [MSR_AMD64_SNP_DEBUG_SWAP_BIT] = "DebugSwap",
74+ [MSR_AMD64_SNP_PREVENT_HOST_IBS_BIT] = "NoHostIBS",
75+ [MSR_AMD64_SNP_BTB_ISOLATION_BIT] = "BTBIsol",
76+ [MSR_AMD64_SNP_VMPL_SSS_BIT] = "VmplSSS",
77+ [MSR_AMD64_SNP_SECURE_TSC_BIT] = "SecureTSC",
78+ [MSR_AMD64_SNP_VMGEXIT_PARAM_BIT] = "VMGExitParam",
79+ [MSR_AMD64_SNP_IBS_VIRT_BIT] = "IBSVirt",
80+ [MSR_AMD64_SNP_VMSA_REG_PROT_BIT] = "VMSARegProt",
81+ [MSR_AMD64_SNP_SMT_PROT_BIT] = "SMTProt",
82+};
83+ 
65/* For early boot hypervisor communication in SEV-ES enabled guests */84/* For early boot hypervisor communication in SEV-ES enabled guests */
66static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);85static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
67 86 
@@ -630,7 +649,7 @@ static u64 __init get_secrets_page(void)
630 649 
631static u64 __init get_snp_jump_table_addr(void)650static u64 __init get_snp_jump_table_addr(void)
632{651{
633- struct snp_secrets_page_layout *layout;652+ struct snp_secrets_page *secrets;
634 void __iomem *mem;653 void __iomem *mem;
635 u64 pa, addr;654 u64 pa, addr;
636 655 
@@ -644,9 +663,9 @@ static u64 __init get_snp_jump_table_addr(void)
644 return 0;663 return 0;
645 }664 }
646 665 
647- layout = (__force struct snp_secrets_page_layout *)mem;666+ secrets = (__force struct snp_secrets_page *)mem;
648 667 
649- addr = layout->os_area.ap_jump_table_pa;668+ addr = secrets->os_area.ap_jump_table_pa;
650 iounmap(mem);669 iounmap(mem);
651 670 
652 return addr;671 return addr;
@@ -2286,3 +2305,19 @@ static int __init snp_init_platform_device(void)
2286 return 0;2305 return 0;
2287}2306}
2288device_initcall(snp_init_platform_device);2307device_initcall(snp_init_platform_device);
2308+ 
2309+void sev_show_status(void)
2310+{
2311+ int i;
2312+ 
2313+ pr_info("Status: ");
2314+ for (i = 0; i < MSR_AMD64_SNP_RESV_BIT; i++) {
2315+ if (sev_status & BIT_ULL(i)) {
2316+ if (!sev_status_feat_names[i])
2317+ continue;
2318+ 
2319+ pr_cont("%s ", sev_status_feat_names[i]);
2320+ }
2321+ }
2322+ pr_cont("\n");
2323+}
March/x86/kvm/Kconfig+22-3
@@ -24,7 +24,7 @@ config KVM
24 depends on HIGH_RES_TIMERS24 depends on HIGH_RES_TIMERS
25 depends on X86_LOCAL_APIC25 depends on X86_LOCAL_APIC
26 select PREEMPT_NOTIFIERS26 select PREEMPT_NOTIFIERS
27- select MMU_NOTIFIER27+ select KVM_GENERIC_MMU_NOTIFIER
28 select HAVE_KVM_IRQCHIP28 select HAVE_KVM_IRQCHIP
29 select HAVE_KVM_PFNCACHE29 select HAVE_KVM_PFNCACHE
30 select HAVE_KVM_IRQFD30 select HAVE_KVM_IRQFD
@@ -49,6 +49,7 @@ config KVM
49 select INTERVAL_TREE49 select INTERVAL_TREE
50 select HAVE_KVM_PM_NOTIFIER if PM50 select HAVE_KVM_PM_NOTIFIER if PM
51 select KVM_GENERIC_HARDWARE_ENABLING51 select KVM_GENERIC_HARDWARE_ENABLING
52+ select KVM_GENERIC_PRE_FAULT_MEMORY
52 help53 help
53 Support hosting fully virtualized guest machines using hardware54 Support hosting fully virtualized guest machines using hardware
54 virtualization extensions. You will need a fairly recent55 virtualization extensions. You will need a fairly recent
@@ -77,6 +78,18 @@ config KVM_WERROR
77 78 
78 If in doubt, say "N".79 If in doubt, say "N".
79 80 
81+config KVM_SW_PROTECTED_VM
82+ bool "Enable support for KVM software-protected VMs"
83+ depends on EXPERT
84+ depends on X86_64
85+ select KVM_GENERIC_PRIVATE_MEM
86+ help
87+ Enable support for KVM software-protected VMs. Currently "protected"
88+ means the VM can be backed with memory provided by
89+ KVM_CREATE_GUEST_MEMFD.
90+ 
91+ If unsure, say "N".
92+ 
80config KVM_INTEL93config KVM_INTEL
81 tristate "KVM for Intel (and compatible) processors support"94 tristate "KVM for Intel (and compatible) processors support"
82 depends on KVM && IA32_FEAT_CTL95 depends on KVM && IA32_FEAT_CTL
@@ -114,9 +127,15 @@ config KVM_AMD_SEV
114 bool "AMD Secure Encrypted Virtualization (SEV) support"127 bool "AMD Secure Encrypted Virtualization (SEV) support"
115 depends on KVM_AMD && X86_64128 depends on KVM_AMD && X86_64
116 depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)129 depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
130+ select ARCH_HAS_CC_PLATFORM
131+ select KVM_GENERIC_PRIVATE_MEM
132+ select HAVE_KVM_ARCH_GMEM_PREPARE
133+ select HAVE_KVM_ARCH_GMEM_INVALIDATE
117 help134 help
118- Provides support for launching Encrypted VMs (SEV) and Encrypted VMs135+ Provides support for launching encrypted VMs which use Secure
119- with Encrypted State (SEV-ES) on AMD processors.136+ Encrypted Virtualization (SEV), Secure Encrypted Virtualization with
137+ Encrypted State (SEV-ES), and Secure Encrypted Virtualization with
138+ Secure Nested Paging (SEV-SNP) technologies on AMD processors.
120 139 
121config KVM_SMM140config KVM_SMM
122 bool "System Management Mode emulation"141 bool "System Management Mode emulation"
March/x86/kvm/Makefile+4-3
@@ -25,9 +25,10 @@ kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \
25kvm-intel-$(CONFIG_X86_SGX_KVM) += vmx/sgx.o25kvm-intel-$(CONFIG_X86_SGX_KVM) += vmx/sgx.o
26kvm-intel-$(CONFIG_KVM_HYPERV) += vmx/hyperv.o vmx/hyperv_evmcs.o26kvm-intel-$(CONFIG_KVM_HYPERV) += vmx/hyperv.o vmx/hyperv_evmcs.o
27 27 
28-kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o \28+kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o
29- svm/sev.o29+ 
30-kvm-amd-$(CONFIG_KVM_HYPERV) += svm/hyperv.o30+kvm-amd-$(CONFIG_KVM_AMD_SEV) += svm/sev.o
31+kvm-amd-$(CONFIG_KVM_HYPERV) += svm/hyperv.o
31 32 
32ifdef CONFIG_HYPERV33ifdef CONFIG_HYPERV
33kvm-y += kvm_onhyperv.o34kvm-y += kvm_onhyperv.o
March/x86/kvm/debugfs.c+1-1
@@ -111,7 +111,7 @@ static int kvm_mmu_rmaps_stat_show(struct seq_file *m, void *v)
111 mutex_lock(&kvm->slots_lock);111 mutex_lock(&kvm->slots_lock);
112 write_lock(&kvm->mmu_lock);112 write_lock(&kvm->mmu_lock);
113 113 
114- for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {114+ for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
115 int bkt;115 int bkt;
116 116 
117 slots = __kvm_memslots(kvm, i);117 slots = __kvm_memslots(kvm, i);
March/x86/kvm/lapic.c+4-1
@@ -2893,7 +2893,10 @@ int kvm_create_lapic(struct kvm_vcpu *vcpu, int timer_advance_ns)
2893 2893 
2894 vcpu->arch.apic = apic;2894 vcpu->arch.apic = apic;
2895 2895 
2896- apic->regs = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT);2896+ if (kvm_x86_ops.alloc_apic_backing_page)
2897+ apic->regs = static_call(kvm_x86_alloc_apic_backing_page)(vcpu);
2898+ else
2899+ apic->regs = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT);
2897 if (!apic->regs) {2900 if (!apic->regs) {
2898 printk(KERN_ERR "malloc apic regs error for vcpu %x\n",2901 printk(KERN_ERR "malloc apic regs error for vcpu %x\n",
2899 vcpu->vcpu_id);2902 vcpu->vcpu_id);
March/x86/kvm/mmu.h+5-5
@@ -129,6 +129,9 @@ void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
129 129 
130static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu)130static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu)
131{131{
132+ if (kvm_check_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu))
133+ kvm_mmu_free_obsolete_roots(vcpu);
134+ 
132 if (likely(vcpu->arch.mmu->root.hpa != INVALID_PAGE))135 if (likely(vcpu->arch.mmu->root.hpa != INVALID_PAGE))
133 return 0;136 return 0;
134 137 
@@ -216,7 +219,7 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
216 */219 */
217 u64 implicit_access = access & PFERR_IMPLICIT_ACCESS;220 u64 implicit_access = access & PFERR_IMPLICIT_ACCESS;
218 bool not_smap = ((rflags & X86_EFLAGS_AC) | implicit_access) == X86_EFLAGS_AC;221 bool not_smap = ((rflags & X86_EFLAGS_AC) | implicit_access) == X86_EFLAGS_AC;
219- int index = (pfec + (not_smap << PFERR_RSVD_BIT)) >> 1;222+ int index = (pfec | (not_smap ? PFERR_RSVD_MASK : 0)) >> 1;
220 u32 errcode = PFERR_PRESENT_MASK;223 u32 errcode = PFERR_PRESENT_MASK;
221 bool fault;224 bool fault;
222 225 
@@ -237,8 +240,7 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
237 pkru_bits = (vcpu->arch.pkru >> (pte_pkey * 2)) & 3;240 pkru_bits = (vcpu->arch.pkru >> (pte_pkey * 2)) & 3;
238 241 
239 /* clear present bit, replace PFEC.RSVD with ACC_USER_MASK. */242 /* clear present bit, replace PFEC.RSVD with ACC_USER_MASK. */
240- offset = (pfec & ~1) +243+ offset = (pfec & ~1) | ((pte_access & PT_USER_MASK) ? PFERR_RSVD_MASK : 0);
241- ((pte_access & PT_USER_MASK) << (PFERR_RSVD_BIT - PT_USER_SHIFT));
242 244 
243 pkru_bits &= mmu->pkru_mask >> offset;245 pkru_bits &= mmu->pkru_mask >> offset;
244 errcode |= -pkru_bits & PFERR_PK_MASK;246 errcode |= -pkru_bits & PFERR_PK_MASK;
@@ -248,8 +250,6 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
248 return -(u32)fault & errcode;250 return -(u32)fault & errcode;
249}251}
250 252 
251-void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end);
252- 
253int kvm_arch_write_log_dirty(struct kvm_vcpu *vcpu);253int kvm_arch_write_log_dirty(struct kvm_vcpu *vcpu);
254 254 
255int kvm_mmu_post_init_vm(struct kvm *kvm);255int kvm_mmu_post_init_vm(struct kvm *kvm);
March/x86/kvm/mmu/mmu.c+327-36
@@ -3061,7 +3061,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
3061 *3061 *
3062 * There are several ways to safely use this helper:3062 * There are several ways to safely use this helper:
3063 *3063 *
3064- * - Check mmu_invalidate_retry_hva() after grabbing the mapping level, before3064+ * - Check mmu_invalidate_retry_gfn() after grabbing the mapping level, before
3065 * consuming it. In this case, mmu_lock doesn't need to be held during the3065 * consuming it. In this case, mmu_lock doesn't need to be held during the
3066 * lookup, but it does need to be held while checking the MMU notifier.3066 * lookup, but it does need to be held while checking the MMU notifier.
3067 *3067 *
March/x86/kvm/mmu/mmu_internal.h+7-15
@@ -190,7 +190,7 @@ static inline bool is_nx_huge_page_enabled(struct kvm *kvm)
190struct kvm_page_fault {190struct kvm_page_fault {
191 /* arguments to kvm_mmu_do_page_fault. */191 /* arguments to kvm_mmu_do_page_fault. */
192 const gpa_t addr;192 const gpa_t addr;
193- const u32 error_code;193+ const u64 error_code;
194 const bool prefetch;194 const bool prefetch;
195 195 
196 /* Derived from error_code. */196 /* Derived from error_code. */
@@ -202,6 +202,7 @@ struct kvm_page_fault {
202 202 
203 /* Derived from mmu and global state. */203 /* Derived from mmu and global state. */
204 const bool is_tdp;204 const bool is_tdp;
205+ const bool is_private;
205 const bool nx_huge_page_workaround_enabled;206 const bool nx_huge_page_workaround_enabled;
206 207 
207 /*208 /*
@@ -279,7 +280,8 @@ enum {
279};280};
280 281 
281static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,282static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
282- u32 err, bool prefetch, int *emulation_type)283+ u64 err, bool prefetch,
284+ int *emulation_type, u8 *level)
283{285{
284 struct kvm_page_fault fault = {286 struct kvm_page_fault fault = {
285 .addr = cr2_or_gpa,287 .addr = cr2_or_gpa,
@@ -297,6 +299,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
297 .max_level = KVM_MAX_HUGEPAGE_LEVEL,299 .max_level = KVM_MAX_HUGEPAGE_LEVEL,
298 .req_level = PG_LEVEL_4K,300 .req_level = PG_LEVEL_4K,
299 .goal_level = PG_LEVEL_4K,301 .goal_level = PG_LEVEL_4K,
302+ .is_private = err & PFERR_PRIVATE_ACCESS,
300 };303 };
301 int r;304 int r;
302 305 
@@ -320,20 +323,9 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
320 323 
321 if (fault.write_fault_to_shadow_pgtable && emulation_type)324 if (fault.write_fault_to_shadow_pgtable && emulation_type)
322 *emulation_type |= EMULTYPE_WRITE_PF_TO_SP;325 *emulation_type |= EMULTYPE_WRITE_PF_TO_SP;
326+ if (level)
327+ *level = fault.goal_level;
323 328 
324- /*
325- * Similar to above, prefetch faults aren't truly spurious, and the
326- * async #PF path doesn't do emulation. Do count faults that are fixed
327- * by the async #PF handler though, otherwise they'll never be counted.
328- */
329- if (r == RET_PF_FIXED)
330- vcpu->stat.pf_fixed++;
331- else if (prefetch)
332- ;
333- else if (r == RET_PF_EMULATE)
334- vcpu->stat.pf_emulate++;
335- else if (r == RET_PF_SPURIOUS)
336- vcpu->stat.pf_spurious++;
337 return r;329 return r;
338}330}
339 331 
March/x86/kvm/mmu/mmutrace.h+1-1
@@ -260,7 +260,7 @@ TRACE_EVENT(
260 TP_STRUCT__entry(260 TP_STRUCT__entry(
261 __field(int, vcpu_id)261 __field(int, vcpu_id)
262 __field(gpa_t, cr2_or_gpa)262 __field(gpa_t, cr2_or_gpa)
263- __field(u32, error_code)263+ __field(u64, error_code)
264 __field(u64 *, sptep)264 __field(u64 *, sptep)
265 __field(u64, old_spte)265 __field(u64, old_spte)
266 __field(u64, new_spte)266 __field(u64, new_spte)
March/x86/kvm/svm/avic.c+6-0
@@ -1210,6 +1210,12 @@ bool avic_hardware_setup(void)
1210 return false;1210 return false;
1211 }1211 }
1212 1212 
1213+ if (cc_platform_has(CC_ATTR_HOST_SEV_SNP) &&
1214+ !boot_cpu_has(X86_FEATURE_HV_INUSE_WR_ALLOWED)) {
1215+ pr_warn("AVIC disabled: missing HvInUseWrAllowed on SNP-enabled system\n");
1216+ return false;
1217+ }
1218+ 
1213 if (boot_cpu_has(X86_FEATURE_AVIC)) {1219 if (boot_cpu_has(X86_FEATURE_AVIC)) {
1214 pr_info("AVIC enabled\n");1220 pr_info("AVIC enabled\n");
1215 } else if (force_avic) {1221 } else if (force_avic) {
March/x86/kvm/svm/nested.c+1-1
@@ -1221,7 +1221,7 @@ int svm_allocate_nested(struct vcpu_svm *svm)
1221 if (svm->nested.initialized)1221 if (svm->nested.initialized)
1222 return 0;1222 return 0;
1223 1223 
1224- vmcb02_page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO);1224+ vmcb02_page = snp_safe_alloc_page(&svm->vcpu);
1225 if (!vmcb02_page)1225 if (!vmcb02_page)
1226 return -ENOMEM;1226 return -ENOMEM;
1227 svm->nested.vmcb02.ptr = page_address(vmcb02_page);1227 svm->nested.vmcb02.ptr = page_address(vmcb02_page);
March/x86/kvm/svm/sev.c+1971-183
March/x86/kvm/svm/svm.c+106-33
@@ -104,6 +104,7 @@ static const struct svm_direct_access_msrs {
104 { .index = MSR_IA32_SPEC_CTRL, .always = false },104 { .index = MSR_IA32_SPEC_CTRL, .always = false },
105 { .index = MSR_IA32_PRED_CMD, .always = false },105 { .index = MSR_IA32_PRED_CMD, .always = false },
106 { .index = MSR_IA32_FLUSH_CMD, .always = false },106 { .index = MSR_IA32_FLUSH_CMD, .always = false },
107+ { .index = MSR_IA32_DEBUGCTLMSR, .always = false },
107 { .index = MSR_IA32_LASTBRANCHFROMIP, .always = false },108 { .index = MSR_IA32_LASTBRANCHFROMIP, .always = false },
108 { .index = MSR_IA32_LASTBRANCHTOIP, .always = false },109 { .index = MSR_IA32_LASTBRANCHTOIP, .always = false },
109 { .index = MSR_IA32_LASTINTFROMIP, .always = false },110 { .index = MSR_IA32_LASTINTFROMIP, .always = false },
@@ -220,7 +221,7 @@ int vgif = true;
220module_param(vgif, int, 0444);221module_param(vgif, int, 0444);
221 222 
222/* enable/disable LBR virtualization */223/* enable/disable LBR virtualization */
223-static int lbrv = true;224+int lbrv = true;
224module_param(lbrv, int, 0444);225module_param(lbrv, int, 0444);
225 226 
226static int tsc_scaling = true;227static int tsc_scaling = true;
@@ -733,7 +734,7 @@ static int svm_cpu_init(int cpu)
733 int ret = -ENOMEM;734 int ret = -ENOMEM;
734 735 
735 memset(sd, 0, sizeof(struct svm_cpu_data));736 memset(sd, 0, sizeof(struct svm_cpu_data));
736- sd->save_area = alloc_page(GFP_KERNEL | __GFP_ZERO);737+ sd->save_area = snp_safe_alloc_page(NULL);
737 if (!sd->save_area)738 if (!sd->save_area)
738 return ret;739 return ret;
739 740 
@@ -1020,7 +1021,7 @@ void svm_copy_lbrs(struct vmcb *to_vmcb, struct vmcb *from_vmcb)
1020 vmcb_mark_dirty(to_vmcb, VMCB_LBR);1021 vmcb_mark_dirty(to_vmcb, VMCB_LBR);
1021}1022}
1022 1023 
1023-static void svm_enable_lbrv(struct kvm_vcpu *vcpu)1024+void svm_enable_lbrv(struct kvm_vcpu *vcpu)
1024{1025{
1025 struct vcpu_svm *svm = to_svm(vcpu);1026 struct vcpu_svm *svm = to_svm(vcpu);
1026 1027 
@@ -1030,6 +1031,9 @@ static void svm_enable_lbrv(struct kvm_vcpu *vcpu)
1030 set_msr_interception(vcpu, svm->msrpm, MSR_IA32_LASTINTFROMIP, 1, 1);1031 set_msr_interception(vcpu, svm->msrpm, MSR_IA32_LASTINTFROMIP, 1, 1);
1031 set_msr_interception(vcpu, svm->msrpm, MSR_IA32_LASTINTTOIP, 1, 1);1032 set_msr_interception(vcpu, svm->msrpm, MSR_IA32_LASTINTTOIP, 1, 1);
1032 1033 
1034+ if (sev_es_guest(vcpu->kvm))
1035+ set_msr_interception(vcpu, svm->msrpm, MSR_IA32_DEBUGCTLMSR, 1, 1);
1036+ 
1033 /* Move the LBR msrs to the vmcb02 so that the guest can see them. */1037 /* Move the LBR msrs to the vmcb02 so that the guest can see them. */
1034 if (is_guest_mode(vcpu))1038 if (is_guest_mode(vcpu))
1035 svm_copy_lbrs(svm->vmcb, svm->vmcb01.ptr);1039 svm_copy_lbrs(svm->vmcb, svm->vmcb01.ptr);
@@ -1039,6 +1043,8 @@ static void svm_disable_lbrv(struct kvm_vcpu *vcpu)
1039{1043{
1040 struct vcpu_svm *svm = to_svm(vcpu);1044 struct vcpu_svm *svm = to_svm(vcpu);
1041 1045 
1046+ KVM_BUG_ON(sev_es_guest(vcpu->kvm), vcpu->kvm);
1047+ 
1042 svm->vmcb->control.virt_ext &= ~LBR_CTL_ENABLE_MASK;1048 svm->vmcb->control.virt_ext &= ~LBR_CTL_ENABLE_MASK;
1043 set_msr_interception(vcpu, svm->msrpm, MSR_IA32_LASTBRANCHFROMIP, 0, 0);1049 set_msr_interception(vcpu, svm->msrpm, MSR_IA32_LASTBRANCHFROMIP, 0, 0);
1044 set_msr_interception(vcpu, svm->msrpm, MSR_IA32_LASTBRANCHTOIP, 0, 0);1050 set_msr_interception(vcpu, svm->msrpm, MSR_IA32_LASTBRANCHTOIP, 0, 0);
@@ -1446,6 +1452,9 @@ static void svm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
1446 svm->spec_ctrl = 0;1452 svm->spec_ctrl = 0;
1447 svm->virt_spec_ctrl = 0;1453 svm->virt_spec_ctrl = 0;
1448 1454 
1455+ if (init_event)
1456+ sev_snp_init_protected_guest_state(vcpu);
1457+ 
1449 init_vmcb(vcpu);1458 init_vmcb(vcpu);
1450 1459 
1451 if (!init_event)1460 if (!init_event)
@@ -1469,7 +1478,7 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
1469 svm = to_svm(vcpu);1478 svm = to_svm(vcpu);
1470 1479 
1471 err = -ENOMEM;1480 err = -ENOMEM;
1472- vmcb01_page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO);1481+ vmcb01_page = snp_safe_alloc_page(vcpu);
1473 if (!vmcb01_page)1482 if (!vmcb01_page)
1474 goto out;1483 goto out;
1475 1484 
@@ -1478,7 +1487,7 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
1478 * SEV-ES guests require a separate VMSA page used to contain1487 * SEV-ES guests require a separate VMSA page used to contain
1479 * the encrypted register state of the guest.1488 * the encrypted register state of the guest.
1480 */1489 */
1481- vmsa_page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO);1490+ vmsa_page = snp_safe_alloc_page(vcpu);
1482 if (!vmsa_page)1491 if (!vmsa_page)
1483 goto error_free_vmcb_page;1492 goto error_free_vmcb_page;
1484 1493 
@@ -1486,14 +1495,6 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
1486 if (csv2_setup_reset_vmsa(svm))1495 if (csv2_setup_reset_vmsa(svm))
1487 goto error_free_vmsa_page;1496 goto error_free_vmsa_page;
1488 }1497 }
1489- 
1490- /*
1491- * SEV-ES guests maintain an encrypted version of their FPU
1492- * state which is restored and saved on VMRUN and VMEXIT.
1493- * Mark vcpu->arch.guest_fpu->fpstate as scratch so it won't
1494- * do xsave/xrstor on it.
1495- */
1496- fpstate_set_confidential(&vcpu->arch.guest_fpu);
1497 }1498 }
1498 1499 
1499 err = avic_init_vcpu(svm);1500 err = avic_init_vcpu(svm);
@@ -1579,7 +1580,7 @@ static void svm_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
1579 struct sev_es_save_area *hostsa;1580 struct sev_es_save_area *hostsa;
1580 hostsa = (struct sev_es_save_area *)(page_address(sd->save_area) + 0x400);1581 hostsa = (struct sev_es_save_area *)(page_address(sd->save_area) + 0x400);
1581 1582 
1582- sev_es_prepare_switch_to_guest(hostsa);1583+ sev_es_prepare_switch_to_guest(svm, hostsa);
1583 }1584 }
1584 1585 
1585 if (tsc_scaling)1586 if (tsc_scaling)
@@ -2107,15 +2108,33 @@ static int pf_interception(struct kvm_vcpu *vcpu)
2107static int npf_interception(struct kvm_vcpu *vcpu)2108static int npf_interception(struct kvm_vcpu *vcpu)
2108{2109{
2109 struct vcpu_svm *svm = to_svm(vcpu);2110 struct vcpu_svm *svm = to_svm(vcpu);
2111+ int rc;
2110 2112 
2111 u64 fault_address = svm->vmcb->control.exit_info_2;2113 u64 fault_address = svm->vmcb->control.exit_info_2;
2112 u64 error_code = svm->vmcb->control.exit_info_1;2114 u64 error_code = svm->vmcb->control.exit_info_1;
2113 2115 
2116+ /*
2117+ * WARN if hardware generates a fault with an error code that collides
2118+ * with KVM-defined sythentic flags. Clear the flags and continue on,
2119+ * i.e. don't terminate the VM, as KVM can't possibly be relying on a
2120+ * flag that KVM doesn't know about.
2121+ */
2122+ if (WARN_ON_ONCE(error_code & PFERR_SYNTHETIC_MASK))
2123+ error_code &= ~PFERR_SYNTHETIC_MASK;
2124+ 
2125+ if (sev_snp_guest(vcpu->kvm) && (error_code & PFERR_GUEST_ENC_MASK))
2126+ error_code |= PFERR_PRIVATE_ACCESS;
2127+ 
2114 trace_kvm_page_fault(vcpu, fault_address, error_code);2128 trace_kvm_page_fault(vcpu, fault_address, error_code);
2115- return kvm_mmu_page_fault(vcpu, fault_address, error_code,2129+ rc = kvm_mmu_page_fault(vcpu, fault_address, error_code,
2116- static_cpu_has(X86_FEATURE_DECODEASSISTS) ?2130+ static_cpu_has(X86_FEATURE_DECODEASSISTS) ?
2117- svm->vmcb->control.insn_bytes : NULL,2131+ svm->vmcb->control.insn_bytes : NULL,
2118- svm->vmcb->control.insn_len);2132+ svm->vmcb->control.insn_len);
2133+ 
2134+ if (rc > 0 && error_code & PFERR_GUEST_RMP_MASK)
2135+ sev_handle_rmp_fault(vcpu, fault_address, error_code);
2136+ 
2137+ return rc;
2119}2138}
2120 2139 
2121static int db_interception(struct kvm_vcpu *vcpu)2140static int db_interception(struct kvm_vcpu *vcpu)
@@ -2886,10 +2905,24 @@ static int svm_get_msr_feature(struct kvm_msr_entry *msr)
2886 return 0;2905 return 0;
2887}2906}
2888 2907 
2908+static bool
2909+sev_es_prevent_msr_access(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
2910+{
2911+ return sev_es_guest(vcpu->kvm) &&
2912+ vcpu->arch.guest_state_protected &&
2913+ svm_msrpm_offset(msr_info->index) != MSR_INVALID &&
2914+ !msr_write_intercepted(vcpu, msr_info->index);
2915+}
2916+ 
2889static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)2917static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
2890{2918{
2891 struct vcpu_svm *svm = to_svm(vcpu);2919 struct vcpu_svm *svm = to_svm(vcpu);
2892 2920 
2921+ if (sev_es_prevent_msr_access(vcpu, msr_info)) {
2922+ msr_info->data = 0;
2923+ return vcpu->kvm->arch_ext.has_protected_state ? -EINVAL : 0;
2924+ }
2925+ 
2893 switch (msr_info->index) {2926 switch (msr_info->index) {
2894 case MSR_AMD64_TSC_RATIO:2927 case MSR_AMD64_TSC_RATIO:
2895 if (!msr_info->host_initiated &&2928 if (!msr_info->host_initiated &&
@@ -3012,11 +3045,7 @@ static int svm_complete_emulated_msr(struct kvm_vcpu *vcpu, int err)
3012 if (!err || !sev_es_guest(vcpu->kvm) || WARN_ON_ONCE(!svm->sev_es.ghcb))3045 if (!err || !sev_es_guest(vcpu->kvm) || WARN_ON_ONCE(!svm->sev_es.ghcb))
3013 return kvm_complete_insn_gp(vcpu, err);3046 return kvm_complete_insn_gp(vcpu, err);
3014 3047 
3015- ghcb_set_sw_exit_info_1(svm->sev_es.ghcb, 1);3048+ svm_vmgexit_inject_exception(svm, X86_TRAP_GP);
3016- ghcb_set_sw_exit_info_2(svm->sev_es.ghcb,
3017- X86_TRAP_GP |
3018- SVM_EVTINJ_TYPE_EXEPT |
3019- SVM_EVTINJ_VALID);
3020 return 1;3049 return 1;
3021}3050}
3022 3051 
@@ -3052,6 +3081,10 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
3052 3081 
3053 u32 ecx = msr->index;3082 u32 ecx = msr->index;
3054 u64 data = msr->data;3083 u64 data = msr->data;
3084+ 
3085+ if (sev_es_prevent_msr_access(vcpu, msr))
3086+ return vcpu->kvm->arch_ext.has_protected_state ? -EINVAL : 0;
3087+ 
3055 switch (ecx) {3088 switch (ecx) {
3056 case MSR_AMD64_TSC_RATIO:3089 case MSR_AMD64_TSC_RATIO:
3057 3090 
@@ -3471,7 +3504,9 @@ static int (*const svm_exit_handlers[])(struct kvm_vcpu *vcpu) = {
3471 [SVM_EXIT_RSM] = rsm_interception,3504 [SVM_EXIT_RSM] = rsm_interception,
3472 [SVM_EXIT_AVIC_INCOMPLETE_IPI] = avic_incomplete_ipi_interception,3505 [SVM_EXIT_AVIC_INCOMPLETE_IPI] = avic_incomplete_ipi_interception,
3473 [SVM_EXIT_AVIC_UNACCELERATED_ACCESS] = avic_unaccelerated_access_interception,3506 [SVM_EXIT_AVIC_UNACCELERATED_ACCESS] = avic_unaccelerated_access_interception,
3507+#ifdef CONFIG_KVM_AMD_SEV
3474 [SVM_EXIT_VMGEXIT] = sev_handle_vmgexit,3508 [SVM_EXIT_VMGEXIT] = sev_handle_vmgexit,
3509+#endif
3475};3510};
3476 3511 
3477static void dump_vmcb(struct kvm_vcpu *vcpu)3512static void dump_vmcb(struct kvm_vcpu *vcpu)
@@ -3695,7 +3730,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
3695 return svm_invoke_exit_handler(vcpu, exit_code);3730 return svm_invoke_exit_handler(vcpu, exit_code);
3696}3731}
3697 3732 
3698-static void pre_svm_run(struct kvm_vcpu *vcpu)3733+static int pre_svm_run(struct kvm_vcpu *vcpu)
3699{3734{
3700 struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu);3735 struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu);
3701 struct vcpu_svm *svm = to_svm(vcpu);3736 struct vcpu_svm *svm = to_svm(vcpu);
@@ -3717,6 +3752,8 @@ static void pre_svm_run(struct kvm_vcpu *vcpu)
3717 /* FIXME: handle wraparound of asid_generation */3752 /* FIXME: handle wraparound of asid_generation */
3718 if (svm->current_vmcb->asid_generation != sd->asid_generation)3753 if (svm->current_vmcb->asid_generation != sd->asid_generation)
3719 new_asid(svm, sd);3754 new_asid(svm, sd);
3755+ 
3756+ return 0;
3720}3757}
3721 3758 
3722static void svm_inject_nmi(struct kvm_vcpu *vcpu)3759static void svm_inject_nmi(struct kvm_vcpu *vcpu)
@@ -4254,6 +4291,9 @@ static void svm_cancel_injection(struct kvm_vcpu *vcpu)
4254 4291 
4255static int svm_vcpu_pre_run(struct kvm_vcpu *vcpu)4292static int svm_vcpu_pre_run(struct kvm_vcpu *vcpu)
4256{4293{
4294+ if (to_kvm_sev_info(vcpu->kvm)->need_init)
4295+ return -EINVAL;
4296+ 
4257 return 1;4297 return 1;
4258}4298}
4259 4299 
@@ -4346,7 +4386,12 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
4346 smp_send_reschedule(vcpu->cpu);4386 smp_send_reschedule(vcpu->cpu);
4347 }4387 }
4348 4388 
4349- pre_svm_run(vcpu);4389+ if (pre_svm_run(vcpu)) {
4390+ vcpu->run->exit_reason = KVM_EXIT_FAIL_ENTRY;
4391+ vcpu->run->fail_entry.hardware_entry_failure_reason = SVM_EXIT_ERR;
4392+ vcpu->run->fail_entry.cpu = vcpu->cpu;
4393+ return EXIT_FASTPATH_EXIT_USERSPACE;
4394+ }
4350 4395 
4351 sync_lapic_to_cr8(vcpu);4396 sync_lapic_to_cr8(vcpu);
4352 4397 
@@ -5109,6 +5154,18 @@ static void svm_vm_destroy(struct kvm *kvm)
5109 5154 
5110static int svm_vm_init(struct kvm *kvm)5155static int svm_vm_init(struct kvm *kvm)
5111{5156{
5157+ int type = kvm->arch_ext.vm_type;
5158+ 
5159+ if (type != KVM_X86_DEFAULT_VM &&
5160+ type != KVM_X86_SW_PROTECTED_VM) {
5161+ kvm->arch_ext.has_protected_state =
5162+ (type == KVM_X86_SEV_ES_VM || type == KVM_X86_SNP_VM);
5163+ to_kvm_sev_info(kvm)->need_init = true;
5164+ 
5165+ kvm->arch_ext.has_private_mem = (type == KVM_X86_SNP_VM);
5166+ kvm->arch_ext.pre_fault_allowed = !kvm->arch_ext.has_private_mem;
5167+ }
5168+ 
5112 if (!pause_filter_count || !pause_filter_thresh)5169 if (!pause_filter_count || !pause_filter_thresh)
5113 kvm->arch.pause_in_guest = true;5170 kvm->arch.pause_in_guest = true;
5114 5171 
@@ -5121,6 +5178,16 @@ static int svm_vm_init(struct kvm *kvm)
5121 return 0;5178 return 0;
5122}5179}
5123 5180 
5181+static void *svm_alloc_apic_backing_page(struct kvm_vcpu *vcpu)
5182+{
5183+ struct page *page = snp_safe_alloc_page(vcpu);
5184+ 
5185+ if (!page)
5186+ return NULL;
5187+ 
5188+ return page_address(page);
5189+}
5190+ 
5124static struct kvm_x86_ops svm_x86_ops __initdata = {5191static struct kvm_x86_ops svm_x86_ops __initdata = {
5125 .name = KBUILD_MODNAME,5192 .name = KBUILD_MODNAME,
5126 5193 
@@ -5238,6 +5305,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
5238 .enable_smi_window = svm_enable_smi_window,5305 .enable_smi_window = svm_enable_smi_window,
5239#endif5306#endif
5240 5307 
5308+#ifdef CONFIG_KVM_AMD_SEV
5309+ .dev_get_attr = sev_dev_get_attr,
5241 .mem_enc_ioctl = sev_mem_enc_ioctl,5310 .mem_enc_ioctl = sev_mem_enc_ioctl,
5242 .mem_enc_register_region = sev_mem_enc_register_region,5311 .mem_enc_register_region = sev_mem_enc_register_region,
5243 .mem_enc_unregister_region = sev_mem_enc_unregister_region,5312 .mem_enc_unregister_region = sev_mem_enc_unregister_region,
@@ -5247,6 +5316,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
5247 .vm_move_enc_context_from = sev_vm_move_enc_context_from,5316 .vm_move_enc_context_from = sev_vm_move_enc_context_from,
5248 5317 
5249 .can_emulate_instruction = svm_can_emulate_instruction,5318 .can_emulate_instruction = svm_can_emulate_instruction,
5319+#endif
5250 5320 
5251 .apic_init_signal_blocked = svm_apic_init_signal_blocked,5321 .apic_init_signal_blocked = svm_apic_init_signal_blocked,
5252 5322 
@@ -5255,6 +5325,11 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
5255 5325 
5256 .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector,5326 .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector,
5257 .vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons,5327 .vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons,
5328+ .alloc_apic_backing_page = svm_alloc_apic_backing_page,
5329+ 
5330+ .gmem_prepare = sev_gmem_prepare,
5331+ .gmem_invalidate = sev_gmem_invalidate,
5332+ .private_max_mapping_level = sev_private_max_mapping_level,
5258};5333};
5259 5334 
5260/*5335/*
@@ -5485,6 +5560,12 @@ static __init int svm_hardware_setup(void)
5485 5560 
5486 nrips = nrips && boot_cpu_has(X86_FEATURE_NRIPS);5561 nrips = nrips && boot_cpu_has(X86_FEATURE_NRIPS);
5487 5562 
5563+ if (lbrv) {
5564+ if (!boot_cpu_has(X86_FEATURE_LBRV))
5565+ lbrv = false;
5566+ else
5567+ pr_info("LBR virtualization supported\n");
5568+ }
5488 /*5569 /*
5489 * Note, SEV setup consumes npt_enabled and enable_mmio_caching (which5570 * Note, SEV setup consumes npt_enabled and enable_mmio_caching (which
5490 * may be modified by svm_adjust_mmio_mask()), as well as nrips.5571 * may be modified by svm_adjust_mmio_mask()), as well as nrips.
@@ -5538,14 +5619,6 @@ static __init int svm_hardware_setup(void)
5538 svm_x86_ops.set_vnmi_pending = NULL;5619 svm_x86_ops.set_vnmi_pending = NULL;
5539 }5620 }
5540 5621 
5541- 
5542- if (lbrv) {
5543- if (!boot_cpu_has(X86_FEATURE_LBRV))
5544- lbrv = false;
5545- else
5546- pr_info("LBR virtualization supported\n");
5547- }
5548- 
5549 if (!enable_pmu)5622 if (!enable_pmu)
5550 pr_info("PMU virtualization is disabled\n");5623 pr_info("PMU virtualization is disabled\n");
5551 5624 
March/x86/kvm/svm/svm.h+123-28
@@ -32,7 +32,7 @@
32 32 
33#define GUEST_PAT_WB_ATTR 0x060606060606060633#define GUEST_PAT_WB_ATTR 0x0606060606060606
34 34 
35-#define MAX_DIRECT_ACCESS_MSRS 4735+#define MAX_DIRECT_ACCESS_MSRS 48
36#define MSRPM_OFFSETS 3236#define MSRPM_OFFSETS 32
37extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly;37extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly;
38extern bool npt_enabled;38extern bool npt_enabled;
@@ -41,6 +41,7 @@ extern int vgif;
41extern bool intercept_smi;41extern bool intercept_smi;
42extern bool x2avic_enabled;42extern bool x2avic_enabled;
43extern bool vnmi;43extern bool vnmi;
44+extern int lbrv;
44 45 
45/*46/*
46 * Clean bits in VMCB.47 * Clean bits in VMCB.
@@ -81,17 +82,24 @@ enum {
81struct kvm_sev_info {82struct kvm_sev_info {
82 bool active; /* SEV enabled guest */83 bool active; /* SEV enabled guest */
83 bool es_active; /* SEV-ES enabled guest */84 bool es_active; /* SEV-ES enabled guest */
85+ bool need_init; /* waiting for SEV_INIT2 */
84 unsigned int asid; /* ASID used for this guest */86 unsigned int asid; /* ASID used for this guest */
85 unsigned int handle; /* SEV firmware handle */87 unsigned int handle; /* SEV firmware handle */
86 int fd; /* SEV device fd */88 int fd; /* SEV device fd */
87 unsigned long pages_locked; /* Number of pages locked */89 unsigned long pages_locked; /* Number of pages locked */
88 struct list_head regions_list; /* List of registered regions */90 struct list_head regions_list; /* List of registered regions */
89 u64 ap_jump_table; /* SEV-ES AP Jump Table address */91 u64 ap_jump_table; /* SEV-ES AP Jump Table address */
92+ u64 vmsa_features;
93+ u16 ghcb_version; /* Highest guest GHCB protocol version allowed */
90 struct kvm *enc_context_owner; /* Owner of copied encryption context */94 struct kvm *enc_context_owner; /* Owner of copied encryption context */
91 struct list_head mirror_vms; /* List of VMs mirroring */95 struct list_head mirror_vms; /* List of VMs mirroring */
92 struct list_head mirror_entry; /* Use as a list entry of mirrors */96 struct list_head mirror_entry; /* Use as a list entry of mirrors */
93 struct misc_cg *misc_cg; /* For misc cgroup accounting */97 struct misc_cg *misc_cg; /* For misc cgroup accounting */
94 atomic_t migration_in_progress;98 atomic_t migration_in_progress;
99+ void *snp_context; /* SNP guest context page */
100+ void *guest_req_buf; /* Bounce buffer for SNP Guest Request input */
101+ void *guest_resp_buf; /* Bounce buffer for SNP Guest Request output */
102+ struct mutex guest_req_mutex; /* Must acquire before using bounce buffers */
95};103};
96 104 
97struct kvm_svm {105struct kvm_svm {
@@ -198,6 +206,7 @@ struct vcpu_sev_es_state {
198 u8 valid_bitmap[16];206 u8 valid_bitmap[16];
199 struct kvm_host_map ghcb_map;207 struct kvm_host_map ghcb_map;
200 bool received_first_sipi;208 bool received_first_sipi;
209+ unsigned int ap_reset_hold_type;
201 210 
202 /* SEV-ES scratch area support */211 /* SEV-ES scratch area support */
203 u64 sw_scratch;212 u64 sw_scratch;
@@ -206,12 +215,24 @@ struct vcpu_sev_es_state {
206 bool ghcb_sa_sync;215 bool ghcb_sa_sync;
207 bool ghcb_sa_free;216 bool ghcb_sa_free;
208 217 
218+ 
209#ifdef CONFIG_HYGON_CSV219#ifdef CONFIG_HYGON_CSV
210 /* migrated ghcb mapping state for HYGON CSV2 */220 /* migrated ghcb mapping state for HYGON CSV2 */
211 bool receiver_ghcb_map_fail;221 bool receiver_ghcb_map_fail;
212 /* CSV2 reboot vmsa */222 /* CSV2 reboot vmsa */
213 struct vmcb_save_area *reset_vmsa;223 struct vmcb_save_area *reset_vmsa;
214#endif224#endif
225+ /* SNP Page-State-Change buffer entries currently being processed */
226+ u16 psc_idx;
227+ u16 psc_inflight;
228+ bool psc_2m;
229+ 
230+ u64 ghcb_registered_gpa;
231+ 
232+ struct mutex snp_vmsa_mutex; /* Used to handle concurrent updates of VMSA. */
233+ gpa_t snp_vmsa_gpa;
234+ bool snp_ap_waiting_for_reset;
235+ bool snp_has_guest_vmsa;
215};236};
216 237 
217struct vcpu_svm {238struct vcpu_svm {
@@ -326,26 +347,39 @@ static __always_inline struct kvm_svm *to_kvm_svm(struct kvm *kvm)
326 return container_of(kvm, struct kvm_svm, kvm);347 return container_of(kvm, struct kvm_svm, kvm);
327}348}
328 349 
329-static __always_inline bool sev_guest(struct kvm *kvm)350+static __always_inline struct kvm_sev_info *to_kvm_sev_info(struct kvm *kvm)
330{351{
331-#ifdef CONFIG_KVM_AMD_SEV352+ return &to_kvm_svm(kvm)->sev_info;
332- struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
333- 
334- return sev->active;
335-#else
336- return false;
337-#endif
338}353}
339 354 
355+#ifdef CONFIG_KVM_AMD_SEV
356+static __always_inline bool sev_guest(struct kvm *kvm)
357+{
358+ return to_kvm_sev_info(kvm)->active;
359+}
340static __always_inline bool sev_es_guest(struct kvm *kvm)360static __always_inline bool sev_es_guest(struct kvm *kvm)
341{361{
342-#ifdef CONFIG_KVM_AMD_SEV362+ struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
343- struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
344 363 
345 return sev->es_active && !WARN_ON_ONCE(!sev->active);364 return sev->es_active && !WARN_ON_ONCE(!sev->active);
365+}
366+ 
367+static __always_inline bool sev_snp_guest(struct kvm *kvm)
368+{
369+ struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
370+ 
371+ return (sev->vmsa_features & SVM_SEV_FEAT_SNP_ACTIVE) &&
372+ !WARN_ON_ONCE(!sev_es_guest(kvm));
373+}
346#else374#else
347- return false;375+#define sev_guest(kvm) false
376+#define sev_es_guest(kvm) false
377+#define sev_snp_guest(kvm) false
348#endif378#endif
379+ 
380+static inline bool ghcb_gpa_is_registered(struct vcpu_svm *svm, u64 val)
381+{
382+ return svm->sev_es.ghcb_registered_gpa == val;
349}383}
350 384 
351static inline void vmcb_mark_all_dirty(struct vmcb *vmcb)385static inline void vmcb_mark_all_dirty(struct vmcb *vmcb)
@@ -539,6 +573,35 @@ static inline bool is_vnmi_enabled(struct vcpu_svm *svm)
539 return false;573 return false;
540}574}
541 575 
576+static inline void svm_vmgexit_set_return_code(struct vcpu_svm *svm,
577+ u64 response, u64 data)
578+{
579+ ghcb_set_sw_exit_info_1(svm->sev_es.ghcb, response);
580+ ghcb_set_sw_exit_info_2(svm->sev_es.ghcb, data);
581+}
582+ 
583+static inline void svm_vmgexit_inject_exception(struct vcpu_svm *svm, u8 vector)
584+{
585+ u64 data = SVM_EVTINJ_VALID | SVM_EVTINJ_TYPE_EXEPT | vector;
586+ 
587+ svm_vmgexit_set_return_code(svm, GHCB_HV_RESP_ISSUE_EXCEPTION, data);
588+}
589+ 
590+static inline void svm_vmgexit_bad_input(struct vcpu_svm *svm, u64 suberror)
591+{
592+ svm_vmgexit_set_return_code(svm, GHCB_HV_RESP_MALFORMED_INPUT, suberror);
593+}
594+ 
595+static inline void svm_vmgexit_success(struct vcpu_svm *svm, u64 data)
596+{
597+ svm_vmgexit_set_return_code(svm, GHCB_HV_RESP_NO_ACTION, data);
598+}
599+ 
600+static inline void svm_vmgexit_no_action(struct vcpu_svm *svm, u64 data)
601+{
602+ svm_vmgexit_set_return_code(svm, GHCB_HV_RESP_NO_ACTION, data);
603+}
604+ 
542/* svm.c */605/* svm.c */
543#define MSR_INVALID 0xffffffffU606#define MSR_INVALID 0xffffffffU
544 607 
@@ -551,6 +614,7 @@ u32 *svm_vcpu_alloc_msrpm(void);
551void svm_vcpu_init_msrpm(struct kvm_vcpu *vcpu, u32 *msrpm);614void svm_vcpu_init_msrpm(struct kvm_vcpu *vcpu, u32 *msrpm);
552void svm_vcpu_free_msrpm(u32 *msrpm);615void svm_vcpu_free_msrpm(u32 *msrpm);
553void svm_copy_lbrs(struct vmcb *to_vmcb, struct vmcb *from_vmcb);616void svm_copy_lbrs(struct vmcb *to_vmcb, struct vmcb *from_vmcb);
617+void svm_enable_lbrv(struct kvm_vcpu *vcpu);
554void svm_update_lbrv(struct kvm_vcpu *vcpu);618void svm_update_lbrv(struct kvm_vcpu *vcpu);
555 619 
556int svm_set_efer(struct kvm_vcpu *vcpu, u64 efer);620int svm_set_efer(struct kvm_vcpu *vcpu, u64 efer);
@@ -674,13 +738,16 @@ void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu);
674 738 
675/* sev.c */739/* sev.c */
676 740 
677-#define GHCB_VERSION_MAX 1ULL741+int pre_sev_run(struct vcpu_svm *svm, int cpu);
678-#define GHCB_VERSION_MIN 1ULL742+void sev_init_vmcb(struct vcpu_svm *svm);
743+void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm);
744+int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in);
745+void sev_es_vcpu_reset(struct vcpu_svm *svm);
746+void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector);
747+void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, struct sev_es_save_area *hostsa);
748+void sev_es_unmap_ghcb(struct vcpu_svm *svm);
679 749 
680- 750+#ifdef CONFIG_KVM_AMD_SEV
681-extern unsigned int max_sev_asid;
682- 
683-void sev_vm_destroy(struct kvm *kvm);
684int sev_mem_enc_ioctl(struct kvm *kvm, void __user *argp);751int sev_mem_enc_ioctl(struct kvm *kvm, void __user *argp);
685int sev_mem_enc_register_region(struct kvm *kvm,752int sev_mem_enc_register_region(struct kvm *kvm,
686 struct kvm_enc_region *range);753 struct kvm_enc_region *range);
@@ -689,21 +756,49 @@ int sev_mem_enc_unregister_region(struct kvm *kvm,
689int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd);756int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd);
690int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd);757int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd);
691void sev_guest_memory_reclaimed(struct kvm *kvm);758void sev_guest_memory_reclaimed(struct kvm *kvm);
759+int sev_handle_vmgexit(struct kvm_vcpu *vcpu);
692 760 
693-void pre_sev_run(struct vcpu_svm *svm, int cpu);761+/* These symbols are used in common code and are stubbed below. */
762+struct page *snp_safe_alloc_page(struct kvm_vcpu *vcpu);
763+void sev_free_vcpu(struct kvm_vcpu *vcpu);
764+void sev_vm_destroy(struct kvm *kvm);
694void __init sev_set_cpu_caps(void);765void __init sev_set_cpu_caps(void);
695void __init sev_hardware_setup(void);766void __init sev_hardware_setup(void);
696void sev_hardware_unsetup(void);767void sev_hardware_unsetup(void);
697int sev_cpu_init(struct svm_cpu_data *sd);768int sev_cpu_init(struct svm_cpu_data *sd);
698-void sev_init_vmcb(struct vcpu_svm *svm);769+int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
699-void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm);770+extern unsigned int max_sev_asid;
700-void sev_free_vcpu(struct kvm_vcpu *vcpu);771+void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
701-int sev_handle_vmgexit(struct kvm_vcpu *vcpu);772+void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu);
702-int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in);773+int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
703-void sev_es_vcpu_reset(struct vcpu_svm *svm);774+void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
704-void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector);775+int sev_private_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn);
705-void sev_es_prepare_switch_to_guest(struct sev_es_save_area *hostsa);776+#else
706-void sev_es_unmap_ghcb(struct vcpu_svm *svm);777+static inline struct page *snp_safe_alloc_page(struct kvm_vcpu *vcpu) {
778+ return alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO);
779+}
780+ 
781+static inline void sev_free_vcpu(struct kvm_vcpu *vcpu) {}
782+static inline void sev_vm_destroy(struct kvm *kvm) {}
783+static inline void __init sev_set_cpu_caps(void) {}
784+static inline void __init sev_hardware_setup(void) {}
785+static inline void sev_hardware_unsetup(void) {}
786+static inline int sev_cpu_init(struct svm_cpu_data *sd) { return 0; }
787+static inline int sev_dev_get_attr(u32 group, u64 attr, u64 *val) { return -ENXIO; }
788+#define max_sev_asid 0
789+static inline void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) {}
790+static inline void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) {}
791+static inline int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)
792+{
793+ return 0;
794+}
795+static inline void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) {}
796+static inline int sev_private_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn)
797+{
798+ return 0;
799+}
800+ 
801+#endif
707 802 
708/* vmenter.S */803/* vmenter.S */
709 804 
March/x86/kvm/trace.h+31-0
@@ -1934,6 +1934,37 @@ TRACE_EVENT(kvm_csv3_sp_remove,
1934 __entry->track_pfn, __entry->track_hva, __entry->order)1934 __entry->track_pfn, __entry->track_hva, __entry->order)
1935);1935);
1936 1936 
1937+/*
1938+ * Tracepoint for #NPFs due to RMP faults.
1939+ */
1940+TRACE_EVENT(kvm_rmp_fault,
1941+ TP_PROTO(struct kvm_vcpu *vcpu, u64 gpa, u64 pfn, u64 error_code,
1942+ int rmp_level, int psmash_ret),
1943+ TP_ARGS(vcpu, gpa, pfn, error_code, rmp_level, psmash_ret),
1944+ 
1945+ TP_STRUCT__entry(
1946+ __field(unsigned int, vcpu_id)
1947+ __field(u64, gpa)
1948+ __field(u64, pfn)
1949+ __field(u64, error_code)
1950+ __field(int, rmp_level)
1951+ __field(int, psmash_ret)
1952+ ),
1953+ 
1954+ TP_fast_assign(
1955+ __entry->vcpu_id = vcpu->vcpu_id;
1956+ __entry->gpa = gpa;
1957+ __entry->pfn = pfn;
1958+ __entry->error_code = error_code;
1959+ __entry->rmp_level = rmp_level;
1960+ __entry->psmash_ret = psmash_ret;
1961+ ),
1962+ 
1963+ TP_printk("vcpu %u gpa %016llx pfn 0x%llx error_code 0x%llx rmp_level %d psmash_ret %d",
1964+ __entry->vcpu_id, __entry->gpa, __entry->pfn,
1965+ __entry->error_code, __entry->rmp_level, __entry->psmash_ret)
1966+);
1967+ 
1937#endif /* _TRACE_KVM_H */1968#endif /* _TRACE_KVM_H */
1938 1969 
1939#undef TRACE_INCLUDE_PATH1970#undef TRACE_INCLUDE_PATH
March/x86/kvm/vmx/vmx.c+5-6
@@ -6793,10 +6793,10 @@ static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu)
6793 return;6793 return;
6794 6794 
6795 /*6795 /*
6796- * Grab the memslot so that the hva lookup for the mmu_notifier retry6796+ * Explicitly grab the memslot using KVM's internal slot ID to ensure
6797- * is guaranteed to use the same memslot as the pfn lookup, i.e. rely6797+ * KVM doesn't unintentionally grab a userspace memslot. It _should_
6798- * on the pfn lookup's validation of the memslot to ensure a valid hva6798+ * be impossible for userspace to create a memslot for the APIC when
6799- * is used for the retry check.6799+ * APICv is enabled, but paranoia won't hurt in this case.
6800 */6800 */
6801 slot = id_to_memslot(slots, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT);6801 slot = id_to_memslot(slots, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT);
6802 if (!slot || slot->flags & KVM_MEMSLOT_INVALID)6802 if (!slot || slot->flags & KVM_MEMSLOT_INVALID)
@@ -6821,8 +6821,7 @@ static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu)
6821 return;6821 return;
6822 6822 
6823 read_lock(&vcpu->kvm->mmu_lock);6823 read_lock(&vcpu->kvm->mmu_lock);
6824- if (mmu_invalidate_retry_hva(kvm, mmu_seq,6824+ if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn)) {
6825- gfn_to_hva_memslot(slot, gfn))) {
6826 kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu);6825 kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu);
6827 read_unlock(&vcpu->kvm->mmu_lock);6826 read_unlock(&vcpu->kvm->mmu_lock);
6828 goto out;6827 goto out;
March/x86/kvm/x86.c+210-118
March/x86/kvm/x86.h+4-4
@@ -25,6 +25,8 @@ struct kvm_caps {
25 bool has_bus_lock_exit;25 bool has_bus_lock_exit;
26 /* notify VM exit supported? */26 /* notify VM exit supported? */
27 bool has_notify_vmexit;27 bool has_notify_vmexit;
28+ /* bit mask of VM types */
29+ u32 supported_vm_types;
28 30 
29 u64 supported_mce_cap;31 u64 supported_mce_cap;
30 u64 supported_xcr0;32 u64 supported_xcr0;
@@ -585,10 +587,8 @@ bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type);
585 __reserved_bits; \587 __reserved_bits; \
586})588})
587 589 
588-int kvm_sev_es_mmio_write(struct kvm_vcpu *vcpu, gpa_t src, unsigned int bytes,590+int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool is_write, gpa_t gpa,
589- void *dst);591+ unsigned int bytes, void *data);
590-int kvm_sev_es_mmio_read(struct kvm_vcpu *vcpu, gpa_t src, unsigned int bytes,
591- void *dst);
592int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size,592int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size,
593 unsigned int port, void *data, unsigned int count,593 unsigned int port, void *data, unsigned int count,
594 int in);594 int in);
March/x86/kvm/xen.c+8-1
@@ -1113,7 +1113,9 @@ int kvm_xen_hvm_config(struct kvm *kvm, struct kvm_xen_hvm_config *xhc)
1113{1113{
1114 /* Only some feature flags need to be *enabled* by userspace */1114 /* Only some feature flags need to be *enabled* by userspace */
1115 u32 permitted_flags = KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL |1115 u32 permitted_flags = KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL |
1116- KVM_XEN_HVM_CONFIG_EVTCHN_SEND;1116+ KVM_XEN_HVM_CONFIG_EVTCHN_SEND |
1117+ KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE;
1118+ u32 old_flags;
1117 1119 
1118 if (xhc->flags & ~permitted_flags)1120 if (xhc->flags & ~permitted_flags)
1119 return -EINVAL;1121 return -EINVAL;
@@ -1134,9 +1136,14 @@ int kvm_xen_hvm_config(struct kvm *kvm, struct kvm_xen_hvm_config *xhc)
1134 else if (!xhc->msr && kvm->arch.xen_hvm_config.msr)1136 else if (!xhc->msr && kvm->arch.xen_hvm_config.msr)
1135 static_branch_slow_dec_deferred(&kvm_xen_enabled);1137 static_branch_slow_dec_deferred(&kvm_xen_enabled);
1136 1138 
1139+ old_flags = kvm->arch.xen_hvm_config.flags;
1137 memcpy(&kvm->arch.xen_hvm_config, xhc, sizeof(*xhc));1140 memcpy(&kvm->arch.xen_hvm_config, xhc, sizeof(*xhc));
1138 1141 
1139 mutex_unlock(&kvm->arch.xen.xen_lock);1142 mutex_unlock(&kvm->arch.xen.xen_lock);
1143+ 
1144+ if ((old_flags ^ xhc->flags) & KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE)
1145+ kvm_make_all_cpus_request(kvm, KVM_REQ_CLOCK_UPDATE);
1146+ 
1140 return 0;1147 return 0;
1141}1148}
1142 1149 
March/x86/mm/fault.c+5-0
@@ -34,6 +34,7 @@
34#include <asm/kvm_para.h> /* kvm_handle_async_pf */34#include <asm/kvm_para.h> /* kvm_handle_async_pf */
35#include <asm/vdso.h> /* fixup_vdso_exception() */35#include <asm/vdso.h> /* fixup_vdso_exception() */
36#include <asm/irq_stack.h>36#include <asm/irq_stack.h>
37+#include <asm/sev.h> /* snp_dump_hva_rmpentry() */
37 38 
38#define CREATE_TRACE_POINTS39#define CREATE_TRACE_POINTS
39#include <asm/trace/exceptions.h>40#include <asm/trace/exceptions.h>
@@ -547,6 +548,7 @@ show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long ad
547 !(error_code & X86_PF_PROT) ? "not-present page" :548 !(error_code & X86_PF_PROT) ? "not-present page" :
548 (error_code & X86_PF_RSVD) ? "reserved bit violation" :549 (error_code & X86_PF_RSVD) ? "reserved bit violation" :
549 (error_code & X86_PF_PK) ? "protection keys violation" :550 (error_code & X86_PF_PK) ? "protection keys violation" :
551+ (error_code & X86_PF_RMP) ? "RMP violation" :
550 "permissions violation");552 "permissions violation");
551 553 
552 if (!(error_code & X86_PF_USER) && user_mode(regs)) {554 if (!(error_code & X86_PF_USER) && user_mode(regs)) {
@@ -579,6 +581,9 @@ show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long ad
579 }581 }
580 582 
581 dump_pagetable(address);583 dump_pagetable(address);
584+ 
585+ if (error_code & X86_PF_RMP)
586+ snp_dump_hva_rmpentry(address);
582}587}
583 588 
584static noinline void589static noinline void
March/x86/mm/mem_encrypt.c+76-26
@@ -12,9 +12,12 @@
12#include <linux/swiotlb.h>12#include <linux/swiotlb.h>
13#include <linux/cc_platform.h>13#include <linux/cc_platform.h>
14#include <linux/mem_encrypt.h>14#include <linux/mem_encrypt.h>
15+#include <linux/virtio_anchor.h>
15 16 
16#include <asm/processor-hygon.h>17#include <asm/processor-hygon.h>
17 18 
19+#include <asm/sev.h>
20+ 
18/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */21/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
19bool force_dma_unencrypted(struct device *dev)22bool force_dma_unencrypted(struct device *dev)
20{23{
@@ -43,43 +46,50 @@ bool force_dma_unencrypted(struct device *dev)
43 46 
44static void print_mem_encrypt_feature_info(void)47static void print_mem_encrypt_feature_info(void)
45{48{
46- pr_info("Memory Encryption Features active:");49+ pr_info("Memory Encryption Features active: ");
47- 
48- if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) {
49- pr_cont(" Intel TDX\n");
50- return;
51- }
52 50 
53 if (is_x86_vendor_hygon()) {51 if (is_x86_vendor_hygon()) {
54 print_hygon_cc_feature_info();52 print_hygon_cc_feature_info();
55 return;53 return;
56 }54 }
57 55 
58- pr_cont(" AMD");56+ switch (cc_vendor) {
57+ case CC_VENDOR_INTEL:
58+ pr_cont("Intel TDX\n");
59+ break;
60+ case CC_VENDOR_AMD:
61+ pr_cont("AMD");
59 62 
60- /* Secure Memory Encryption */63+ /* Secure Memory Encryption */
61- if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) {64+ if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) {
62 /*65 /*
63 * SME is mutually exclusive with any of the SEV66 * SME is mutually exclusive with any of the SEV
64 * features below.67 * features below.
65- */68+ */
66- pr_cont(" SME\n");69+ pr_cont(" SME\n");
67- return;70+ return;
71+ }
72+ 
73+ /* Secure Encrypted Virtualization */
74+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
75+ pr_cont(" SEV");
76+ 
77+ /* Encrypted Register State */
78+ if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT))
79+ pr_cont(" SEV-ES");
80+ 
81+ /* Secure Nested Paging */
82+ if (cc_platform_has(CC_ATTR_GUEST_SEV_SNP))
83+ pr_cont(" SEV-SNP");
84+ 
85+ pr_cont("\n");
86+ 
87+ sev_show_status();
88+ 
89+ break;
90+ default:
91+ pr_cont("Unknown\n");
68 }92 }
69- 
70- /* Secure Encrypted Virtualization */
71- if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
72- pr_cont(" SEV");
73- 
74- /* Encrypted Register State */
75- if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT))
76- pr_cont(" SEV-ES");
77- 
78- /* Secure Nested Paging */
79- if (cc_platform_has(CC_ATTR_GUEST_SEV_SNP))
80- pr_cont(" SEV-SNP");
81- 
82- pr_cont("\n");
83}93}
84 94 
85/* Architecture __weak replacement functions */95/* Architecture __weak replacement functions */
@@ -93,3 +103,43 @@ void __init mem_encrypt_init(void)
93 103 
94 print_mem_encrypt_feature_info();104 print_mem_encrypt_feature_info();
95}105}
106+ 
107+void __init mem_encrypt_setup_arch(void)
108+{
109+ phys_addr_t total_mem = memblock_phys_mem_size();
110+ unsigned long size;
111+ 
112+ /*
113+ * Do RMP table fixups after the e820 tables have been setup by
114+ * e820__memory_setup().
115+ */
116+ if (cc_platform_has(CC_ATTR_HOST_SEV_SNP))
117+ snp_fixup_e820_tables();
118+ 
119+ if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
120+ return;
121+ 
122+ /*
123+ * For SEV and TDX, all DMA has to occur via shared/unencrypted pages.
124+ * Kernel uses SWIOTLB to make this happen without changing device
125+ * drivers. However, depending on the workload being run, the
126+ * default 64MB of SWIOTLB may not be enough and SWIOTLB may
127+ * run out of buffers for DMA, resulting in I/O errors and/or
128+ * performance degradation especially with high I/O workloads.
129+ *
130+ * Adjust the default size of SWIOTLB using a percentage of guest
131+ * memory for SWIOTLB buffers. Also, as the SWIOTLB bounce buffer
132+ * memory is allocated from low memory, ensure that the adjusted size
133+ * is within the limits of low available memory.
134+ *
135+ * The percentage of guest memory used here for SWIOTLB buffers
136+ * is more of an approximation of the static adjustment which
137+ * 64MB for <1G, and ~128M to 256M for 1G-to-4G, i.e., the 6%
138+ */
139+ size = total_mem * 6 / 100;
140+ size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
141+ swiotlb_adjust_size(size);
142+ 
143+ /* Set restricted memory access for virtio. */
144+ virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc);
145+}
March/x86/mm/mem_encrypt_amd.c+0-35
@@ -20,7 +20,6 @@
20#include <linux/bitops.h>20#include <linux/bitops.h>
21#include <linux/dma-mapping.h>21#include <linux/dma-mapping.h>
22#include <linux/virtio_config.h>22#include <linux/virtio_config.h>
23-#include <linux/virtio_anchor.h>
24#include <linux/cc_platform.h>23#include <linux/cc_platform.h>
25 24 
26#include <asm/tlbflush.h>25#include <asm/tlbflush.h>
@@ -217,40 +216,6 @@ void __init sme_map_bootdata(char *real_mode_data)
217 __sme_early_map_unmap_mem(__va(cmdline_paddr), COMMAND_LINE_SIZE, true);216 __sme_early_map_unmap_mem(__va(cmdline_paddr), COMMAND_LINE_SIZE, true);
218}217}
219 218 
220-void __init sev_setup_arch(void)
221-{
222- phys_addr_t total_mem = memblock_phys_mem_size();
223- unsigned long size;
224- 
225- if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
226- return;
227- 
228- /*
229- * For SEV, all DMA has to occur via shared/unencrypted pages.
230- * SEV uses SWIOTLB to make this happen without changing device
231- * drivers. However, depending on the workload being run, the
232- * default 64MB of SWIOTLB may not be enough and SWIOTLB may
233- * run out of buffers for DMA, resulting in I/O errors and/or
234- * performance degradation especially with high I/O workloads.
235- *
236- * Adjust the default size of SWIOTLB for SEV guests using
237- * a percentage of guest memory for SWIOTLB buffers.
238- * Also, as the SWIOTLB bounce buffer memory is allocated
239- * from low memory, ensure that the adjusted size is within
240- * the limits of low available memory.
241- *
242- * The percentage of guest memory used here for SWIOTLB buffers
243- * is more of an approximation of the static adjustment which
244- * 64MB for <1G, and ~128M to 256M for 1G-to-4G, i.e., the 6%
245- */
246- size = total_mem * 6 / 100;
247- size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
248- swiotlb_adjust_size(size);
249- 
250- /* Set restricted memory access for virtio. */
251- virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc);
252-}
253- 
254static unsigned long pg_level_to_pfn(int level, pte_t *kpte, pgprot_t *ret_prot)219static unsigned long pg_level_to_pfn(int level, pte_t *kpte, pgprot_t *ret_prot)
255{220{
256 unsigned long pfn = 0;221 unsigned long pfn = 0;
Aarch/x86/virt/svm/Makefile+3-0
@@ -0,0 +1,3 @@
1+# SPDX-License-Identifier: GPL-2.0
2+ 
3+obj-$(CONFIG_KVM_AMD_SEV) += sev.o
Aarch/x86/virt/svm/sev.c+1071-0
Mdrivers/crypto/ccp/Kconfig+1-1
@@ -40,7 +40,7 @@ config CRYPTO_DEV_CCP_CRYPTO
40config CRYPTO_DEV_SP_PSP40config CRYPTO_DEV_SP_PSP
41 bool "Platform Security Processor (PSP) device"41 bool "Platform Security Processor (PSP) device"
42 default y42 default y
43- depends on CRYPTO_DEV_CCP_DD && X86_6443+ depends on CRYPTO_DEV_CCP_DD && X86_64 && AMD_IOMMU
44 help44 help
45 Provide support for the AMD Platform Security Processor (PSP).45 Provide support for the AMD Platform Security Processor (PSP).
46 The PSP is a dedicated processor that provides support for key46 The PSP is a dedicated processor that provides support for key
Mdrivers/crypto/ccp/sev-dev.c+1126-93
Mdrivers/crypto/ccp/sev-dev.h+5-0
@@ -54,9 +54,14 @@ struct sev_device {
54 u8 build;54 u8 build;
55 55 
56 void *cmd_buf;56 void *cmd_buf;
57+ void *cmd_buf_backup;
58+ bool cmd_buf_active;
59+ bool cmd_buf_backup_active;
57 60 
58 /* Management of the Hygon RING BUFFER mode */61 /* Management of the Hygon RING BUFFER mode */
59 struct csv_ringbuffer_queue ring_buffer[CSV_COMMAND_PRIORITY_NUM];62 struct csv_ringbuffer_queue ring_buffer[CSV_COMMAND_PRIORITY_NUM];
63+
64+ bool snp_initialized;
60};65};
61 66 
62int sev_dev_init(struct psp_device *psp);67int sev_dev_init(struct psp_device *psp);
Mdrivers/iommu/amd/amd_iommu.h+0-1
@@ -173,7 +173,6 @@ struct dev_table_entry *amd_iommu_get_ivhd_dte_flags(u16 segid, u16 devid);
173void amd_iommu_domain_set_pgtable(struct protection_domain *domain,173void amd_iommu_domain_set_pgtable(struct protection_domain *domain,
174 u64 *root, int mode);174 u64 *root, int mode);
175struct dev_table_entry *get_dev_table(struct amd_iommu *iommu);175struct dev_table_entry *get_dev_table(struct amd_iommu *iommu);
176-extern bool amd_iommu_snp_en;
177struct iommu_dev_data *search_dev_data(struct amd_iommu *iommu, u16 devid);176struct iommu_dev_data *search_dev_data(struct amd_iommu *iommu, u16 devid);
178 177 
179#endif /* AMD_IOMMU_H */178#endif /* AMD_IOMMU_H */
Mdrivers/iommu/amd/init.c+135-31
@@ -29,6 +29,7 @@
29#include <asm/io_apic.h>29#include <asm/io_apic.h>
30#include <asm/irq_remapping.h>30#include <asm/irq_remapping.h>
31#include <asm/set_memory.h>31#include <asm/set_memory.h>
32+#include <asm/sev.h>
32 33 
33#include <linux/crash_dump.h>34#include <linux/crash_dump.h>
34 35 
@@ -3268,6 +3269,47 @@ static bool __init detect_ivrs(void)
3268 return true;3269 return true;
3269}3270}
3270 3271 
3272+static __init void iommu_snp_enable(void)
3273+{
3274+#ifdef CONFIG_KVM_AMD_SEV
3275+ if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP))
3276+ return;
3277+ /*
3278+ * The SNP support requires that IOMMU must be enabled, and is
3279+ * configured with V1 page table (DTE[Mode] = 0 is not supported).
3280+ */
3281+ if (no_iommu || iommu_default_passthrough()) {
3282+ pr_warn("SNP: IOMMU disabled or configured in passthrough mode, SNP cannot be supported.\n");
3283+ goto disable_snp;
3284+ }
3285+ 
3286+ if (amd_iommu_pgtable != AMD_IOMMU_V1) {
3287+ pr_warn("SNP: IOMMU is configured with V2 page table mode, SNP cannot be supported.\n");
3288+ goto disable_snp;
3289+ }
3290+ 
3291+ amd_iommu_snp_en = check_feature(FEATURE_SNP);
3292+ if (!amd_iommu_snp_en) {
3293+ pr_warn("SNP: IOMMU SNP feature not enabled, SNP cannot be supported.\n");
3294+ goto disable_snp;
3295+ }
3296+ 
3297+ /*
3298+ * Enable host SNP support once SNP support is checked on IOMMU.
3299+ */
3300+ if (snp_rmptable_init()) {
3301+ pr_warn("SNP: RMP initialization failed, SNP cannot be supported.\n");
3302+ goto disable_snp;
3303+ }
3304+ 
3305+ pr_info("IOMMU SNP support enabled.\n");
3306+ return;
3307+ 
3308+disable_snp:
3309+ cc_platform_clear(CC_ATTR_HOST_SEV_SNP);
3310+#endif
3311+}
3312+ 
3271/****************************************************************************3313/****************************************************************************
3272 *3314 *
3273 * AMD IOMMU Initialization State Machine3315 * AMD IOMMU Initialization State Machine
@@ -3303,6 +3345,7 @@ static int __init state_next(void)
3303 break;3345 break;
3304 case IOMMU_ENABLED:3346 case IOMMU_ENABLED:
3305 register_syscore_ops(&amd_iommu_syscore_ops);3347 register_syscore_ops(&amd_iommu_syscore_ops);
3348+ iommu_snp_enable();
3306 ret = amd_iommu_init_pci();3349 ret = amd_iommu_init_pci();
3307 init_state = ret ? IOMMU_INIT_ERROR : IOMMU_PCI_INIT;3350 init_state = ret ? IOMMU_INIT_ERROR : IOMMU_PCI_INIT;
3308 break;3351 break;
@@ -3358,6 +3401,19 @@ static int __init iommu_go_to_state(enum iommu_init_state state)
3358 ret = state_next();3401 ret = state_next();
3359 }3402 }
3360 3403 
3404+ /*
3405+ * SNP platform initilazation requires IOMMUs to be fully configured.
3406+ * If the SNP support on IOMMUs has NOT been checked, simply mark SNP
3407+ * as unsupported. If the SNP support on IOMMUs has been checked and
3408+ * host SNP support enabled but RMP enforcement has not been enabled
3409+ * in IOMMUs, then the system is in a half-baked state, but can limp
3410+ * along as all memory should be Hypervisor-Owned in the RMP. WARN,
3411+ * but leave SNP as "supported" to avoid confusing the kernel.
3412+ */
3413+ if (ret && cc_platform_has(CC_ATTR_HOST_SEV_SNP) &&
3414+ !WARN_ON_ONCE(amd_iommu_snp_en))
3415+ cc_platform_clear(CC_ATTR_HOST_SEV_SNP);
3416+ 
3361 return ret;3417 return ret;
3362}3418}
3363 3419 
@@ -3461,25 +3517,28 @@ static bool amd_iommu_sme_check(void)
3461 * IOMMUs3517 * IOMMUs
3462 *3518 *
3463 ****************************************************************************/3519 ****************************************************************************/
3464-int __init amd_iommu_detect(void)3520+void __init amd_iommu_detect(void)
3465{3521{
3466 int ret;3522 int ret;
3467 3523 
3468 if (no_iommu || (iommu_detected && !gart_iommu_aperture))3524 if (no_iommu || (iommu_detected && !gart_iommu_aperture))
3469- return -ENODEV;3525+ goto disable_snp;
3470 3526 
3471 if (!amd_iommu_sme_check())3527 if (!amd_iommu_sme_check())
3472- return -ENODEV;3528+ goto disable_snp;
3473 3529 
3474 ret = iommu_go_to_state(IOMMU_IVRS_DETECTED);3530 ret = iommu_go_to_state(IOMMU_IVRS_DETECTED);
3475 if (ret)3531 if (ret)
3476- return ret;3532+ goto disable_snp;
3477 3533 
3478 amd_iommu_detected = true;3534 amd_iommu_detected = true;
3479 iommu_detected = 1;3535 iommu_detected = 1;
3480 x86_init.iommu.iommu_init = amd_iommu_init;3536 x86_init.iommu.iommu_init = amd_iommu_init;
3537+ return;
3481 3538 
3482- return 1;3539+disable_snp:
3540+ if (cc_platform_has(CC_ATTR_HOST_SEV_SNP))
3541+ cc_platform_clear(CC_ATTR_HOST_SEV_SNP);
3483}3542}
3484 3543 
3485/****************************************************************************3544/****************************************************************************
@@ -3822,40 +3881,85 @@ int amd_iommu_pc_set_reg(struct amd_iommu *iommu, u8 bank, u8 cntr, u8 fxn, u64
3822 return iommu_pc_get_set_reg(iommu, bank, cntr, fxn, value, true);3881 return iommu_pc_get_set_reg(iommu, bank, cntr, fxn, value, true);
3823}3882}
3824 3883 
3825-#ifdef CONFIG_AMD_MEM_ENCRYPT3884+#ifdef CONFIG_KVM_AMD_SEV
3826-int amd_iommu_snp_enable(void)3885+static int iommu_page_make_shared(void *page)
3827{3886{
3828- /*3887+ unsigned long paddr, pfn;
3829- * The SNP support requires that IOMMU must be enabled, and is3888+ 
3830- * not configured in the passthrough mode.3889+ paddr = iommu_virt_to_phys(page);
3831- */3890+ /* Cbit maybe set in the paddr */
3832- if (no_iommu || iommu_default_passthrough()) {3891+ pfn = __sme_clr(paddr) >> PAGE_SHIFT;
3833- pr_err("SNP: IOMMU is disabled or configured in passthrough mode, SNP cannot be supported");3892+ 
3834- return -EINVAL;3893+ if (!(pfn % PTRS_PER_PMD)) {
3894+ int ret, level;
3895+ bool assigned;
3896+ 
3897+ ret = snp_lookup_rmpentry(pfn, &assigned, &level);
3898+ if (ret) {
3899+ pr_warn("IOMMU PFN %lx RMP lookup failed, ret %d\n", pfn, ret);
3900+ return ret;
3901+ }
3902+ 
3903+ if (!assigned) {
3904+ pr_warn("IOMMU PFN %lx not assigned in RMP table\n", pfn);
3905+ return -EINVAL;
3906+ }
3907+ 
3908+ if (level > PG_LEVEL_4K) {
3909+ ret = psmash(pfn);
3910+ if (!ret)
3911+ goto done;
3912+ 
3913+ pr_warn("PSMASH failed for IOMMU PFN %lx huge RMP entry, ret: %d, level: %d\n",
3914+ pfn, ret, level);
3915+ return ret;
3916+ }
3835 }3917 }
3836 3918 
3837- /*3919+done:
3838- * Prevent enabling SNP after IOMMU_ENABLED state because this process3920+ return rmp_make_shared(pfn, PG_LEVEL_4K);
3839- * affect how IOMMU driver sets up data structures and configures3921+}
3840- * IOMMU hardware.
3841- */
3842- if (init_state > IOMMU_ENABLED) {
3843- pr_err("SNP: Too late to enable SNP for IOMMU.\n");
3844- return -EINVAL;
3845- }
3846 3922 
3847- amd_iommu_snp_en = check_feature(FEATURE_SNP);3923+static int iommu_make_shared(void *va, size_t size)
3848- if (!amd_iommu_snp_en)3924+{
3849- return -EINVAL;3925+ void *page;
3926+ int ret;
3850 3927 
3851- pr_info("SNP enabled\n");3928+ if (!va)
3929+ return 0;
3852 3930 
3853- /* Enforce IOMMU v1 pagetable when SNP is enabled. */3931+ for (page = va; page < (va + size); page += PAGE_SIZE) {
3854- if (amd_iommu_pgtable != AMD_IOMMU_V1) {3932+ ret = iommu_page_make_shared(page);
3855- pr_warn("Force to using AMD IOMMU v1 page table due to SNP\n");3933+ if (ret)
3856- amd_iommu_pgtable = AMD_IOMMU_V1;3934+ return ret;
3857 }3935 }
3858 3936 
3859 return 0;3937 return 0;
3860}3938}
3939+ 
3940+int amd_iommu_snp_disable(void)
3941+{
3942+ struct amd_iommu *iommu;
3943+ int ret;
3944+ 
3945+ if (!amd_iommu_snp_en)
3946+ return 0;
3947+ 
3948+ for_each_iommu(iommu) {
3949+ ret = iommu_make_shared(iommu->evt_buf, EVT_BUFFER_SIZE);
3950+ if (ret)
3951+ return ret;
3952+ 
3953+ ret = iommu_make_shared(iommu->ppr_log, PPR_LOG_SIZE);
3954+ if (ret)
3955+ return ret;
3956+ 
3957+ ret = iommu_make_shared((void *)iommu->cmd_sem, PAGE_SIZE);
3958+ if (ret)
3959+ return ret;
3960+ }
3961+ 
3962+ return 0;
3963+}
3964+EXPORT_SYMBOL_GPL(amd_iommu_snp_disable);
3861#endif3965#endif
Mdrivers/virt/coco/sev-guest/Kconfig+1-0
@@ -5,6 +5,7 @@ config SEV_GUEST
5 select CRYPTO5 select CRYPTO
6 select CRYPTO_AEAD26 select CRYPTO_AEAD2
7 select CRYPTO_GCM7 select CRYPTO_GCM
8+ select TSM_REPORTS
8 help9 help
9 SEV-SNP firmware provides the guest a mechanism to communicate with10 SEV-SNP firmware provides the guest a mechanism to communicate with
10 the PSP without risk from a malicious hypervisor who wishes to read,11 the PSP without risk from a malicious hypervisor who wishes to read,
Mfs/anon_inodes.c+1-0
Mmm/compaction.c+31-12
Mmm/migrate.c+2-0
Mmm/truncate.c+2-1
Mtools/testing/selftests/kvm/kvm_page_table_test.c+1-1
Mvirt/kvm/Kconfig+29-0
Mvirt/kvm/kvm_main.c+584-82
Mvirt/kvm/kvm_mm.h+26-0