diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 2ca9e01ad0e8..317e0f41ac9b 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -526,6 +526,13 @@ config RISCV_COVE_GUEST help Enables support for running TVMs on platforms supporting CoVE. +config RISCV_COVE_GUEST_PROMOTE + bool "Automatic promotion of VM to TVM for Confidential VM Extension(CoVE)" + default n + select RISCV_COVE_GUEST + help + VM requests promotion to TVM during early boot on platforms supporting CoVE. + endmenu # "Confidential VM Extension(CoVE) Support" endmenu # "Platform type" diff --git a/arch/riscv/include/asm/kvm_cove.h b/arch/riscv/include/asm/kvm_cove.h index afaea7c621bb..811b1fd4b90a 100644 --- a/arch/riscv/include/asm/kvm_cove.h +++ b/arch/riscv/include/asm/kvm_cove.h @@ -19,6 +19,13 @@ #include #include +#define KVM_COVE_TSM_CAP_PROMOTE_TVM 0x0 +#define KVM_COVE_TSM_CAP_ATTESTATION_LOCAL 0x1 +#define KVM_COVE_TSM_CAP_ATTESTATION_REMOTE 0x2 +#define KVM_COVE_TSM_CAP_AIA 0x3 +#define KVM_COVE_TSM_CAP_MRIF 0x4 +#define KVM_COVE_TSM_CAP_MEMORY_ALLOCATION 0x5 + #define KVM_COVE_PAGE_SIZE_4K (1UL << 12) #define KVM_COVE_PAGE_SIZE_2MB (1UL << 21) #define KVM_COVE_PAGE_SIZE_1GB (1UL << 30) @@ -126,11 +133,15 @@ static inline bool is_cove_vcpu(struct kvm_vcpu *vcpu) #ifdef CONFIG_RISCV_COVE_HOST bool kvm_riscv_cove_enabled(void); +bool kvm_riscv_cove_capability(unsigned long cap); int kvm_riscv_cove_init(void); /* TVM related functions */ void kvm_riscv_cove_vm_destroy(struct kvm *kvm); -int kvm_riscv_cove_vm_init(struct kvm *kvm); +int kvm_riscv_cove_vm_single_step_init(struct kvm_vcpu *vcpu, + unsigned long fdt_address, + unsigned long tap_addr); +int kvm_riscv_cove_vm_multi_step_init(struct kvm *kvm); /* TVM VCPU related functions */ void kvm_riscv_cove_vcpu_destroy(struct kvm_vcpu *vcpu); @@ -158,13 +169,20 @@ int kvm_riscv_cove_aia_convert_imsic(struct kvm_vcpu *vcpu, phys_addr_t imsic_pa int kvm_riscv_cove_vcpu_imsic_addr(struct kvm_vcpu *vcpu); #else static inline bool kvm_riscv_cove_enabled(void) {return false; }; +static inline bool kvm_riscv_cove_capability(unsigned long cap) { return false; }; static inline int kvm_riscv_cove_init(void) { return -1; } static inline void kvm_riscv_cove_hardware_disable(void) {} static inline int kvm_riscv_cove_hardware_enable(void) {return 0; } /* TVM related functions */ static inline void kvm_riscv_cove_vm_destroy(struct kvm *kvm) {} -static inline int kvm_riscv_cove_vm_init(struct kvm *kvm) {return -1; } +static inline int kvm_riscv_cove_vm_single_step_init(struct kvm_vcpu *vcpu, + unsigned long fdt_address, + unsigned long tap_addr) +{ + return -1; +} +static inline int kvm_riscv_cove_vm_multi_step_init(struct kvm *kvm) {return -1; } /* TVM VCPU related functions */ static inline void kvm_riscv_cove_vcpu_destroy(struct kvm_vcpu *vcpu) {} diff --git a/arch/riscv/include/asm/kvm_cove_sbi.h b/arch/riscv/include/asm/kvm_cove_sbi.h index c9302650adc8..1314227adfad 100644 --- a/arch/riscv/include/asm/kvm_cove_sbi.h +++ b/arch/riscv/include/asm/kvm_cove_sbi.h @@ -78,6 +78,10 @@ int sbi_covh_tvm_demote_page(unsigned long tvmid, int sbi_covh_tvm_remove_pages(unsigned long tvmid, unsigned long tvm_base_page_addr, unsigned long len); +int sbi_covh_tsm_promote_to_tvm(unsigned long fdt_address, + unsigned long tap_addr, + unsigned long sepc, + unsigned long *tvmid); /* Functions related to CoVE Interrupt Management(COVI) Extension */ int sbi_covi_tvm_aia_init(unsigned long tvm_gid, struct sbi_cove_tvm_aia_params *tvm_aia_params); diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h index 5b37a12337b1..763a931407f3 100644 --- a/arch/riscv/include/asm/kvm_vcpu_sbi.h +++ b/arch/riscv/include/asm/kvm_vcpu_sbi.h @@ -68,6 +68,7 @@ extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_experimental; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_vendor; #ifdef CONFIG_RISCV_COVE_HOST extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_covg; +extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_covh; #endif #endif /* __RISCV_KVM_VCPU_SBI_H__ */ diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 03b0cc871242..01e9e5b1d7a2 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -374,6 +374,7 @@ enum sbi_ext_covh_fid { SBI_EXT_COVH_TVM_PROMOTE_PAGE, SBI_EXT_COVH_TVM_DEMOTE_PAGE, SBI_EXT_COVH_TVM_REMOVE_PAGES, + SBI_EXT_COVH_PROMOTE_TO_TVM, }; enum sbi_ext_covi_fid { @@ -410,9 +411,15 @@ struct sbi_cove_tsm_info { /* Current state of the TSM */ enum sbi_cove_tsm_state tstate; + /* TSM implementation identifier */ + uint32_t impl_id; + /* Version of the loaded TSM */ uint32_t version; + /* Capabilities of the TSM */ + unsigned long capabilities; + /* Number of 4K pages required per TVM */ unsigned long tvm_pages_needed; diff --git a/arch/riscv/include/uapi/asm/kvm.h b/arch/riscv/include/uapi/asm/kvm.h index 2a2434136e39..679a6727a143 100644 --- a/arch/riscv/include/uapi/asm/kvm.h +++ b/arch/riscv/include/uapi/asm/kvm.h @@ -149,6 +149,7 @@ enum KVM_RISCV_SBI_EXT_ID { KVM_RISCV_SBI_EXT_VENDOR, KVM_RISCV_SBI_EXT_DBCN, KVM_RISCV_SBI_EXT_COVG, + KVM_RISCV_SBI_EXT_COVH, KVM_RISCV_SBI_EXT_MAX, }; diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 4bf6c449d78b..a5eaff780616 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -198,6 +198,20 @@ ENTRY(_start_kernel) csrw CSR_IE, zero csrw CSR_IP, zero +#if defined(CONFIG_RISCV_COVE_GUEST_PROMOTE) && !defined(CONFIG_RISCV_M_MODE) + mv s0, a0 + mv s1, a1 + /* Request hypervisor to promote to TVM */ + li a7, 0x434F5648 /* SBI_EXT_COVH */ + li a6, 0x15 /* SBI_EXT_COVH_PROMOTE_TO_TVM */ + mv a0, a1 /* DTB physical address */ + la a1, __cove_tap_start /* TAP physical address */ + ecall + /* Attestation reflects the result of promotion, so ignore it */ + mv a0, s0 + mv a1, s1 +#endif /* CONFIG_RISCV_COVE_GUEST_PROMOTE */ + #ifdef CONFIG_RISCV_M_MODE /* flush the instruction cache */ fence.i diff --git a/arch/riscv/kernel/vmlinux.lds.S b/arch/riscv/kernel/vmlinux.lds.S index 53a8ad65b255..f65731bdcf1d 100644 --- a/arch/riscv/kernel/vmlinux.lds.S +++ b/arch/riscv/kernel/vmlinux.lds.S @@ -113,6 +113,18 @@ SECTIONS } __init_end = .; +#ifdef CONFIG_RISCV_COVE_GUEST_PROMOTE + . = ALIGN(4096); + .cove_tvm_attestation_payload : { + __cove_tap_start = .; + LONG(0xace0ace0) + SHORT(0x0FFA) + FILL(0x00) + . += 4090; + __cove_tap_end = .; + } +#endif + /* Start of data section */ _sdata = .; RO_DATA(SECTION_ALIGN) diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile index 31f4dbd97b03..fba7ebd0cd72 100644 --- a/arch/riscv/kvm/Makefile +++ b/arch/riscv/kvm/Makefile @@ -31,4 +31,4 @@ kvm-y += aia.o kvm-y += aia_device.o kvm-y += aia_aplic.o kvm-y += aia_imsic.o -kvm-$(CONFIG_RISCV_COVE_HOST) += cove_sbi.o cove.o vcpu_sbi_covg.o +kvm-$(CONFIG_RISCV_COVE_HOST) += cove_sbi.o cove.o vcpu_sbi_covg.o vcpu_sbi_covh.o diff --git a/arch/riscv/kvm/cove.c b/arch/riscv/kvm/cove.c index ba596b7f2240..95359b63e6f7 100644 --- a/arch/riscv/kvm/cove.c +++ b/arch/riscv/kvm/cove.c @@ -150,6 +150,11 @@ __always_inline bool kvm_riscv_cove_enabled(void) return riscv_cove_enabled; } +__always_inline bool kvm_riscv_cove_capability(unsigned long cap) +{ + return tinfo.capabilities & BIT(cap); +} + static void kvm_cove_imsic_clone(void *info) { int rc; @@ -589,9 +594,9 @@ void noinstr kvm_riscv_cove_vcpu_switchto(struct kvm_vcpu *vcpu, struct kvm_cpu_ /* * Bind the vsfile here instead during the new vsfile allocation because - * COVH bind call requires the TVM to be in finalized state. + * COVI bind call requires the TVM to be in finalized state. */ - if (tvcpuc->imsic.bind_required) { + if (kvm_riscv_cove_capability(KVM_COVE_TSM_CAP_AIA) && tvcpuc->imsic.bind_required) { tvcpuc->imsic.bind_required = false; rc = kvm_riscv_cove_vcpu_imsic_bind(vcpu, BIT(tvcpuc->imsic.vsfile_hgei)); if (rc) { @@ -628,12 +633,12 @@ void kvm_riscv_cove_vcpu_destroy(struct kvm_vcpu *vcpu) int kvm_riscv_cove_vcpu_init(struct kvm_vcpu *vcpu) { - int rc; - struct kvm *kvm; struct kvm_cove_tvm_vcpu_context *tvcpuc; struct kvm_cove_tvm_context *tvmc; - struct page *vcpus_page; unsigned long vcpus_phys_addr; + struct page *vcpus_page; + struct kvm *kvm; + int rc; if (!vcpu) return -EINVAL; @@ -654,15 +659,20 @@ int kvm_riscv_cove_vcpu_init(struct kvm_vcpu *vcpu) if (!tvcpuc) return -ENOMEM; + tvcpuc->vcpu = vcpu; + tvcpuc->vcpu_state.npages = tinfo.tvcpu_pages_needed; + vcpu->arch.tc = tvcpuc; + + if (!kvm_riscv_cove_capability(KVM_COVE_TSM_CAP_MEMORY_ALLOCATION)) { + return 0; + } + vcpus_page = alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order_num_pages(tinfo.tvcpu_pages_needed)); if (!vcpus_page) { rc = -ENOMEM; goto alloc_page_failed; } - - tvcpuc->vcpu = vcpu; - tvcpuc->vcpu_state.npages = tinfo.tvcpu_pages_needed; tvcpuc->vcpu_state.page = vcpus_page; vcpus_phys_addr = page_to_phys(vcpus_page); @@ -674,8 +684,6 @@ int kvm_riscv_cove_vcpu_init(struct kvm_vcpu *vcpu) if (rc) goto vcpu_create_failed; - vcpu->arch.tc = tvcpuc; - return 0; vcpu_create_failed: @@ -686,6 +694,7 @@ int kvm_riscv_cove_vcpu_init(struct kvm_vcpu *vcpu) __free_pages(vcpus_page, get_order_num_pages(tinfo.tvcpu_pages_needed)); alloc_page_failed: + vcpu->arch.tc = NULL; kfree(tvcpuc); return rc; } @@ -877,7 +886,7 @@ void kvm_riscv_cove_vm_destroy(struct kvm *kvm) kvm_err("Memory reclaim failed with rc %d\n", rc); } -int kvm_riscv_cove_vm_init(struct kvm *kvm) +int kvm_riscv_cove_vm_multi_step_init(struct kvm *kvm) { struct kvm_cove_tvm_context *tvmc; struct page *tvms_page, *pgt_page; @@ -885,6 +894,10 @@ int kvm_riscv_cove_vm_init(struct kvm *kvm) unsigned long gstage_pgd_size = kvm_riscv_gstage_pgd_size(); int rc = 0; + // Multi-step TVM creation requires TSM that supports dynamic page conversion + if (!kvm_riscv_cove_capability(KVM_COVE_TSM_CAP_MEMORY_ALLOCATION)) + return -EOPNOTSUPP; + tvmc = kzalloc(sizeof(*tvmc), GFP_KERNEL); if (!tvmc) return -ENOMEM; @@ -980,6 +993,65 @@ int kvm_riscv_cove_vm_init(struct kvm *kvm) return rc; } +int kvm_riscv_cove_vm_single_step_init(struct kvm_vcpu *vcpu, unsigned long fdt_address, + unsigned long tap_addr) +{ + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + unsigned long tvm_gid, target_vcpuid; + struct kvm_cove_tvm_context *tvmc; + struct kvm_vcpu *target_vcpu; + struct kvm *kvm = vcpu->kvm; + void *nshmem = nacl_shmem(); + int rc = 0, gpr_id, offset; + + if (!kvm_riscv_cove_capability(KVM_COVE_TSM_CAP_PROMOTE_TVM)) + return -EOPNOTSUPP; + + tvmc = kzalloc(sizeof(*tvmc), GFP_KERNEL); + if (!tvmc) + return -ENOMEM; + + for (gpr_id = 1; gpr_id < 32; gpr_id++) { + offset = KVM_ARCH_GUEST_ZERO + gpr_id * sizeof(unsigned long); + nacl_shmem_gpr_write_cove(nshmem, offset, + ((unsigned long *)cp)[gpr_id]); + } + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + rc = sbi_covh_tsm_promote_to_tvm(fdt_address, tap_addr, cp->sepc, &tvm_gid); + if (rc) + goto done; + + INIT_LIST_HEAD(&tvmc->measured_pages); + INIT_LIST_HEAD(&tvmc->zero_pages); + INIT_LIST_HEAD(&tvmc->shared_pages); + INIT_LIST_HEAD(&tvmc->reclaim_pending_pages); + + tvmc->tvm_guest_id = tvm_gid; + tvmc->kvm = kvm; + kvm->arch.tvmc = tvmc; + + kvm_for_each_vcpu(target_vcpuid, target_vcpu, kvm) { + rc = kvm_riscv_cove_vcpu_init(target_vcpu); + if (rc) + goto vcpus_allocated; + + target_vcpu->requests = 0; + } + + tvmc->finalized_done = true; + kvm_info("Guest VM creation successful with guest id %lx\n", tvm_gid); + return 0; + +vcpus_allocated: + kvm_for_each_vcpu(target_vcpuid, target_vcpu, kvm) + if (target_vcpu->arch.tc) + kfree(target_vcpu->arch.tc); + +done: + kfree(tvmc); + return rc; +} + int kvm_riscv_cove_init(void) { int rc; diff --git a/arch/riscv/kvm/cove_sbi.c b/arch/riscv/kvm/cove_sbi.c index 4759b4920226..2325ee0f2a15 100644 --- a/arch/riscv/kvm/cove_sbi.c +++ b/arch/riscv/kvm/cove_sbi.c @@ -488,3 +488,23 @@ int sbi_covh_tvm_remove_pages(unsigned long tvmid, return 0; } + +int sbi_covh_tsm_promote_to_tvm(unsigned long fdt_address, + unsigned long tap_addr, + unsigned long sepc, + unsigned long *tvmid) +{ + struct sbiret ret; + int rc = 0; + + ret = sbi_ecall(SBI_EXT_COVH, SBI_EXT_COVH_PROMOTE_TO_TVM, fdt_address, + tap_addr, sepc, 0, 0, 0); + if (ret.error) { + rc = sbi_err_map_linux_errno(ret.error); + goto done; + } + + *tvmid = ret.value; +done: + return rc; +} diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index a05941420307..9a9625f9c7a9 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -31,12 +31,11 @@ int kvm_arch_hardware_enable(void) return rc; /* - * We just need to invoke aia enable for CoVE if host is in VS mode - * However, if the host is running in HS mode, we need to initialize - * other CSRs as well for legacy VMs. - * TODO: Handle host in HS mode use case. + * We just need to invoke aia enable for CoVE if host is in VS mode and TSM + * supports AIA (COVI extension). However, if the host is running in HS mode, + * we need to initialize other CSRs as well for legacy VMs. */ - if (unlikely(kvm_riscv_cove_enabled())) + if (unlikely(kvm_riscv_cove_enabled()) && kvm_riscv_cove_capability(KVM_COVE_TSM_CAP_AIA)) goto enable_aia; hedeleg = 0; diff --git a/arch/riscv/kvm/tlb.c b/arch/riscv/kvm/tlb.c index b007c027baed..5a3ef6ea01e9 100644 --- a/arch/riscv/kvm/tlb.c +++ b/arch/riscv/kvm/tlb.c @@ -199,7 +199,7 @@ void kvm_riscv_hfence_gvma_vmid_all_process(struct kvm_vcpu *vcpu) struct kvm_vmid *v = &vcpu->kvm->arch.vmid; unsigned long vmid = READ_ONCE(v->vmid); - if (kvm_riscv_nacl_available()) + if (kvm_riscv_nacl_sync_hfence_available()) nacl_shmem_hfence_gvma_vmid_all(nacl_shmem(), vmid); else kvm_riscv_local_hfence_gvma_vmid_all(vmid); @@ -210,7 +210,7 @@ void kvm_riscv_hfence_vvma_all_process(struct kvm_vcpu *vcpu) struct kvm_vmid *v = &vcpu->kvm->arch.vmid; unsigned long vmid = READ_ONCE(v->vmid); - if (kvm_riscv_nacl_available()) + if (kvm_riscv_nacl_sync_hfence_available()) nacl_shmem_hfence_vvma_all(nacl_shmem(), vmid); else kvm_riscv_local_hfence_vvma_all(vmid); @@ -277,7 +277,7 @@ void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu) break; case KVM_RISCV_HFENCE_GVMA_VMID_GPA: vmid = READ_ONCE(v->vmid); - if (kvm_riscv_nacl_available()) + if (kvm_riscv_nacl_sync_hfence_available()) nacl_shmem_hfence_gvma_vmid( nacl_shmem(), vmid, d.addr, d.size, d.order); @@ -288,7 +288,7 @@ void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu) case KVM_RISCV_HFENCE_VVMA_ASID_GVA: kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_RCVD); vmid = READ_ONCE(v->vmid); - if (kvm_riscv_nacl_available()) + if (kvm_riscv_nacl_sync_hfence_available()) nacl_shmem_hfence_vvma_asid( nacl_shmem(), vmid, d.asid, d.addr, d.size, d.order); @@ -300,7 +300,7 @@ void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu) case KVM_RISCV_HFENCE_VVMA_ASID_ALL: kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_RCVD); vmid = READ_ONCE(v->vmid); - if (kvm_riscv_nacl_available()) + if (kvm_riscv_nacl_sync_hfence_available()) nacl_shmem_hfence_vvma_asid_all( nacl_shmem(), vmid, d.asid); else @@ -310,7 +310,7 @@ void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu) case KVM_RISCV_HFENCE_VVMA_GVA: kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_RCVD); vmid = READ_ONCE(v->vmid); - if (kvm_riscv_nacl_available()) + if (kvm_riscv_nacl_sync_hfence_available()) nacl_shmem_hfence_vvma(nacl_shmem(), vmid, d.addr, d.size, d.order); else diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index 005c7c93536d..62153d6ca579 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -730,8 +730,8 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp, if (ioctl == KVM_INTERRUPT) { struct kvm_interrupt irq; - /* We do not support user space emulated IRQCHIP for TVMs yet */ - if (is_cove_vcpu(vcpu)) + /* We do not support user space emulated IRQCHIP for TVMs that utilize AIA yet */ + if (is_cove_vcpu(vcpu) && kvm_riscv_cove_capability(KVM_COVE_TSM_CAP_AIA)) return -ENXIO; if (copy_from_user(&irq, argp, sizeof(irq))) @@ -997,7 +997,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) goto skip_load; } - if (kvm_riscv_nacl_sync_csr_available()) { + if (unlikely(kvm_riscv_cove_enabled()) || kvm_riscv_nacl_sync_csr_available()) { nshmem = nacl_shmem(); nacl_shmem_csr_write(nshmem, CSR_VSSTATUS, csr->vsstatus); nacl_shmem_csr_write(nshmem, CSR_VSIE, csr->vsie); @@ -1061,7 +1061,7 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) kvm_riscv_vcpu_timer_save(vcpu); - if (kvm_riscv_nacl_available()) { + if (kvm_riscv_nacl_sync_csr_available()) { /** * For TVMs, we don't need a separate case as TSM only updates * the required CSRs during the world switch. All other CSR @@ -1325,8 +1325,11 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) */ kvm_riscv_vcpu_flush_interrupts(vcpu); - /* Update HVIP CSR for current CPU only for non TVMs */ - if (!is_cove_vcpu(vcpu)) + /* + * Do not update HVIP CSR for TVMs with AIA because AIA + * provides alternative method to inject interrupts. + */ + if (!is_cove_vcpu(vcpu) || !kvm_riscv_cove_capability(KVM_COVE_TSM_CAP_AIA)) kvm_riscv_update_hvip(vcpu); if (ret <= 0 || diff --git a/arch/riscv/kvm/vcpu_sbi.c b/arch/riscv/kvm/vcpu_sbi.c index 8bc7d7398349..9399cf5a3062 100644 --- a/arch/riscv/kvm/vcpu_sbi.c +++ b/arch/riscv/kvm/vcpu_sbi.c @@ -40,6 +40,11 @@ static const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_covg = { .extid_end = -1UL, .handler = NULL, }; +static const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_covh = { + .extid_start = -1UL, + .extid_end = -1UL, + .handler = NULL, +}; #endif struct kvm_riscv_sbi_extension_entry { @@ -96,6 +101,10 @@ static const struct kvm_riscv_sbi_extension_entry sbi_ext[] = { .dis_idx = KVM_RISCV_SBI_EXT_COVG, .ext_ptr = &vcpu_sbi_ext_covg, }, + { + .dis_idx = KVM_RISCV_SBI_EXT_COVH, + .ext_ptr = &vcpu_sbi_ext_covh, + }, }; void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run) diff --git a/arch/riscv/kvm/vcpu_sbi_covg.c b/arch/riscv/kvm/vcpu_sbi_covg.c index 44a3b06d0593..42f3571361a0 100644 --- a/arch/riscv/kvm/vcpu_sbi_covg.c +++ b/arch/riscv/kvm/vcpu_sbi_covg.c @@ -55,7 +55,7 @@ static int cove_share_converted_page(struct kvm_vcpu *vcpu, gpa_t gpa, } static int cove_share_page(struct kvm_vcpu *vcpu, gpa_t gpa, - unsigned long *sbi_err) + struct kvm_vcpu_sbi_return *retdata) { unsigned long hva = gfn_to_hva(vcpu->kvm, gpa >> PAGE_SHIFT); struct kvm_cove_tvm_context *tvmc = vcpu->kvm->arch.tvmc; @@ -66,7 +66,7 @@ static int cove_share_page(struct kvm_vcpu *vcpu, gpa_t gpa, if (kvm_is_error_hva(hva)) { /* Address is out of the guest ram memory region. */ - *sbi_err = SBI_ERR_INVALID_PARAM; + retdata->err_val = SBI_ERR_INVALID_PARAM; return 0; } @@ -95,6 +95,7 @@ static int cove_share_page(struct kvm_vcpu *vcpu, gpa_t gpa, list_add(&tpage->link, &tvmc->shared_pages); spin_unlock(&vcpu->kvm->mmu_lock); + retdata->out_val = page_to_phys(tpage->page); return 0; free_tpage: @@ -104,7 +105,7 @@ static int cove_share_page(struct kvm_vcpu *vcpu, gpa_t gpa, } static int kvm_riscv_cove_share_page(struct kvm_vcpu *vcpu, gpa_t gpa, - unsigned long *sbi_err) + struct kvm_vcpu_sbi_return *retdata) { struct kvm_cove_tvm_context *tvmc = vcpu->kvm->arch.tvmc; struct kvm_riscv_cove_page *tpage, *next; @@ -129,7 +130,7 @@ static int kvm_riscv_cove_share_page(struct kvm_vcpu *vcpu, gpa_t gpa, if (converted) return cove_share_converted_page(vcpu, gpa, tpage); - return cove_share_page(vcpu, gpa, sbi_err); + return cove_share_page(vcpu, gpa, retdata); } static int kvm_riscv_cove_unshare_page(struct kvm_vcpu *vcpu, gpa_t gpa) @@ -189,7 +190,7 @@ static int kvm_sbi_ext_covg_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, case SBI_EXT_COVG_SHARE_MEMORY: for (i = 0; i < num_pages; i++) { ret = kvm_riscv_cove_share_page( - vcpu, cp->a0 + i * PAGE_SIZE, err_val); + vcpu, cp->a0 + i * PAGE_SIZE, retdata); if (ret || *err_val != SBI_SUCCESS) return ret; } diff --git a/arch/riscv/kvm/vcpu_sbi_covh.c b/arch/riscv/kvm/vcpu_sbi_covh.c new file mode 100644 index 000000000000..226c3d6ffd05 --- /dev/null +++ b/arch/riscv/kvm/vcpu_sbi_covh.c @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2024 IBM. + * + * Authors: + * Wojciech Ozga + */ +#include +#include +#include +#include +#include +#include + +static int preload_pages(struct kvm_vcpu *vcpu) { + unsigned long hva, fault_addr, page; + struct kvm_memory_slot *memslot; + bool writable; + int bkt; + + kvm_for_each_memslot(memslot, bkt, kvm_memslots(vcpu->kvm)) { + for (page = 0; page < memslot->npages; page++) { + fault_addr = gfn_to_gpa(memslot->base_gfn) + + page * PAGE_SIZE; + hva = gfn_to_hva_memslot_prot(memslot, + gpa_to_gfn(fault_addr), + &writable); + if (!kvm_is_error_hva(hva)) + kvm_riscv_gstage_map(vcpu, memslot, fault_addr, + hva, NULL); + } + } + + return 0; +} + +static int kvm_riscv_cove_promote_to_tvm(struct kvm_vcpu *vcpu, + unsigned long fdt_address, + unsigned long tap_addr) { + int rc; + + preload_pages(vcpu); + rc = kvm_riscv_cove_vm_single_step_init(vcpu, fdt_address, tap_addr); + if (rc) + goto done; + + vcpu->kvm->arch.vm_type = KVM_VM_TYPE_RISCV_COVE; +done: + return rc; +} + +static int kvm_sbi_ext_covh_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, + struct kvm_vcpu_sbi_return *retdata) +{ + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + unsigned long funcid = cp->a6; + int ret; + + switch (funcid) { + case SBI_EXT_COVH_PROMOTE_TO_TVM: + ret = kvm_riscv_cove_promote_to_tvm(vcpu, cp->a0, cp->a1); + return ret; + + default: + kvm_err("%s: Unsupported guest SBI %ld.\n", __func__, funcid); + retdata->err_val = SBI_ERR_NOT_SUPPORTED; + return -EOPNOTSUPP; + } +} + +const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_covh = { + .extid_start = SBI_EXT_COVH, + .extid_end = SBI_EXT_COVH, + .handler = kvm_sbi_ext_covh_handler, +}; diff --git a/arch/riscv/kvm/vm.c b/arch/riscv/kvm/vm.c index 8a1460dba76c..6df73ea8e16f 100644 --- a/arch/riscv/kvm/vm.c +++ b/arch/riscv/kvm/vm.c @@ -44,15 +44,15 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) if (unlikely(type == KVM_VM_TYPE_RISCV_COVE)) { if (!kvm_riscv_cove_enabled()) { - kvm_err("Unable to init CoVE VM because cove is not enabled\n"); + kvm_err("Unable to init CoVE VM because CoVE extension is not enabled\n"); return -EPERM; } - r = kvm_riscv_cove_vm_init(kvm); + r = kvm_riscv_cove_vm_multi_step_init(kvm); if (r) return r; kvm->arch.vm_type = type; - kvm_info("Trusted VM instance init successful\n"); + kvm_info("CoVE VM instance init successful\n"); } kvm_riscv_aia_init_vm(kvm); diff --git a/arch/riscv/mm/mem_encrypt.c b/arch/riscv/mm/mem_encrypt.c index 8523c508c3a5..498fbf5c6c9b 100644 --- a/arch/riscv/mm/mem_encrypt.c +++ b/arch/riscv/mm/mem_encrypt.c @@ -25,25 +25,44 @@ bool force_dma_unencrypted(struct device *dev) int set_memory_encrypted(unsigned long addr, int numpages) { + int i, rc; + if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return 0; if (!PAGE_ALIGNED(addr)) return -EINVAL; - return sbi_covg_unshare_memory(__pa(addr), numpages * PAGE_SIZE); + rc = sbi_covg_unshare_memory(__pa(addr), numpages * PAGE_SIZE); + if (rc) { + rc = 0; + for (i = 0; i < numpages && rc == 0; i++) + rc = sbi_covg_unshare_memory(__pa(addr + i * PAGE_SIZE), PAGE_SIZE); + } + + return rc; } EXPORT_SYMBOL_GPL(set_memory_encrypted); int set_memory_decrypted(unsigned long addr, int numpages) { + int i, rc; + if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return 0; if (!PAGE_ALIGNED(addr)) return -EINVAL; - return sbi_covg_share_memory(__pa(addr), numpages * PAGE_SIZE); + rc = sbi_covg_share_memory(__pa(addr), numpages * PAGE_SIZE); + if (rc) { + rc = 0; + /* Try page by page if TSM cannot share all pages at once */ + for (i = 0; i < numpages && rc == 0; i++) + rc = sbi_covg_share_memory(__pa(addr + i * PAGE_SIZE), PAGE_SIZE); + } + + return rc; } EXPORT_SYMBOL_GPL(set_memory_decrypted); diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c index d1a68b6d03b3..46e2ce22c729 100644 --- a/drivers/of/fdt.c +++ b/drivers/of/fdt.c @@ -1167,6 +1167,7 @@ int __init early_init_dt_scan_chosen(char *cmdline) early_init_dt_check_for_initrd(node); early_init_dt_check_for_elfcorehdr(node); +#ifndef CONFIG_RISCV_COVE_GUEST rng_seed = of_get_flat_dt_prop(node, "rng-seed", &l); if (rng_seed && l > 0) { add_bootloader_randomness(rng_seed, l); @@ -1178,6 +1179,7 @@ int __init early_init_dt_scan_chosen(char *cmdline) of_fdt_crc32 = crc32_be(~0, initial_boot_params, fdt_totalsize(initial_boot_params)); } +#endif /* Retrieve command line */ p = of_get_flat_dt_prop(node, "bootargs", &l);