crypto: qat - enable SRIOV VF live migration
From: | Xin Zeng <xin.zeng-AT-intel.com> | |
To: | linux-crypto-AT-vger.kernel.org, kvm-AT-vger.kernel.org | |
Subject: | [RFC 0/5] crypto: qat - enable SRIOV VF live migration | |
Date: | Fri, 30 Jun 2023 21:12:59 +0800 | |
Message-ID: | <20230630131304.64243-1-xin.zeng@intel.com> | |
Cc: | giovanni.cabiddu-AT-intel.com, andriy.shevchenko-AT-linux.intel.com, Xin Zeng <xin.zeng-AT-intel.com> | |
Archive-link: | Article |
This set enables live migration for Intel QAT GEN4 SRIOV Virtual Functions (VFs). It is composed of 5 patches. The first is a pre-requisite. It adds logic to the QAT PF driver that allows to save and restore the state of a bank (a virtual function is a wrapper around a bank) and drain a ring pair. The second patch adds to the QAT PF driver a set of interfaces to allow to save and restore the state of a VF that will be called by the modules qat_vfio_pci which will be introduced in the last patch. The third adds HZ_PER_GHZ which will be required by the fourth one. The fourth one implements the defined interfaces. The last one adds a vfio pci extension specific for QAT which intercepts the vfio device operations for a QAT VF to allow live migration. Here are the steps required to test the live migration of a QAT GEN4 VF: 1. Bind one or more QAT GEN4 VF devices to the module qat_vfio_pci.ko 2. Assign the VFs to the virtual machine and enable device live migration 3. Run a workload using a QAT VF inside the VM, for example using qatlib (https://github.com/intel/qatlib) 4. Migrate the VM from the source node to a destination node For P2P states support and AER support, we are going to implement these in the final version. Any feedback is appreciated! Andy Shevchenko (1): units: Add HZ_PER_GHZ Siming Wan (1): crypto: qat - add bank save/restore and RP drain Xin Zeng (3): crypto: qat - add interface for live migration crypto: qat - implement interface for live migration vfio/qat: Add vfio_pci driver for Intel QAT VF devices .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c | 9 +- .../intel/qat/qat_4xxx/adf_4xxx_hw_data.h | 3 +- .../intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c | 2 +- .../qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c | 2 +- .../intel/qat/qat_c62x/adf_c62x_hw_data.c | 2 +- .../intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c | 2 +- drivers/crypto/intel/qat/qat_common/Makefile | 4 +- .../intel/qat/qat_common/adf_accel_devices.h | 78 ++- .../intel/qat/qat_common/adf_gen2_hw_data.c | 17 +- .../intel/qat/qat_common/adf_gen2_hw_data.h | 10 +- .../intel/qat/qat_common/adf_gen4_hw_data.c | 415 +++++++++++- .../intel/qat/qat_common/adf_gen4_hw_data.h | 152 ++++- .../intel/qat/qat_common/adf_gen4_pfvf.c | 7 +- .../intel/qat/qat_common/adf_gen4_pfvf.h | 7 + .../intel/qat/qat_common/adf_gen4_vf_mig.c | 609 ++++++++++++++++++ .../intel/qat/qat_common/adf_mstate_mgr.c | 267 ++++++++ .../intel/qat/qat_common/adf_mstate_mgr.h | 99 +++ .../intel/qat/qat_common/adf_transport.c | 11 +- .../crypto/intel/qat/qat_common/adf_vf_isr.c | 2 +- .../crypto/intel/qat/qat_common/qat_vf_mig.c | 106 +++ .../qat/qat_dh895xcc/adf_dh895xcc_hw_data.c | 2 +- .../qat_dh895xccvf/adf_dh895xccvf_hw_data.c | 2 +- drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 1 + drivers/vfio/pci/qat/Kconfig | 13 + drivers/vfio/pci/qat/Makefile | 4 + drivers/vfio/pci/qat/qat_vfio_pci_main.c | 518 +++++++++++++++ include/linux/qat/qat_vf_mig.h | 15 + include/linux/units.h | 6 +- 29 files changed, 2332 insertions(+), 35 deletions(-) create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h create mode 100644 drivers/crypto/intel/qat/qat_common/qat_vf_mig.c create mode 100644 drivers/vfio/pci/qat/Kconfig create mode 100644 drivers/vfio/pci/qat/Makefile create mode 100644 drivers/vfio/pci/qat/qat_vfio_pci_main.c create mode 100644 include/linux/qat/qat_vf_mig.h -- 2.18.2