This series implements the QEMU/KVM live migration support for the ice E800 series hardware. It is a long-overdue followup to the version from Yahui back in 2023. That version itself is an evolution of the original v1 protocol from some years before that. This version is a significant rework and complete replaces the virtchnl message based approach with one based on the PF serializing state via a custom Type-Length-Value format. This solution significantly reduces the impact a VF can have on the payload size, as the size is no longer based directly on messages sent by the VF. It also ensures that other data such as host-based RSS and MSI-X is sent, which previously was ignored since there were no equivalent virtchnl messages. Finally, replay can now be ordered to safely restore Tx and Rx queues without needing hacks to the virtchnl initialization flows, and the driver now only holds migration data for a short limited time while suspending, rather than requiring we save every virtchnl message the VF sends for the life time of the VF. To test this, I used QEMU/KVM: echo 2 >/sys/class/net/enp175s0f0np0/device/sriov_numvfs echo "0000:af:01.0" >/sys/bus/pci/drivers/iavf/unbind echo "0000:af:01.1" >/sys/bus/pci/drivers/iavf/unbind modprobe ice_vfio_pci echo "8086 1889" >/sys/bus/pci/drivers/ice-vfio-pci/new_id I've tested with QEMU using the "enable-migration=on" and "x-pre-copy-dirty-page-tracking=off" settings, as we do not currently support dirty page tracking. The initial host QEMU instance is launched as usual, while the target QEMU instance is launched with the -incoming tcp:localhost:4444 option. To initiate migration you can issue the migration command from the QEMU console: migrate tcp:localhost:4444 I've tested with and without traffic, and I tried to cover a wide variety of VF settings and configuration. REVIEWER NOTES AND REQUESTS: I am sending this as RFC to the netdev and VFIO mailing lists, as I am uncertain what the preferred path for merging is. I am also awaiting testing from Intel's virtualization team. I've managed to reduce the overall patch series size as much as possible by sending many of the cleanups ahead of time. These have finally all merged into net-next. This work is based on the original live migration patches from Yahui. However, the ice driver implementation is entirely rewritten. The VFIO driver code is still primarily Yahui's, with some minor alterations and cleanups applied. I decided to separate the deferred reset logic in the VFIO driver to its own patch, and am open to alternative suggestions for resolving this potential deadlock. I saw a similar deferred logic in other drivers. I have thus far not come up with a better solution. The biggest remaining gap on the ice driver side is that I don't have a good idea how to best plan for future VF enhancements. Currently, all the existing configuration and features now work. However, a future feature might require new migration data, and I don't have a good idea how to make the driver safely disable such features until they're supported within the migration. The TLV format does allow for extension, (both with a full version field and with passing the set of known TLVs). However, the driver likely could use some sort of infrastructure so that new VF virtchnl commands or features get blocked by default until they are confirmed to work with migration. Suggestions on how best to implement that are welcome. Link: https://lore.kernel.org/netdev/20231121025111.257597-1-yahui.cao@xxxxxxxxx/ Signed-off-by: Jacob Keller <jacob.e.keller@xxxxxxxxx> --- Jacob Keller (7): ice: add basic skeleton and TLV framework for live migration support ice: implement device suspension for live migration ice: add migration TLV for basic VF information ice: add migration TLVs for queue and interrupt state ice: add remaining migration TLVs ice-vfio-pci: add ice VFIO PCI live migration driver ice-vfio-pci: implement PCI .reset_done handling drivers/net/ethernet/intel/ice/ice.h | 2 + drivers/net/ethernet/intel/ice/ice_hw_autogen.h | 8 + drivers/net/ethernet/intel/ice/ice_vf_lib.h | 2 + .../net/ethernet/intel/ice/virt/migration_tlv.h | 495 +++++ include/linux/net/intel/ice_migration.h | 49 + drivers/net/ethernet/intel/ice/ice_main.c | 16 + drivers/net/ethernet/intel/ice/ice_vf_lib.c | 3 + drivers/net/ethernet/intel/ice/virt/migration.c | 2147 ++++++++++++++++++++ drivers/net/ethernet/intel/ice/virt/queues.c | 21 + drivers/vfio/pci/ice/main.c | 764 +++++++ MAINTAINERS | 7 + drivers/net/ethernet/intel/ice/Makefile | 1 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/ice/Kconfig | 8 + drivers/vfio/pci/ice/Makefile | 4 + 16 files changed, 3531 insertions(+) --- base-commit: 3b4296f5893d3a4e19edfc3800cb79381095e55f change-id: 20250130-e810-live-migration-jk-migration-tlv-33ce5c1d1034 Best regards, -- Jacob Keller <jacob.e.keller@xxxxxxxxx>
