On Fri, Jun 06, 2025 at 01:34:55PM -0300, Jason Gunthorpe wrote: > On Fri, Jun 06, 2025 at 11:40:30PM +0800, Xu Yilun wrote: > > On Wed, Jun 04, 2025 at 09:37:47AM -0300, Jason Gunthorpe wrote: > > > On Wed, Jun 04, 2025 at 11:47:14AM +1000, Alexey Kardashevskiy wrote: > > > > > > > If it is the case of killing QEMU - then there is no usespace and > > > > then it is a matter of what fd is closed first - VFIO-PCI or > > > > IOMMUFD, we could teach IOMMUFD to hold an VFIO-PCI fd to guarantee > > > > the order. Thanks, > > > > > > It is the other way around VFIO holds the iommufd and it always > > > destroys first. > > > > > > We are missing a bit where the vfio destruction of the idevice does > > > not clean up the vdevice too. > > > > Seems there is still problem, the suggested flow is: > > > > 1. VFIO fops release > > 2. vfio_pci_core_close_device() > > 3. vfio_df_iommufd_unbind() > > 4. iommufd_device_unbind() > > 5. iommufd_device_destroy() > > 6. iommufd_vdevice_destroy() (not implemented) > > 7. iommufd_vdevice_tsm_unbind() > > > > In step 2, vfio pci does all cleanup, including invalidate MMIO. > > In step 7 vdevice does tsm unbind, this is not correct. TSM unbind > > should be done before invalidating MMIO. > > > > TSM unbind should always the first thing to do to release lockdown, > > then cleanup physical device configuration. > > I think you'd have to re-order things so that vfio_df_iommufd_unbind() > happens before the mmio invalidate.. That seems an easier solution. > > And if you can succeed in moving the MMIO mapping to bind/unbind then > the invalidate from vfio won't matter. It still matters. If VFIO tries to invalidate MMIOs before Unbind, DMA Silent Drop protection still triggers. Ensuring a correct bind/unbind operation in IOMMUFD cannot make VFIO agnostic to bind/unbind. BTW: Let me summarize what I've got about Bind-MMIO interaction. 1. All vendors need TSM Unbind first, then invalidate MMIOs, this is to avoid DMA Silent Drop protection which exists on all vendors. 2. TDX Connect additionally requires finer operation sequence during TSM Unbind, i.e. invalidate MMIOs after TDI stop and before TDI metadata free, this is for TDX firmware internal TDI management logic. Thanks, Yilun > > Jason
