On Fri, Jul 18, 2025 at 09:26:46AM -0300, Jason Gunthorpe wrote: > On Fri, Jul 18, 2025 at 05:15:42PM +0800, Xu Yilun wrote: > > > I would say this discussion is irrelevant to this case since we are > > > not doing any kind of in-place conversion. > > > > > > 1) At VM start the VMM gets the shared DMABUF and maps it to IOAS and > > > EPT > > > 2) Bind request comes in, VMM unmaps shared DMABUF from IOAS and EPT > > > then closes it. > > > 3) Bind is done > > > 4) VMM opens a shared and private DMABUF FD and learns the valid > > > ranges in both DMABUFs (ie what PFNS are private/shared) > > > > Who decides the dmabuf be shared or private? I assume you mean > > userspace, VFIO doesn't know about the shareability of each dmabuf? > > I imagined userspace would put a flag in the ioctl to VFIO to get the > dmabuf kind it wants. VFIO would have to be told from iommufd what the > per-pfn shared/private layout is and can use that to manage the > dmabufs. > > > > 5) VMM maps the shared DMABUF fragments to the EPT and IOAS > > > > Userspace could set a shared only memory slot. > > > > > 6) VMM maps the private DMABUF fragments to the S-EPT > > > > Userspace could set a private only memory slot, but maybe the concern > > is KVM can't trust userspace about the assertion of private. > > I imagined kvm can query the dmabuf and learn it is all private from > VFIO. The whole dmabuf, not per-pfn. Same for #5 it can check it is > shared memory too. Yes. On KVM side, userspace should firstly indicate the slot type in the ioctl and KVM verifies. Nowadays, KVM accepts 'private slot', which can be private or shared. And legacy slot, which can only be shared. We need to introduce a new type 'private only slot', then KVM verifies against dma-buf. We will see how KVM folks think about it. > > I sort of imagine using the same mechanism for p2p where we can mark > memory in the dmabuf with a 'provider' indicating these details. We > will have to see. Yes. > > > If VFIO should be aware of the shareability, IOMMUFD to VFIO callback is > > needed after TSM bind. > > Yes, whever this list of per-PFN shared/private must be shared between > VFIO and the TSM. > > > It is simpler on kernel side. FYR, may introduce some complexity in > > userspace, this may result in 3+ dmabufs. There may be shared holes > > in the BAR, and kvm slot can't be gfn sparse. This may not be a big > > problem. > > Userspace probably needs to create multiple slots for sparsity > > Again this seems much better if userspace handles it and just uses > simple mostly existing KVM interfaces. > > Why 3+ dmabufs? Isn't just one shared and one private per BAR? Sorry, we must have 3+ KVM slots, but can have 2 dmabufs. We can map the file offset to pfn ranges. KVM slot can't be for sparse GFNs. If there is a shared hole inside private space in the same bar, e.g. 0x10000000 - 0x10001fff private 0x10002000 - 0x10002fff shared 0x10003000 - 0x101fffff private We need 3 slots. Thanks, Yilun > > Jason
