<dan.j.williams@xxxxxxxxx> writes:
> Jason Gunthorpe wrote:
>> On Thu, Jul 31, 2025 at 07:07:17PM -0700, dan.j.williams@xxxxxxxxx wrote:
>> > Aneesh Kumar K.V (Arm) wrote:
>> > > Host:
>> > > step 1.
>> > > echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind
>> > > echo vfio-pci > /sys/bus/pci/devices/${DEVICE}/driver_override
>> > > echo ${DEVICE} > /sys/bus/pci/drivers_probe
>> > >
>> > > step 2.
>> > > echo 1 > /sys/bus/pci/devices/$DEVICE/tsm/connect
>> >
>> > Just for my own understanding... presumably there is no ordering
>> > constraint for ARM CCA between step1 and step2, right? I.e. The connect
>> > state is independent of the bind state.
>> >
>> > In the v4 PCI/TSM scheme the connect command is now:
>> >
>> > echo $tsm_dev > /sys/bus/pci/devices/$DEVICE/tsm/connect
>>
>> What does this do on the host? It seems to somehow prep it for VM
>> assignment? Seems pretty strange this is here in sysfs and not part of
>> creating the vPCI function in the VM through VFIO and iommufd?
>
> vPCI is out of the picture at this phase.
>
> On the host this establishes an SPDM session and sets up link encryption
> (IDE) with the physical device. Leave VMs out of the picture, this
> capability in isolation is a useful property. It addresses the similar
> threat model that Intel Total Memory Encryption (TME) or AMD Secure
> Memory Encryption (SME) go after, i.e. interposer on a physical link
> capturing data in flight.
>
> With that established then one can go futher to do the full TDISP dance.
>
>> Frankly, I'm nervous about making any uAPI whatsoever for the
>> hypervisor side at this point. I don't think we have enough of the
>> solution even in draft format. I'd really like your first merged TSM
>> series to only have uAPI for the guest side where things are hopefully
>> closer to complete.
>
> Aligned. I am not comfortable merging any of this until we have that end
> to end reliably stable for a kernel cycle or 2. The proposal is soak all
> the vendor solutions together in tsm.git#staging.
>
> Now, if the guest side graduates out of that staging before the host
> side, I am ok with that.
>
>> > > step 1:
>> > > echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind
>> > >
>> > > step 2: Move the device to TDISP LOCK state
>> > > echo 1 > /sys/bus/pci/devices/${DEVICE}/tsm/lock
>> >
>> > Ok, so my stance has recently picked up some nuance here. As Jason
>> > mentions here:
>> >
>> > http://lore.kernel.org/20250410235008.GC63245@xxxxxxxx
>> >
>> > "However it works, it should be done before the driver is probed and
>> > remain stable for the duration of the driver attachment. From the
>> > iommu side the correct iommu domain, on the correct IOMMU instance to
>> > handle the expected traffic should be setup as the DMA API's iommu
>> > domain."
>>
>> I think it is not just the dma api, but also the MMIO registers may
>> move location (form shared to protected IPA space for
>> example). Meaning any attached driver is completely wrecked.
>
> True.
>
>> > I agree with that up until the point where the implication is userspace
>> > control of the UNLOCKED->LOCKED transition. That transition requires
>> > enabling bus-mastering (BME),
>>
>> Why? That's sad. BME should be controlled by the VM driver not the
>> TSM, and it should be set only when a VM driver is probed to the RUN
>> state device?
>
> To me it is an unfortunate PCI specification wrinkle that writing to the
> command register drops the device from RUN to ERROR. So you can LOCK
> without setting BME, but then no DMA.
>
This is only w.r.t clearing BME isn't ?
According to section 11.2.6 DSM Tracking and Handling of Locked TDI Configurations
Clearing any of the following bits causes the TDI hosted
by the Function to transition to ERROR:
• Memory Space Enable
• Bus Master Enable
Which implies the flow described in the cover-letter where driver enable the BME works?
However clearing BME may be problematic? I did have a FIXME!!/comment in [1]
vfio_pci_core_close_device():
#if 0
/*
* destroy vdevice which involves tsm unbind before we disable pci disable
* A MSE/BME clear will transition the device to error state.
*/
if (core_vdev->iommufd_device)
iommufd_device_tombstone_vdevice(core_vdev->iommufd_device);
#endif
vfio_pci_core_disable(vdev);
Currently, we destroy (TSM unbind) the vdevice after calling
vfio_pci_core_disable(), which means BME is cleared before unbinding,
and the TDI transitions to the ERROR state.
[1] https://lore.kernel.org/all/20250728135216.48084-9-aneesh.kumar@xxxxxxxxxx/
-aneesh
