The PCIe 6.1 specification, section 11, introduces the Trusted Execution
Environment (TEE) Device Interface Security Protocol (TDISP). This
protocol definition builds upon Component Measurement and Authentication
(CMA), and link Integrity and Data Encryption (IDE). It adds support for
assigning devices (PCI physical or virtual function) to a confidential
VM such that the assigned device is enabled to access guest private
memory protected by technologies like Intel TDX, AMD SEV-SNP, RISCV
COVE, or ARM CCA.
The "TSM" (TEE Security Manager) is a concept in the TDISP specification
of an agent that mediates between a "DSM" (Device Security Manager) and
system software in both a VMM and a confidential VM. A VMM uses TSM ABIs
to setup link security and assign devices. A confidential VM uses TSM
ABIs to transition an assigned device into the TDISP "RUN" state and
validate its configuration. From a Linux perspective the TSM abstracts
many of the details of TDISP, IDE, and CMA. Some of those details leak
through at times, but for the most part TDISP is an internal
implementation detail of the TSM.
CONFIG_PCI_TSM adds an "authenticated" attribute and "tsm/" subdirectory
to pci-sysfs. Consider that the TSM driver may itself be a PCI driver.
Userspace can watch for the arrival of the "TSM" core device,
/sys/class/tsm/tsm0/uevent, to know when the PCI core has initialized
TSM services.
The common verbs that the low-level TSM drivers implement are defined by
'struct pci_tsm_ops'. For now only 'connect' and 'disconnect' are
defined for secure session and IDE establishment. The 'probe' and
'remove' operations setup per-device context objects starting with
'struct pci_tsm_pf0', the device Physical Function 0 that mediates
communication to the device's Security Manager (DSM).
The locking allows for multiple devices to be executing commands
simultaneously, one outstanding command per-device and an rwsem
synchronizes the implementation relative to TSM
registration/unregistration events.
Thanks to Wu Hao for his work on an early draft of this support.
Cc: Lukas Wunner <lukas@xxxxxxxxx>
Cc: Samuel Ortiz <sameo@xxxxxxxxxxxx>
Cc: Alexey Kardashevskiy <aik@xxxxxxx>
Acked-by: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>
Co-developed-by: Xu Yilun <yilun.xu@xxxxxxxxxxxxxxx>
Signed-off-by: Xu Yilun <yilun.xu@xxxxxxxxxxxxxxx>
Signed-off-by: Dan Williams <dan.j.williams@xxxxxxxxx>
---
Documentation/ABI/testing/sysfs-bus-pci | 45 +++
MAINTAINERS | 2 +
drivers/pci/Kconfig | 14 +
drivers/pci/Makefile | 1 +
drivers/pci/pci-sysfs.c | 4 +
drivers/pci/pci.h | 10 +
drivers/pci/probe.c | 1 +
drivers/pci/remove.c | 3 +
drivers/pci/tsm.c | 437 ++++++++++++++++++++++++
drivers/virt/coco/host/tsm-core.c | 19 +-
include/linux/pci-tsm.h | 138 ++++++++
include/linux/pci.h | 3 +
include/linux/tsm.h | 4 +-
include/uapi/linux/pci_regs.h | 1 +
14 files changed, 679 insertions(+), 3 deletions(-)
create mode 100644 drivers/pci/tsm.c
create mode 100644 include/linux/pci-tsm.h
diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci
index 69f952fffec7..1d38e0d3a6be 100644
--- a/Documentation/ABI/testing/sysfs-bus-pci
+++ b/Documentation/ABI/testing/sysfs-bus-pci
@@ -612,3 +612,48 @@ Description:
# ls doe_features
0001:01 0001:02 doe_discovery
+
+What: /sys/bus/pci/devices/.../tsm/
+Date: July 2024
+Contact: linux-coco@xxxxxxxxxxxxxxx
+Description:
+ This directory only appears if a physical device function
+ supports authentication (PCIe CMA-SPDM), interface security
+ (PCIe TDISP), and is accepted for secure operation by the
+ platform TSM driver. This attribute directory appears
+ dynamically after the platform TSM driver loads. So, only after
+ the /sys/class/tsm/tsm0 device arrives can tools assume that
+ devices without a tsm/ attribute directory will never have one,
+ before that, the security capabilities of the device relative to
+ the platform TSM are unknown. See
+ Documentation/ABI/testing/sysfs-class-tsm.
+
+What: /sys/bus/pci/devices/.../tsm/connect
+Date: July 2024
+Contact: linux-coco@xxxxxxxxxxxxxxx
+Description:
+ (RW) Writing "1" to this file triggers the platform TSM (TEE
+ Security Manager) to establish a connection with the device.
+ This typically includes an SPDM (DMTF Security Protocols and
+ Data Models) session over PCIe DOE (Data Object Exchange) and
+ may also include PCIe IDE (Integrity and Data Encryption)
+ establishment.
+
+What: /sys/bus/pci/devices/.../authenticated
+Date: July 2024
+Contact: linux-pci@xxxxxxxxxxxxxxx
+Description:
+ When the device's tsm/ directory is present device
+ authentication (PCIe CMA-SPDM) and link encryption (PCIe IDE)
+ are handled by the platform TSM (TEE Security Manager). When the
+ tsm/ directory is not present this attribute reflects only the
+ native CMA-SPDM authentication state with the kernel's
+ certificate store.
+
+ If the attribute is not present, it indicates that
+ authentication is unsupported by the device, or the TSM has no
+ available authentication methods for the device.
+
+ When present and the tsm/ attribute directory is present, the
+ authenticated attribute is an alias for the device 'connect'
+ state. See the 'tsm/connect' attribute for more details.
diff --git a/MAINTAINERS b/MAINTAINERS
index 09bf7b45708b..2f92623b4de5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -24560,8 +24560,10 @@ M: Dan Williams <dan.j.williams@xxxxxxxxx>
L: linux-coco@xxxxxxxxxxxxxxx
S: Maintained
F: Documentation/ABI/testing/configfs-tsm-report
+F: drivers/pci/tsm.c
F: drivers/virt/coco/guest/
F: drivers/virt/coco/host/
+F: include/linux/pci-tsm.h
F: include/linux/tsm.h
TRUSTED SERVICES TEE DRIVER
diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 0c662f9813eb..5c3f896ac9f4 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -135,6 +135,20 @@ config PCI_IDE_STREAM_MAX
track the maximum possibility of 256 streams per host bridge
in the typical case.
+config PCI_TSM
+ bool "PCI TSM: Device security protocol support"
+ select PCI_IDE
+ select PCI_DOE
+ help
+ The TEE (Trusted Execution Environment) Device Interface
+ Security Protocol (TDISP) defines a "TSM" as a platform agent
+ that manages device authentication, link encryption, link
+ integrity protection, and assignment of PCI device functions
+ (virtual or physical) to confidential computing VMs that can
+ access (DMA) guest private memory.
+
+ Enable a platform TSM driver to use this capability.
+
config PCI_DOE
bool "Enable PCI Data Object Exchange (DOE) support"
help
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index 6612256fd37d..2c545f877062 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -35,6 +35,7 @@ obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
obj-$(CONFIG_VGA_ARB) += vgaarb.o
obj-$(CONFIG_PCI_DOE) += doe.o
obj-$(CONFIG_PCI_IDE) += ide.o
+obj-$(CONFIG_PCI_TSM) += tsm.o
obj-$(CONFIG_PCI_DYNAMIC_OF_NODES) += of_property.o
obj-$(CONFIG_PCI_NPEM) += npem.o
obj-$(CONFIG_PCIE_TPH) += tph.o
diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index c6cda56ca52c..6bd16a110916 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -1811,6 +1811,10 @@ const struct attribute_group *pci_dev_attr_groups[] = {
#endif
#ifdef CONFIG_PCI_DOE
&pci_doe_sysfs_group,
+#endif
+#ifdef CONFIG_PCI_TSM
+ &pci_tsm_auth_attr_group,
+ &pci_tsm_pf0_attr_group,
#endif
NULL,
};
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 10be2ce5e5d5..7f763441f658 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -517,6 +517,16 @@ void pci_ide_init(struct pci_dev *dev);
static inline void pci_ide_init(struct pci_dev *dev) { }
#endif
+#ifdef CONFIG_PCI_TSM
+void pci_tsm_init(struct pci_dev *pdev);
+void pci_tsm_destroy(struct pci_dev *pdev);
+extern const struct attribute_group pci_tsm_pf0_attr_group;
+extern const struct attribute_group pci_tsm_auth_attr_group;
+#else
+static inline void pci_tsm_init(struct pci_dev *pdev) { }
+static inline void pci_tsm_destroy(struct pci_dev *pdev) { }
+#endif
+
/**
* pci_dev_set_io_state - Set the new error state if possible.
*
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 1b597b6e946c..c090289b70be 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -2620,6 +2620,7 @@ static void pci_init_capabilities(struct pci_dev *dev)
pci_tph_init(dev); /* TLP Processing Hints */
pci_rebar_init(dev); /* Resizable BAR */
pci_ide_init(dev); /* Link Integrity and Data Encryption */
+ pci_tsm_init(dev); /* TEE Security Manager connection */
pcie_report_downtraining(dev);
pci_init_reset_methods(dev);
diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c
index 445afdfa6498..21851c13becd 100644
--- a/drivers/pci/remove.c
+++ b/drivers/pci/remove.c
@@ -55,6 +55,9 @@ static void pci_destroy_dev(struct pci_dev *dev)
pci_doe_sysfs_teardown(dev);
pci_npem_remove(dev);
+ /* before device_del() to keep config cycle access */
+ pci_tsm_destroy(dev);
+
device_del(&dev->dev);
down_write(&pci_bus_sem);
diff --git a/drivers/pci/tsm.c b/drivers/pci/tsm.c
new file mode 100644
index 000000000000..d00a8e471340
--- /dev/null
+++ b/drivers/pci/tsm.c
@@ -0,0 +1,437 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * TEE Security Manager for the TEE Device Interface Security Protocol
+ * (TDISP, PCIe r6.1 sec 11)
+ *
+ * Copyright(c) 2024 Intel Corporation. All rights reserved.
+ */
+
+#define dev_fmt(fmt) "TSM: " fmt
+
+#include <linux/bitfield.h>
+#include <linux/xarray.h>
+#include <linux/sysfs.h>
+
+#include <linux/pci.h>
+#include <linux/pci-doe.h>
+#include <linux/pci-tsm.h>
+#include "pci.h"
+
+/*
+ * Provide a read/write lock against the init / exit of pdev tsm
+ * capabilities and arrival/departure of a tsm instance
+ */
+static DECLARE_RWSEM(pci_tsm_rwsem);
+static const struct pci_tsm_ops *tsm_ops;
+
+/* supplemental attributes to surface when pci_tsm_attr_group is active */
+static const struct attribute_group *pci_tsm_owner_attr_group;
+
+static struct pci_tsm_pf0 *to_pci_tsm_pf0(struct pci_tsm *pci_tsm)
+{
+ struct pci_dev *pdev = pci_tsm->pdev;
+
+ if (!is_pci_tsm_pf0(pdev) || pci_tsm->type != PCI_TSM_PF0) {
+ dev_WARN_ONCE(&pdev->dev, 1, "invalid context object\n");
+ return NULL;
+ }
+
+ return container_of(pci_tsm, struct pci_tsm_pf0, tsm);
+}
+
+/* TODO: switch to ACQUIRE() and ACQUIRE_ERR() */
+static struct mutex *tsm_ops_lock(struct pci_tsm_pf0 *tsm)
+{
+ lockdep_assert_held(&pci_tsm_rwsem);
+
+ if (mutex_lock_interruptible(&tsm->lock) != 0)
+ return NULL;
+ return &tsm->lock;
+}
+DEFINE_FREE(tsm_ops_unlock, struct mutex *, if (_T) mutex_unlock(_T))
+
+static int pci_tsm_disconnect(struct pci_dev *pdev)
+{
+ struct pci_tsm_pf0 *tsm = to_pci_tsm_pf0(pdev->tsm);
+
+ struct mutex *lock __free(tsm_ops_unlock) = tsm_ops_lock(tsm);
+ if (!lock)
+ return -EINTR;
+
+ if (tsm->state < PCI_TSM_INIT)
+ return -ENXIO;
+ if (tsm->state < PCI_TSM_CONNECT)
+ return 0;
+
+ tsm_ops->disconnect(pdev);
+ tsm->state = PCI_TSM_INIT;
+
+ return 0;
+}
+
+static int pci_tsm_connect(struct pci_dev *pdev)
+{
+ struct pci_tsm_pf0 *tsm = to_pci_tsm_pf0(pdev->tsm);
+ int rc;
+
+ struct mutex *lock __free(tsm_ops_unlock) = tsm_ops_lock(tsm);
+ if (!lock)
+ return -EINTR;
+
+ if (tsm->state < PCI_TSM_INIT)
+ return -ENXIO;
+ if (tsm->state >= PCI_TSM_CONNECT)
+ return 0;
+
+ rc = tsm_ops->connect(pdev);
+ if (rc)
+ return rc;
+ tsm->state = PCI_TSM_CONNECT;
+ return 0;
+}
+
+/* TODO: switch to ACQUIRE() and ACQUIRE_ERR() */
+static struct rw_semaphore *tsm_read_lock(void)
+{
+ if (down_read_interruptible(&pci_tsm_rwsem))
+ return NULL;
+ return &pci_tsm_rwsem;
+}
+DEFINE_FREE(tsm_read_unlock, struct rw_semaphore *, if (_T) up_read(_T))
+
+static ssize_t connect_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t len)
+{
+ int rc;
+ bool connect;
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+ rc = kstrtobool(buf, &connect);
+ if (rc)
+ return rc;
+
+ struct rw_semaphore *lock __free(tsm_read_unlock) = tsm_read_lock();
+ if (!lock)
+ return -EINTR;
+
+ if (connect)
+ rc = pci_tsm_connect(pdev);
+ else
+ rc = pci_tsm_disconnect(pdev);
+ if (rc)
+ return rc;
+ return len;
+}
+
+static ssize_t connect_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ struct pci_tsm_pf0 *tsm;
+
+ struct rw_semaphore *lock __free(tsm_read_unlock) = tsm_read_lock();
+ if (!lock)
+ return -EINTR;
+
+ if (!pdev->tsm)
+ return -ENXIO;
+
+ tsm = to_pci_tsm_pf0(pdev->tsm);
+ return sysfs_emit(buf, "%d\n", tsm->state >= PCI_TSM_CONNECT);
+}
+static DEVICE_ATTR_RW(connect);
+
+static bool pci_tsm_pf0_group_visible(struct kobject *kobj)
+{
+ struct device *dev = kobj_to_dev(kobj);
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+ if (pdev->tsm && is_pci_tsm_pf0(pdev))
+ return true;
+ return false;
+}
+DEFINE_SIMPLE_SYSFS_GROUP_VISIBLE(pci_tsm_pf0);
+
+static struct attribute *pci_tsm_pf0_attrs[] = {
+ &dev_attr_connect.attr,
+ NULL
+};
+
+const struct attribute_group pci_tsm_pf0_attr_group = {
+ .name = "tsm",
+ .attrs = pci_tsm_pf0_attrs,
+ .is_visible = SYSFS_GROUP_VISIBLE(pci_tsm_pf0),
+};
+
+static ssize_t authenticated_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ /*
+ * When device authentication is TSM owned, 'authenticated' is
+ * identical to the connect state.
+ */
+ return connect_show(dev, attr, buf);
+}
+static DEVICE_ATTR_RO(authenticated);
+
+static struct attribute *pci_tsm_auth_attrs[] = {
+ &dev_attr_authenticated.attr,
+ NULL
+};
+
+const struct attribute_group pci_tsm_auth_attr_group = {
+ .attrs = pci_tsm_auth_attrs,
+ .is_visible = SYSFS_GROUP_VISIBLE(pci_tsm_pf0),
+};
+
+/*
+ * Retrieve physical function0 device whether it has TEE capability or not
+ */
+static struct pci_dev *pf0_dev_get(struct pci_dev *pdev)
+{
+ struct pci_dev *pf_dev = pci_physfn(pdev);
+
+ if (PCI_FUNC(pf_dev->devfn) == 0)
+ return pci_dev_get(pf_dev);
+
+ return pci_get_slot(pf_dev->bus,
+ pf_dev->devfn - PCI_FUNC(pf_dev->devfn));
+}
+
+static bool is_pci_tsm_downstream(struct pci_dev *pdev)
+{
+ struct pci_dev *uport;
+
+ if (pci_pcie_type(pdev) != PCI_EXP_TYPE_ENDPOINT)
+ return false;
+
+ /* "grandparent" of an endpoint is an Upstream Port (or Root Port) */
+ if (!pdev->dev.parent)
+ return false;
+ if (!pdev->dev.parent->parent)
+ return false;
+
+ uport = to_pci_dev(pdev->dev.parent->parent);
+ if (pci_pcie_type(uport) != PCI_EXP_TYPE_UPSTREAM)
+ return false;
+
+ if (!uport->tsm)
+ return false;
+
+ /* Upstream Port has a 'tsm' context, probe downstream devices. */
+ return true;
+}
+
+static enum pci_tsm_type pci_tsm_type(struct pci_dev *pdev)
+{
+ if (is_pci_tsm_pf0(pdev))
+ return PCI_TSM_PF0;
+
+ struct pci_dev *pf0 __free(pci_dev_put) = pf0_dev_get(pdev);
+ if (!pf0)
+ return PCI_TSM_INVALID;
+
+ if (pf0->tsm && pf0->tsm->type == PCI_TSM_PF0) {
+ if (pdev->is_virtfn)
+ return PCI_TSM_VIRTFN;
+ else
+ return PCI_TSM_MFD;
+ }
+
+ /*
+ * Allow for Device Security Managers (DSMs) at a Switch level
+ * to host TDISP services for downstream devices
+ */
+ if (is_pci_tsm_downstream(pdev))
+ return PCI_TSM_DOWNSTREAM;
+ return PCI_TSM_INVALID;
+}
+
+/**
+ * pci_tsm_initialize() - base 'struct pci_tsm' initialization
+ * @pdev: The PCI device
+ * @tsm: context to initialize
+ */
+void pci_tsm_initialize(struct pci_dev *pdev, struct pci_tsm *tsm)
+{
+ tsm->type = pci_tsm_type(pdev);
+ tsm->pdev = pdev;
+}
+EXPORT_SYMBOL_GPL(pci_tsm_initialize);
+
+/**
+ * pci_tsm_pf0_initialize() - common 'struct pci_tsm_pf0' initialization
+ * @pdev: Physical Function 0 PCI device
+ * @tsm: context to initialize
+ */
+int pci_tsm_pf0_initialize(struct pci_dev *pdev, struct pci_tsm_pf0 *tsm)
+{
+ mutex_init(&tsm->lock);
+ tsm->doe_mb = pci_find_doe_mailbox(pdev, PCI_VENDOR_ID_PCI_SIG,
+ PCI_DOE_PROTO_CMA);
+ if (!tsm->doe_mb) {
+ pci_warn(pdev, "TSM init failure, no CMA mailbox\n");
+ return -ENODEV;
+ }
+
+ tsm->state = PCI_TSM_INIT;
+ pci_tsm_initialize(pdev, &tsm->tsm);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(pci_tsm_pf0_initialize);
+
+static void __pci_tsm_pf0_destroy(struct pci_tsm_pf0 *tsm)
+{
+ mutex_destroy(&tsm->lock);
+}
+
+static void tsm_remove(struct pci_tsm *tsm)
+{
+ if (!tsm)
+ return;
+ tsm_ops->remove(tsm);
+}
+DEFINE_FREE(tsm_remove, struct pci_tsm *, if (_T) tsm_remove(_T))
+
+static void pci_tsm_pf0_init(struct pci_dev *pdev)
+{
+ bool tee_cap;
+
+ tee_cap = pdev->devcap & PCI_EXP_DEVCAP_TEE;
+
+ if (!(pdev->ide_cap || tee_cap))
+ return;
+
+ lockdep_assert_held_write(&pci_tsm_rwsem);
+ if (!tsm_ops)
+ return;
+
+ /*
+ * If a physical device has any security capabilities it may be
+ * a candidate to connect with the platform TSM
+ */
+ struct pci_tsm *pci_tsm __free(tsm_remove) = tsm_ops->probe(pdev);
+
+ pci_dbg(pdev, "Device security capabilities detected (%s%s ), TSM %s\n",
+ pdev->ide_cap ? " ide" : "", tee_cap ? " tee" : "",
+ pci_tsm ? "attach" : "skip");
+
+ if (!pci_tsm)
+ return;
+
+ pdev->tsm = no_free_ptr(pci_tsm);
+ sysfs_update_group(&pdev->dev.kobj, &pci_tsm_auth_attr_group);
+ sysfs_update_group(&pdev->dev.kobj, &pci_tsm_pf0_attr_group);
+ if (pci_tsm_owner_attr_group)
+ sysfs_merge_group(&pdev->dev.kobj, pci_tsm_owner_attr_group);
+}
+
+static void __pci_tsm_init(struct pci_dev *pdev)
+{
+ enum pci_tsm_type type = pci_tsm_type(pdev);
+
+ switch (type) {
+ case PCI_TSM_PF0:
+ pci_tsm_pf0_init(pdev);
+ break;
+ case PCI_TSM_VIRTFN:
+ case PCI_TSM_MFD:
+ case PCI_TSM_DOWNSTREAM:
+ pdev->tsm = tsm_ops->probe(pdev);
+ break;
+ case PCI_TSM_INVALID:
+ default:
+ break;
+ }
+}
+
+void pci_tsm_init(struct pci_dev *pdev)
+{
+ guard(rwsem_write)(&pci_tsm_rwsem);
+ __pci_tsm_init(pdev);
+}
+
+int pci_tsm_core_register(const struct pci_tsm_ops *ops, const struct attribute_group *grp)
+{
+ struct pci_dev *pdev = NULL;
+
+ if (!ops)
+ return 0;
+ guard(rwsem_write)(&pci_tsm_rwsem);
+ if (tsm_ops)
+ return -EBUSY;
+ tsm_ops = ops;
+ pci_tsm_owner_attr_group = grp;
+ for_each_pci_dev(pdev)
+ __pci_tsm_init(pdev);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(pci_tsm_core_register);
+
+static void pci_tsm_pf0_destroy(struct pci_dev *pdev)
+{
+ struct pci_tsm_pf0 *tsm = to_pci_tsm_pf0(pdev->tsm);
+
+ if (tsm->state > PCI_TSM_INIT)
+ pci_tsm_disconnect(pdev);
+ pdev->tsm = NULL;
+ if (pci_tsm_owner_attr_group)
+ sysfs_unmerge_group(&pdev->dev.kobj, pci_tsm_owner_attr_group);
+ sysfs_update_group(&pdev->dev.kobj, &pci_tsm_pf0_attr_group);
+ sysfs_update_group(&pdev->dev.kobj, &pci_tsm_auth_attr_group);
+ __pci_tsm_pf0_destroy(tsm);
+}
+
+static void __pci_tsm_destroy(struct pci_dev *pdev)
+{
+ struct pci_tsm *pci_tsm = pdev->tsm;
+
+ if (!pci_tsm)
+ return;
+
+ lockdep_assert_held_write(&pci_tsm_rwsem);
+
+ if (is_pci_tsm_pf0(pdev))
+ pci_tsm_pf0_destroy(pdev);
+ tsm_ops->remove(pci_tsm);
+}
+
+void pci_tsm_destroy(struct pci_dev *pdev)
+{
+ guard(rwsem_write)(&pci_tsm_rwsem);
+ __pci_tsm_destroy(pdev);
+}
+
+void pci_tsm_core_unregister(const struct pci_tsm_ops *ops)
+{
+ struct pci_dev *pdev = NULL;
+
+ if (!ops)
+ return;
+ guard(rwsem_write)(&pci_tsm_rwsem);
+ if (ops != tsm_ops)
+ return;
+ for_each_pci_dev(pdev)
+ __pci_tsm_destroy(pdev);
+ tsm_ops = NULL;
+}
+EXPORT_SYMBOL_GPL(pci_tsm_core_unregister);
+
+int pci_tsm_doe_transfer(struct pci_dev *pdev, enum pci_doe_proto type,
+ const void *req, size_t req_sz, void *resp,
+ size_t resp_sz)
