The following patches were made over Linus's tree. They implement a virtual PCI NVMe device using mdev/vfio. The device can be used by QEMU and in the guest will look like a normal old local PCI NVMe drive. They are based on Maxim Levitsky's mdev patches: https://lore.kernel.org/lkml/20190506125752.GA5288@xxxxxx/t/ but instead of trying to export a physical NVMe device to a guest, they are only focused on exporting a virtual device using the nvmet layer. Why another driver when we have so many? Performance. ===================================================== Without any tuning and major locks still in the main IO path, 4K IOPS for a single controller with a single namespace are higher than the kernel vhost-scsi driver and SPDK vhost-scsi/blk user when using lower number of queues/cpus/jobs. At just 2 queues, we are able to hit 1M IOPS: Note: the nvme mdev values below have the shadow doorbell enabled mdev vhost-scsi vhost-scsi-usr vhost-blk-usr numjobs 1 518K 198K 332K 301K 2 1037K 363K 609K 664K 4 974K 633K 1369K 1383K 8 813K 1788K 1358K 1363K However, by default we can't scale. But, tuning mdev to pre-pin pages (this requires patches to the vfio layer to support) then it also performs better at lower and higher number of queues/cpus/jobs used with it reaching 2.3M IOPS woth only 4 cpus/queues used: mdev numjobs 1 505K 2 1037K 4 2375K 8 2162K If we agree on a new virtual NVMe driver being ok, why mdev vs vhost? ===================================================================== The problem with a vhost nvme is: 2.1. If we do a fully vhost nvmet solution, it will require new guest drivers that present NVMe interfaces to userspace then perform the vhost spec on the backend like how vhost-scsi does. I don't want to implement a windows or even a linux nvme vhost driver. I don't think anyone wants the extra headache. 2.2. We can do a hybrid approach where in the guest it looks like we are a normal old local NVMe drive and use the guest's native NVMe driver. However in QEMU we would have a vhost nvme module that instead of using vhost virtqueues handles virtual PCI memory accesses as well as a vhost nvme kernel or user driver to process IO. So not as much extra code as option 1 since we don't have to worry about the guest but still extra QEMU code. 3. The mdev based solution does not have these drawbacks as it can look like a normal old local NVMe drive to the guest and can use QEMU's existing vfio layer. So it just requires the kernel driver. Why not a new blk driver or why not vdpa blk? ============================================= Applications want standardized interfaces for things like persistent reservations. They have to support them with SCSI and NVMe already and don't want to have to support a new virtio block interface. Also the nvmet-mdev-pci driver in this patchset can perform was well as SPDK vhost blk so that doesn't have the perf advantage like it used to. Status ====== This patchset is RFC quality only. You can discover a drive and do IO but it's not stable. There's several TODO items mentioned in the last patch. However, I think the patches are at the point where I wanted to get some feedback about if this even acceptable because the last time they were posted some people did not like how they hooked into drivers/nvme/host (this has been fixed in this posting). There's some other issues like: 1. Should the driver integrate with pci-epf (the drivers work very differently but could share some code)? 2. Should it try to fit into the existing configfs interface or implement it's own like how pci-epf did? I did an attempt for this but it feels wrong.
