On Wed, Sep 10, 2025 at 8:49 AM Robert Sander
<r.sander@xxxxxxxxxxxxxxxxxxx> wrote:
>
> Hi,
>
> Am 9/9/25 um 17:34 schrieb Yuri Weinstein:
>
> > * Dashboard: Added support for NVMe/TCP (gateway groups, multiple
> > namespaces),
>
> I have just upgraded a test cluster and NVMEoF does not work any more.
>
> The previous version running was quay.ceph.io/ceph-ci/ceph:squid-nvmeof and NVMEoF was working there.
>
> The gateways seem to work and subsystem and namespaces are still configured but an initiator does not see namespaces.
>
> # ceph orch ps --daemon_type nvmeof
> NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
> nvmeof.hosa.b236asbrce101.ccaujq b236asbrce101 *:5500,4420,8009,10008 running (13h) 36s ago 13h 128M - 1.5.7 2c3e7ef350b8 74672e87a945
> nvmeof.hosa.b236asbrce102.gnbwsc b236asbrce102 *:5500,4420,8009,10008 running (13h) 36s ago 13h 120M - 1.5.7 2c3e7ef350b8 1929979ab7e1
> nvmeof.hosa.b236asbrce103.bthewi b236asbrce103 *:5500,4420,8009,10008 running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8 14885171d1a5
> nvmeof.hosa.b236asbrce104.vcgzvg b236asbrce104 *:5500,4420,8009,10008 running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8 7b00b0f3356a
> # ceph nvmeof subsystem list
> +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
> |Nqn |Serial Number |Model Number |Namespace Count|Subtype|Max Namespaces|Has Dhchap Key|Allow Any Host|Created Without Key|
> +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
> |nqn.2025-08.com.example:HOSA.hosa|Ceph30844860597194|Ceph bdev Controller|2 |NVMe |128 |False |True |False |
> +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
> # ceph nvmeof namespace list nqn.2025-08.com.example:HOSA.hosa
> +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+
> |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only|
> +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+
> |bdev_57f2c81a-0b07-406b-84aa-c62d66b5f10d|ns01 |nvmeof |5 |100GB |512B |0 |0 |0 |0 |True |[] |1 |57f2c81a-0b07-406b-84aa-c62d66b5f10d|nqn.2025-08.com.example:HOSA.hosa|False |False |False |
> |bdev_b4bf328a-8c67-4879-bf13-a799884d6dbe|ns02 |nvmeof |5 |200GB |512B |0 |0 |0 |0 |True |[] |2 |b4bf328a-8c67-4879-bf13-a799884d6dbe|nqn.2025-08.com.example:HOSA.hosa|False |False |False |
> +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+
> # ceph nvmeof listener list nqn.2025-08.com.example:HOSA.hosa
> +-------------+------+--------------+------+------+-------+
> |Host Name |Trtype|Traddr |Secure|Adrfam|Trsvcid|
> +-------------+------+--------------+------+------+-------+
> |b236asbrce101|TCP |10.105.141.195|False |ipv4 |4420 |
> |b236asbrce102|TCP |10.105.141.196|False |ipv4 |4420 |
> |b236asbrce103|TCP |10.105.141.197|False |ipv4 |4420 |
> |b236asbrce104|TCP |10.105.141.198|False |ipv4 |4420 |
> +-------------+------+--------------+------+------+-------+
>
> On the initiator host:
>
> # nvme discover -t tcp -a 10.105.141.195 -s 8009 -v
> warning: use hostid which does not match uuid in hostnqn
> nvme0: nqn.2014-08.org.nvmexpress.discovery connected
>
> Discovery Log Number of Records 4, Generation counter 6
> =====Discovery Log Entry 0======
> trtype: tcp
> adrfam: ipv4
> subtype: nvme subsystem
> treq: not required
> portid: 0
> trsvcid: 4420
> subnqn: nqn.2025-08.com.example:HOSA.hosa
> traddr: 10.105.141.195
> eflags: none
> sectype: none
> =====Discovery Log Entry 1======
> trtype: tcp
> adrfam: ipv4
> subtype: nvme subsystem
> treq: not required
> portid: 1
> trsvcid: 4420
> subnqn: nqn.2025-08.com.example:HOSA.hosa
> traddr: 10.105.141.196
> eflags: none
> sectype: none
> =====Discovery Log Entry 2======
> trtype: tcp
> adrfam: ipv4
> subtype: nvme subsystem
> treq: not required
> portid: 2
> trsvcid: 4420
> subnqn: nqn.2025-08.com.example:HOSA.hosa
> traddr: 10.105.141.197
> eflags: none
> sectype: none
> =====Discovery Log Entry 3======
> trtype: tcp
> adrfam: ipv4
> subtype: nvme subsystem
> treq: not required
> portid: 3
> trsvcid: 4420
> subnqn: nqn.2025-08.com.example:HOSA.hosa
> traddr: 10.105.141.198
> eflags: none
> sectype: none
> nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected
>
> # nvme connect-all -t tcp -a 10.105.141.195 -s 8009 -v
> warning: use hostid which does not match uuid in hostnqn
> nvme0: nqn.2014-08.org.nvmexpress.discovery connected
> nvme1: nqn.2025-08.com.example:HOSA.hosa connected
> nvme2: nqn.2025-08.com.example:HOSA.hosa connected
> nvme3: nqn.2025-08.com.example:HOSA.hosa connected
> nvme4: nqn.2025-08.com.example:HOSA.hosa connected
> nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected
>
> Looking good so far, but:
>
> # nvme list
> Node Generic SN Model Namespace Usage Format FW Rev
> --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
> # nvme list -v
> Subsystem Subsystem-NQN Controllers
> ---------------- -------------------------------------------------------------------------------------------- ----------------
> nvme-subsys1 nqn.2025-08.com.example:HOSA.hosa nvme1, nvme2, nvme3, nvme4
>
> Device SN MN FR TxPort Address Slot Subsystem Namespaces
> -------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------
> nvme1 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.195,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1
> nvme2 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.196,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1
> nvme3 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.197,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1
> nvme4 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.198,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1
>
> Device Generic NSID Usage Format Controllers
> ------------ ------------ ---------- -------------------------- ---------------- ----------------
>
> # nvme list-ns /dev/nvme1 -v -a
> opcode : 06
> flags : 00
> rsvd1 : 0000
> nsid : 00000000
> cdw2 : 00000000
> cdw3 : 00000000
> data_len : 00001000
> metadata_len : 00000000
> addr : 616b6af40000
> metadata : 0
> cdw10 : 00000010
> cdw11 : 00000000
> cdw12 : 00000000
> cdw13 : 00000000
> cdw14 : 00000000
> cdw15 : 00000000
> timeout_ms : 00000000
> result : 00000000
> err : 2
> latency : 569 us
> NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x2)
>
> How do I make it work again?
Hi Robert,
Aviv is going to take a look at this.
Thanks,
Ilya
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
