Hi, Am 9/9/25 um 17:34 schrieb Yuri Weinstein:
* Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces),
I have just upgraded a test cluster and NVMEoF does not work any more. The previous version running was quay.ceph.io/ceph-ci/ceph:squid-nvmeof and NVMEoF was working there. The gateways seem to work and subsystem and namespaces are still configured but an initiator does not see namespaces. # ceph orch ps --daemon_type nvmeof NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID nvmeof.hosa.b236asbrce101.ccaujq b236asbrce101 *:5500,4420,8009,10008 running (13h) 36s ago 13h 128M - 1.5.7 2c3e7ef350b8 74672e87a945 nvmeof.hosa.b236asbrce102.gnbwsc b236asbrce102 *:5500,4420,8009,10008 running (13h) 36s ago 13h 120M - 1.5.7 2c3e7ef350b8 1929979ab7e1 nvmeof.hosa.b236asbrce103.bthewi b236asbrce103 *:5500,4420,8009,10008 running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8 14885171d1a5 nvmeof.hosa.b236asbrce104.vcgzvg b236asbrce104 *:5500,4420,8009,10008 running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8 7b00b0f3356a # ceph nvmeof subsystem list +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |Nqn |Serial Number |Model Number |Namespace Count|Subtype|Max Namespaces|Has Dhchap Key|Allow Any Host|Created Without Key| +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |nqn.2025-08.com.example:HOSA.hosa|Ceph30844860597194|Ceph bdev Controller|2 |NVMe |128 |False |True |False | +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ # ceph nvmeof namespace list nqn.2025-08.com.example:HOSA.hosa +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+ |bdev_57f2c81a-0b07-406b-84aa-c62d66b5f10d|ns01 |nvmeof |5 |100GB |512B |0 |0 |0 |0 |True |[] |1 |57f2c81a-0b07-406b-84aa-c62d66b5f10d|nqn.2025-08.com.example:HOSA.hosa|False |False |False | |bdev_b4bf328a-8c67-4879-bf13-a799884d6dbe|ns02 |nvmeof |5 |200GB |512B |0 |0 |0 |0 |True |[] |2 |b4bf328a-8c67-4879-bf13-a799884d6dbe|nqn.2025-08.com.example:HOSA.hosa|False |False |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+ # ceph nvmeof listener list nqn.2025-08.com.example:HOSA.hosa +-------------+------+--------------+------+------+-------+ |Host Name |Trtype|Traddr |Secure|Adrfam|Trsvcid| +-------------+------+--------------+------+------+-------+ |b236asbrce101|TCP |10.105.141.195|False |ipv4 |4420 | |b236asbrce102|TCP |10.105.141.196|False |ipv4 |4420 | |b236asbrce103|TCP |10.105.141.197|False |ipv4 |4420 | |b236asbrce104|TCP |10.105.141.198|False |ipv4 |4420 | +-------------+------+--------------+------+------+-------+ On the initiator host: # nvme discover -t tcp -a 10.105.141.195 -s 8009 -v warning: use hostid which does not match uuid in hostnqn nvme0: nqn.2014-08.org.nvmexpress.discovery connected Discovery Log Number of Records 4, Generation counter 6 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 0 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.195 eflags: none sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 1 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.196 eflags: none sectype: none =====Discovery Log Entry 2====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 2 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.197 eflags: none sectype: none =====Discovery Log Entry 3====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 3 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.198 eflags: none sectype: none nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected # nvme connect-all -t tcp -a 10.105.141.195 -s 8009 -v warning: use hostid which does not match uuid in hostnqn nvme0: nqn.2014-08.org.nvmexpress.discovery connected nvme1: nqn.2025-08.com.example:HOSA.hosa connected nvme2: nqn.2025-08.com.example:HOSA.hosa connected nvme3: nqn.2025-08.com.example:HOSA.hosa connected nvme4: nqn.2025-08.com.example:HOSA.hosa connected nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected Looking good so far, but: # nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- # nvme list -v Subsystem Subsystem-NQN Controllers ---------------- -------------------------------------------------------------------------------------------- ---------------- nvme-subsys1 nqn.2025-08.com.example:HOSA.hosa nvme1, nvme2, nvme3, nvme4 Device SN MN FR TxPort Address Slot Subsystem Namespaces -------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ---------------- nvme1 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.195,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme2 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.196,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme3 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.197,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme4 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.198,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 Device Generic NSID Usage Format Controllers ------------ ------------ ---------- -------------------------- ---------------- ---------------- # nvme list-ns /dev/nvme1 -v -a opcode : 06 flags : 00 rsvd1 : 0000 nsid : 00000000 cdw2 : 00000000 cdw3 : 00000000 data_len : 00001000 metadata_len : 00000000 addr : 616b6af40000 metadata : 0 cdw10 : 00000010 cdw11 : 00000000 cdw12 : 00000000 cdw13 : 00000000 cdw14 : 00000000 cdw15 : 00000000 timeout_ms : 00000000 result : 00000000 err : 2 latency : 569 us NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x2) How do I make it work again? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
