On 5/8/25 6:36 PM, Mira Limbeck wrote: > On 4/30/25 15:39, Damien Le Moal wrote: >> On 2025/04/30 7:13, Friedrich Weber wrote: >>> Hi, >>> >>> One of our users reports that, in their setup, hotplugging new disks doesn't >>> work anymore with recent kernels (details below). The issue appeared somewhere >>> between kernels 6.4 and 6.5, and they bisected the change to this patch: >>> >>> 624885209f31 (scsi: core: Detect support for command duration limits) >>> >>> The issue is also reproducible on a mainline kernel 6.14.4 build from [1]. When >>> hotplugging a disk under 6.14.4, the following is logged (I've redacted some >>> identifiers, let me know in case I've been too overzealous with that): >>> >>> Apr 28 16:41:13 pbs-disklab kernel: mpt3sas_cm0: handle(0xa) sas_address(0xREDACTED_SAS_ADDR) port_type(0x1) >>> Apr 28 16:41:13 pbs-disklab kernel: scsi 5:0:1:0: Direct-Access WDC REDACTED_SN C5C0 PQ: 0 ANSI: 7 >>> Apr 28 16:41:13 pbs-disklab kernel: scsi 5:0:1:0: SSP: handle(0x000a), sas_addr(0xREDACTED_SAS_ADDR), phy(2), device_name(REDACTED_DEVICE_NAME) >>> Apr 28 16:41:13 pbs-disklab kernel: scsi 5:0:1:0: enclosure logical id (REDACTED_LOGICAL_ID), slot(0) >>> Apr 28 16:41:13 pbs-disklab kernel: scsi 5:0:1:0: enclosure level(0x0000), connector name( ) >>> Apr 28 16:41:13 pbs-disklab kernel: scsi 5:0:1:0: qdepth(254), tagged(1), scsi_level(8), cmd_que(1) >>> Apr 28 16:41:13 pbs-disklab kernel: scsi 5:0:1:0: Power-on or device reset occurred >>> Apr 28 16:41:16 pbs-disklab kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) >> >> This decodes to: >> >> Code: 00110000h PL_LOGINFO_CODE_RESET See Sub-Codes below (PL_LOGINFO_SUB_CODE) >> Sub Code: 00000E00h PL_LOGINFO_SUB_CODE_DISCOVERY_SATA_ERR >> >>> Apr 28 16:41:18 pbs-disklab kernel: mpt3sas_cm0: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000) >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: Attached scsi generic sg1 type 0 >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] Test Unit Ready failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] Sense not available. >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] Sense not available. >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] 0-byte physical blocks >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] Test WP failed, assume Write Enabled >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] Asking for cache data failed >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] Assuming drive cache: write through >>> Apr 28 16:41:18 pbs-disklab kernel: end_device-5:1: add: handle(0x000a), sas_addr(0xREDACTED_SAS_ADDR) >>> Apr 28 16:41:18 pbs-disklab kernel: mpt3sas_cm0: handle(0x000a), ioc_status(0x0022) failure at drivers/scsi/mpt3sas/mpt3sas_transport.c:225/_transport_set_identify()! >>> Apr 28 16:41:18 pbs-disklab kernel: sd 5:0:1:0: [sdb] Attached SCSI disk >>> Apr 28 16:41:18 pbs-disklab kernel: mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0xREDACTED_SAS_ADDR) >>> Apr 28 16:41:18 pbs-disklab kernel: mpt3sas_cm0: removing handle(0x000a), sas_addr(0xREDACTED_SAS_ADDR) >>> Apr 28 16:41:18 pbs-disklab kernel: mpt3sas_cm0: enclosure logical id(REDACTED_LOGICAL_ID), slot(0) >>> Apr 28 16:41:18 pbs-disklab kernel: mpt3sas_cm0: enclosure level(0x0000), connector name( ) >>> >>> and the block device isn't accessible afterwards. It does seem to be visible >>> after a reboot. >>> >>> lspci on this host shows: >>> >>> 02:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02) >>> Subsystem: Broadcom / LSI SAS9300-8i [1000:30e0] >>> Kernel driver in use: mpt3sas >>> Kernel modules: mpt3sas >>> >>> The HBA is placed on a PCIe 3.0 x8 slot (not bifurcated) and connected via >>> SFF-8643 to a simple 2U 12xLFF SAS3 Supermicro box. The user can also reproduce >>> the issue with other HBAs with e.g. the SAS3108 and SAS3816 chipsets. >>> >>> The device doesn't seem to support CDL. So if I see correctly, the only >>> effective change introduced by the patch are the four scsi_cdl_check_cmd (and >>> thus scsi_report_opcode) calls to check for CDL support. Hence we wondered >>> whether may be the cause of the issue. We ran a few tests to verify: >>> >>> - disabling "REPORT SUPPORTED OPERATION CODES" by passing >>> `scsi_mod.dev_flags=WDC:REDACTED_SN:536870912` (the flag being >>> BLIST_NO_RSOC) resolves the issue (hotplug works again), but I imagine >>> disabling RSOC altogether isn't a good workaround. This test was not done >>> on a mainline kernel, but I don't think it would make a difference. >> >> So it seems that the HBA SAT is choking on the report supported opcode command. >> I have several mpt3sas HBAs and I have never seen this issue running the latest >> FW version for these (EOL) HBAs. So I am tempted to say that an HBA FW update >> should resolve the issue, BUT, I do not recall doing any drive hotplug tests >> though. This issue may trigger only with hotplug and not with a cold start... >> Can you confirm that ? >> > Yes, a cold boot works. With hotplug it enters a broken state and any > subsequent reboots don't fix the issue. > Removing power is needed to fix the issue again. > > They mentioned the following tests: > > - Get the 20TB disk in a faulty state by booting kernels 6.5 and above > (6.14.X in this case, diskcaddy light on server keeps blinking, dmesg > shows power-reset) > - Reboot server, reboot into same kernel (6.14.X) > - Disk remains in faulty state, does not attached to system or show up > under any path (lsblk, df, blkid) > > and > > - Get 20TB disk into faulty state by hotswapping on kern 6.5 and above. > - Shut off machine, remove from power & reattach. > - Start machine > - 20TB disk mounts during boot, accessible in OS as block-device after. > > >>> >>> - we patched out the four calls to scsi_cdl_check_cmd and unconditionally set >>> cdl_supported to 0, see [2] for the patch (on top of 6.14.4). This resolves >>> the issue. >>> >>> - I suspected that particularly the two latter scsi_cdl_check_cmd calls with a >>> nonzero service action might be problematic, so we patched them out >>> specifically but kept the other two calls without a service action, see [3] >>> for the patch (on top of 6.14.4). But with this patch, hotplug still does >>> not work. >>> >>> - the RSOC commands themselves don't seem to be problematic per se. We asked >>> the user to boot a (non-mainline) kernel with the `scsi_mod.dev_flags` >>> parameter to disable RSOC as above, hotplug the disk (this succeeds), and >>> then query the four opcodes/service actions using `sg_opcodes`, and this >>> looks okay [4] (reporting that CDL is not supported). >>> >>> I wonder whether these results might suggest the RSOC queries are problematic >>> not in general, but at this particular point (during device initialization) in >>> this particular hardware setup? If this turns out to be the case -- would it be >>> feasible to suppress these RSOC queries if CDL is not enabled via sysfs? >> >> I would be tempted to say that indeed it is the RSOC command handling in the HBA >> SAT that has issues. But your command line checks [4] tend to indicate >> otherwise. The issue may trigger only with timing differences with hotplug though. >> >> The other possible problem may be that the RSOC command translation is actually >> fine but ends up generating an ATA command that the drive is not happy about, >> either because of a drive FW bug or because of the timing the drive receives >> that command. Given that this is a WD drive, I can probably check that if you >> can send to me the drive model and FW rev (sending that information off-list is >> fine). >> >>> If you have any ideas for further troubleshooting, we're happy to gather more >>> data. I'll be AFK for a few weeks, but Mira (in CC) will take over in the >>> meantime. >> >> Checking the HBA FW version would be a start, and also if you can confirm if >> this issue happens only on hotplug or also during cold boot would be nice. I am >> traveling right now and will not be able to test hot-plugging drives on my >> setups until end of next week. >> > > They provided controller information via `sas3ircu` and `storcli`: > > sas3ircu: > > Controller type : SAS3008 > BIOS version : 8.37.00.00 > Firmware version : 16.00.16.00 Is this the latest available FW for this HBA ? (see below) > > storcli: > > Firmware Package Build = 24.18.0-0021 > Firmware Version = 4.670.00-6500 > CPLD Version = 26515-00A > Bios Version = 6.34.01.0_4.19.08.00_0x06160200 > HII Version = 03.23.06.00 > Ctrl-R Version = 5.18-0400 > Preboot CLI Version = 01.07-05:#%0000 > NVDATA Version = 3.1611.00-0005 > Boot Block Version = 3.07.00.00-0003 > Driver Name = megaraid_sas > Driver Version = 07.727.03.00-rc1 Unfortunately, I do not have any megaraid model so I cannot test/recreate. I only have mpt3sas (9300, 9400 and 9500 series HBAs) and mpi3mr models (9600 HBA series). > > And the disk information from `smartctl --xall` > > 20T: > > === START OF INFORMATION SECTION === > Vendor: WDC > Product: WUH722020BL5204 > Revision: C5C0 > Compliance: SPC-5 > User Capacity: 20,000,588,955,648 bytes [20.0 TB] > Logical block size: 512 bytes > Physical block size: 4096 bytes > LU is fully provisioned > Rotation Rate: 7200 rpm > Form Factor: 3.5 inches > Logical Unit id: <id> > Serial number: <S/N> > Device type: disk > Transport protocol: SAS (SPL-4) > Local Time is: Thu May 1 15:23:35 2025 CEST > SMART support is: Available - device has SMART capability. > SMART support is: Enabled > Temperature Warning: Enabled > Read Cache is: Enabled > Writeback Cache is: Enabled > > 18T: > > === START OF INFORMATION SECTION === > Vendor: WDC > Product: WUH721818AL5204 > Revision: C8C2 > Compliance: SPC-5 > User Capacity: 18,000,207,937,536 bytes [18.0 TB] > Logical block size: 512 bytes > Physical block size: 4096 bytes > LU is fully provisioned > Rotation Rate: 7200 rpm > Form Factor: 3.5 inches > Logical Unit id: <id> > Serial number: <S/N> > Device type: disk > Transport protocol: SAS (SPL-4) > Local Time is: Thu May 1 15:25:27 2025 CEST > SMART support is: Available - device has SMART capability. > SMART support is: Enabled > Temperature Warning: Enabled > Read Cache is: Enabled > Writeback Cache is: Enabled > > The 18T disk is not affected by this issue. Hotplug works as expected > with it. I do not think that the drives are relevant for this issue. How the HBA react to a command error from the drive resulting from the HBA command translation likely is the issue. > If you need any additional information, please let us know! Adding the Broadcom folks to this thread, since as suspected, this seems to be an HBA issue. I strongly suspect that it relates to a recent very similar issue I have seen with the mpi3mr driver and a 9600 Broadcom HBA: any hotplug of a drive would completely crash the HBA and a full power cycle was needed to recover. A simple reboot would not be sufficient. I think the latest HBA FW version fixes that problem. Broadcom team, Any comment ? -- Damien Le Moal Western Digital Research
