On 8/30/25 1:38 PM, Mike Snitzer wrote: > From: Mike Snitzer <snitzer@xxxxxxxxxxxxxxx> > > Chuck Lever advised that allocating a single start_extra_page, to > avoid RDMA corruption on client, definitely shouldn't be needed: > > "There's nothing I can think of in the RDMA or RPC/RDMA protocols that > mandates that the first page offset must always be zero. Moving data > at one address on the server to an entirely different address and > alignment on the client is exactly what RDMA is supposed to do. > > It sounds like an implementation omission because the server's upper > layers have never needed it before now. If TCP already handles it, I'm > guessing it's going to be straightforward to fix." > > So avoid papering over what seems to be an rpcrdma bug, remove the > allocation and use of an extra start_extra_page. > > With this patch applied ontop of v8 patchset [0], I get the following > data mismatch errors at the end [3] when using the NFS RDMA client > with reproducer documented in associated patch header since v2 [1]: > > "Must allocate and use a bounce-buffer page (called 'start_extra_page') > if/when expanding the misaligned READ requires reading extra partial > page at the start of the READ so that its DIO-aligned. Otherwise that > extra page at the start will make its way back to the NFS client and > corruption will occur. As found, and then this fix of using an extra > page verified, using the 'dt' utility: > dt of=/mnt/share1/dt_a.test passes=1 bs=47008 count=2 \ > iotype=sequential pattern=iot onerr=abort oncerr=abort > see: https://github.com/RobinTMiller/dt.git " > > I really did try to call attention to this misaligned DIO READ > alloc_page hack to make RDMA work, see [2], but I didn't frame it as > RDMA specific and definitely should've been clearer on that important > detail: > > "Also, I think its worth calling out this > nfsd_complete_misaligned_read_dio function for its remapping/shifting > of the READ payload reflected in rqstp->rq_bvec[]." > > Signed-off-by: Mike Snitzer <snitzer@xxxxxxxxxx> > > [0]: https://lore.kernel.org/linux-nfs/20250826185718.5593-1-snitzer@xxxxxxxxxx/ > [1]: https://lore.kernel.org/linux-nfs/20250708160619.64800-9-snitzer@xxxxxxxxxx/ > [2]: https://lore.kernel.org/linux-nfs/aG2MDVyyCbjTpgOv@xxxxxxxxxx/ > [3]: partial output of dt utility that shows NFS client READ data mismatch: > ++ COUNT=3 > ++ IOSIZE=47008 > ++ dt of=/mnt/hs_test/dt_thisisa.test passes=1 bs=47008 count=3 iotype=sequential pattern=iot onerr=abort oncerr=abort > dt (j:1 t:1): > dt (j:1 t:1): Write Statistics: > dt (j:1 t:1): Job Information Reported: Job 1, Thread 1 > dt (j:1 t:1): Last IOT seed value used: 0x01010101 > dt (j:1 t:1): Total records processed: 3 @ 47008 bytes/record (45.906 Kbytes) > dt (j:1 t:1): Total bytes transferred: 141024 (137.719 Kbytes, 0.134 Mbytes) > dt (j:1 t:1): Average transfer rates: 1004137 bytes/sec, 980.602 Kbytes/sec, 0.958 Mbytes/sec > dt (j:1 t:1): Number I/O's per second: 21.361 > dt (j:1 t:1): Number seconds per I/O: 0.0468 (46.81ms) > dt (j:1 t:1): Total passes completed: 0/1 > dt (j:1 t:1): Total errors detected: 0/1 > dt (j:1 t:1): Total elapsed time: 00m00.14s > dt (j:1 t:1): Total system time: 00m00.00s > dt (j:1 t:1): Total user time: 00m00.00s > dt (j:1 t:1): Starting time: Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): Ending time: Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): Warning: The bytes written 141024, is less than the data limit 1880320000 requested! > dt (j:1 t:1): ERROR: Error number 1 occurred on Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): > dt (j:1 t:1): Error Number: 1 > dt (j:1 t:1): Time of Current Error: Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): Read Pass Start Time: Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): Write Pass Start Time: Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): Pass Number: 1 > dt (j:1 t:1): Pass Elapsed Time: 00m00.10s > dt (j:1 t:1): Test Elapsed Time: 00m00.24s > dt (j:1 t:1): File Name: /mnt/hs_test/dt_thisisa.test > dt (j:1 t:1): File Inode: 1199 (0x4af) > dt (j:1 t:1): Directory Inode: 1 (0x1) > dt (j:1 t:1): File Size: 1880320000 (0x70136800) > dt (j:1 t:1): Operation: miscompare > dt (j:1 t:1): Record Number: 2 > dt (j:1 t:1): Request Size: 47008 (0xb7a0) > dt (j:1 t:1): Block Length: 91 (0x5b) > dt (j:1 t:1): I/O Mode: read > dt (j:1 t:1): I/O Type: sequential > dt (j:1 t:1): File Type: output > dt (j:1 t:1): Direct I/O: disabled (caching data) > dt (j:1 t:1): Device Size: 512 (0x200) > dt (j:1 t:1): Starting File Offset: 47008 (0xb7a0) > dt (j:1 t:1): Starting LBA: 91 (0x5b) > dt (j:1 t:1): Ending File Offset: 94016 (0x16f40) > dt (j:1 t:1): Ending LBA: 182 (0xb6) > dt (j:1 t:1): Error File Offset: 47008 (0xb7a0) > dt (j:1 t:1): Error Offset Modulos: %8 = 0, %512 = 416, %4096 = 1952 > dt (j:1 t:1): Starting Relative Error LBA: 91 (0x5b) > dt (j:1 t:1): Relative 4096 byte Error LBA: 11 (0xb) > dt (j:1 t:1): Corruption Buffer Index: 0 (byte index into read buffer) > dt (j:1 t:1): Corruption Block Index: 0 (byte index in miscompare block) > dt (j:1 t:1): > dt (j:1 t:1): > dt (j:1 t:1): Data compare error at byte 0 in record number 2 > dt (j:1 t:1): Relative block number where the error occurred is 91, offset 47008 > dt (j:1 t:1): Block expected = 91 (0x0000005b), block found = 1919311731 (0x72665f73), count = 47008 > dt (j:1 t:1): The correct data starts at memory address 0x000000003c589000 (marked by asterisk '*') > dt (j:1 t:1): Dumping Pattern Buffer (base = 0x3c589000, mismatch offset = 0, limit = 512 bytes): > dt (j:1 t:1): / Buffer > dt (j:1 t:1): Memory Address / Index > dt (j:1 t:1): 0x000000003c589000/ 0 |*5b 00 00 00 5c 01 01 01 5d 02 02 02 5e 03 03 03 "[ \ ] ^ " > dt (j:1 t:1): 0x000000003c589010/ 16 | 5f 04 04 04 60 05 05 05 61 06 06 06 62 07 07 07 "_ ` a b " > dt (j:1 t:1): 0x000000003c589020/ 32 | 63 08 08 08 64 09 09 09 65 0a 0a 0a 66 0b 0b 0b "c d e f " > dt (j:1 t:1): 0x000000003c589030/ 48 | 67 0c 0c 0c 68 0d 0d 0d 69 0e 0e 0e 6a 0f 0f 0f "g h i j " > dt (j:1 t:1): 0x000000003c589040/ 64 | 6b 10 10 10 6c 11 11 11 6d 12 12 12 6e 13 13 13 "k l m n " > dt (j:1 t:1): 0x000000003c589050/ 80 | 6f 14 14 14 70 15 15 15 71 16 16 16 72 17 17 17 "o p q r " > dt (j:1 t:1): 0x000000003c589060/ 96 | 73 18 18 18 74 19 19 19 75 1a 1a 1a 76 1b 1b 1b "s t u v " > dt (j:1 t:1): 0x000000003c589070/ 112 | 77 1c 1c 1c 78 1d 1d 1d 79 1e 1e 1e 7a 1f 1f 1f "w x y z " > dt (j:1 t:1): 0x000000003c589080/ 128 | 7b 20 20 20 7c 21 21 21 7d 22 22 22 7e 23 23 23 "{ |!!!}"""~###" > dt (j:1 t:1): 0x000000003c589090/ 144 | 7f 24 24 24 80 25 25 25 81 26 26 26 82 27 27 27 " $$$ %%% &&& '''" > dt (j:1 t:1): 0x000000003c5890a0/ 160 | 83 28 28 28 84 29 29 29 85 2a 2a 2a 86 2b 2b 2b " ((( ))) *** +++" > dt (j:1 t:1): 0x000000003c5890b0/ 176 | 87 2c 2c 2c 88 2d 2d 2d 89 2e 2e 2e 8a 2f 2f 2f " ,,, --- ... ///" > dt (j:1 t:1): 0x000000003c5890c0/ 192 | 8b 30 30 30 8c 31 31 31 8d 32 32 32 8e 33 33 33 " 000 111 222 333" > dt (j:1 t:1): 0x000000003c5890d0/ 208 | 8f 34 34 34 90 35 35 35 91 36 36 36 92 37 37 37 " 444 555 666 777" > dt (j:1 t:1): 0x000000003c5890e0/ 224 | 93 38 38 38 94 39 39 39 95 3a 3a 3a 96 3b 3b 3b " 888 999 ::: ;;;" > dt (j:1 t:1): 0x000000003c5890f0/ 240 | 97 3c 3c 3c 98 3d 3d 3d 99 3e 3e 3e 9a 3f 3f 3f " <<< === >>> ???" > dt (j:1 t:1): 0x000000003c589100/ 256 | 9b 40 40 40 9c 41 41 41 9d 42 42 42 9e 43 43 43 " @@@ AAA BBB CCC" > dt (j:1 t:1): 0x000000003c589110/ 272 | 9f 44 44 44 a0 45 45 45 a1 46 46 46 a2 47 47 47 " DDD EEE FFF GGG" > dt (j:1 t:1): 0x000000003c589120/ 288 | a3 48 48 48 a4 49 49 49 a5 4a 4a 4a a6 4b 4b 4b " HHH III JJJ KKK" > dt (j:1 t:1): 0x000000003c589130/ 304 | a7 4c 4c 4c a8 4d 4d 4d a9 4e 4e 4e aa 4f 4f 4f " LLL MMM NNN OOO" > dt (j:1 t:1): 0x000000003c589140/ 320 | ab 50 50 50 ac 51 51 51 ad 52 52 52 ae 53 53 53 " PPP QQQ RRR SSS" > dt (j:1 t:1): 0x000000003c589150/ 336 | af 54 54 54 b0 55 55 55 b1 56 56 56 b2 57 57 57 " TTT UUU VVV WWW" > dt (j:1 t:1): 0x000000003c589160/ 352 | b3 58 58 58 b4 59 59 59 b5 5a 5a 5a b6 5b 5b 5b " XXX YYY ZZZ [[[" > dt (j:1 t:1): 0x000000003c589170/ 368 | b7 5c 5c 5c b8 5d 5d 5d b9 5e 5e 5e ba 5f 5f 5f " \\\ ]]] ^^^ ___" > dt (j:1 t:1): 0x000000003c589180/ 384 | bb 60 60 60 bc 61 61 61 bd 62 62 62 be 63 63 63 " ``` aaa bbb ccc" > dt (j:1 t:1): 0x000000003c589190/ 400 | bf 64 64 64 c0 65 65 65 c1 66 66 66 c2 67 67 67 " ddd eee fff ggg" > dt (j:1 t:1): 0x000000003c5891a0/ 416 | c3 68 68 68 c4 69 69 69 c5 6a 6a 6a c6 6b 6b 6b " hhh iii jjj kkk" > dt (j:1 t:1): 0x000000003c5891b0/ 432 | c7 6c 6c 6c c8 6d 6d 6d c9 6e 6e 6e ca 6f 6f 6f " lll mmm nnn ooo" > dt (j:1 t:1): 0x000000003c5891c0/ 448 | cb 70 70 70 cc 71 71 71 cd 72 72 72 ce 73 73 73 " ppp qqq rrr sss" > dt (j:1 t:1): 0x000000003c5891d0/ 464 | cf 74 74 74 d0 75 75 75 d1 76 76 76 d2 77 77 77 " ttt uuu vvv www" > dt (j:1 t:1): 0x000000003c5891e0/ 480 | d3 78 78 78 d4 79 79 79 d5 7a 7a 7a d6 7b 7b 7b " xxx yyy zzz {{{" > dt (j:1 t:1): 0x000000003c5891f0/ 496 | d7 7c 7c 7c d8 7d 7d 7d d9 7e 7e 7e da 7f 7f 7f " ||| }}} ~~~ " > dt (j:1 t:1): > dt (j:1 t:1): The incorrect data starts at memory address 0x000000003c596000 (for Robin's debug! :) > dt (j:1 t:1): The incorrect data starts at file offset 000000000000047008 (marked by asterisk '*') > dt (j:1 t:1): Dumping Data File offsets (base = 47008, mismatch offset = 0, limit = 512 bytes): > dt (j:1 t:1): / Block > dt (j:1 t:1): File Offset / Index > dt (j:1 t:1): 000000000000047008/ 0 |*73 5f 66 72 65 65 5f 63 6f 6d 6d 69 74 5f 61 72 "s_free_commit_ar" > dt (j:1 t:1): 000000000000047024/ 16 | 72 61 79 00 54 43 50 5f 54 49 4d 45 5f 57 41 49 "ray TCP_TIME_WAI" > dt (j:1 t:1): 000000000000047040/ 32 | 54 00 42 50 46 5f 50 52 4f 47 5f 54 59 50 45 5f "T BPF_PROG_TYPE_" > dt (j:1 t:1): 000000000000047056/ 48 | 43 47 52 4f 55 50 5f 53 59 53 43 54 4c 00 4c 41 "CGROUP_SYSCTL LA" > dt (j:1 t:1): 000000000000047072/ 64 | 59 4f 55 54 5f 46 4c 45 58 5f 46 49 4c 45 53 00 "YOUT_FLEX_FILES " > dt (j:1 t:1): 000000000000047088/ 80 | 4e 46 53 45 52 52 5f 4a 55 4b 45 42 4f 58 00 72 "NFSERR_JUKEBOX r" > dt (j:1 t:1): 000000000000047104/ 96 | 78 5f 63 70 75 5f 72 6d 61 70 00 6d 69 67 72 61 "x_cpu_rmap migra" > dt (j:1 t:1): 000000000000047120/ 112 | 74 69 6f 6e 5f 64 69 73 61 62 6c 65 64 00 5f 5f "tion_disabled __" > dt (j:1 t:1): 000000000000047136/ 128 | 64 61 74 61 00 6e 64 6f 5f 64 65 6c 5f 73 6c 61 "data ndo_del_sla" > dt (j:1 t:1): 000000000000047152/ 144 | 76 65 00 6e 66 73 5f 63 6f 6d 6d 69 74 5f 64 61 "ve nfs_commit_da" > dt (j:1 t:1): 000000000000047168/ 160 | 74 61 00 65 78 74 5f 6d 75 74 65 78 00 63 6f 6e "ta ext_mutex con" > dt (j:1 t:1): 000000000000047184/ 176 | 6e 65 63 74 5f 63 6f 6f 6b 69 65 00 54 43 50 5f "nect_cookie TCP_" > dt (j:1 t:1): 000000000000047200/ 192 | 43 4c 4f 53 45 5f 57 41 49 54 00 6d 65 6d 63 6d "CLOSE_WAIT memcm" > dt (j:1 t:1): 000000000000047216/ 208 | 70 00 52 50 4d 5f 52 45 51 5f 53 55 53 50 45 4e "p RPM_REQ_SUSPEN" > dt (j:1 t:1): 000000000000047232/ 224 | 44 00 63 72 6d 61 74 63 68 00 63 61 6e 63 65 6c "D crmatch cancel" > dt (j:1 t:1): 000000000000047248/ 240 | 5f 66 6f 72 6b 00 70 67 70 72 6f 74 5f 74 00 74 "_fork pgprot_t t" > dt (j:1 t:1): 000000000000047264/ 256 | 72 61 63 65 70 6f 69 6e 74 5f 70 74 72 5f 74 00 "racepoint_ptr_t " > dt (j:1 t:1): 000000000000047280/ 272 | 66 6f 72 5f 72 65 63 6c 61 69 6d 00 4e 46 53 45 "for_reclaim NFSE" > dt (j:1 t:1): 000000000000047296/ 288 | 52 52 5f 42 41 44 43 48 41 52 00 5f 73 6b 62 5f "RR_BADCHAR _skb_" > dt (j:1 t:1): 000000000000047312/ 304 | 72 65 66 64 73 74 00 70 68 79 73 69 63 61 6c 5f "refdst physical_" > dt (j:1 t:1): 000000000000047328/ 320 | 6c 6f 63 61 74 69 6f 6e 00 6e 75 6d 5f 72 65 71 "location num_req" > dt (j:1 t:1): 000000000000047344/ 336 | 73 00 5f 5f 53 43 54 5f 5f 74 70 5f 66 75 6e 63 "s __SCT__tp_func" > dt (j:1 t:1): 000000000000047360/ 352 | 5f 70 6e 66 73 5f 6d 64 73 5f 66 61 6c 6c 62 61 "_pnfs_mds_fallba" > dt (j:1 t:1): 000000000000047376/ 368 | 63 6b 5f 77 72 69 74 65 5f 64 6f 6e 65 00 74 61 "ck_write_done ta" > dt (j:1 t:1): 000000000000047392/ 384 | 73 6b 5f 63 6c 65 61 6e 75 70 00 65 78 70 61 6e "sk_cleanup expan" > dt (j:1 t:1): 000000000000047408/ 400 | 64 5f 72 65 61 64 61 68 65 61 64 00 6c 6f 63 6b "d_readahead lock" > dt (j:1 t:1): 000000000000047424/ 416 | 5f 6d 61 6e 61 67 65 72 5f 6f 70 65 72 61 74 69 "_manager_operati" > dt (j:1 t:1): 000000000000047440/ 432 | 6f 6e 73 00 73 72 63 5f 72 65 67 00 63 72 64 65 "ons src_reg crde" > dt (j:1 t:1): 000000000000047456/ 448 | 73 74 72 6f 79 00 63 68 69 6c 64 72 65 6e 5f 6c "stroy children_l" > dt (j:1 t:1): 000000000000047472/ 464 | 6f 77 5f 75 73 61 67 65 00 6e 75 6d 5f 76 66 00 "ow_usage num_vf " > dt (j:1 t:1): 000000000000047488/ 480 | 73 63 72 61 74 63 68 00 50 49 44 54 59 50 45 5f "scratch PIDTYPE_" > dt (j:1 t:1): 000000000000047504/ 496 | 4d 41 58 00 70 72 65 70 61 72 65 5f 77 72 69 74 "MAX prepare_writ" > dt (j:1 t:1): > dt (j:1 t:1): > dt (j:1 t:1): Analyzing IOT Record Data: (Note: Block #'s are relative to start of record!) > dt (j:1 t:1): > dt (j:1 t:1): IOT block size: 512 > dt (j:1 t:1): Total number of blocks: 91 (47008 bytes) > dt (j:1 t:1): Current IOT seed value: 0x01010101 (pass 1) > dt (j:1 t:1): Range of corrupted blocks: 0 - 90 > dt (j:1 t:1): Length of corrupted blocks: 91 (46592 bytes) > dt (j:1 t:1): Corrupted blocks file offset: 47008 (LBA 91) > dt (j:1 t:1): Number of corrupted blocks: 91 > dt (j:1 t:1): Number of good blocks found: 0 > dt (j:1 t:1): Number of zero blocks found: 0 > dt (j:1 t:1): > dt (j:1 t:1): Record #: 2 > dt (j:1 t:1): Starting Record Offset: 47008 > dt (j:1 t:1): Transfer Count: 47008 (0xb7a0) > dt (j:1 t:1): Ending Record Offset: 94016 > dt (j:1 t:1): Relative Record Block Range: 91 - 182 > dt (j:1 t:1): Read Buffer Address: 0x3c596000 > dt (j:1 t:1): Pattern Base Address: 0x3c589000 > dt (j:1 t:1): Note: Incorrect data is marked with asterisk '*' > dt (j:1 t:1): > dt (j:1 t:1): Record Block: 0 (BAD data) > dt (j:1 t:1): Record Block Offset: 47008 (LBA 91) > dt (j:1 t:1): Record Buffer Index: 0 (0x0) > dt (j:1 t:1): Expected Block Number: 91 (0x0000005b) > dt (j:1 t:1): Received Block Number: 1919311731 (0x72665f73) > dt (j:1 t:1): Received Block Offset: 982687606272 > dt (j:1 t:1): > dt (j:1 t:1): Byte Expected: address 0x3c589000 Received: address 0x3c596000 > dt (j:1 t:1): 0000 0000005b 0101015c 0202025d 0303035e * 72665f73 635f6565 696d6d6f 72615f74 > dt (j:1 t:1): 0010 0404045f 05050560 06060661 07070762 * 00796172 5f504354 454d4954 4941575f > dt (j:1 t:1): 0020 08080863 09090964 0a0a0a65 0b0b0b66 * 50420054 52505f46 545f474f 5f455059 > dt (j:1 t:1): 0030 0c0c0c67 0d0d0d68 0e0e0e69 0f0f0f6a * 4f524743 535f5055 54435359 414c004c > dt (j:1 t:1): 0040 1010106b 1111116c 1212126d 1313136e * 54554f59 454c465f 49465f58 0053454c > dt (j:1 t:1): 0050 1414146f 15151570 16161671 17171772 * 4553464e 4a5f5252 42454b55 7200584f > dt (j:1 t:1): 0060 18181873 19191974 1a1a1a75 1b1b1b76 * 70635f78 6d725f75 6d007061 61726769 > dt (j:1 t:1): 0070 1c1c1c77 1d1d1d78 1e1e1e79 1f1f1f7a * 6e6f6974 7369645f 656c6261 5f5f0064 > dt (j:1 t:1): 0080 2020207b 2121217c 2222227d 2323237e * 61746164 6f646e00 6c65645f 616c735f > dt (j:1 t:1): 0090 2424247f 25252580 26262681 27272782 * 6e006576 635f7366 696d6d6f 61645f74 > dt (j:1 t:1): 00a0 28282883 29292984 2a2a2a85 2b2b2b86 * 65006174 6d5f7478 78657475 6e6f6300 > dt (j:1 t:1): 00b0 2c2c2c87 2d2d2d88 2e2e2e89 2f2f2f8a * 7463656e 6f6f635f 0065696b 5f504354 > dt (j:1 t:1): 00c0 3030308b 3131318c 3232328d 3333338e * 534f4c43 41575f45 6d005449 6d636d65 > dt (j:1 t:1): 00d0 3434348f 35353590 36363691 37373792 * 50520070 45525f4d 55535f51 4e455053 > dt (j:1 t:1): 00e0 38383893 39393994 3a3a3a95 3b3b3b96 * 72630044 6374616d 61630068 6c65636e > dt (j:1 t:1): 00f0 3c3c3c97 3d3d3d98 3e3e3e99 3f3f3f9a * 726f665f 6770006b 746f7270 7400745f > dt (j:1 t:1): 0100 4040409b 4141419c 4242429d 4343439e * 65636172 6e696f70 74705f74 00745f72 > dt (j:1 t:1): 0110 4444449f 454545a0 464646a1 474747a2 * 5f726f66 6c636572 006d6961 4553464e > dt (j:1 t:1): 0120 484848a3 494949a4 4a4a4aa5 4b4b4ba6 * 425f5252 48434441 5f005241 5f626b73 > dt (j:1 t:1): 0130 4c4c4ca7 4d4d4da8 4e4e4ea9 4f4f4faa * 64666572 70007473 69737968 5f6c6163 > dt (j:1 t:1): 0140 505050ab 515151ac 525252ad 535353ae * 61636f6c 6e6f6974 6d756e00 7165725f > dt (j:1 t:1): 0150 545454af 555555b0 565656b1 575757b2 * 5f5f0073 5f544353 5f70745f 636e7566 > dt (j:1 t:1): 0160 585858b3 595959b4 5a5a5ab5 5b5b5bb6 * 666e705f 646d5f73 61665f73 61626c6c > dt (j:1 t:1): 0170 5c5c5cb7 5d5d5db8 5e5e5eb9 5f5f5fba * 775f6b63 65746972 6e6f645f 61740065 > dt (j:1 t:1): 0180 606060bb 616161bc 626262bd 636363be * 635f6b73 6e61656c 65007075 6e617078 > dt (j:1 t:1): 0190 646464bf 656565c0 666666c1 676767c2 * 65725f64 68616461 00646165 6b636f6c > dt (j:1 t:1): 01a0 686868c3 696969c4 6a6a6ac5 6b6b6bc6 * 6e616d5f 72656761 65706f5f 69746172 > dt (j:1 t:1): 01b0 6c6c6cc7 6d6d6dc8 6e6e6ec9 6f6f6fca * 00736e6f 5f637273 00676572 65647263 > dt (j:1 t:1): 01c0 707070cb 717171cc 727272cd 737373ce * 6f727473 68630079 72646c69 6c5f6e65 > dt (j:1 t:1): 01d0 747474cf 757575d0 767676d1 777777d2 * 755f776f 65676173 6d756e00 0066765f > dt (j:1 t:1): 01e0 787878d3 797979d4 7a7a7ad5 7b7b7bd6 * 61726373 00686374 54444950 5f455059 > dt (j:1 t:1): 01f0 7c7c7cd7 7d7d7dd8 7e7e7ed9 7f7f7fda * 0058414d 70657270 5f657261 74697277 > ... > dt (j:1 t:1): Reread data does NOT match previous data or expected data! > dt (j:1 t:1): Writing reread data to file dt_thisisa.test-REREAD3-j1t1, from buffer 0x7f12bc004000, 47008 bytes... > dt (j:1 t:1): Command line to re-read the corrupted data: > dt (j:1 t:1): # dt if=/mnt/hs_test/dt_thisisa.test bs=47008 count=1 offset=47008 pattern=iot disable=retryDC,savecorrupted,trigdefaults > dt (j:1 t:1): > dt (j:1 t:1): Command line to re-read the data: > dt (j:1 t:1): # dt if=/mnt/hs_test/dt_thisisa.test bs=47008 dsize=512 iotype=sequential iodir=forward limit=94016 records=1 pattern=iot disable=retryDC,savecorrupted,trigdefaults > dt (j:1 t:1): > dt (j:1 t:1): > dt (j:1 t:1): Read Statistics: > dt (j:1 t:1): Job Information Reported: Job 1, Thread 1 > dt (j:1 t:1): Last IOT seed value used: 0x01010101 > dt (j:1 t:1): Total records processed: 2 @ 47008 bytes/record (45.906 Kbytes) > dt (j:1 t:1): Total bytes transferred: 94016 (91.812 Kbytes, 0.090 Mbytes) > dt (j:1 t:1): Average transfer rates: 656857 bytes/sec, 641.462 Kbytes/sec, 0.626 Mbytes/sec > dt (j:1 t:1): Number I/O's per second: 13.973 > dt (j:1 t:1): Number seconds per I/O: 0.0716 (71.56ms) > dt (j:1 t:1): Total passes completed: 1/1 > dt (j:1 t:1): Total errors detected: 1/1 > dt (j:1 t:1): Total elapsed time: 00m00.15s > dt (j:1 t:1): Total system time: 00m00.00s > dt (j:1 t:1): Total user time: 00m00.00s > dt (j:1 t:1): Starting time: Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): Ending time: Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): > dt (j:1 t:1): Operating System Information: > dt (j:1 t:1): Host name: plsm121c-06.perf.hammer.space (192.168.1.106) > dt (j:1 t:1): User name: root > dt (j:1 t:1): Process ID: 31703 > dt (j:1 t:1): OS information: Linux 6.12.24.17.hs.snitm+ #34 SMP PREEMPT_DYNAMIC Fri Aug 15 22:03:10 UTC 2025 x86_64 > dt (j:1 t:1): > dt (j:1 t:1): File System Information: > dt (j:1 t:1): Mounted from device: 192.168.0.105:/hs_test > dt (j:1 t:1): Mounted on directory: /mnt/hs_test > dt (j:1 t:1): Filesystem type: nfs4 > dt (j:1 t:1): Filesystem options: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,nconnect=16,port=20491,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.106,local_lock=none,addr=192.168.0.105 > dt (j:1 t:1): Filesystem block size: 1048576 > dt (j:1 t:1): Filesystem free space: 60990430380032 (58165007.000 Mbytes, 56801.765 Gbytes, 55.470 Tbytes) > dt (j:1 t:1): Filesystem total space: 60992310476800 (58166800.000 Mbytes, 56803.516 Gbytes, 55.472 Tbytes) > dt (j:1 t:1): > dt (j:1 t:1): Total Statistics: > dt (j:1 t:1): Output device/file name: /mnt/hs_test/dt_thisisa.test (device type=regular) > dt (j:1 t:1): Type of I/O's performed: sequential (forward) > dt (j:1 t:1): Job Information Reported: Job 1, Thread 1 > dt (j:1 t:1): Data pattern string used: 'IOT Pattern' (blocking is 512 bytes) > dt (j:1 t:1): Last IOT seed value used: 0x01010101 > dt (j:1 t:1): Total records read: 2 > dt (j:1 t:1): Total bytes read: 94016 (91.812 Kbytes, 0.090 Mbytes, 0.000 Gbytes) > dt (j:1 t:1): Total records written: 3 > dt (j:1 t:1): Total bytes written: 141024 (137.719 Kbytes, 0.134 Mbytes, 0.000 Gbytes) > dt (j:1 t:1): Total records processed: 5 @ 47008 bytes/record (45.906 Kbytes) > dt (j:1 t:1): Total bytes transferred: 235040 (229.531 Kbytes, 0.224 Mbytes) > dt (j:1 t:1): Average transfer rates: 828023 bytes/sec, 808.616 Kbytes/sec, 0.790 Mbytes/sec > dt (j:1 t:1): Number I/O's per second: 17.615 > dt (j:1 t:1): Number seconds per I/O: 0.0568 (56.77ms) > dt (j:1 t:1): Total passes completed: 1/1 > dt (j:1 t:1): Total errors detected: 1/1 > dt (j:1 t:1): Total elapsed time: 00m00.29s > dt (j:1 t:1): Total system time: 00m00.00s > dt (j:1 t:1): Total user time: 00m00.00s > dt (j:1 t:1): Starting time: Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): Ending time: Sat Aug 30 16:14:08 2025 > dt (j:1 t:1): > dt (j:1 t:1): Command line to re-read the data: > dt (j:1 t:1): # dt if=/mnt/hs_test/dt_thisisa.test bs=47008 dsize=512 iotype=sequential iodir=forward limit=141024 records=3 pattern=iot > dt (j:1 t:1): > dt (j:1 t:1): Command Line: > dt (j:1 t:1): > dt (j:1 t:1): # dt of=/mnt/hs_test/dt_thisisa.test passes=1 bs=47008 count=3 iotype=sequential pattern=iot onerr=abort oncerr=abort > dt (j:1 t:1): > dt (j:1 t:1): --> Date: September 21st, 2023, Version: 25.05, Author: Robin T. Miller <-- > dt (j:1 t:1): > dt (j:1 t:1): onerr=abort, so stopping all threads for job 1... > dt (j:0 t:0): Job 1 is being stopped (1 thread) > dt (j:0 t:0): Program is exiting with status -1... > --- > fs/nfsd/vfs.c | 25 ++++++------------------- > 1 file changed, 6 insertions(+), 19 deletions(-) > > diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c > index f8975ee262b5c..762d745b1b15d 100644 > --- a/fs/nfsd/vfs.c > +++ b/fs/nfsd/vfs.c > @@ -1079,13 +1079,11 @@ struct nfsd_read_dio { > loff_t end; > unsigned long start_extra; > unsigned long end_extra; > - struct page *start_extra_page; > }; > > static void init_nfsd_read_dio(struct nfsd_read_dio *read_dio) > { > memset(read_dio, 0, sizeof(*read_dio)); > - read_dio->start_extra_page = NULL; > } > > #define NFSD_READ_DIO_MIN_KB (32 << 10) > @@ -1121,9 +1119,8 @@ static bool nfsd_analyze_read_dio(struct svc_rqst *rqstp, struct svc_fh *fhp, > > /* > * Any misaligned READ less than NFSD_READ_DIO_MIN_KB won't be expanded > - * to be DIO-aligned (this heuristic avoids excess work, like allocating > - * start_extra_page, for smaller IO that can generally already perform > - * well using buffered IO). > + * to be DIO-aligned (this heuristic avoids excess work, for smaller IO > + * that can generally already perform well using buffered IO). > */ > if ((read_dio->start_extra || read_dio->end_extra) && > (len < NFSD_READ_DIO_MIN_KB)) { > @@ -1131,15 +1128,6 @@ static bool nfsd_analyze_read_dio(struct svc_rqst *rqstp, struct svc_fh *fhp, > return false; > } > > - if (read_dio->start_extra) { > - read_dio->start_extra_page = alloc_page(GFP_KERNEL); > - if (WARN_ONCE(read_dio->start_extra_page == NULL, > - "%s: Unable to allocate start_extra_page\n", __func__)) { > - init_nfsd_read_dio(read_dio); > - return false; > - } > - } > - > /* Show original offset and count, and how it was expanded for DIO */ > middle_end = read_dio->end - read_dio->end_extra; > trace_nfsd_analyze_read_dio(rqstp, fhp, offset, len, > @@ -1162,11 +1150,10 @@ static ssize_t nfsd_complete_misaligned_read_dio(struct svc_rqst *rqstp, > if (!read_dio->start_extra && !read_dio->end_extra) > return host_err; > > - /* If nfsd_analyze_read_dio() allocated a start_extra_page it must > - * be removed from rqstp->rq_bvec[] to avoid returning unwanted data. > + /* If nfsd_analyze_read_dio() found start_extra (front-pad) page needed it > + * must be removed from rqstp->rq_bvec[] to avoid returning unwanted data. > */ > - if (read_dio->start_extra_page) { > - __free_page(read_dio->start_extra_page); > + if (read_dio->start_extra) { > *rq_bvec_numpages -= 1; > v = *rq_bvec_numpages; > memmove(rqstp->rq_bvec, rqstp->rq_bvec + 1, > @@ -1276,7 +1263,7 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp, > if (read_dio.start_extra) { > len = read_dio.start_extra; > bvec_set_page(&rqstp->rq_bvec[v], > - read_dio.start_extra_page, > + *(rqstp->rq_next_page++), > len, PAGE_SIZE - len); > total -= len; > ++v; Thank you, Mike. This will help me reproduce the problem. Saves me a bunch of time! -- Chuck Lever
