Hello,
Recently I encountered an issue in kernel 6.1.123, when writing to a
file after disk exhaustion, it will report EFSCORRUPTED. I think it is
un-expected behavior.
Could you help clarify:
1. Why writing to file after disk exhaust will cause "Error while
async write back metadata"? Assume it might be inode or block metadata
is corrupted there?
2. Why would the file system corrupt, like "Aborting journal on device"?
Thank you~
Detailed reproduction steps are:
# 1. Create ext4 file system in mydisk
root@testbed:/tmp# touch mydisk
root@testbed:/tmp# ls -l mydisk
-rw-r--r-- 1 root root 0 Jul 8 05:36 mydisk
root@testbed:/tmp# truncate -s 128M mydisk
root@testbed:/tmp# ls -lh mydisk
-rw-r--r-- 1 root root 128M Jul 8 05:36 mydisk
root@testbed:/tmp# mkfs.ext4 mydisk
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 131072 1k blocks and 32768 inodes
Filesystem UUID: b0b12002-d497-436e-b89d-d0e02f53b46d
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729
Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
# 2. Mount mydisk to /mnt/test_fs
root@testbed:/tmp# mkdir /mnt/test_fs
root@testbed:/tmp# mount mydisk /mnt/test_fs/
root@testbed:/tmp# findmnt /mnt/test_fs
TARGET SOURCE FSTYPE OPTIONS
/mnt/test_fs /dev/loop2 ext4 rw,relatime
root@testbed:/mnt/test_fs# file /tmp/mydisk
/tmp/mydisk: Linux rev 1.0 ext4 filesystem data,
UUID=b0b12002-d497-436e-b89d-d0e02f53b46d (needs journal recovery)
(extents) (64bit) (large files) (huge files)
# 3. Exhaust disk in /mnt/test_fs with 32G test_file
root@testbed:/mnt/test_fs# fallocate -l 32716560K /mnt/test_fs/test_file
fallocate: fallocate failed: No space left on device
root@testbed:/mnt/test_fs# ls
lost+found test_file
root@testbed:/mnt/test_fs# journalctl -f
Jul 08 05:43:07 testbed kernel: loop: Write error at byte offset
9178112, length 1024.
Jul 08 05:43:07 testbed kernel: loop: Write error at byte offset
274432, length 1024.
Jul 08 05:43:07 testbed kernel: loop: Write error at byte offset
274432, length 1024.
Jul 08 05:43:07 testbed kernel: loop: Write error at byte offset
274432, length 1024.
Jul 08 05:43:07 testbed kernel: loop: Write error at byte offset
274432, length 1024.
Jul 08 05:43:07 testbed kernel: loop: Write error at byte offset
274432, length 1024.
Jul 08 05:43:07 testbed kernel: loop: Write error at byte offset
274432, length 1024.
Jul 08 05:43:07 testbed kernel: loop: Write error at byte offset
274432, length 1024.
Jul 08 05:43:07 testbed kernel: loop: Write error at byte offset
274432, length 1024.
Jul 08 05:43:07 testbed kernel: I/O error, dev loop2, sector 17926 op
0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 2
Jul 08 05:43:07 testbed kernel: Buffer I/O error on dev loop2, logical
block 8963, lost async page write
Jul 08 05:43:07 testbed kernel: I/O error, dev loop2, sector 518 op
0x1:(WRITE) flags 0x103000 phys_seg 17 prio class 2
Jul 08 05:43:07 testbed kernel: Buffer I/O error on dev loop2, logical
block 259, lost async page write
Jul 08 05:43:07 testbed kernel: Buffer I/O error on dev loop2, logical
block 260, lost async page write
Jul 08 05:43:07 testbed kernel: Buffer I/O error on dev loop2, logical
block 261, lost async page write
Jul 08 05:43:07 testbed kernel: Buffer I/O error on dev loop2, logical
block 262, lost async page write
# 4. Write to /mnt/test_fs/file.dat with dd cmd, I/O error appears.
root@testbed:/mnt/test_fs# dd if=/dev/zero of=/mnt/test_fs/file.dat
bs=1M count=64
root@testbed:/mnt/test_fs# journalctl -f
Jul 08 05:49:24 testbed kernel: Buffer I/O error on dev loop2, logical
block 268, lost async page write
Jul 08 05:49:26 testbed kernel: EXT4-fs error (device loop2):
ext4_check_bdev_write_error:217: comm dd: Error while async write back
metadata
Jul 08 05:49:26 testbed kernel: I/O error, dev loop2, sector 20482 op
0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 2
# Can see First error function is ext4_check_bdev_write_error.
root@testbed:/mnt/test_fs# dumpe2fs -h /dev/loop2
dumpe2fs 1.47.0 (5-Feb-2023)
Filesystem volume name: <none>
Last mounted on: /mnt/test_fs
Filesystem UUID: b0b12002-d497-436e-b89d-d0e02f53b46d
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype needs_recovery extent 64bit flex_bg sparse_super large_file
huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 32768
Block count: 131072
Reserved block count: 6553
Overhead clusters: 13869
Free blocks: 42236
Free inodes: 32754
First block: 1
Block size: 1024
Fragment size: 1024
Group descriptor size: 64
Reserved GDT blocks: 256
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 2048
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Tue Jul 8 05:37:11 2025
Last mount time: Tue Jul 8 05:37:37 2025
Last write time: Tue Jul 8 05:50:36 2025
Mount count: 1
Maximum mount count: -1
Last checked: Tue Jul 8 05:37:11 2025
Check interval: 0 (<none>)
Lifetime writes: 74 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 32
Desired extra isize: 32
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: bf9009a6-ff19-41d5-8abc-a4d9cd65eeb4
Journal backup: inode blocks
FS Error count: 4
First error time: Tue Jul 8 05:45:22 2025
First error function: ext4_check_bdev_write_error
First error line #: 217
First error err: EIO
Last error time: Tue Jul 8 05:50:36 2025
Last error function: ext4_check_bdev_write_error
Last error line #: 217
Last error err: EIO
Checksum type: crc32c
Checksum: 0x0583faaa
Journal features: journal_incompat_revoke journal_64bit
journal_checksum_v3
Total journal size: 4096k
Total journal blocks: 4096
Max transaction length: 4096
Fast commit length: 0
Journal sequence: 0x00000002
Journal start: 1
Journal checksum type: crc32c
Journal checksum: 0xebf7b874
# 5. unmount the filesystem, file system became read-only, result show
EFSCORRUPTED
Jul 08 06:44:17 testbed kernel: EXT4-fs (loop2): unmounting filesystem.
Jul 08 06:44:17 testbed kernel: Aborting journal on device loop2-8.
Jul 08 06:44:17 testbed kernel: EXT4-fs error (device loop2):
ext4_put_super:1232: comm umount: Couldn't clean up the journal
Jul 08 06:44:17 testbed kernel: EXT4-fs (loop2): Remounting filesystem read-only
root@testbed:/tmp# dumpe2fs -h /dev/loop2
...
FS Error count: 9
First error time: Tue Jul 8 05:45:22 2025
First error function: ext4_check_bdev_write_error
First error line #: 217
First error err: EIO
Last error time: Tue Jul 8 06:46:30 2025
Last error function: ext4_validate_block_bitmap
Last error line #: 420
Last error err: EFSCORRUPTED
...
Thank you~
Best regards,
Jianyue Wu
