Hi All: As discussed in [1], we need to use mmap mechanism to open large "packed_refs" file to save the memory usage. This patch mainly does the following things: 1: Fix an issue that we would report an error when the "packed-refs" file is empty, which does not align with the runtime behavior. 2-4: Extract some logic from the existing code and then use these created helper functions to let fsck code to use mmap necessarily [1] https://lore.kernel.org/git/20250503133158.GA4450@xxxxxxxxxxxxxxxxxxxxxxx Really thank Peff and Patrick to suggest me to do above change. --- Change in v2: 1. Update the commit message of [PATCH 1/4]. And use redirection to create an empty file instead of using `touch`. 2. Don't use if for the refactored function in [PATCH 3/4] and then update the commit message to align with the new function name. 3. Enhance the commit message of [PATCH 4/4]. --- Change in v3: 1. Drop the patch which creates a new function "munmap_temporary_snapshot". As discussed, there is no need to munmap the file during fsck. 2. Allocate snapshot variable in the stack instead of heap. 3. Fix rebase issue, remove unneeded code to check the file size explicitly. --- Change in v4: 1. Report the "emptyPackedRefsFile(INFO)" to the user when the "packed-refs" is empty instead of ONLY skipping checking the content and update the shell script and commit message. 2. Apply Peff's advice to make [PATCH v3 2/3] more clear. --- Change in v5: 1. Improve the commit message in the first patch to be more clear: 1. Talk about the current behavior, what error we would report if "packed-refs" is empty. 2. To align with the runtime behavior, we should skip checking the content of "packed-refs". 3. Why do we need to report to the user when the "packed-refs" is empty 2. Fix grammar issue in the last patch. Thanks, Jialuo shejialuo (3): packed-backend: fsck should warn when "packed-refs" file is empty packed-backend: extract snapshot allocation in `load_contents` packed-backend: mmap large "packed-refs" file during fsck Documentation/fsck-msgids.adoc | 6 +++ fsck.h | 1 + refs/packed-backend.c | 73 ++++++++++++++++++++-------------- t/t0602-reffiles-fsck.sh | 17 ++++++++ 4 files changed, 67 insertions(+), 30 deletions(-) Range-diff against v4: 1: 75636c9c85 ! 1: 3487692a03 packed-backend: fsck should warn when "packed-refs" file is empty @@ Metadata ## Commit message ## packed-backend: fsck should warn when "packed-refs" file is empty - During fsck, an empty "packed-refs" gives an error; this is unwarranted. - The runtime code paths would accept an empty "packed-refs" file, such as - "create_snapshot" would simply return the "snapshot" without checking - the content of "packed-refs". + We assume the "packed-refs" won't be empty and instead has at least one + line in it (even when there are no refs packed, there is the file header + line). Because there is no terminating LF in the empty file, we will + report "packedRefEntryNotTerminated(ERROR)" to the user. - And we should also skip checking the content of "packed-refs" when it is - empty during fsck. However, we should think about whether we need to - report something to the users in this case. + However, the runtime code paths would accept an empty "packed-refs" + file, for example, "create_snapshot" would simply return the "snapshot" + without checking the content of "packed-refs". So, we should skip + checking the content of "packed-refs" when it is empty during fsck. After 694b7a1999 (repack_without_ref(): write peeled refs in the rewritten file, 2013-04-22), we would always write a header into the - "packed-refs" file where we would never create empty file since then. - Because we only create empty "packed-refs" in the very early versions, - we may tighten this rule in the future. In order to notify the users - about this, we should at least report an warning to the users. + "packed-refs" file. So, versions of Git that are not too ancient never + write such an empty "packed-refs" file. - But we need to consider the fsck message type carefully, it is not - appropriate that we use "FSCK_ERROR". This is because we would - definitely break the compatibility. Let's create a "FSCK_INFO" message - id EMPTY_PACKED_REFS_FILE" to indicate that "packed-refs" is empty. + As an empty file often indicates a sign of a filesystem-level issue, the + way we want to resolve this inconsistency is not make everybody totally + silent but notice and report the anomaly. + + Let's create a "FSCK_INFO" message id "EMPTY_PACKED_REFS_FILE" to report + to the users that "packed-refs" is empty. Signed-off-by: shejialuo <shejialuo@xxxxxxxxx> 2: 1a5893379d = 2: 0d050849bc packed-backend: extract snapshot allocation in `load_contents` 3: 31e272db7e ! 3: fe5ffec8fb packed-backend: mmap large "packed-refs" file during fsck @@ Commit message current codebase. As we have introduced the helper function "allocate_snapshot_buffer", we - could simple use this function to use mmap mechanism. + can simply use this function to use mmap mechanism. Suggested-by: Jeff King <peff@xxxxxxxx> Suggested-by: Patrick Steinhardt <ps@xxxxxx> -- 2.49.0
