Hi All: As discussed in [1], we need to use mmap mechanism to open large "packed_refs" file to save the memory usage. This patch mainly does the following things: 1: Fix an issue that we would report an error when the "packed-refs" file is empty, which does not align with the runtime behavior. 2-4: Extract some logic from the existing code and then use these created helper functions to let fsck code to use mmap necessarily [1] https://lore.kernel.org/git/20250503133158.GA4450@xxxxxxxxxxxxxxxxxxxxxxx Really thank Peff and Patrick to suggest me to do above change. --- Change in v2: 1. Update the commit message of [PATCH 1/4]. And use redirection to create an empty file instead of using `touch`. 2. Don't use if for the refactored function in [PATCH 3/4] and then update the commit message to align with the new function name. 3. Enhance the commit message of [PATCH 4/4]. --- Change in v3: 1. Drop the patch which creates a new function "munmap_temporary_snapshot". As discussed, there is no need to munmap the file during fsck. 2. Allocate snapshot variable in the stack instead of heap. 3. Fix rebase issue, remove unneeded code to check the file size explicitly. --- Change in v4: 1. Report the "emptyPackedRefsFile(INFO)" to the user when the "packed-refs" is empty instead of ONLY skipping checking the content and update the shell script and commit message. 2. Apply Peff's advice to make [PATCH v3 2/3] more clear. Thanks, Jialuo shejialuo (3): packed-backend: fsck should warn when "packed-refs" file is empty packed-backend: extract snapshot allocation in `load_contents` packed-backend: mmap large "packed-refs" file during fsck Documentation/fsck-msgids.adoc | 6 +++ fsck.h | 1 + refs/packed-backend.c | 73 ++++++++++++++++++++-------------- t/t0602-reffiles-fsck.sh | 17 ++++++++ 4 files changed, 67 insertions(+), 30 deletions(-) Range-diff against v3: 1: 26c3fd55a8 ! 1: 75636c9c85 packed-backend: fsck should allow an empty "packed-refs" file @@ Metadata Author: shejialuo <shejialuo@xxxxxxxxx> ## Commit message ## - packed-backend: fsck should allow an empty "packed-refs" file + packed-backend: fsck should warn when "packed-refs" file is empty During fsck, an empty "packed-refs" gives an error; this is unwarranted. - We should just skip checking the content of "packed-refs" just like the - runtime code paths such as "create_snapshot" which simply returns the - "snapshot" without checking the content of "packed-refs". + The runtime code paths would accept an empty "packed-refs" file, such as + "create_snapshot" would simply return the "snapshot" without checking + the content of "packed-refs". + + And we should also skip checking the content of "packed-refs" when it is + empty during fsck. However, we should think about whether we need to + report something to the users in this case. + + After 694b7a1999 (repack_without_ref(): write peeled refs in the + rewritten file, 2013-04-22), we would always write a header into the + "packed-refs" file where we would never create empty file since then. + Because we only create empty "packed-refs" in the very early versions, + we may tighten this rule in the future. In order to notify the users + about this, we should at least report an warning to the users. + + But we need to consider the fsck message type carefully, it is not + appropriate that we use "FSCK_ERROR". This is because we would + definitely break the compatibility. Let's create a "FSCK_INFO" message + id EMPTY_PACKED_REFS_FILE" to indicate that "packed-refs" is empty. Signed-off-by: shejialuo <shejialuo@xxxxxxxxx> + ## Documentation/fsck-msgids.adoc ## +@@ + `emptyName`:: + (WARN) A path contains an empty name. + ++`emptyPackedRefsFile`:: ++ (INFO) "packed-refs" file is empty. Report to the ++ git@xxxxxxxxxxxxxxx mailing list if you see this error. As only ++ very early versions of Git would create such an empty ++ "packed_refs" file, we might tighten this rule in the future. ++ + `extraHeaderEntry`:: + (IGNORE) Extra headers found after `tagger`. + + + ## fsck.h ## +@@ fsck.h: enum fsck_msg_type { + FUNC(LARGE_PATHNAME, WARN) \ + /* infos (reported as warnings, but ignored by default) */ \ + FUNC(BAD_FILEMODE, INFO) \ ++ FUNC(EMPTY_PACKED_REFS_FILE, INFO) \ + FUNC(GITMODULES_PARSE, INFO) \ + FUNC(GITIGNORE_SYMLINK, INFO) \ + FUNC(GITATTRIBUTES_SYMLINK, INFO) \ + ## refs/packed-backend.c ## @@ refs/packed-backend.c: static int packed_fsck(struct ref_store *ref_store, goto cleanup; } -+ if (!st.st_size) ++ if (!st.st_size) { ++ struct fsck_ref_report report = { 0 }; ++ report.path = "packed-refs"; ++ ret = fsck_report_ref(o, &report, ++ FSCK_MSG_EMPTY_PACKED_REFS_FILE, ++ "file is empty"); + goto cleanup; ++ } + if (strbuf_read(&packed_ref_content, fd, 0) < 0) { ret = error_errno(_("unable to read '%s'"), refs->path); @@ t/t0602-reffiles-fsck.sh: test_expect_success SYMLINKS 'the filetype of packed-r ) ' -+test_expect_success 'empty packed-refs should not be reported' ' ++test_expect_success 'empty packed-refs should be reported' ' + test_when_finished "rm -rf repo" && + git init repo && + ( @@ t/t0602-reffiles-fsck.sh: test_expect_success SYMLINKS 'the filetype of packed-r + + >.git/packed-refs && + git refs verify 2>err && -+ test_must_be_empty err ++ cat >expect <<-EOF && ++ warning: packed-refs: emptyPackedRefsFile: file is empty ++ EOF ++ rm .git/packed-refs && ++ test_cmp expect err + ) +' + 2: 4604be8b51 ! 2: 1a5893379d packed-backend: extract snapshot allocation in `load_contents` @@ refs/packed-backend.c: static int refname_contains_nul(struct strbuf *refname) struct stat st; - size_t size; - ssize_t bytes_read; -+ int ret = 1; ++ int ret; + int fd; fd = open(snapshot->refs->path, O_RDONLY); @@ refs/packed-backend.c: static int load_contents(struct snapshot *snapshot) - snapshot->start = snapshot->buf; - snapshot->eof = snapshot->buf + size; -+ if (!allocate_snapshot_buffer(snapshot, fd, &st)) -+ ret = 0; ++ ret = allocate_snapshot_buffer(snapshot, fd, &st); - return 1; + close(fd); 3: 82e19a65c6 ! 3: 31e272db7e packed-backend: mmap large "packed-refs" file during fsck @@ refs/packed-backend.c: static int packed_fsck(struct ref_store *ref_store, goto cleanup; } -- if (!st.st_size) -+ if (!allocate_snapshot_buffer(&snapshot, fd, &st)) +- if (!st.st_size) { ++ if (!allocate_snapshot_buffer(&snapshot, fd, &st)) { + struct fsck_ref_report report = { 0 }; + report.path = "packed-refs"; + ret = fsck_report_ref(o, &report, +@@ refs/packed-backend.c: static int packed_fsck(struct ref_store *ref_store, goto cleanup; + } - if (strbuf_read(&packed_ref_content, fd, 0) < 0) { - ret = error_errno(_("unable to read '%s'"), refs->path); -- 2.49.0
