Hi brian, Thank you for your quick feedback! I was not thinking of the immutable file aspect, that's new for me and makes sense. Your feedback enabled us to press the storage team and we are discovering we had a single nic ontap that works and a dual nic ontap storage that doesn't, so it appears your assessment of it being a storage failure is right on the money. Thank you again for your detailed response. It was very helpful! Tim On Thu, Jul 24, 2025 at 7:14 PM brian m. carlson <sandals@xxxxxxxxxxxxxxxxxxxx> wrote: > > On 2025-07-24 at 21:49:41, Tim Cederquist wrote: > > What did you expect to happen? (Expected behavior) > > git clone https://github.com/githubtraining/hellogitworld.git > > Command should have cloned a public repository the NFS hosted home folder > > > > What happened instead? (Actual behavior) > > > > Cloning into 'hellogitworld'... > > remote: Enumerating objects: 306, done. > > remote: Counting objects: 100% (32/32), done. > > remote: Compressing objects: 100% (13/13), done. > > remote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1) > > Receiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done. > > Resolving deltas: 100% (70/70), done. > > fatal: fsync error on > > '/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY': > > Permission denied > > This is definitely a bug in your NFS server. EACCES is only a code you > should see before you have a file descriptor. Once you have a file > descriptor (e.g., open(2) succeeded), the proper status code if it is > not suitable for your purpose is EBADF. (For instance, if you attempt > to write(2) to a file open only for reading.) > > However, fsync(2) should never return EBADF or EACCES on a file open for > writing. > > > fatal: fetch-pack: invalid index-pack output > > > > What's different between what you expected and what actually happened? > > fsync error - permission denied > > > > Anything else you want to add: > > From Ontap storage host sectrace events command, it indicates the > > failure is due to the user not having 'Append' permissions to the file > > and generated the fsync error. > > Running an strace on the git clone command shows the process runs > > "openat" with 444 file permission but with O_RDWR flag. > > Ontap creates the file with 444 (read only) posix permission > > git continues on to write() into the read only file - ontap rejects > > it and fails the command due to read only status of the file > > Yup, this is explicitly allowed by POSIX. > > > I've tested with linux "instruction" command to set permission and > > copy a file in a similar fashion: > > strace -f -o trace_install install -m 444 src.txt test/a/test4.txt > > This command opens the file handle 600, writes into it, and then > > chmods to 444 << not using 444 to start the file as git clone is > > trying to do. > > Additionally, I've added an inherited non-intrinsic permission to > > the user of A:FD:tcederquist@domain:wa << this appends the required > > write + append attribute and the FD means it is inherited by all files > > and not overridden by posix permissions. This is not a solution but a > > testable method. With this permission added, the git clone works as > > expected. > > Suggestion is to use 600 on the openat/fopen for the pack index file > > instead of 444. This is how 'install' sets up the file. However, I > > don't know if this was an attempt at a cross platform mutex? I cannot > > imagine any other reason why 444 would have been used for a file that > > would have content written after opening with read only permissions. > > The goal is to create a file which has permissions honoured by the umask > but is not at all writable. There is no reason to write to a pack file > or loose object once it's written: the file is immutable until it's no > longer useful, at which point it's removed (which does not require write > permission on the file). POSIX requires that the restrictions set by > the file mode be ignored when determining whether the file is open for > writing, so your NFS server is not following the POSIX spec correctly. > > Note this is possible to do correctly over NFS, and many servers do so, > but there are also unfortunately a large number of servers which do not > honour the POSIX standard correctly. In addition, this problem not only > affects Git, but a wide variety of other software as well, including zsh > and Emacs, as well as every other Git implementation I'm aware of, so us > trying to work around it would still leave you with a server that didn't > work properly with lots of software. > -- > brian m. carlson (they/them) > Toronto, Ontario, CA
