Hi, We are attempting to push a bare repository on our local disk to Azure DevOps using the command "git push -mirror <URL>". Git version being used: 2.48.1 and no global git configurations exist in the machine. Issue: command progresses for 5-10 mins and then the git processes get hung after we see the output "Storing index (done)" and remains hung indefinitely. Steps to reproduce: i) Bare clone open source repo (elasticsearch or node) to local disk. ii) Run "git lfs push -mirror" to any repository in an Azure DevOps organization using PAT authentication. iii) Issue happens intermittently, so alternating the push operation between the local elasticsearch and node repo to the same remote repository should help recreate the issue. Workaround steps taken by us to not face this issue: i) Since our repository is quite large (1.5 GB), we set http.postBuffer to 50 MB, it did not help in resolving the issue. ii) We have set GIT_CURL_TIMEOUT=180 along with http.lowSpeedLimit=1 http.lowSpeedTime=180. But nothing seems to make the process exit after it gets hung. iii) We set the http.maxRequests=1 to avoid concurrent requests and that did not help either. iv) We used the -no-thin option, did not work either, process still hung. v) We have tried with different values as well for the above git config options and nothing seemed to bring us out from the hung state. GIT_TRACE=2 GIT_CURL_VERBOSE=2 outputs: remote: Storing packfile... done (43758 ms) 08:26:52.444122 http.c:878 == Info: TLSv1.3 (IN), TLS app data, [no content] (0): 08:26:57.457679 http.c:878 == Info: TLSv1.3 (IN), TLS app data, [no content] (0): 08:26:58.309172 http.c:878 == Info: TLSv1.3 (IN), TLS app data, [no content] (0): remote: Storing index... done (5865 ms) 08:28:09.549750 http.c:878 == Info: TLSv1.3 (IN), TLS app data, [no content] (0): 08:28:09.550215 http.c:878 == Info: TLSv1.3 (IN), TLS app data, [no content] (0): 08:28:09.550751 http.c:878 == Info: TLSv1.3 (IN), TLS app data, [no content] (0): 08:28:09.551245 http.c:878 == Info: TLSv1.3 (IN), TLS app data, [no content] (0): 08:28:09.551768 http.c:878 == Info: TLSv1.3 (IN), TLS app data, [no content] (0): 08:28:09.551850 http.c:878 == Info: Connection #0 to host dev.azure.com left intact Processes launched after running git push -mirror: [root@dop02108c1eu06 ~]# ps -aef | grep Token root 1043353 1043352 0 08:17 pts/10 00:00:00 /usr/local/bin/git -c http.maxRequests=1 -c http.postBuffer=50000000 -c http.lowSpeedLimit=1 -c http.lowSpeedTime=30 push --mirror --no-thin https://Token:%3cTOKEN%3e@xxxxxxxxxxxxx/TestADOPATPolicy/cv-restores/_git/cv-restores root 1043354 1043353 0 08:17 pts/10 00:00:00 /usr/local/libexec/git-core/git remote-https https://Token:%3cTOKEN%3e@xxxxxxxxxxxxx/TestADOPATPolicy/cv-restores/_git/cv-restores https://Token:%3cTOKEN%3e@xxxxxxxxxxxxx/TestADOPATPolicy/cv-restores/_git/cv-restores root 1043355 1043354 0 08:17 pts/10 00:00:05 /usr/local/libexec/git-core/git-remote-https https://Token:%3cTOKEN%3e@xxxxxxxxxxxxx/TestADOPATPolicy/cv-restores/_git/cv-restores https://Token:%3cTOKEN%3e@xxxxxxxxxxxxx/TestADOPATPolicy/cv-restores/_git/cv-restores root 1043377 1043355 0 08:17 pts/10 00:00:03 /usr/local/libexec/git-core/git send-pack --stateless-rpc --helper-status --thin --progress https://Token:%3cTOKEN%3e@xxxxxxxxxxxxx/TestADOPATPolicy/cv-restores/_git/cv-restores/ --stdin root 1052699 1049394 0 08:32 pts/14 00:00:00 grep --color=auto Token pstack of all above processes : [root@dop02108c1eu06 ~]# pstack 1043353 #0 0x00007f7face375a5 in read () from /lib64/libc.so.6 #1 0x00007f7facdc7418 in __GI__IO_file_underflow () from /lib64/libc.so.6 #2 0x00007f7facdbbdc8 in getdelim () from /lib64/libc.so.6 #3 0x0000000000635eb8 in strbuf_getwholeline (sb=sb@entry=0x7ffe7067cbf0, fp=0x1cf02c0, term=term@entry=10) at strbuf.c:645 #4 0x00000000006360c0 in strbuf_getdelim_strip_crlf (sb=0x7ffe7067cbf0, fp=<optimized out>, term=10) at strbuf.c:727 #5 0x000000000064dacc in recvline_fh (helper=0x1cf02c0, buffer=0x7ffe7067cbf0) at transport-helper.c:76 #6 0x000000000064ec6e in recvline (helper=0x1cf8570, buffer=0x7ffe7067cbf0) at transport-helper.c:923 #7 push_update_refs_status (data=0x1cf8570, remote_refs=0x1d2b3a0, flags=10) at transport-helper.c:923 #8 0x0000000000650c3f in push_refs_with_push (flags=10, remote_refs=0x1d2b3a0, transport=0x1cf82c0) at transport-helper.c:1088 #9 push_refs (transport=0x1cf82c0, remote_refs=0x1d2b3a0, flags=10) at transport-helper.c:1189 #10 0x0000000000653faf in transport_push (r=0x993960 <the_repo>, transport=transport@entry=0x1cf82c0, rs=rs@entry=0x98f1f0 <rs>, flags=flags@entry=10, reject_reasons=reject_reasons@entry=0x7ffe7067cfa0) at transport.c:1502 #11 0x0000000000491cef in push_with_options (flags=10, rs=0x98f1f0 <rs>, transport=0x1cf82c0) at builtin/push.c:387 #12 do_push (remote=0x1cf7b50, push_options=0x97aba0 <push_options_config>, flags=10) at builtin/push.c:442 #13 cmd_push (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>, repository=<optimized out>) at builtin/push.c:664 #14 0x0000000000404eae in run_builtin (repo=0x993960 <the_repo>, argv=<optimized out>, argc=<optimized out>, p=0x971fe8 <commands+2184>) at git.c:480 #15 handle_builtin (args=args@entry=0x7ffe7067dbe0) at git.c:740 #16 0x0000000000405f1f in run_argv (args=args@entry=0x7ffe7067dbe0) at git.c:807 #17 0x00000000004068d6 in cmd_main (argc=<optimized out>, argc@entry=13, argv=<optimized out>, argv@entry=0x7ffe7067ddb8) at git.c:947 #18 0x0000000000404a67 in main (argc=13, argv=0x7ffe7067ddb8) at common-main.c:64 [root@dop02108c1eu06 ~]# [root@dop02108c1eu06 ~]# pstack 1043354 #0 0x00007f64b602b468 in waitpid () from /lib64/libpthread.so.0 #1 0x000000000060fc0b in wait_or_whine (pid=1043355, argv0=0x298f5f0 "git-remote-https", in_signal=in_signal@entry=0) at run-command.c:559 #2 0x00000000006115c9 in finish_command (cmd=cmd@entry=0x7ffe5685e600) at run-command.c:989 #3 0x0000000000611679 in run_command (cmd=cmd@entry=0x7ffe5685e600) at run-command.c:1015 #4 0x0000000000405fe7 in execv_dashed_external (argv=<optimized out>) at git.c:777 #5 run_argv (args=args@entry=0x7ffe5685e6d0) at git.c:844 #6 0x00000000004068d6 in cmd_main (argc=<optimized out>, argc@entry=4, argv=<optimized out>, argv@entry=0x7ffe5685e8a8) at git.c:947 #7 0x0000000000404a67 in main (argc=4, argv=0x7ffe5685e8a8) at common-main.c:64 [root@dop02108c1eu06 ~]# [root@dop02108c1eu06 ~]# pstack 1043355 #No Pstack strace [root@dop02108c1eu06 ~]# pstack 1043377 Thread 2 (Thread 0x7fa246a18700 (LWP 1043384)): #0 0x00007fa24ce568e4 in read () from /lib64/libpthread.so.0 #1 0x000000000066a07e in xread (fd=fd@entry=0, buf=buf@entry=0x7fa246a07d7c, len=len@entry=4) at wrapper.c:234 #2 0x000000000066a1eb in read_in_full (fd=0, buf=buf@entry=0x7fa246a07d7c, count=count@entry=4) at wrapper.c:292 #3 0x00000000005c4d32 in get_packet_data (fd=fd@entry=0, src_buf=src_buf@entry=0x0, src_size=src_size@entry=0x0, dst=dst@entry=0x7fa246a07d7c, size=size@entry=4, options=options@entry=1) at pkt-line.c:354 #4 0x00000000005c5827 in packet_read_with_status (fd=fd@entry=0, src_buffer=src_buffer@entry=0x0, src_len=src_len@entry=0x0, buffer=buffer@entry=0x7fa246a07e20 "\001\060\060\061aok refs/tags/v0.10.45\n", size=size@entry=65520, pktlen=pktlen@entry=0x7fa246a07df8, options=1) at pkt-line.c:422 #5 0x00000000005c5fa9 in recv_sideband (me=me@entry=0x6e35ea "send-pack", in_stream=0, out=out@entry=5) at pkt-line.c:586 #6 0x0000000000612bf3 in sideband_demux (in=<optimized out>, out=5, data=0x7ffcfb7a3b08) at send-pack.c:283 #7 0x000000000060fd8e in run_thread (data=0x7ffcfb7a3930) at run-command.c:1040 #8 0x00007fa24ce4d14a in start_thread () from /lib64/libpthread.so.0 #9 0x00007fa24c974dc3 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fa24d49cb80 (LWP 1043377)): #0 0x00007fa24ce568e4 in read () from /lib64/libpthread.so.0 #1 0x000000000066a07e in xread (fd=fd@entry=4, buf=buf@entry=0x7ffcfb7a36cc, len=len@entry=4) at wrapper.c:234 #2 0x000000000066a1eb in read_in_full (fd=4, buf=buf@entry=0x7ffcfb7a36cc, count=count@entry=4) at wrapper.c:292 #3 0x00000000005c4d32 in get_packet_data (fd=fd@entry=4, src_buf=src_buf@entry=0x7ffcfb7a3968, src_size=src_size@entry=0x7ffcfb7a3970, dst=dst@entry=0x7ffcfb7a36cc, size=size@entry=4, options=options@entry=6) at pkt-line.c:354 #4 0x00000000005c5827 in packet_read_with_status (fd=4, src_buffer=src_buffer@entry=0x7ffcfb7a3968, src_len=src_len@entry=0x7ffcfb7a3970, buffer=0x99af00 <packet_buffer> "ok", size=65520, pktlen=pktlen@entry=0x7ffcfb7a398c, options=6) at pkt-line.c:422 #5 0x00000000005c5d10 in packet_reader_read (reader=reader@entry=0x7ffcfb7a3960) at pkt-line.c:640 #6 0x00000000005c60ac in packet_reader_read (reader=reader@entry=0x7ffcfb7a3960) at pkt-line.c:626 #7 0x0000000000612d98 in receive_status (reader=reader@entry=0x7ffcfb7a3960, refs=refs@entry=0x1583c90) at send-pack.c:183 #8 0x0000000000613bbb in send_pack (args=args@entry=0x98f5c0 <args>, fd=fd@entry=0x7ffcfb7a3b08, conn=conn@entry=0x0, remote_refs=<optimized out>, extra_have=extra_have@entry=0x7ffcfb7a3b70) at send-pack.c:746 #9 0x00000000004b64de in cmd_send_pack (argc=<optimized out>, argv=0x153e660, prefix=<optimized out>, repo=<optimized out>) at builtin/send-pack.c:320 #10 0x0000000000404eae in run_builtin (repo=0x993960 <the_repo>, argv=<optimized out>, argc=<optimized out>, p=0x9721c8 <commands+2664>) at git.c:480 #11 handle_builtin (args=args@entry=0x7ffcfb7a4580) at git.c:740 #12 0x0000000000405f1f in run_argv (args=args@entry=0x7ffcfb7a4580) at git.c:807 #13 0x00000000004068d6 in cmd_main (argc=<optimized out>, argc@entry=8, argv=<optimized out>, argv@entry=0x7ffcfb7a4758) at git.c:947 #14 0x0000000000404a67 in main (argc=8, argv=0x7ffcfb7a4758) at common-main.c:64 strace of all above processes: [root@dop02108c1eu06 ~]# strace -p 1043353 strace: Process 1043353 attached read(3, [root@dop02108c1eu06 workingDir]# strace -p 1043354 strace: Process 1043354 attached wait4(1043355, [root@dop02108c1eu06 workingDir]# strace -p 1043355 rt_sigaction(SIGPIPE, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0 select(4, [3], [], [], {tv_sec=0, tv_usec=795000}) = 1 (in [3], left {tv_sec=0, tv_usec=794998}) rt_sigaction(SIGPIPE, NULL, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, 8) = 0 rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}]) read(3, "\27\3\3\0006", 5) = 5 read(3, "\216v\301[\367\322\25\326\25\266p6\322\211%\2237\305$W\21\r.oK\241\207f\306\232~\23"..., 54) = 54 write(2, "08:28:09.551245 http.c:878 "..., 95) = 95 write(5, "001f\001001aok refs/tags/v0.10.45\n", 31) = 31 rt_sigaction(SIGPIPE, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0 select(4, [3], [], [], {tv_sec=0, tv_usec=795000}) = 1 (in [3], left {tv_sec=0, tv_usec=794998}) rt_sigaction(SIGPIPE, NULL, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, 8) = 0 rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}]) read(3, "\27\3\3\0\26", 5) = 5 read(3, "\346\302h\301\6g\216s^:\301\352\274\341G\6\245\3AT+7", 22) = 22 write(2, "08:28:09.551768 http.c:878 "..., 95) = 95 write(2, "08:28:09.551850 http.c:878 "..., 97) = 97 rt_sigaction(SIGPIPE, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0 read(6, [root@dop02108c1eu06 ~]# strace -p 1043377 strace: Process 1043377 attached read(4, Thanks, Akash
