401 Unauthorized - GIT LFS and Nginx reverse proxy auth problem

Projects NGINX Commercial Products
, ,
1

Hello team,

I have small self-hosted platform for software development lifecycle management Azure DevOps on-prem. On the backend site I have 2 application tiers (server1, server2), one SQL Server (server3) and one elastic search service (server4). Entire setup is hidden behind Nginx Plus Load Balancer (ver 1.27.4)

Problem summary:
Whenever we try to rut GIT clone operation on LFS files Nginx keeps asking for credentials. I have tried to use credential-manager. Tried to store locally the creds in cache but each time the same result.

Here is the output:

$ GIT_TRACE=1 GIT_CURL_VERBOSE=1 GIT_TRACE_TRANSFER=1 git clone https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2
12:41:31.980408 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
12:41:31.991399 git.c:476               trace: built-in: git clone 'https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2'
Cloning into 'LFSTest2'...
12:41:32.036401 run-command.c:674       trace: run_command: git remote-https origin 'https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2'
12:41:32.036401 run-command.c:935       trace: start_command: git remote-https origin 'https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2'
12:41:32.064401 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core

12:41:32.162402 http.c:914              == Info: Connected to qa.cpazuredevops.domain.com (172.21.24.25) port 443
12:41:32.163402 http.c:914              == Info: using HTTP/1.x
12:41:32.163402 http.c:861              => Send header, 0000000250 bytes (0x000000fa)
12:41:32.163402 http.c:873              => Send header: GET /PLM/PLM%20Solutions/_git/LFSTest2/info/refs?service=git-upload-pack HTTP/1.1
12:41:32.163402 http.c:873              => Send header: Host: qa.cpazuredevops.domain.com
12:41:32.163402 http.c:873              => Send header: User-Agent: git/2.50.0.windows.2
12:41:32.163402 http.c:873              => Send header: Accept: */*
12:41:32.163402 http.c:873              => Send header: Accept-Encoding: deflate, gzip, br, zstd
12:41:32.175402 http.c:861              <= Recv header, 0000000029 bytes (0x0000001d)
12:41:32.175402 http.c:873              <= Recv header: WWW-Authenticate: Negotiate
12:41:32.179401 http.c:861              <= Recv header, 0000000024 bytes (0x00000018)
12:41:32.179401 http.c:873              <= Recv header: WWW-Authenticate: NTLM
12:41:32.179401 http.c:861              <= Recv header, 0000000023 bytes (0x00000017)
12:41:32.179401 http.c:873              <= Recv header: X-Powered-By: ASP.NET
12:41:32.179401 http.c:861              <= Recv header, 0000000124 bytes (0x0000007c)
12:41:32.179401 http.c:873              <= Recv header: P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
12:41:32.179401 http.c:861              <= Recv header, 0000000024 bytes (0x00000018)
12:41:32.180402 http.c:873              <= Recv header: Lfs-Authenticate: NTLM
12:41:32.180402 http.c:861              <= Recv header, 0000000033 bytes (0x00000021)
12:41:32.180402 http.c:873              <= Recv header: X-Content-Type-Options: nosniff
12:41:32.180402 http.c:861              <= Recv header, 0000000002 bytes (0x00000002)
12:41:32.180402 http.c:873              <= Recv header:
12:41:32.180402 http.c:914              == Info: Connection #0 to host qa.cpazuredevops.domain.com left intact
12:41:32.181439 run-command.c:674       trace: run_command: 'git credential-cache get'
12:41:32.181439 run-command.c:935       trace: start_command: 'C:/Program Files/Git/usr/bin/sh.exe' -c 'git credential-cache get' 'git credential-cache get'
12:41:32.347404 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:41:32.359403 git.c:476               trace: built-in: git credential-cache get
12:41:32.370404 run-command.c:674       trace: run_command: 'C:/Program Files/Git/mingw64/bin/git-askpass.exe' 'Username for '\''https://qa.cpazuredevops.domain.com'\'': '
12:41:32.370404 run-command.c:935       trace: start_command: 'C:/Program Files/Git/mingw64/bin/git-askpass.exe' 'Username for '\''https://qa.cpazuredevops.domain.com'\'': '
12:41:39.804261 run-command.c:674       trace: run_command: 'C:/Program Files/Git/mingw64/bin/git-askpass.exe' 'Password for '\''https://ch00-TCTST@qa.cpazuredevops.domain.com'\'': '
12:41:39.805258 run-command.c:935       trace: start_command: 'C:/Program Files/Git/mingw64/bin/git-askpass.exe' 'Password for '\''https://ch00-TCTST@qa.cpazuredevops.domain.com'\'': '
12:41:45.238619 http.c:914              == Info: Re-using existing https: connection with host qa.cpazuredevops.domain.com
12:41:45.238619 http.c:861              => Send header, 0000000250 bytes (0x000000fa)
12:41:45.238619 http.c:873              => Send header: GET /PLM/PLM%20Solutions/_git/LFSTest2/info/refs?service=git-upload-pack HTTP/1.1
12:41:45.239623 http.c:873              => Send header: Host: qa.cpazuredevops.domain.com
n: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3
Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3C
soap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%
3ABody%3E%3C%2Fsoap%3AEnvelope%3E
12:41:45.250624 http.c:861              <= Recv header, 0000000128 bytes (0x00000080)
12:41:45.250624 http.c:873              <= Recv header: X-TFS-ServiceError: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
12:41:45.251622 http.c:861              <= Recv header, 0000000026 bytes (0x0000001a)
12:41:45.251622 http.c:873              <= Recv header: WWW-Authenticate: Bearer
12:41:45.251622 http.c:861              <= Recv header, 0000000068 bytes (0x00000044)
12:41:45.251622 http.c:873              <= Recv header: WWW-Authenticate: Basic realm="https://qa.cpazuredevops.domain.com/"
12:41:45.251622 http.c:861              <= Recv header, 0000000029 bytes (0x0000001d)
12:41:45.251622 http.c:873              <= Recv header: WWW-Authenticate: Negotiate
12:41:45.251622 http.c:861              <= Recv header, 0000000024 bytes (0x00000018)
12:41:45.251622 http.c:873              <= Recv header: WWW-Authenticate: NTLM
12:41:45.252623 http.c:861              <= Recv header, 0000000023 bytes (0x00000017)
12:41:45.252623 http.c:873              <= Recv header: X-Powered-By: ASP.NET
12:41:45.252623 http.c:861              <= Recv header, 0000000124 bytes (0x0000007c)
12:41:45.252623 http.c:873              <= Recv header: P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
12:41:45.252623 http.c:861              <= Recv header, 0000000024 bytes (0x00000018)
12:41:45.252623 http.c:873              <= Recv header: Lfs-Authenticate: NTLM
12:41:45.252623 http.c:861              <= Recv header, 0000000033 bytes (0x00000021)
12:41:45.253623 http.c:873              <= Recv header: X-Content-Type-Options: nosniff
12:41:45.253623 http.c:914              == Info: Ignoring the response-body
12:41:45.253623 http.c:914              == Info: setting size while ignoring
12:41:45.253623 http.c:861              <= Recv header, 0000000002 bytes (0x00000002)
12:41:45.253623 http.c:873              <= Recv header:
12:41:45.253623 http.c:914              == Info: Connection #0 to host qa.cpazuredevops.domain.com left intact
12:41:45.253623 http.c:914              == Info: Issue another request to this URL: 'https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2/info/refs?service=git-upload-pack'
12:41:45.254622 http.c:914              == Info: Re-using existing https: connection with host qa.cpazuredevops.domain.com
12:41:45.254622 http.c:914              == Info: Server auth using NTLM with user 'ch00-TCTST'
12:41:45.254622 http.c:861              => Send header, 0000000328 bytes (0x00000148)
12:41:45.254622 http.c:873              => Send header: GET /PLM/PLM%20Solutions/_git/LFSTest2/info/refs?service=git-upload-pack HTTP/1.1
12:41:45.255620 http.c:873              => Send header: Host: qa.cpazuredevops.domain.com
12:41:45.255620 http.c:873              => Send header: Authorization: NTLM <redacted>
12:41:45.255620 http.c:873              => Send header: User-Agent: git/2.50.0.windows.2
12:41:45.255620 http.c:873              => Send header: Accept: */*
12:41:45.255620 http.c:873              => Send header: Accept-Encoding: deflate, gzip, br, zstd
12:41:45.255620 http.c:873              => Send header: Pragma: no-cache
12:41:45.255620 http.c:873              => Send header: Git-Protocol: version=2
12:41:45.255620 http.c:873              => Send header:
12:41:45.256623 http.c:914              == Info: Request completely sent off
12:41:45.257622 http.c:861              <= Recv header, 0000000027 bytes (0x0000001b)
12:41:45.257622 http.c:873              <= Recv header: HTTP/1.1 401 Unauthorized
12:41:45.257622 http.c:861              <= Recv header, 0000000022 bytes (0x00000016)
12:41:45.257622 http.c:873              <= Recv header: Server: nginx/1.27.4
12:41:45.257622 http.c:861              <= Recv header, 0000000037 bytes (0x00000025)
12:41:45.257622 http.c:873              <= Recv header: Date: Wed, 09 Jul 2025 10:41:44 GMT
12:41:45.257622 http.c:861              <= Recv header, 0000000043 bytes (0x0000002b)
12:41:45.258622 http.c:873              <= Recv header: Content-Type: text/html; charset=us-ascii
12:41:45.402624 run-command.c:674       trace: run_command: 'git credential-cache store'
12:41:45.402624 run-command.c:935       trace: start_command: 'C:/Program Files/Git/usr/bin/sh.exe' -c 'git credential-cache store' 'git credential-cache store'
12:41:45.553627 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:41:45.565625 git.c:476               trace: built-in: git credential-cache store
12:41:45.569625 run-command.c:674       trace: run_command: git credential-cache--daemon 'C:\Users\ch00-tctst/.cache/git/credential/socket'
12:41:45.569625 run-command.c:935       trace: start_command: git credential-cache--daemon 'C:\Users\ch00-tctst/.cache/git/credential/socket'
12:41:45.596625 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:41:45.608625 git.c:476               trace: built-in: git credential-cache--daemon 'C:\Users\ch00-tctst/.cache/git/credential/socket'
12:41:45.629629 run-command.c:674       trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning 'https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2/'
12:41:45.630632 run-command.c:935       trace: start_command: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning 'https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2/'
12:41:45.655624 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:41:45.668626 git.c:476               trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning 'https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2/'
12:41:45.670625 http.c:914              == Info: Re-using existing https: connection with host qa.cpazuredevops.domain.com
12:41:45.670625 http.c:914              == Info: Server auth using NTLM with user 'ch00-TCTST'
12:41:45.670625 http.c:861              => Send header, 0000000297 bytes (0x00000129)
12:41:45.670625 http.c:873              => Send header: POST /PLM/PLM%20Solutions/_git/LFSTest2/git-upload-pack HTTP/1.1
12:41:45.670625 http.c:873              => Send header: Host: qa.cpazuredevops.domain.com
12:41:45.671628 http.c:873              => Send header: User-Agent: git/2.50.0.windows.2
12:41:45.671628 http.c:873              => Send header: Accept-Encoding: deflate, gzip, br, zstd
12:41:45.671628 http.c:873              => Send header: Content-Type: application/x-git-upload-pack-request
12:41:45.671628 http.c:873              => Send header: Accept: application/x-git-upload-pack-result
12:41:45.671628 http.c:873              => Send header: Content-Length: 114
12:41:45.671628 http.c:873              => Send header:
12:41:45.671628 http.c:914              == Info: upload completely sent off: 114 bytes
12:41:45.690627 http.c:861              <= Recv header, 0000000017 bytes (0x00000011)
12:41:45.690627 http.c:873              <= Recv header: HTTP/1.1 200 OK
12:41:45.690627 http.c:861              <= Recv header, 0000000022 bytes (0x00000016)
12:41:45.690627 http.c:873              <= Recv header: Server: nginx/1.27.4
12:41:45.690627 http.c:861              <= Recv header, 0000000037 bytes (0x00000025)
12:41:45.691627 http.c:873              <= Recv header: Date: Wed, 09 Jul 2025 10:41:45 GMT
12:41:45.691627 http.c:861              <= Recv header, 0000000052 bytes (0x00000034)
12:41:45.691627 http.c:873              <= Recv header: Content-Type: application/x-git-upload-pack-result
qa.cpazuredevops.domain.com left intact
remote: Azure Repos
remote: Found 26 objects to send. (4 ms)
12:41:45.697627 run-command.c:674       trace: run_command: git unpack-objects --pack_header=2,26
12:41:45.697627 run-command.c:935       trace: start_command: git unpack-objects --pack_header=2,26
12:41:45.725628 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:41:45.737625 git.c:476               trace: built-in: git unpack-objects --pack_header=2,26
Unpacking objects: 100% (26/26), 5.36 KiB | 73.00 KiB/s, done.
12:41:45.818628 run-command.c:674       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
12:41:45.819634 run-command.c:935       trace: start_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
12:41:45.846628 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
12:41:45.858628 git.c:476               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
12:41:45.905631 run-command.c:674       trace: run_command: 'git-lfs filter-process'
12:41:45.906631 run-command.c:935       trace: start_command: 'C:/Program Files/Git/usr/bin/sh.exe' -c 'git-lfs filter-process' 'git-lfs filter-process'
12:41:46.238916 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
12:41:46.279922 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
12:41:46.318735 trace git-lfs: exec: git 'config' '--includes' '--local' 'lfs.repositoryformatversion'
12:41:46.361318 trace git-lfs: exec: git 'config' '--includes' '--replace-all' 'lfs.repositoryformatversion' '0'
12:41:46.404408 trace git-lfs: exec: git 'config' '--includes' '-l'
12:41:46.444324 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
12:41:46.484227 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
12:41:46.524825 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
12:41:46.566138 trace git-lfs: Install hook: pre-push, force=false, path=C:\Users\ch00-tctst\LFSTest2\.git\hooks\pre-push
12:41:46.567766 trace git-lfs: Install hook: post-checkout, force=false, path=C:\Users\ch00-tctst\LFSTest2\.git\hooks\post-checkout
12:41:46.568856 trace git-lfs: Install hook: post-commit, force=false, path=C:\Users\ch00-tctst\LFSTest2\.git\hooks\post-commit
12:41:46.569984 trace git-lfs: Install hook: post-merge, force=false, path=C:\Users\ch00-tctst\LFSTest2\.git\hooks\post-merge
12:41:46.571072 trace git-lfs: Initialize filter-process
12:41:46.572145 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
12:41:46.614409 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
12:41:46.661568 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
12:41:46.707431 trace git-lfs: tq: running as batched queue, batch size of 100
12:41:46.709082 trace git-lfs: filepathfilter: accepting "certificates - Copy - Copy.7z"
12:41:46.710178 trace git-lfs: filepathfilter: accepting "certificates - Copy.7z"
12:41:46.710178 trace git-lfs: already transferring "5529a290d231d19f7a64d1a9b57f966f48ccdc5dc656f05a108e5a83e914e35a", skipping duplicate
12:41:46.710722 trace git-lfs: filepathfilter: accepting "certificates.7z"
12:41:46.711266 trace git-lfs: already transferring "5529a290d231d19f7a64d1a9b57f966f48ccdc5dc656f05a108e5a83e914e35a", skipping duplicate
12:41:46.711814 trace git-lfs: tq: sending batch of size 1
12:41:46.712365 trace git-lfs: api: batch 1 files
12:41:46.714014 trace git-lfs: HTTP: POST https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2.git/info/lfs/objects/batch
> POST /PLM/PLM%20Solutions/_git/LFSTest2.git/info/lfs/objects/batch HTTP/1.1
> Host: qa.cpazuredevops.domain.com
> Accept: application/vnd.git-lfs+json
> Content-Length: 227
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.7.0 (GitHub; windows amd64; go 1.24.4; git 92dddf56)
>
{"operation":"download","objects":[{"oid":"5529a290d231d19f7a64d1a9b57f966f48ccdc5dc656f05a108e5a83e914e35a","size":3940}],"transfers":["lfs-standalone-file","basic","ssh"],"ref":{"name":"refs/heads/main"},"hash_algo":"sha256"}12:41:46.7
38574 trace git-lfs: HTTP: 401


< HTTP/1.1 401 Unauthorized
< Content-Length: 341
< Connection: keep-alive
< Content-Type: text/html; charset=us-ascii
< Date: Wed, 09 Jul 2025 10:41:46 GMT
< Server: nginx/1.27.4
< Www-Authenticate: Negotiate oXwweqADCgEBoQsGCSqGSIL3EgECAqJmBGRgYgYJKoZIhvcSAQICAwB+UzBRoAMCAQWhAwIBHqQRGA8yMDI1MDcwOTEwNDE0NlqlBQIDCDAbpgMCASmpDRsLRVUuTVQuTVRORVSqFzAVoAMCAQGhDjAMGwpjaDAwc2E2MjEk
<
12:41:46.739642 trace git-lfs: setting repository access to basic
12:41:46.740175 trace git-lfs: exec: git 'config' '--includes' '--replace-all' 'lfs.https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2.git/info/lfs.access' 'basic'
12:41:46.784700 trace git-lfs: api error: Authentication required: Authorization error: https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2.git/info/lfs/objects/batch
Check that you have proper access to the repository
12:41:46.785238 trace git-lfs: tq: running as batched queue, batch size of 100
12:41:46.785780 trace git-lfs: filepathfilter: accepting "certificates - Copy - Copy.7z"
Downloading certificates - Copy - Copy.7z (3.9 KB)
12:41:46.785780 trace git-lfs: tq: running as batched queue, batch size of 100
12:41:46.786327 trace git-lfs: tq: sending batch of size 1
12:41:46.786327 trace git-lfs: api: batch 1 files
12:41:46.786870 trace git-lfs: creds: git credential fill ("https", "qa.cpazuredevops.domain.com", "")
12:41:46.787411 trace git-lfs: exec: git 'credential' 'fill'
12:41:47.003492 trace git-lfs: Filled credentials for https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2
12:41:47.004032 trace git-lfs: HTTP: POST https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2.git/info/lfs/objects/batch
> POST /PLM/PLM%20Solutions/_git/LFSTest2.git/info/lfs/objects/batch HTTP/1.1
> Host: qa.cpazuredevops.domain.com
> Accept: application/vnd.git-lfs+json
> Authorization: Basic * * * * *
> Content-Length: 227
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.7.0 (GitHub; windows amd64; go 1.24.4; git 92dddf56)
>
{"operation":"download","objects":[{"oid":"5529a290d231d19f7a64d1a9b57f966f48ccdc5dc656f05a108e5a83e914e35a","size":3940}],"transfers":["lfs-standalone-file","basic","ssh"],"ref":{"name":"refs/heads/main"},"hash_algo":"sha256"}12:41:47.0
13161 trace git-lfs: HTTP: 401


< HTTP/1.1 401 Unauthorized
< Content-Length: 86
< Activityid: 28106c52-737a-48b1-9031-60dc560da215
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Wed, 09 Jul 2025 10:41:46 GMT
< Lfs-Authenticate: NTLM
< P3p: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
< Server: nginx/1.27.4
< Www-Authenticate: Bearer
< Www-Authenticate: Basic realm="https://qa.cpazuredevops.domain.com/"
< Www-Authenticate: Negotiate
< Www-Authenticate: NTLM
< X-Content-Type-Options: nosniff
< X-Powered-By: ASP.NET
< X-Tfs-Processid: a696cd9a-6ae3-4cba-8f51-ff3d17af70b7
< X-Tfs-Serviceerror: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
< X-Tfs-Session: 28106c52-737a-48b1-9031-60dc560da215
< X-Tfs-Soapexception: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Cs
oap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22
%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
< X-Vss-E2eid: 28106c52-737a-48b1-9031-60dc560da215
< X-Vss-Senderdeploymentid: cd7c01d8-9e6c-446f-b544-d6833c636053
<
12:41:47.016416 trace git-lfs: exec: git 'credential' 'reject'
12:41:47.224832 trace git-lfs: api: http response indicates "basic" authentication. Resubmitting...
12:41:47.224832 trace git-lfs: creds: git credential fill ("https", "qa.cpazuredevops.domain.com", "")
12:41:47.225379 trace git-lfs: exec: git 'credential' 'fill'
error: unable to read askpass response from 'C:/Program Files/Git/mingw64/bin/git-askpass.exe'
Password for 'https://Taem%230394~@qa.cpazuredevops.domain.com':
13:18:14.827740 trace git-lfs: Filled credentials for https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2
13:18:14.827740 trace git-lfs: HTTP: POST https://qa.cpazuredevops.domain.com/PLM/PLM%20Solutions/_git/LFSTest2.git/info/lfs/objects/batch
> POST /PLM/PLM%20Solutions/_git/LFSTest2.git/info/lfs/objects/batch HTTP/1.1
> Host: qa.cpazuredevops.domain.com
> Accept: application/vnd.git-lfs+json
> Authorization: Basic * * * * *
> Content-Length: 227
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.7.0 (GitHub; windows amd64; go 1.24.4; git 92dddf56)
>
{"operation":"download","objects":[{"oid":"5529a290d231d19f7a64d1a9b57f966f48ccdc5dc656f05a108e5a83e914e35a","size":3940}],"transfers":["lfs-standalone-file","basic","ssh"],"ref":{"name":"refs/heads/main"},"hash_algo":"sha256"}{"operatio
n":"download","objects":[{"oid":"5529a290d231d19f7a64d1a9b57f966f48ccdc5dc656f05a108e5a83e914e35a","size":3940}],"transfers":["lfs-standalone-file","basic","ssh"],"ref":{"name":"refs/heads/main"},"hash_algo":"sha256"}13:18:14.846632 trac
e git-lfs: HTTP: 401


< HTTP/1.1 401 Unauthorized
< Content-Length: 86
< Activityid: 28105a54-737a-48b1-9031-60dc560da215
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Wed, 09 Jul 2025 11:18:14 GMT
< Lfs-Authenticate: NTLM
< P3p: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
< Server: nginx/1.27.4
< Www-Authenticate: Bearer
< Www-Authenticate: Basic realm="https://qa.cpazuredevops.domain.com/"
< Www-Authenticate: Negotiate
< Www-Authenticate: NTLM
< X-Content-Type-Options: nosniff
< X-Powered-By: ASP.NET
< X-Tfs-Processid: a696cd9a-6ae3-4cba-8f51-ff3d17af70b7
< X-Tfs-Serviceerror: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
< X-Tfs-Session: 28105a54-737a-48b1-9031-60dc560da215
< X-Tfs-Soapexception: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Cs
oap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22
%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
< X-Vss-E2eid: 28105a54-737a-48b1-9031-60dc560da215
< X-Vss-Senderdeploymentid: cd7c01d8-9e6c-446f-b544-d6833c636053
<
13:18:14.850415 trace git-lfs: exec: git 'credential' 'reject'
13:18:15.063357 trace git-lfs: api: http response indicates "basic" authentication. Resubmitting...
13:18:15.063357 trace git-lfs: creds: git credential fill ("https", "qa.cpazuredevops.domain.com", "")
13:18:15.063905 trace git-lfs: exec: git 'credential' 'fill'

What’s more interesting when I try to use PAT instead of basic authentication (negotiate/NTLM) it works fine without issues. But not if opposite.

That’s my Nginx conf below:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
error_log  /var/log/nginx/error.log info;
#error_log  /var/log/nginx/error.log debug;

pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

#SSH Block

stream {
    upstream ssh_backend {
    least_conn;
    server ch00sa621.domain.com:22;
    server ch00sa700.domain.com:22;
    }

    server {
    listen                      22;
    proxy_pass                  ssh_backend;
    proxy_timeout               1h;
    proxy_connect_timeout       600s;
    }
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" "http_user_agent" '
                    '$request_length $request_time $upstream_addr '
                      '$upstream_response_length $upstream_response_time $upstream_status ';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout           65;
    proxy_read_timeout          1800s;
    proxy_send_timeout          1800s;
    proxy_connect_timeout       60s;

    proxy_next_upstream error timeout http_502 http_504;
    proxy_next_upstream_tries 3;

    #gzip  on;
    server_names_hash_bucket_size               128;
    client_max_body_size                        200M;

    include /etc/nginx/conf.d/*.conf;


    upstream qa_azure_devops {
        ip_hash;
#       least_conn;
        server ch00sa621.domain.com:443 max_fails=3 fail_timeout=30s;
        server ch00sa700.domain.com:443 max_fails=3 fail_timeout=30s;
        ntlm;
        }

    server {
        listen       443 ssl;
        server_name  qa.cpazuredevops.domain.com;
        add_header      Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
        client_body_buffer_size 200M;

        ssl_certificate /etc/pki/tls/certs/qa-cpazuredevops-swdc-bundle.cer;
        ssl_certificate_key /etc/pki/tls/private/qa-cpazuredevops-swdc.key;
        ssl_protocols   TLSv1.2 TLSv1.3;
        ssl_session_timeout 4h;
        ssl_session_cache builtin:1000 shared:SSL:50m;
        ssl_session_tickets off;
        proxy_ssl_trusted_certificate /etc/pki/tls/misc/bundled.pem;
        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;
        proxy_request_buffering off;
        proxy_buffering off;
        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            proxy_pass https://qa_azure_devops;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Authorization $http_authorization;
            proxy_set_header Proxy-Connection 'Keep-Alive';
            proxy_pass_header Authorization;
            proxy_set_header X-Forwarded-Proto $scheme;
            add_header X-Upstream-Server $upstream_addr;
            proxy_ssl_server_name on;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_pass_request_headers on;
        #New settings below!!!
            client_max_body_size 100G;
            proxy_read_timeout 300;
            proxy_send_timeout 300;

            proxy_buffer_size 128k;
            proxy_buffers 4 256k;
            proxy_busy_buffers_size 256k;
            lingering_close on;
            proxy_request_buffering off;
            proxy_buffering off;
        }
}
}

Can someone please advice what might be wrong?

2

Hey @dabrowsk1! I don’t really see where the issue might be. If PATs work and ntlm does not, there might be some underlying limitation that I am unaware of. Given you are using NGINX Plus I would suggest you reach out to F5 support for assistance :slightly_smiling_face:

3

Hi @alessandro

Thank you so much for quick reply.
The problem is that they have proposed the solution which I tried already so I entered here :frowning:

Problem is related with Nginx as when I try to run GIT LFS clone bahind load balancer everything gets completed without any issues (no prompts for credentials).

Perhaps problem is related in authentication headers that are coming through Nginx but no idea what is really going despite of tracing enabled :frowning:

4

Hey again! Did you let the support team know that their proposed solution doesn’t work?

5

Hi. Yes I did. So far no reply.

6

I’ve sent you a DM to follow up on more details. In the meantime, could you try removing your load balancing setup and proxy to a single upstream server? Just in case the connection to the GIT LFS upstream is not being handled properly and the credentials are getting cached for one of the upstreams but not the other, which in turn is causing some sort of auth loop.

7

Hi @alessandro

Thank you so much !

So when I only run GIT LFS clone directly via FQDN of my backend server (I have two servers as mentioned above) everything works basically OK. I don’t get any checkout issues. Of course during first attempts he is being bounced with 401 but that’s because he tries first to authenticate as anonymous what of course won’t work but finally when using negotiate he is accesing the resources properly and clone operation is ended as expected.
Please follow the log below:

GIT_Backend_log.txt (103.8 KB)

8

That is good to know and confirms that there is some issue with authenticating and load balancing between both servers.