NGINX uploads entire 2GB request body even though auth_request returns 401

Please use this template for troubleshooting questions.

My issue:

I have a service which accepts file uploads up to 25GB. Uploads work as expected with one exception. We use the auth_request module which returns 200 if a token in the request is still valid and 401 otherwise. An error_page location receives the request that got the 401 auth response and that tries to use a refresh token if its present and returns 401 if not. The problem I’m having is that the entire file uploads even though the auth_request endpoint knows with just the headers that the request is not authenticated and to return the 401 status.

I’m hoping someone can identify a mistake in my configuration that would explain why the entire request body uploads regardless of the authentication status of the request.

How I encountered the problem:

If i make a request with a deliberately expired token I notice that the entire file uploads to nginx before the 401 response is returned.

Solutions I’ve tried:

Version of NGINX or NGINX adjacent software (e.g. NGINX Gateway Fabric):

I run the official docker container tag nginx:1.28.0-bookworm in kubernetes.

Deployment environment:

kubernetes and AWS

Minimal NGINX config to reproduce your issue (preferably running on https://tech-playground.com/playgrounds/nginx for ease of debugging, and if not as a code block): (Tip → Run nginx -T to print your entire NGINX config to your terminal.)

These are the most relevant parts of my nginx.conf.

    location = /auth {
        internal;
        client_max_body_size 0;
        access_log off;
        set $gateway_service_uri http://localhost:8080;
        proxy_pass $gateway_service_uri;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
        proxy_set_header X-Original-URI https://$http_host$request_uri;
        proxy_set_header X-Original-Method $request_method;
      }

    location @unauthorized {
        internal;
        client_max_body_size 0;
        access_log off;
        set $gateway_service_uri http://localhost:8080;
        proxy_pass $gateway_service_uri/auth/unauthorized;
        include proxy-host-header.conf;
        include forwarded-request-headers.conf;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
        proxy_set_header X-Original-URI https://$http_host$request_uri;
        proxy_set_header X-Original-Method $request_method;
      }

    location = /api/v1/fs/upload {
        client_max_body_size  25g;
        proxy_read_timeout 1200s;
        proxy_send_timeout 1200s;
        proxy_request_buffering off;
        auth_request /auth;
        error_page 401 = @unauthorized;
        set $service_uri http://filestore-service.default.svc.cluster.local:8080;
        proxy_pass $service_uri;
      }

NGINX access/error log: (Tip → You can usually find the logs in the /var/log/nginx directory.)

hello @Dan_Finucane for token validation by auth_request what you are using keycloak or your service also in nginx.conf change log info to log debug and share the log

Thanks for taking a look!

The /auth and /auth/unauthorized are both Java services running on the same host as nginx. They are well tested and make no network requests so they are fast. /auth basically just validates the token which is either a Bearer token in the authorization header or a cookie that contains the same token. /auth/unauthorized checks if there is a refresh token in a cookie and if so it attempts to use that to get new tokens so this does make a network call if there is a refresh token but that is very fast and that does hit a keycloak endpoint off host. All this is working fine but I wanted you to have the details since you asked.

I enabled debug logging with error_log stderr debug; but it doesn’t provide much so maybe i’m not doing everything needed to enable debug logging. I get this on startup

"2025/11/21 13:50:05 [notice] 1#1: start worker process 7"
"2025/11/21 13:50:05 [notice] 1#1: start worker processes"
"2025/11/21 13:50:05 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65536:1048576"
"2025/11/21 13:50:05 [notice] 1#1: OS: Linux 6.1.124"
"2025/11/21 13:50:05 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)"
"2025/11/21 13:50:05 [notice] 1#1: nginx/1.28.0"
"2025/11/21 13:50:05 [notice] 1#1: using the ""epoll"" event method"

and this line every second or so

2025/11/21 14:05:32 [info] 7#7: *2719 client closed connection while waiting for request, client: 10.251.205.108, server: 0.0.0.0:8081

but nothing else even when i push files through. Am I missing something?

I’m uploading my complete nginx.conf and the supporting include files.

Hopefully there is some explanation why proxy_request_buffering off; is being ignored.

proxy-host-header.conf (424 Bytes)

original-request-headers.conf (115 Bytes)

nginx.conf (13.3 KB)

forwarded-request-headers.conf (584 Bytes)

common-response-headers.conf (275 Bytes)

The nginx.conf was named nginx.conf.template and it has its environment variables substituted to become nginx.conf but i just renamed it so i could upload it.

OK i see why debug logging isn’t working. I’ll have to enable it in the docker container.

nginx version: nginx/1.28.0
built by gcc 12.2.0 (Debian 12.2.0-14) 
built with OpenSSL 3.0.15 3 Sep 2024 (running with OpenSSL 3.0.17 1 Jul 2025)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/run/nginx.pid --lock-path=/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/home/builder/debuild/nginx-1.28.0/debian/debuild-base/nginx-1.28.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

I’ll work on that.

Heya!

Can you try setting proxy_request_buffering to on and let me know if anything changes? Per the docs, When buffering is disabled, the request body is sent to the proxied server immediately as it is received., so there is a chance this might be why the request is being sent/the file is getting uploaded even on a 401.

Let us know how it goes!

I want it to send the request body immediately because the /auth and /auth/unauthorized endpoints will see the headers in the first few thousand bytes and then return the 401 or 200. What it is doing is uploading the entire request body to disk i guess before it passes anything to the upstreams. So with that a client uploads a 2GB or 20GB file only to find they are not logged on or to receive a 307 request with a new token and then upload the file again.

Here is a debug log. I’m still trying to make sense of it but i figured I’d give it to you guys since you have more experience with it.

nginx-debug.log (7.6 MB)

With all of these on the http scope I expect them to be inherited by every location but I’m wondering of something I’m doing is making that not happen

  proxy_http_version      1.1;
  proxy_set_header        Connection "";
  proxy_request_buffering off;
  proxy_buffer_size             128k;
  proxy_buffers                 4 256k;
  proxy_busy_buffers_size       256k;
  large_client_header_buffers   4 16k;

I’ve noticed that the Connection header is not being cleared on the upstream requests and remains Connection: close as well so maybe this is the problem.

The change to put the proxy_ directives on every location did solve the problem where now /auth and @authorize no longer have a Connection header but unfortunately nginx is still buffering the request body before passing to the upstream.

I have asked internally and I have been told that this is ultimately because NGINX needs to process the entire request before being able to return a status code, which is why the upload goes through even when the request auth fails.

Finding a solution might be a bit tricky. Ideally you’d be able to use something such as the Expect header but NGINX does not support it natively. I’ll keep asking around and let you know if I get any further suggestions!

Thank you so much. It has been so hard to find any conclusive information on this so i really appreciate your efforts. If it is not possible to return an error without processing the entire request that’s a little surprising but at least knowing that allows me to stop trying to get it to work. I would be interested in alternatives or workarounds so if you do come by any please pass it along.

I have some more information that clarifies things a bit. I see now that the auth_request endpoint /auth is called right at the beginning of the request and within milliseconds it returns its 401 status response but the error_page @unauthorized is not invoked for over 4 minutes presumably because nginx is first uploading the entire request body of the 20GB file.

My postman request is submitted at 8:39:53 AM EST. The auth_request /auth handler is invoked at 8:39:53 AM EST and completes in 3ms returning a 401 status

2025-11-22 13:39:53,897 | grizzly-http-server-11              | TRACE | server.jersey.EventSource:624 | The server successfully processed a REST request.  Context{resource='AuthResource.verify', latency(ms)=3}
Request:
	Request{method=GET, uri='http://filestore-dev.xylem-vue.com:80/auth'}
Request Headers:
	host: [filestore-dev.xylem-vue.com]
	x-forwarded-for: [136.61.7.72]
	x-forwarded-proto: [https]
	x-forwarded-port: [443]
	x-original-uri: [https://filestore-dev.xylem-vue.com/api/v1/fs/upload]
	x-original-method: [PUT]
	x-amzn-trace-id: [Root=1-6921bd29-430a1b4a585af9b8745078bd]
	content-type: [multipart/form-data; boundary=--------------------------292971603214225699847952]
	accept: [application/json]
	authorization: *** REDACTED ***
	user-agent: [PostmanRuntime/7.49.1]
	postman-token: [516a2819-ccd1-4251-ae5b-9343c21d8dca]
	accept-encoding: [gzip, deflate, br]
Response:
	Response{status=401, reason=Unauthorized}

Here is the /api/v1/fs/upload location

    location = /api/v1/fs/upload {
        proxy_http_version        1.1;
        proxy_set_header          Connection "";
        proxy_request_buffering   off;
        proxy_buffering           off;
        proxy_max_temp_file_size  0;
        proxy_buffer_size         128k;
        proxy_buffers             4 256k;
        proxy_busy_buffers_size   256k;

        client_max_body_size  25g;
        proxy_read_timeout 1200s;
        proxy_send_timeout 1200s;

        auth_request /auth;
        error_page 401 = @unauthorized;
        set $service_uri ${FILESTORE_SERVICE_URI};
        proxy_pass $service_uri;
        include proxy-host-header.conf;
        include forwarded-request-headers.conf;
        include common-response-headers.conf;
      }

so the auth_request 401 is to send the request to the @unauthorized error_page and that location is configured similarly to the /auth endpoint so that the request body is not included. Here is that location

    location @unauthorized {
        internal;

        proxy_http_version        1.1;
        proxy_set_header          Connection "";
        proxy_request_buffering   off;
        proxy_buffering           off;
        proxy_max_temp_file_size  0;
        proxy_buffer_size         128k;
        proxy_buffers             4 256k;
        proxy_busy_buffers_size   256k;

        client_max_body_size 0;
        access_log off;
        set $gateway_service_uri ${FILESTORE_GATEWAY_SERVICE_URI};
        proxy_pass $gateway_service_uri/auth/unauthorized;
        include proxy-host-header.conf;
        include forwarded-request-headers.conf;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
        include original-request-headers.conf;
      }

but that service is not invoked for 4 minutes as this log message shows

2025-11-22 13:44:25,162 | grizzly-http-server-11              | TRACE | server.jersey.EventSource:624 | The server successfully processed a REST request.  Context{resource='AuthResource.unauthorizedPut', latency(ms)=0}
Request:
	Request{method=PUT, uri='http://filestore-dev.xylem-vue.com:80/auth/unauthorized'}
Request Headers:
	host: [filestore-dev.xylem-vue.com]
	x-forwarded-for: [136.61.7.72]
	x-forwarded-proto: [https]
	x-forwarded-port: [443]
	x-original-uri: [https://filestore-dev.xylem-vue.com/api/v1/fs/upload]
	x-original-method: [PUT]
	x-amzn-trace-id: [Root=1-6921bd29-430a1b4a585af9b8745078bd]
	content-type: [multipart/form-data; boundary=--------------------------292971603214225699847952]
	accept: [application/json]
	authorization: *** REDACTED ***
	user-agent: [PostmanRuntime/7.49.1]
	postman-token: [516a2819-ccd1-4251-ae5b-9343c21d8dca]
	accept-encoding: [gzip, deflate, br]
Response:
	Response{status=401, reason=Unauthorized}

when it is invoked it takes less than 1ms to complete and returns a 401 status at which point nginx returns the 401 to postman.

Why is nginx reading the whole request (i’m assuming that is what is happing during those 4 minutes) when the @unauthorized location is configured to not get the body?

I have a hard time believing that nginx wants to read the entire request body when it knows right away that the upstream is never going to be sent it. The HTTP spec doesn’t require that the whole request be read so I don’t understand why this is happening.