Http3 with stream proxies?

Projects NGINX
1

Please use this template for troubleshooting questions.

My issue:

I am trying to configure http3 on my webserver.

How I encountered the problem:

Solutions I’ve tried:

First, my webserver listens on port 443 using stream module and dispatches based on sni to one of three variants:

  • sites served by nginx locally (all these listen in fact on 127.0.0.1:444, and both ends of the local connection use proxy_protocol), using static information or proxying upstream,
  • connecting to other services in my network by passing the tcp stream without terminating tls.

None of the second flavor does support http3.
Now when I add listen 443 quic; to one those site served locally, restart nginx, and test using https://http3checker.com/ it reports success.

Now Support for QUIC and HTTP/3 tells “Along with the quic parameter it is also possible to specify the reuseport parameter to make it work properly with multiple workers.” However if I add reuseport (i.e. listen 443 quic reuseport;) I get the error message
nginx: [emerg] duplicate listen options for 0.0.0.0:443 in /etc/nginx/nginx.conf:433

I suspect, listen 443 quic does not tell nginx to listen on tcp only, correct? but then why does it work without reuseport?

In another iteration I am trying to mirror the sni (though unneeded). In my stream section I use

server {
    listen [::]:443 udp ipv6only=off reuseport;
    proxy_timeout 20s;
    proxy_pass web;
}

where web is 127.0.0.1:444, and I changed the listen above to listen 444 quic reuseport;
https://http3checker.com/ again reports success. But at least the proxy_timeout above feels strange, and I think I am loosing the proxy_protocol and thus ip addresses in logs, correct?

Version of NGINX or NGINX adjacent software (e.g. NGINX Gateway Fabric):

nginx 1.29.7

Deployment environment:

ubuntu 24.04.4

Minimal NGINX config to reproduce your issue (preferably running on https://tech-playground.com/playgrounds/nginx for ease of debugging, and if not as a code block): (Tip → Run nginx -T to print your entire NGINX config to your terminal.)

haven´t tried this so far.

NGINX access/error log: (Tip → You can usually find the logs in the /var/log/nginx directory.)

Ideally I don´t want to have this proxying step as it is unnecessary and looses the correct IP address. I am dreaming of something like listen 443 quiconly reuseport;

Any thoughts?

1 Like
2

Some general info…

I suspect, listen 443 quic does not tell nginx to listen on tcp only, correct?

QUIC is a UDP based protocol, so yeah, there is no TCP involved. E.g.

nginx with two worker processes with http/2.0 and http/3.0 (QUIC) enabled with reuseport listening on [::1]:8080


$ ss -tunlp | grep nginx
udp   UNCONN 0      0                                [::1]:8080          [::]:*    users:(("nginx",pid=348134,fd=7),("nginx",pid=348133,fd=7),("nginx",pid=348132,fd=7))
udp   UNCONN 0      0                                [::1]:8080          [::]:*    users:(("nginx",pid=348134,fd=9),("nginx",pid=348133,fd=9),("nginx",pid=348132,fd=9))
tcp   LISTEN 0      511                              [::1]:8080          [::]:*    users:(("nginx",pid=348134,fd=8),("nginx",pid=348133,fd=8),("nginx",pid=348132,fd=8))

There we see the two QUIC sockets listening on UDP (two due to reuseport) as well as a single TCP socket for HTTP/1.1/2.0

nginx: [emerg] duplicate listen options for 0.0.0.0:443 in /etc/nginx/nginx.conf:433

Kind of means what it says, in fact let me just quote the documentation

The listen directive can have several additional parameters specific to socket-related system calls. These parameters can be specified in any listen directive, but only once for a given address:port pair.

but only once for a given address:port pair.

Is the key bit here. So for example do you have ipv6only=off or reuseport in multiple listen directives for a given address:port pair?

3

Thanks for clarification. But I still don´t get, why just adding “reuseport” to “listen 443 quic;” in a single location - the one and only listen 443 quic - creates an issue.

4

If you only added it to a single listen directive then it shouldn’t be an issue. Here’s the config I tested with

daemon off;

events {}

pid nginx.pid;

worker_processes 2;

http {
    access_log /dev/stdout;
    error_log /dev/stderr;

    server {
	    server_name	localhost;
	    listen		[::1]:8080 quic reuseport;
	    listen		[::1]:8080 ssl;

	    http2 on;

	    ssl_certificate		/path/to/cert.pem;
	    ssl_certificate_key	/path/to/key.pem;

        location / {
		    add_header	Alt-Svc 'h3=":8080"; ma=86400';

		    root		/srv/nginx;
        }
    }
}
5

which declares only one http server using quic. As I explained initially, I am running multiple using SNI, all on the same port.

The issue is caused by https://stackoverflow.com/questions/76348128/enabling-quic-http-3-on-multiple-domains-with-nginx-1-25.

6

It sounds like the issue there was using reuseport multiple times. Seeing as it’s a socket option it should only be specified once per address:port pair.

Here’s an example config with two server sections each doing quic.

daemon off;

worker_processes 2;

events {}

pid nginx.pid;

http {
    access_log /dev/stdout;
    error_log /dev/stderr;

    ssl_certificate		    /etc/letsencrypt/live/fullchain.pem;
    ssl_certificate_key		/etc/letsencrypt/live/privkey.pem;

    server {
        server_name	server-1;
	    listen		[::]:443 quic reuseport;

	    root		/srv/nginx/test1;
    }

    server {
	    server_name	server-2;
	    listen      [::]:443 quic;

        root		/srv/nginx/test2;
   }
}

So we have two worker processes

nobody      4417    4416  0 Apr08 pts/1    00:00:00 nginx: worker process
nobody      4418    4416  0 Apr08 pts/1    00:00:00 nginx: worker process

Each has two UDP listen QUIC sockets (due to reuseport)

$ sudo ss -tunlp | grep nginx
udp   UNCONN 0      0                               [::]:443          [::]:*    users:(("nginx",pid=4418,fd=7),("nginx",pid=4417,fd=7),("nginx",pid=4416,fd=7))
udp   UNCONN 0      0                               [::]:443          [::]:*    users:(("nginx",pid=4418,fd=8),("nginx",pid=4417,fd=8),("nginx",pid=4416,fd=8))

For comparison, without reuseport

$ sudo ss -tunlp | grep nginx
udp   UNCONN 0      0                               [::]:443          [::]:*    users:(("nginx",pid=4554,fd=7),("nginx",pid=4553,fd=7),("nginx",pid=4552,fd=7))

There is just a single listen socket shared between them.

7

I think that is a bad approach. I want to declare what I want for the specific server rather than think about what is implemented and optimize for that. Therefore nginx should enforce the same settings on all listens where necessary and warn about inconsistent usage of all this attributes rather than provide a misleading error message.

I got this to working in the meantime, but am struggling with testing. https://http3checker.com/ doesn´t allow me to test all my addresses, https://http3check.net/ sometimes fails without specific reason.
Any recommendation for better diagnostics?