HTTP/3 QUIC - Very poor throughput with high latency, high bandwidth connections

Troubleshooting
1

Hello,

I have been experimenting with the new HTTP/3 quic feature in nginx,

I have noticed that as latency increases, the throughput of download speed seems to exponentially decrease. H2 traffic over TCP does not have the same effect.

At a ping of around 270ms, I see nginx’s quic implementation crawl to a download speed of 200KB per second.

Here is my example: Two servers in data centers, both with 10 gigabit internet connections. One in Australia, the other in the UK. iperf3 tests over UDP yields 700mbits per second between them both ways.

H2 over TCP = 42 MBytes per second. Downloads 1G in 27 seconds.

curl https://www.afamsterdam.nl/1GB.iso > /dev/null

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1024M 100 1024M 0 0 36.8M 0 0:00:27 0:00:27 --:–:-- 42.5M

H3 over UDP = 240KB per second. Did not wait for test to finish (1.5 hours remaining).

curl --http3 https://www.afamsterdam.nl/1GB.iso > /dev/null

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 1024M 0 7460k 0 0 226k 0 1:16:59 0:00:32 1:16:27 244k^C

When I run a http3 test from a low latency, the speed is similar, though still a bit slower than TCP.

This slowness presents itself in chrome too (1MB images take 9 seconds to download over 1 second on H2). Nothing was seemingly unusual in a chrome export quic JSON dump.

Any ideas?

nginx version: nginx/1.25.2
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 3.0.10+quic 1 Aug 2023
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-http_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-compat --add-dynamic-module=…/ngx_brotli --add-module=/tmp/modsecurity_nginx/modsecurity-nginx-v1.0.3 --with-openssl=…/openssl-openssl-3.0.10-quic1 --with-cc-opt=‘-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.25.2/debian/debuild-base/nginx-1.25.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC’ --with-ld-opt=‘-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,–as-needed -pie’

3 Likes
2

Yeah been many reports of the same online. I still consider Nginx QUIC HTTP/3 implement experimental, so still needs improvements.

Might want to first update to Nginx 1.26.3 or 1.27.4 for bug fixes and security fixes including QUIC fixes

Changes with nginx 1.27.4                                        05 Feb 2025

    *) Security: insufficient check in virtual servers handling with TLSv1.3
       SNI allowed to reuse SSL sessions in a different virtual server, to
       bypass client SSL certificates verification (CVE-2025-23419).

    *) Feature: the "ssl_object_cache_inheritable", "ssl_certificate_cache",
       "proxy_ssl_certificate_cache", "grpc_ssl_certificate_cache", and
       "uwsgi_ssl_certificate_cache" directives.

    *) Feature: the "keepalive_min_timeout" directive.

    *) Workaround: "gzip filter failed to use preallocated memory" alerts
       appeared in logs when using zlib-ng.

    *) Bugfix: nginx could not build libatomic library using the library
       sources if the --with-libatomic=DIR option was used.

    *) Bugfix: QUIC connection might not be established when using 0-RTT;
       the bug had appeared in 1.27.1.

    *) Bugfix: nginx now ignores QUIC version negotiation packets from
       clients.

    *) Bugfix: nginx could not be built on Solaris 10 and earlier with the
       ngx_http_v3_module.

    *) Bugfixes in HTTP/3.


Changes with nginx 1.27.3                                        26 Nov 2024

    *) Feature: the "server" directive in the "upstream" block supports the
       "resolve" parameter.

    *) Feature: the "resolver" and "resolver_timeout" directives in the
       "upstream" block.

    *) Feature: SmarterMail specific mode support for IMAP LOGIN with
       untagged CAPABILITY response in the mail proxy module.

    *) Change: now TLSv1 and TLSv1.1 protocols are disabled by default.

    *) Change: an IPv6 address in square brackets and no port can be
       specified in the "proxy_bind", "fastcgi_bind", "grpc_bind",
       "memcached_bind", "scgi_bind", and "uwsgi_bind" directives, and as
       client address in ngx_http_realip_module.

    *) Bugfix: in the ngx_http_mp4_module.
       Thanks to Nils Bars.

    *) Bugfix: the "so_keepalive" parameter of the "listen" directive might
       be handled incorrectly on DragonFly BSD.

    *) Bugfix: in the "proxy_store" directive.


Changes with nginx 1.27.2                                        02 Oct 2024

    *) Feature: SSL certificates, secret keys, and CRLs are now cached on
       start or during reconfiguration.

    *) Feature: client certificate validation with OCSP in the stream
       module.

    *) Feature: OCSP stapling support in the stream module.

    *) Feature: the "proxy_pass_trailers" directive in the
       ngx_http_proxy_module.

    *) Feature: the "ssl_client_certificate" directive now supports
       certificates with auxiliary information.

    *) Change: now the "ssl_client_certificate" directive is not required
       for client SSL certificates verification.


Changes with nginx 1.27.1                                        14 Aug 2024

    *) Security: processing of a specially crafted mp4 file by the
       ngx_http_mp4_module might cause a worker process crash
       (CVE-2024-7347).
       Thanks to Nils Bars.

    *) Change: now the stream module handler is not mandatory.

    *) Bugfix: new HTTP/2 connections might ignore graceful shutdown of old
       worker processes.
       Thanks to Kasei Wang.

    *) Bugfixes in HTTP/3.


Changes with nginx 1.27.0                                        29 May 2024

    *) Security: when using HTTP/3, processing of a specially crafted QUIC
       session might cause a worker process crash, worker process memory
       disclosure on systems with MTU larger than 4096 bytes, or might have
       potential other impact (CVE-2024-32760, CVE-2024-31079,
       CVE-2024-35200, CVE-2024-34161).
       Thanks to Nils Bars of CISPA.

    *) Feature: variables support in the "proxy_limit_rate",
       "fastcgi_limit_rate", "scgi_limit_rate", and "uwsgi_limit_rate"
       directives.

    *) Bugfix: reduced memory consumption for long-lived requests if "gzip",
       "gunzip", "ssi", "sub_filter", or "grpc_pass" directives are used.

    *) Bugfix: nginx could not be built by gcc 14 if the --with-libatomic
       option was used.
       Thanks to Edgar Bonet.

    *) Bugfixes in HTTP/3.


Changes with nginx 1.26.3                                        05 Feb 2025

    *) Security: insufficient check in virtual servers handling with TLSv1.3
       SNI allowed to reuse SSL sessions in a different virtual server, to
       bypass client SSL certificates verification (CVE-2025-23419).

    *) Bugfix: in the ngx_http_mp4_module.
       Thanks to Nils Bars.

    *) Workaround: "gzip filter failed to use preallocated memory" alerts
       appeared in logs when using zlib-ng.

    *) Bugfix: nginx could not build libatomic library using the library
       sources if the --with-libatomic=DIR option was used.

    *) Bugfix: nginx now ignores QUIC version negotiation packets from
       clients.

    *) Bugfix: nginx could not be built on Solaris 10 and earlier with the
       ngx_http_v3_module.

    *) Bugfixes in HTTP/3.


Changes with nginx 1.26.2                                        14 Aug 2024

    *) Security: processing of a specially crafted mp4 file by the
       ngx_http_mp4_module might cause a worker process crash
       (CVE-2024-7347).
       Thanks to Nils Bars.


Changes with nginx 1.26.1                                        29 May 2024

    *) Security: when using HTTP/3, processing of a specially crafted QUIC
       session might cause a worker process crash, worker process memory
       disclosure on systems with MTU larger than 4096 bytes, or might have
       potential other impact (CVE-2024-32760, CVE-2024-31079,
       CVE-2024-35200, CVE-2024-34161).
       Thanks to Nils Bars of CISPA.

    *) Bugfix: reduced memory consumption for long-lived requests if "gzip",
       "gunzip", "ssi", "sub_filter", or "grpc_pass" directives are used.

    *) Bugfix: nginx could not be built by gcc 14 if the --with-libatomic
       option was used.
       Thanks to Edgar Bonet.

    *) Bugfix: in HTTP/3.


Changes with nginx 1.26.0                                        23 Apr 2024

    *) 1.26.x stable branch.


Changes with nginx 1.25.5                                        16 Apr 2024

    *) Feature: virtual servers in the stream module.

    *) Feature: the ngx_stream_pass_module.

    *) Feature: the "deferred", "accept_filter", and "setfib" parameters of
       the "listen" directive in the stream module.

    *) Feature: cache line size detection for some architectures.
       Thanks to Piotr Sikora.

    *) Feature: support for Homebrew on Apple Silicon.
       Thanks to Piotr Sikora.

    *) Bugfix: Windows cross-compilation bugfixes and improvements.
       Thanks to Piotr Sikora.

    *) Bugfix: unexpected connection closure while using 0-RTT in QUIC.
       Thanks to Vladimir Khomutov.


Changes with nginx 1.25.4                                        14 Feb 2024

    *) Security: when using HTTP/3 a segmentation fault might occur in a
       worker process while processing a specially crafted QUIC session
       (CVE-2024-24989, CVE-2024-24990).

    *) Bugfix: connections with pending AIO operations might be closed
       prematurely during graceful shutdown of old worker processes.

    *) Bugfix: socket leak alerts no longer logged when fast shutdown was
       requested after graceful shutdown of old worker processes.

    *) Bugfix: a socket descriptor error, a socket leak, or a segmentation
       fault in a worker process (for SSL proxying) might occur if AIO was
       used in a subrequest.

    *) Bugfix: a segmentation fault might occur in a worker process if SSL
       proxying was used along with the "image_filter" directive and errors
       with code 415 were redirected with the "error_page" directive.

    *) Bugfixes and improvements in HTTP/3.


Changes with nginx 1.25.3                                        24 Oct 2023

    *) Change: improved detection of misbehaving clients when using HTTP/2.

    *) Feature: startup speedup when using a large number of locations.
       Thanks to Yusuke Nojima.

    *) Bugfix: a segmentation fault might occur in a worker process when
       using HTTP/2 without SSL; the bug had appeared in 1.25.1.

    *) Bugfix: the "Status" backend response header line with an empty
       reason phrase was handled incorrectly.

    *) Bugfix: memory leak during reconfiguration when using the PCRE2
       library.
       Thanks to ZhenZhong Wu.

    *) Bugfixes and improvements in HTTP/3.


Changes with nginx 1.25.2                                        15 Aug 2023

    *) Feature: path MTU discovery when using HTTP/3.

    *) Feature: TLS_AES_128_CCM_SHA256 cipher suite support when using
       HTTP/3.

    *) Change: now nginx uses appname "nginx" when loading OpenSSL
       configuration.

    *) Change: now nginx does not try to load OpenSSL configuration if the
       --with-openssl option was used to built OpenSSL and the OPENSSL_CONF
       environment variable is not set.

    *) Bugfix: in the $body_bytes_sent variable when using HTTP/3.

    *) Bugfix: in HTTP/3.

Tried playing with http3_stream_buffer_size yet ?

Curious if you tried newer OpenSSL 3.2+ QUIC as OpenSSL 3.0 does have performance regressions.

Also how was curl’s HTTP/3 support built ? Which crypto library and version used ? Might want to ensure curl’s HTTP/3 binary is up to date too.

1 Like
3

Looks like you posted the question a year ago on https://forum.nginx.org/read.php?11,298079,298079 too?

and other folks are reporting the same HTTP3 speed tuning

1 Like
4

Hi George,

Yep I posted the question a year ago and wanted to move it to here on the new forum. Because its still an issue.

The web server in the test above is running 1.27.3 with OpenSSL 3.4 already.
Here is my current Dockerfile that this server is running:

# Compile nginx (need to compiled so we can add the brotli and mod security module)
RUN apt-get install build-essential zlib1g zlib1g-dev libssl-dev libgd-dev libxml2 libxml2-dev uuid-dev libxslt-dev -y
WORKDIR /tmp/nginx-src
RUN wget https://nginx.org/download/nginx-1.27.3.tar.gz
RUN tar -xvzf nginx-*.tar.gz
RUN git clone https://github.com/google/ngx_brotli.git --recursive

# OpenSSL
RUN wget https://github.com/openssl/openssl/releases/download/openssl-3.4.0/openssl-3.4.0.tar.gz
RUN tar -xvzf openssl-*.tar.gz && rm openssl-*.tar.gz

RUN cd nginx-* && ./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module \
 --with-http_v3_module --with-http_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-compat \
 --add-dynamic-module=../ngx_brotli --add-module=/tmp/modsecurity_nginx/modsecurity-nginx-v1.0.3 \
 --with-openssl=../openssl-3.4.0 \
 --with-pcre-jit \
 --with-cc-opt='-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.27.3/debian/debuild-base/nginx-1.27.3=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wp,-fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
RUN cd nginx-* && make && make install
RUN cd nginx-* && make modules
RUN cd nginx-* && cp objs/ngx_http_brotli_filter_module.so /usr/lib/nginx/modules
RUN cd nginx-* && cp objs/ngx_http_brotli_static_module.so /usr/lib/nginx/modules
RUN chmod -x /etc/systemd/system/nginx.service
RUN systemctl enable nginx

The problem still persists.

On H3 with latency, it is slow as. I get 200KB/sec.
On H2 its always fast. It’s a 10gbps connection in UK.

I have played with http3_stream_buffer_size. No measurable impact. Still slow as.

Right now this address has H3 enabled, without the headers to upgrade. So the curl tests will work:

Some other points:

  • I used to use openlitespeed and it was always fast
  • I briefly was using the old cloudflare quic implementation and do not believe it had performance issues like this

Thank you for your attention on this as I’d really love to get H3 working well.

1 Like
5

@scott0x Thanks for taking the time to shift this over here, it is much appreciated. I am a bit slammed at current prepping for our company event at the end of the month, but I am going to add this to my task list for replication.

Once that part is done I can bubble it up to the dev team and PM to see what they say. No promises for an outcome but I can get it on their radar and see how it ends up being prioritized.

Also @George thanks for jumping in on this thread to offer support also greatly appreciated.

2 Likes