Very uneven distribution of long-lived WebSocket connections between nginx workers

[Christouilhe]

Please use this template for troubleshooting questions.

My issue:

We are seeing an extremely uneven distribution of long-lived WebSocket connections between NGINX workers.
We currently have about 9000–10000 devices maintaining long-lived WebSocket connections through NGINX.
With 9253 connected devices, we observed:
worker 911 : 461 FDs
worker 912 : 39 FDs
worker 913 : 18226 FDs
worker 914 : 47 FDs
So almost all connections are handled by a single worker.
Each device creates approximately two NGINX-side TCP sockets:
device → NGINX
NGINX → WebSocket application server
The practical consequence is that one worker approaches worker_connections while the other workers are nearly idle.

How I encountered the problem:

We initially had: worker_connections 20000;
One worker reached that limit and NGINX started logging: worker_connections are not enough while connecting to upstream
This caused production issues for other proxied applications as NGINX could no longer open new upstream connections.
The server was previously running Debian 11 / NGINX 1.18.0.
We upgraded the server to Debian 12 / NGINX 1.22.1 and performed a full reboot, forcing all WebSocket clients to reconnect from scratch.
The same extreme worker imbalance is still present after the upgrade.

Solutions I’ve tried:

Increased worker_connections from 20000 to 22000 as a temporary safety margin.
Verified worker_rlimit_nofile 100000.
Upgraded from Debian 11 / NGINX 1.18.0 to Debian 12 / NGINX 1.22.1.
Performed a full server restart.
Monitored file descriptors per worker during the complete reconnection of the devices. Verified that RAM and CPU are not currently exhausted.
We have not yet changed multi_accept or enabled reuseport.
We are considering testing multi_accept off next.

Version of NGINX or NGINX adjacent software (e.g. NGINX Gateway Fabric):

Debian GNU/Linux 12 (bookworm)
nginx version: nginx/1.22.1
built with OpenSSL 3.0.20
Linux epoll
4 worker processes

Relevant limits:
Max open files: 100000 per worker
worker_connections: 22000

Deployment environment:

The architecture is:
~10000 devices
|
| WSS / TCP 443
v
NGINX frontend
Debian 12
4 vCPU / 8 GB RAM
|
| proxied WebSocket connection
v
Application server
WebSocket service on TCP port 9014
NGINX terminates TLS and proxies the WebSocket connection to one application server.
The WebSocket connections are long-lived, typically remaining established for hours or days.
There is no load balancer in front of NGINX.

Minimal NGINX config to reproduce your issue :

worker_processes auto;
worker_rlimit_nofile 100000;
events {
use epoll;
worker_connections 22000;
multi_accept on;
}
http {
upstream websocket_backend {
server 10.x.x.x:9014;
}
server {
listen 443 ssl http2;
location / {
proxy_pass http://websocket_backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
proxy_read_timeout 620s;
proxy_send_timeout 620s;
}
}
}

NGINX access/error log:

When the busy worker reached the configured limit, we observed:
worker_connections are not enough while connecting to upstream
After increasing the limit to 22000, the service is currently running correctly, but the worker distribution is still extremely uneven.
For example:
911 : 461 FDs
912 : 39 FDs
913 : 18226 FDs
914 : 47 FDs
Our main questions are:
Can multi_accept on cause such an extreme imbalance with epoll?
Is testing multi_accept off the recommended next step?
Would reuseport help distribute the initial WebSocket connections more evenly?
Are there additional NGINX/Linux settings we should inspect to understand why almost all connections are accepted by one worker?

Thanks for any advice.

[Christouilhe]

Update after additional testing

We have now been able to reproduce the issue consistently in our staging environment.

The staging server runs Debian 12 / nginx 1.22.1 with 8 workers, epoll, worker_connections 40000, and a WebSocket reverse-proxy setup.

We generated long-lived WebSocket connections from a separate machine and compared the worker FD distribution while changing only multi_accept.

100 WebSockets

multi_accept off: 66 / 66 / 66 / 66 / 66 / 34 / 66 / 34
multi_accept on : 32 / 259 / 32 / 32 / 34 / 32 / 32 / 32

500 WebSockets

multi_accept off: 162 / 162 / 162 / 162 / 162 / 162 / 162 / 130
multi_accept on : 1036 / 32 / 36 / 32 / 32 / 32 / 32 / 32

1000 WebSockets

multi_accept off: 284 / 280 / 290 / 288 / 290 / 286 / 290 / 258
multi_accept on : 32 / 2017 / 32 / 32 / 76 / 36 / 32 / 32

Each WebSocket generates approximately two nginx FDs in our setup (client : nginx + nginx : upstream), so the totals are consistent with the expected number of connections.

With multi_accept on, almost all newly established WebSockets consistently end up on a single worker. With multi_accept off, the same load is distributed quite evenly across the workers.

We restarted nginx between configurations, so long-lived connections from previous worker generations do not affect the comparison.

Questions:

• Does this behaviour match what you would expect from multi_accept on with epoll
• For a workload made mostly of many long-lived WebSocket connections, is multi_accept off generally the recommended choice?
• Is there any downside or side effect we should specifically monitor when switching from multi_accept on to off?
• Are there other nginx or Linux parameters you would recommend checking before considering this issue understood?

Thanks for any feedback.

[oxpa]

Hey Christouilhe,

The behavior you observe is expected. In general, I wouldn’t recommend using “multi_accept” without “reuse_port” option to the “listen” directive.

There are no real downsides to not enabling multi_accept. You can try it with reuse_port but keep in mind that reuse port flag brings some security considerations with it.

I’m not quite sure why people constantly try using multi_accept (: It probably gave some advantage at one point in a synthetic test. But it never helped in any real world setups with request bursts.

[Christouilhe]

Hello Oxpa,

Thanks a lot for your feedback, that matches very well with what we observed in our staging tests.

With multi_accept on, almost all long-lived websockets connections consistently ended up on a single worker. With multi_accept off, the same load was distributed quite evenly across all workers.

We tested this with 100, 500 and 1000 websockets connections, restarting nginx between each configuration, and the result was reproducible every time.

We are now leaving staging with multi_accept off for a while to validate the application behaviour before considering the same change in production.

We’ll keep reuse_port as a possible additional test, but given the current results, disabling multi_accept already seems to address our issue.

Thanks again for confirming that this behaviour is expected.

[ac000]

This is largely for posterity.

oxpa’s answer is spot on, but since you have such a clean reproduction it might be worth spelling out why multi_accept does this, because the mechanism explains your numbers rather precisely.

Since 1.11.3, accept_mutex is off by default, and on Linux that means nginx adds the listening socket to each worker’s epoll instance with EPOLLEXCLUSIVE. The kernel then tends to wake whichever worker registered the socket first, over and over. nginx knows this and works around it — from src/event/ngx_event_accept.c:

/*
 * Linux with EPOLLEXCLUSIVE usually notifies only the process which
 * was first to add the listening socket to the epoll instance.  As
 * a result most of the connections are handled by the first worker
 * process.  To fix this, we re-add the socket periodically, so other
 * workers will get a chance to accept connections.
 */

That is ngx_reorder_accept_events(), added in 1.21.6, so your 1.22.1 has it. The re-add is rate-limited:

if (c->requests++ % 16 != 0
    && ngx_accept_disabled <= 0)
{
    return;
}

Here is the crux: that counter is bumped once per accept event, not once per connection. ngx_reorder_accept_events() is called at the very end of ngx_event_accept(), after the accept loop has finished:

do {
    ...
} while (ev->available);

ngx_reorder_accept_events(ls);

and ev->available is simply multi_accept:

ev->available = ecf->multi_accept;

with the only ev->available-- in the loop being under NGX_USE_KQUEUE_EVENT. So on epoll:

• multi_accept off — the loop accepts exactly one connection per event, so the socket is re-added every 16 connections. The balancing works.
• multi_accept on — the loop drains the entire accept queue in one go, so one event can be hundreds or thousands of connections. The socket is now re-added every 16 bursts, which during a reconnection storm may be never. The favoured worker takes essentially everything.

That is exactly the pattern in your staging figures. Allowing for the ~32 baseline FDs each worker has regardless, your 500-connection run with multi_accept on is 1036 - 32 = 1004 of ~1000 connection FDs on one worker, i.e. effectively all of them, while the other seven are at their floor. With it off you get 162/162/…/130, which is what a re-add every 16 connections should look like.

Two further things make it stick for you specifically:

1. Long-lived connections don’t let the skew decay. With ordinary HTTP the imbalance is continuously washed out as connections complete. WebSockets that live for hours or days mean whatever distribution you got at accept time is the distribution you keep.

2. The full reboot was the worst possible case. Forcing ~10000 devices to reconnect at once is precisely the burst that keeps the accept queue permanently non-empty and therefore keeps the re-add from firing. The remediation produced the pathology.

It is also worth knowing that the other balancing mechanism cannot save you here. The ngx_accept_disabled back-pressure only engages at 7/8 of worker_connections (19250 of your 22000 — you measured 18226, so you were close but not over), and in any case the code that acts on it in ngx_process_events_and_timers() sits inside if (ngx_use_accept_mutex). With accept_mutex off it never sheds load; its only residual effect is the ngx_accept_disabled <= 0 term in the re-add condition above. It is accept_mutex-era machinery that the current defaults have left stranded, which is why it is still fenced off that way.

On reuseport: ngx_reorder_accept_events() returns early for a reuseport listener, because there is nothing to rebalance — each worker has its own listening socket and the kernel distributes incoming connections by 4-tuple hash. That is why multi_accept is harmless there and why oxpa’s advice is “not without reuseport”. One caveat if you do test it: on reload or binary upgrade, a shutting-down worker closes its own listening socket (ngx_close_listening_sockets()), and any connections still sitting in that socket’s accept queue are reset rather than picked up by another worker. Without reuseport the socket is shared, so that does not happen.

A practical note on rolling the change out: multi_accept off only affects connections accepted from that point on. Your existing skew will persist for the lifetime of the connections already established, and a reload will not clear it quickly either — the old workers hang around holding their WebSockets until they close. worker_shutdown_timeout will bound that if you need it to happen on a schedule, at the cost of cutting those sessions.

A history aside

On oxpa’s “I’m not quite sure why people constantly try using multi_accept” — the history is mildly interesting. It is not a later tuning knob; it is in the first public code drop, a4b16df728ab (“nginx-0.0.2-2004-02-03-00:19:52 import”), and the original context shows what it was for:

} else if (!(ngx_event_flags & NGX_HAVE_KQUEUE_EVENT)) {
    ev->available = ecf->multi_accept;
}

With kqueue the kernel reports how many connections are queued, in the kevent data field, and nginx drains them. multi_accept was simply the way to ask for that same behaviour on the event methods that cannot report it. That is still the code today, with only the flag test renamed.

The real-world case for it was accept_mutex, which was on by default until 1.11.3. Under the mutex only one worker may accept at a time, and it is taken and released around each loop iteration, so accepting a single connection per acquisition is wasteful — draining the queue amortised the lock. Then, on a single day in July 2016, two commits landed in 1.11.3:

46dd747c9e2b Events: the "accept_mutex" directive is turned off by default.
5c2dd3913aad Events: support for EPOLLEXCLUSIVE.

The first says:

Now it is believed that the accept mutex brings more harm than benefits.
Especially in various benchmarks it often results in situation where only
one worker grabs all connections.

and the second:

This flag appeared in Linux 4.5 and is useful for avoiding thundering herd
problem.

Avoiding the thundering herd was the mutex’s entire job, so between EPOLLEXCLUSIVE and reuseport (1.9.1) the kernel had taken it over. The code still says as much — the two are strictly mutually exclusive in ngx_event_process_init():

if (ngx_use_accept_mutex) {
    continue;
}

#if (NGX_HAVE_EPOLLEXCLUSIVE)

    if ((ngx_event_flags & NGX_USE_EPOLL_EVENT)
        && ccf->worker_processes > 1)
    {
        ngx_use_exclusive_accept = 1;

So multi_accept’s one genuine benefit disappeared in 2016. What did not disappear is the chain it left behind: the mutex was dropped because one worker grabbed everything; EPOLLEXCLUSIVE fixed the thundering herd but reintroduced the first-worker bias in the kernel instead; that needed ngx_reorder_accept_events() five years later in 1.21.6; and multi_accept is precisely the thing that stops that mitigation from firing. You have ended up back at the 2016 symptom by a longer route.