My issue:
Hi,
I have the following upstream block configuration
upstream https_origin_78e2e843 {
keepalive 64;
zone https_origin_78e2e843 64k;
server 10.50.30.33:8743 resolve;
}
Using Nginx 1.28.0 version and using the dynamic upstream resolution feature. The zone statement fails with nginx: [emerg] zone "http_origin_78e2e843" is too small error.
Solutions I’ve tried:
Running nginx in a container. When I use the above configuration in an amd64 container image, it doesn’t throw the above error. However, it gives the error for the arm64 image. I have to increase the zone size from 64k to 512k, for it to not throw the error for the arm image. Both are same images using the same software built using same pipeline. Only difference is they are different arch. images.
Version of NGINX or NGINX adjacent software (e.g. NGINX Gateway Fabric):
1.28.0
Deployment environment:
Container images: arm64 and amd64
Minimal NGINX config to reproduce your issue (preferably running on https://tech-playground.com/playgrounds/nginx for ease of debugging, and if not as a code block): (Tip → Run nginx -T to print your entire NGINX config to your terminal.)
worker_processes 1;
pid /opt/nginx/nginx.pid;
error_log /opt/nginx/log/error.log warn;
events {
worker_connections 16384;
}
http {
map_hash_bucket_size 128;
include /opt/nginx/mime.types;
log_format main '[$time_local] $remote_addr $remote_port $uri $request_uri $status $body_bytes_sent "$http_user_agent" $request_time "$upstream_connect_time" "$upstream_header_time" "$upstream_response_time" "$upstream_status" "$connection" "$connection_requests" "$upstream_addr" "$http_host"';
default_type application/octet-stream;
server_tokens off;
resolver 8.8.8.8;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
underscores_in_headers on;
uninitialized_variable_warn off;
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 4;
gzip_http_version 1.0;
gzip_min_length 512;
gzip_proxied no-cache no-store private expired auth;
gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
gzip_vary on;
access_log /opt/nginx/log/access.log main;
server {
listen 8080 reuseport;
server_name _;
client_header_buffer_size 8k;
large_client_header_buffers 4 64k;
location / {
proxy_pass http://$http_origin;
proxy_ssl_server_name on;
proxy_http_version 1.1;
proxy_ssl_name $host;
proxy_set_header Host $host;
proxy_set_header Connection "";
}
}
server {
listen 8443 ssl reuseport;
server_name _;
http2 on;
client_header_buffer_size 8k;
large_client_header_buffers 4 64k;
ssl_certificate /opt/nginx/certs/cert.pem;
ssl_certificate_key /opt/nginx/certs/key.pem;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1.2;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK;
ssl_prefer_server_ciphers on;
location / {
proxy_pass https://$https_origin;
proxy_ssl_server_name on;
proxy_http_version 1.1;
proxy_ssl_name $host;
proxy_set_header Host $host;
proxy_set_header Connection "";
}
}
upstream http_origin_3e063a24 {
keepalive 64;
#zone http_origin_3e063a24 256k; <== FAILS
zone http_origin_3e063a24 512k; # WORKS
server 10.50.30.33:8890 resolve;
}
upstream https_origin_78e2e843 {
keepalive 64;
#zone http_origin_3e063a24 256k; <== FAILS
zone https_origin_78e2e843 512k;
server 10.50.30.33:8743 resolve;
}
map $host $http_origin {
hostnames;
default http_origin_3e063a24;
}
map $host $https_origin {
hostnames;
default https_origin_78e2e843;
}
}
NGINX access/error log: (Tip → You can usually find the logs in the /var/log/nginx directory.)
tmp# /usr/sbin/nginx -p . -c t.conf -t
nginx: [emerg] zone "http_origin_3e063a24" is too small in ./t.conf:74
nginx: configuration file ./t.conf test failed
Hey @dnj0496! Can you let me know which Docker image are you using as your base image? Can you also let me know what is your base OS? Are you only running into this issue when using Docker, or is it also when using a VM? I wonder if it might be related to some limitation with the amd64/arm64 emulation.
1 Like
I am using Redhat ubi9 (9.5 to be exact). Base OS RHEL8 (Fedora 8.6). We only run it container which run on a aws ec2 machine. One is aarch64 ec2 instance and the other is an amd64 ec2 instance. Not if sure if there’s any emulation because, when we build the container images, they get built on aarch64 and amd64 machines respectively and then we create the docker manifest combining them into a single pkg. The docker pull takes care of pulling the correct arch image based on machine architecture where docker pull is run.
Is there any chance you could test out using the images we build (https://hub.docker.com/_/nginx) and seeing if you run into the same issue?
I get the same error. Here’s the error and the config used.
root@ab87de961923:/# /usr/sbin/nginx -p . -c nginx.conf -t
2025/06/25 20:37:57 [emerg] 351#351: zone "http_origin_3e063a24" is too small in ./nginx.conf:74
nginx: [emerg] zone "http_origin_3e063a24" is too small in ./nginx.conf:74
nginx: configuration file ./nginx.conf test failed
root@ab87de961923:/# cat nginx.conf
worker_processes 1;
pid /var/log/nginx/nginx.pid;
error_log /var/log/nginx/error.log warn;
events {
worker_connections 16384;
}
http {
map_hash_bucket_size 128;
include /etc/nginx/mime.types;
log_format main '[$time_local] $remote_addr $remote_port $uri $request_uri $status $body_bytes_sent "$http_user_agent" $request_time "$upstream_connect_time" "$upstream_header_time" "$upstream_response_time" "$upstream_status" "$connection" "$connection_requests" "$upstream_addr" "$http_host"';
default_type application/octet-stream;
server_tokens off;
resolver 8.8.8.8;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
underscores_in_headers on;
uninitialized_variable_warn off;
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 4;
gzip_http_version 1.0;
gzip_min_length 512;
gzip_proxied no-cache no-store private expired auth;
gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
gzip_vary on;
access_log /var/log/nginx/access.log main;
server {
listen 8080 reuseport;
server_name _;
client_header_buffer_size 8k;
large_client_header_buffers 4 64k;
location / {
proxy_pass http://$http_origin;
proxy_ssl_server_name on;
proxy_http_version 1.1;
proxy_ssl_name $host;
proxy_set_header Host $host;
proxy_set_header Connection "";
}
}
server {
listen 8443 ssl reuseport;
server_name _;
http2 on;
client_header_buffer_size 8k;
large_client_header_buffers 4 64k;
ssl_certificate /etc/nginx/cert.pem;
ssl_certificate_key /etc/nginx/key.pem;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1.2;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK;
ssl_prefer_server_ciphers on;
location / {
proxy_pass https://$https_origin;
proxy_ssl_server_name on;
proxy_http_version 1.1;
proxy_ssl_name $host;
proxy_set_header Host $host;
proxy_set_header Connection "";
}
}
upstream http_origin_3e063a24 {
keepalive 64;
zone http_origin_3e063a24 256k; #<== FAILS
#zone http_origin_3e063a24 512k; # WORKS
server 10.50.30.33:8890 resolve;
}
upstream https_origin_78e2e843 {
keepalive 64;
zone http_origin_3e063a24 256k; #<== FAILS
#zone https_origin_78e2e843 512k;
server 10.50.30.33:8743 resolve;
}
map $host $http_origin {
hostnames;
default http_origin_3e063a24;
}
map $host $https_origin {
hostnames;
default https_origin_78e2e843;
}
}
[cequence@6c8b517d86b7] certs# uname -a
Linux 6c8b517d86b7 4.18.0-372.32.1.el8_6.aarch64 #1 SMP Fri Oct 7 11:33:30 EDT 2022 aarch64 aarch64 aarch64 GNU/Linux
[cequence@6c8b517d86b7] certs# cat /etc/*release*
NAME="Red Hat Enterprise Linux"
VERSION="9.6 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.6"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.6 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.6"
Red Hat Enterprise Linux release 9.6 (Plow)
Red Hat Enterprise Linux release 9.6 (Plow)
cpe:/o:redhat:enterprise_linux:9::baseos
[cequence@6c8b517d86b7] certs#
@alessandro
The same thing works even with zone size of 64k
root@0d7b018ca315:/# /usr/sbin/nginx -p . -c nginx.conf -t
nginx: the configuration file ./nginx.conf syntax is ok
nginx: configuration file ./nginx.conf test is successful
root@0d7b018ca315:/# cat nginx.conf
worker_processes 1;
pid /var/log/nginx/nginx.pid;
error_log /var/log/nginx/error.log warn;
events {
worker_connections 16384;
}
http {
map_hash_bucket_size 128;
include /etc/nginx/mime.types;
log_format main '[$time_local] $remote_addr $remote_port $uri $request_uri $status $body_bytes_sent "$http_user_agent" $request_time "$upstream_connect_time" "$upstream_header_time" "$upstream_response_time" "$upstream_status" "$connection" "$connection_requests" "$upstream_addr" "$http_host"';
default_type application/octet-stream;
server_tokens off;
resolver 8.8.8.8;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
underscores_in_headers on;
uninitialized_variable_warn off;
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 4;
gzip_http_version 1.0;
gzip_min_length 512;
gzip_proxied no-cache no-store private expired auth;
gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
gzip_vary on;
access_log /var/log/nginx/access.log main;
server {
listen 8080 reuseport;
server_name _;
client_header_buffer_size 8k;
large_client_header_buffers 4 64k;
location / {
proxy_pass http://$http_origin;
proxy_ssl_server_name on;
proxy_http_version 1.1;
proxy_ssl_name $host;
proxy_set_header Host $host;
proxy_set_header Connection "";
}
}
server {
listen 8443 ssl reuseport;
server_name _;
http2 on;
client_header_buffer_size 8k;
large_client_header_buffers 4 64k;
ssl_certificate /etc/nginx/cert.pem;
ssl_certificate_key /etc/nginx/key.pem;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1.2;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK;
ssl_prefer_server_ciphers on;
location / {
proxy_pass https://$https_origin;
proxy_ssl_server_name on;
proxy_http_version 1.1;
proxy_ssl_name $host;
proxy_set_header Host $host;
proxy_set_header Connection "";
}
}
upstream http_origin_3e063a24 {
keepalive 64;
zone http_origin_3e063a24 64k; # WORKS
#zone http_origin_3e063a24 512k; # WORKS
server 10.50.30.33:8890 resolve;
}
upstream https_origin_78e2e843 {
keepalive 64;
zone http_origin_3e063a24 64k; # WORKS
#zone https_origin_78e2e843 512k;
server 10.50.30.33:8743 resolve;
}
map $host $http_origin {
hostnames;
default http_origin_3e063a24;
}
map $host $https_origin {
hostnames;
default https_origin_78e2e843;
}
}
root@0d7b018ca315:/# uname -a
Linux 0d7b018ca315 4.9.215-36.el7.x86_64 #1 SMP Mon Mar 2 11:42:52 UTC 2020 x86_64 GNU/Linux
root@0d7b018ca315:/# cat /etc/*release*
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@0d7b018ca315:/#
I’ve gone ahead and recreated your config on both the amd64 and arm64 images we build and distribute and a zone size of 64k works just fine on both.
I think this might be something to do with either how you build your images or some of the memory defaults that ship on the RHEL AWS instances. I’d suggest looking at things such as page file sizes and what not, but sadly there isn’t much more I can do to help.
Thanks for verifying. I don’t think how we build our images matters as much because I was able to reproduce the issue with your nginx docker container image. The behavior with your nginx container is exactly the same as our own image using the same nginx.conf file.
Could you elaborate bit on the memory defaults or page files size etc. What can look or settings can I play with to debug this further?
Page file size might have been a bit of misnomer on my part, sorry! Page tables is the correct name. We do have some docs on our .org side detailing how NGINX handles the shared memory zone, but there is nothing system specific Development guide.
Since everything seems to run fine when running on an amd64 system with an amd64 container, I would probably start by looking at any differences in system defaults for system memory between the RHEL amd64 and arm64 OSes and double check that both container runtimes use the same defaults too.
Good luck! And maybe someone else from the community that’s been running into a similar issue has some suggestions!
Jumping back real quick to send this doc your way ARM Paging - OSDev Wiki – it goes a little bit more in depth into memory allocations and it seems as if arm64 arches can have bigger default sizes than amd64, which might be related to your issue. Hoping this helps you debug your issue!
Thank you! will review that.
Bumping this thread again one last time! Seems like by default arm64 RHEL images have a 64k page size by default, and it is a known issue that this might break some configs that are ported without any further tweaks.
@alessandro Thanks for responding. Your response gave me a clue. I tried reducing the zone size on amd to 32k and 16k. The 32k one passed but not the 16k one. Looks like zone size needs to be at least 8x of the page size. Like you mentioned, the page size on my arm machine was 64k. Hence 64k*8=512k worked and anything lower did not work. Similarly, amd also needed 32k = 4k*8. It make sense now and I am glad that it is not something arbitrary between the architectures.
Not sure why it is tied to page size though. 512k seems way too much memory for an upstream zone. Hopefully, that’s something Nginx team can look at.
If you look through the dev guide I shared some posts ago you’ll see how/why it is tied to page size.