NGINX reverse proxy issues, services unable to communicate between them

My issue: I cannot get two services running behind Nginx Reverse Proxy (RVP for short) to talk to each other while also having the browser to communicate with the backend through the frontend

How I encountered the problem: I’m currently working on a personal project involving proxmox containers, where one container has nginx set as a reverse proxy. The other one is the pterodactyl panel container that will eventually (once everything is known to be stable and working). The PVE has a special NAT to forward the containers ports.

Solutions I’ve tried: setting the node (wings) FQDN in the pterodactyl panel as localhost and the panel to connect to in wings config to be localhost too. Both connect and communicate, but the browser cannot communicate to the node to see and manage the demo servers, for example.

I’ve also tried to just use the dedicated RVP wildcard domain setup for each of them to communicate through, but this what’s failing, and I’m stuck here. I can either have the browser communicating with the panel and potentially the node (at least there are no errors in the console), but then wings cannot run since it cannot connect. Or I can have them both running, and I can’t do anything with the node on the user end.

This is for a demo panel system made using a discord bot btw, if anyone is curious about what i’m making

Version of NGINX or NGINX adjacent software (e.g. NGINX Gateway Fabric):

nginx 1.18.0 on Ubuntu 20.04 LTS x86_64 LXC container

Deployment environment:

This is my config i’m using, and i cannot put on playground since it cannot handle the map things:

# configuration file /etc/nginx/sites-enabled/nebula.conf:
map $host $ip_suffix {
    ~^nebula([0-9]+)\.demo\.system-breached\.xyz$ $1;
    ~^node([0-9]+)\.demo\.system-breached\.xyz$ $1;
    default 2;
}

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    include /nginx/errors;

    server_name nebula.demo.system-breached.xyz ~^nebula([0-9]+)\.demo\.system-breached\.xyz$;

    ssl_certificate /certificates/demo.system-breached.xyz/fullchain.pem;
    ssl_certificate_key /certificates/demo.system-breached.xyz/privkey.pem;
    ssl_trusted_certificate /certificates/demo.system-breached.xyz/chain.pem;

    client_max_body_size 100M;
    proxy_max_temp_file_size 0;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
        proxy_buffering off;
        proxy_request_buffering off;
        proxy_pass http://10.10.10.$ip_suffix:80;
        proxy_read_timeout 3600s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    include /nginx/errors;

    server_name node.demo.system-breached.xyz ~^node([0-9]+)\.demo\.system-breached\.xyz$;

    ssl_certificate /certificates/demo.system-breached.xyz/fullchain.pem;
    ssl_certificate_key /certificates/demo.system-breached.xyz/privkey.pem;
    ssl_trusted_certificate /certificates/demo.system-breached.xyz/chain.pem;

    client_max_body_size 1G;
    proxy_max_temp_file_size 0;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://10.10.10.$ip_suffix:443;
        proxy_read_timeout 3600s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

server {
    listen 80;
    server_name demo.system-breached.xyz *.demo.system-breached.xyz;

    return 301 https://$host$request_uri;
}

# configuration file /nginx/errors:
error_page 404 /custom_404.html;
error_page 500 502 503 504 /custom_50x.html;

location = /custom_404.html {
    root /usr/share/nginx/html;
    internal;
}

location = /custom_50x.html {
    root /usr/share/nginx/html;
    internal;
}

If anybody can help me with my issue, but you are missing some information please do not hesitate to ask me for it, it’s my first time asking for support with nginx like this so maybe i forgot some bits, thanks

1 Like

Hey @Tr3yWay996! Few things:

  • You are running a very old version of NGINX! NGINX 1.18 is over 5 years old at this point. I would suggest upgrading to the latest mainline release, NGINX 1.28. You will need to make a few tweaks to your config (e.g. http2 is now its own directive) but beyond that it should be relatively painless.
  • Can you share your access and error logs?
  • I am not entirely sure I understand the architecture of your environment. At broad scale I assume it’s something like “Browser → NGINX → pterodactyl” but what is this panel and nodes you talk about? Is pterodactyl composed of two different systems?

Hello @alessandro

I cannot remember when i first installed proxmox for this project and the reverse proxy but if it’s 5 years old then yeah i should update it for sure even if it means having to edit the config a little

I found a workaround to the issue by editing the hosts file to automatically resolve, for example nebula65.demo.system-breached.xyz to the local IP of the reverse proxy, and it worked, the node is healthy on the panel and i can make servers (but only after i redo the entire setup on an unprivileged container with keyctl that is) but yes i can give you the error and access logs, i tried looking at them but couldn’t find anything related to wings trying to connect tho.

I can give you a map of my environment to help you understand:

Pterodactyl panel is indeed made out of two separate things, the web server itself and Wings the daemon that handles everything about servers, creating, starting, stoping, deleting them and their file system with SFTP.

Let me know if you need anything else, thanks

From what I understand, you managed to fix your issue right? If the issue was with automatically resolving the domain name to an IP, you should be able to address this issue from within NGINX Open Source itself as of last November Dynamic DNS Resolution Open Sourced in NGINX – NGINX Community Blog.

I found a workaround yeah, but the main issue was that wings wasn’t able at all to contact the web panel, in this case it is, https://nebula65.demo.system-breached.xyz/api/remote/servers?page=0&per_page=50 and was always throwing that it couldn’t contact the server IP at the 443 port, now is this a workaround that only me had to do i do not know, since i just know the system I’m replicating is working perfectly fine so maybe he did it too since he told me he was also using external scripts, most probably a scriphook like i did to edit the /var/www/pterodactyl.env APP_URL part (“nebula“ thing of the HTTPS url to the right number, here it would be and is nebula65).

So now i think i just need to take the modification i did manually and to automate it using the scripthook so whenever a demo panel gets requested the script runs and replace the needed parts of the right files

I’m about to redo the entire container setup entirelly right now, would you mind helping me with the transition if i have some issues with the config in the older format on the newest nginx RvP ?

Thanks

The only major change in your config should be the http2 directive I mentioned earlier – and your current config should still work even then! Using listen xxx http2; has only been deprecated for now :slightly_smiling_face:

For your DNS to IP issue do both the DNS and IP change frequently? If your IP is static, couldn’t your “wings” simply link back to an IP instead of using a domain (nebula in your example)?

Oh alright thanks, it should be easy then

By “your IP“ you mean the private or public one ? Public one always stays the same, as for th private/local one a new IP will be assigned automatically by the underlying typescript code controlling the container creation from a preconfigured template, if it assign 10065 as the CTID then the last bit of the IP will be 65 as in 10.10.10.65 and so the pterodactyl end will be nebula65 and wings will be node65, then my hookscript will compute the last bit of the private IP and deduct the right number to replace nebulaX by inside of the .env of the panel, here it is nebula65, and this hookscript behavior will also be used in the future in order to edit the hosts file in the container so it applies the workaround i found
Hopes that this clear things up

In theory you could set things up through NGINX so that your wings would connect via the public IP, but if you want to keep everything internal, it seems as if your script is indeed the way to go :slightly_smiling_face:

Hey @alessandro,
I tried to update Nginx on the reverse proxy container and i get that 1.18 is the latest, do you know why ? It’s running Ubuntu 20.04.2 LTS — so maybe that is why i never used anything newer, if my setup is working fine currently can i just keep it on U20 and N1.18 ?

root@rvp:~# apt install nginx
Reading package lists… Done
Building dependency tree
Reading state information… Done
nginx is already the newest version (1.18.0-0ubuntu1.7).
0 upgraded, 0 newly installed, 0 to remove and 197 not upgraded.

I don’t exactly know why it’s like this in Ubuntu, but hopefully someone else can chime in! Historically, any version of Debian and Ubuntu has had outdated NGINX packages even upon release. We always recommend users use our NGINX repository and package builds (https://nginx.org/en/linux_packages.html#Ubuntu). This is the only guaranteed way to always get the latest version of NGINX.

That being said, Ubuntu 20.04 LTS went EoL in May so we no longer build packages for it. You should still be able to follow the instructions and get a much newer version of NGINX though! (I would also suggest upgrading Ubuntu if you get the chance!)

Hm i see, i will see what i can do, maybe i can in-place upgrade Ubuntu itself then upgrading Nginx in it, i have a backup either ways

Hey @alessandro i’ve updated to Ubuntu 22.04.2 LTS and used the official nginx repo to install 1.28.0 but now it doesn’t serve my config anymore, i tried everything, im out of ideas, it just worked before and now it just doesn’t do its reverse proxy job anymore

Did you start the NGINX service? Do the logs show anything if you check under /var/log/nginx/? Did you modify the original nginx.conf in your setup? If you didn’t that could also be the issue. You would have to move your config file from /etc/nginx/sites-enabled/nebula.conf to somewhere in /etc/nginx/conf.d/.

Nginx is running yes, there’s nothing in the logs and about nginx.conf no I’m using another one that was located at /etc/nginx/sites-availible/nebula.conf, i moved the config file over /etc/nginx/conf.d/ and yet nginx is still not doing anything its supposed to

There always is a file under etc/nginx/nginx.conf since that is the core NGINX config file. If there is absolutely nothing in the logs then NGINX shouldn’t be running. How did you test it? Can you paste the output of both nginx -T and nginx -V? That should help debug what’s going on :slightly_smiling_face:

Nono i know that nginx.conf is always there, in fact this is the content of it:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Here is your commands outputs:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    include /nginx/errors;

    server_name nebula.demo.system-breached.xyz ~^nebula([0-9]+)\.demo\.system-breached\.xyz$;

    ssl_certificate /certificates/demo.system-breached.xyz/fullchain.pem;
    ssl_certificate_key /certificates/demo.system-breached.xyz/privkey.pem;
    ssl_trusted_certificate /certificates/demo.system-breached.xyz/chain.pem;

    client_max_body_size 100M;
    proxy_max_temp_file_size 0;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
        proxy_buffering off;
        proxy_request_buffering off;
        proxy_pass http://10.10.10.$ip_suffix:80;
        proxy_read_timeout 3600s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    include /nginx/errors;

    server_name node.demo.system-breached.xyz ~^node([0-9]+)\.demo\.system-breached\.xyz$;

    ssl_certificate /certificates/demo.system-breached.xyz/fullchain.pem;
    ssl_certificate_key /certificates/demo.system-breached.xyz/privkey.pem;
    ssl_trusted_certificate /certificates/demo.system-breached.xyz/chain.pem;

    client_max_body_size 1G;
    proxy_max_temp_file_size 0;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://10.10.10.$ip_suffix:8080;
        proxy_read_timeout 3600s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

server {
    listen 80;
    server_name demo.system-breached.xyz *.demo.system-breached.xyz;

    return 301 https://$host$request_uri;
}

# configuration file /nginx/errors:
error_page 404 /custom_404.html;
error_page 500 502 503 504 /custom_50x.html;

location = /custom_404.html {
    root /usr/share/nginx/html;
    internal;
}

location = /custom_50x.html {
    root /usr/share/nginx/html;
    internal;
}

root@rvp:/etc/nginx/conf.d# 
root@rvp:/etc/nginx/conf.d# 
root@rvp:/etc/nginx/conf.d# nginx -V
nginx version: nginx/1.28.0
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/run/nginx.pid --lock-path=/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/home/builder/debuild/nginx-1.28.0/debian/debuild-base/nginx-1.28.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

I assume you split the output of nginx -T to put your nginx.confseparately right? If that’s what you did, NGINX is correctly parsing your whole config. If you visit your browser at port 80, what do you see or what error do you get? Are you sure both the access and error log are empty?

Yeah i’ve split output of nginx -T and -V from the nginx.conf, as for what happen when i go to port 80 (or whatever i set in the DNAT) i — what, i end up on the currently running demo ptero panel, and yes i’m sure that the logs weren’t used at all when i tried to make the RvP functional again, now though probably since i accessed the CT 103

So what exactly is it that isn’t running then?