NGINX mail reverse proxy

[rockinride]

Searching for some gurus to clarify and discuss some of the procedures involved in Nginx Reverse Mail Proxy, its limitations and best practices for upstream servers, load balancing and other applications. There seems to be very little accurate discussion on this topic which is an excellent feature of Nginx that many other servers do not offer. The nginx documentation is essentially an overview lacking depth. Perhaps the expertise found in this forum can clarify:

1. Compiling Nginx with the required modules for reverse mail proxy.
2. Loading the modules correctly in the configuration.
3. Creating the required authentication server or script & advantages, disadvantages.
4. Upstream server TLS options. Termination at proxy versus upstream MTA server.

[rockinride]

Here is my guide for compiling Nginx from source to get things started:

1. apt update && sudo apt upgrade
2. apt install build-essential libpcre3-dev libssl-dev zlib1g-dev libgd-dev
3. wget http://nginx.org/download/nginx-1.29.1.tar.gz *Use current version
4. tar -xzvf nginx-1.29.1.tar.gz *Extract file
5. cd nginx-1.29.1 *Navigate to directory
6. Configure build: *Included Mail Modules

./configure --prefix=/var/www/html --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --with-pcre --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-http_ssl_module --with-http_image_filter_module=dynamic --modules-path=/etc/nginx/modules --with-http_v2_module --with-stream=dynamic --with-http_addition_module --with-http_mp4_module --with-mail --with-mail_ssl_module

7. make *runs configuration
8. make install *installs configuration
9. nano /etc/systemd/system/nginx.service *create systemd file with contents and save:

[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

10. systemctl daemon-reload
11. systemctl enable nginx
12. systemctl start nginx
13. verify by accessing http://“vps server ip” to see nginx default page.

*nginx compiled from source does not create conf.d, sites-available, or sites-enabled directories. To use the modern conf.d structure:

14. mkdir /etc/nginx/conf.d
15. nano /etc/nginx/nginx.conf
16. include /etc/nginx/conf.d/.conf; * add to http block:
17. systemctl restart nginx
18. nginx -V *Lists Nginx version and compiled modules.
19. The response should include the required mail modules included in the configuration.
20. --with-mail --with-mail_ssl_module are not dynamic and shouldn’t need to be loaded in /etc/nginx/nginx.conf

[rockinride]

Alright that should cover steps 1 & 2. Let’s move on to Creating the required authentication server or script & advantages, disadvantages. Although many have used basic authentication (auth_basic) in nginx to restrict access to websites, specific directories, or pages, few have created a dedicated authentication server or created local scripts for a reverse mail proxy using the auth_http directive. This is where further discussion would be helpful in both concept, clarification and procedural methods. It is my hope someone with experience in this area can add to the topic.

[oxpa]

do you have a specific task or is more of a exploration project? nginx is meant to be used in front of postfix/dovecot. They create processes per user session. So to reduce the load on mail daemon nginx is introduced to allow only authenticated traffic.

Without authentication nginx won’t know where to proxy a connection.

As for TLS - you can’t terminate client TLS at MTA, cause you have to authenticate the session. And as with most other upstreams - MTA is trusted. So if it behaves - nginx may fail.

I haven’t used mail module too much but try me, I can probably help you with some basics.

[rockinride]

Awesome thanks for your response. It’s more of a clarification project to enhance the official documentation. I think one of the more common scenarios is the use of nginx (mail modules enabled) on an edge server that terminates the ssl and then proxies the SMTP, IMAP, or POP protocols to one or more MTA servers (load balancing etc). The confusion seems to lie in why there is an additional authentication server required by nginx to simply proxy the traffic mail protocols when this isn’t the case for normal HTTP proxies. The nginx mail directive in the server config directs the traffic to the correct MTA location and the MTA authenticates its own connections ( traffic sent from edge to MTA servers is unencrypted ). It would be great to have clear examples of creating the required authentication server, or alternatively creating a script that handles it as mentioned in the documentation which simply mentions the necessity. What are the advantages of an authentication server vs a script? The Nginx authentication seems redundant. I’ve always wanted to try the Nginx mail proxy, but instead have simply used iptables to forward to the MTA’s that handle their own SSL termination.

[Mason_Brooks]

@Codexa

nginx reverse mail proxy is powerful but the tricky part is not the basic config it’s avoiding bottlenecks in the chain the main ones i’ve seen are in authentication lookups and tls handling if the proxy does all the heavy tls termination without tuning it becomes the choke point same with poorly optimized upstream balancing configs to get best performance you want to offload where possible spread auth across multiple servers and tune buffer sizes otherwise the proxy itself turns into the bottleneck instead of improving throughput

thank you this makes it much clearer how bottlenecks can happen with nginx reverse mail proxy

[Ethan_Carter]

thanks this really helped me understand where bottlenecks can happen in nginx reverse mail proxy and how to avoid them

[alessandro]

Hey all! Just popping in here to say that if you have any suggestions on how to improve the documentation please feel free to open an issue or PR in our nginx.org repo!

[rockinride]

yes I think we need much better documentation and walk through on creating authentication server/script requirements. The Mail server will always be the actual authentication for its imap and smtp protocols. The official documentation does not cover this in detail.