My issue:
I have a client with IP 10.70.0.7 making an SMPP call to port 14200 towards an Azure Load Balancer with frontend IP 10.70.0.4. The Azure Load Balancer balances traffic over two NGINX instances (10.70.0.5 and 10.70.0.6) using a floating IP rule.
I’ve configured the lo interface on the NGINX instances with IP 10.70.0.4/32 to accept traffic from the load balancer.
The NGINX instances are configured with the following code to balance traffic listening on port 14200 to the final server 10.80.0.132:2775 using proxy_bind 10.70.0.4 (frontend IP of the load balancer):
in the ‘stream’ context
upstream stream_proveedor_1_smpp {
server 10.80.0.132:2775;
zone proveedor_1 1M;
}
server {
listen 10.70.0.4:14200;
proxy_bind 10.70.0.4;
proxy_pass stream_proveedor_1_smpp;
status_zone proveedor_1;
error_log /etc/nginx/upstream_logs/core_error.log warn;
access_log /etc/nginx/upstream_logs/core_access.log main;
}
How I encountered the problem:
When I capture traffic on any of the NGINX instances, I see that the outgoing traffic still uses the source IP of the eth0 interface of the NGINX instances.
tcpdump: verbose output suppressed, use -v[v]… for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
07:23:34.258562 IP ngx-oss-app1-1.internal.cloudapp.net.60134 > 10.70.0.4.14200: Flags [S], seq 1901296326, win 64240, options [mss 1410,sackOK,TS val 3385207481 ecr 0,nop,wscale 7], length 0
07:23:34.258616 IP 10.70.0.4.14200 > ngx-oss-app1-1.internal.cloudapp.net.60134: Flags [S.], seq 932270532, ack 1901296327, win 43440, options [mss 1460,sackOK,TS val 2019128601 ecr 3385207481,nop,wscale 9], length 0
07:23:34.259337 IP ngx-oss-app1-1.internal.cloudapp.net.60134 > 10.70.0.4.14200: Flags [.], ack 1, win 502, options [nop,nop,TS val 3385207482 ecr 2019128601], length 0
07:23:34.259337 IP ngx-oss-app1-1.internal.cloudapp.net.60134 > 10.70.0.4.14200: Flags [P.], seq 1:81, ack 1, win 502, options [nop,nop,TS val 3385207482 ecr 2019128601], length 80
07:23:34.259385 IP 10.70.0.4.14200 > ngx-oss-app1-1.internal.cloudapp.net.60134: Flags [.], ack 81, win 85, options [nop,nop,TS val 2019128602 ecr 3385207482], length 0
07:23:34.259510 IP ngx-plus-1.internal.cloudapp.net.14694 > 10.80.0.132.2775: Flags [S], seq 1454567838, win 42340, options [mss 1460,sackOK,TS val 4279807172 ecr 0,nop,wscale 9], length 0
07:23:34.264803 IP 10.80.0.132.2775 > ngx-plus-1.internal.cloudapp.net.14694: Flags [S.], seq 1478375283, ack 1454567839, win 65535, options [mss 1310,nop,wscale 8,nop,nop,sackOK], length 0
07:23:34.264833 IP ngx-plus-1.internal.cloudapp.net.14694 > 10.80.0.132.2775: Flags [.], ack 1, win 83, length 0
07:23:34.264890 IP ngx-plus-1.internal.cloudapp.net.14694 > 10.80.0.132.2775: Flags [P.], seq 1:81, ack 1, win 83, length 80: 2775: HEAD / 2775/1.1
07:23:34.272806 IP 10.80.0.132.2775 > ngx-plus-1.internal.cloudapp.net.14694: Flags [P.], seq 1:225, ack 81, win 16415, length 224: 2775: 2775/1.1 200 OK
07:23:34.272833 IP ngx-plus-1.internal.cloudapp.net.14694 > 10.80.0.132.2775: Flags [.], ack 225, win 83, length 0
07:23:34.275962 IP 10.80.0.132.2775 > ngx-plus-1.internal.cloudapp.net.14694: Flags [F.], seq 225, ack 82, win 16415, length 0
07:23:34.272868 IP 10.70.0.4.14200 > ngx-oss-app1-1.internal.cloudapp.net.60134: Flags [P.], seq 1:225, ack 81, win 85, options [nop,nop,TS val 2019128616 ecr 3385207482], length 224
07:23:34.274130 IP ngx-oss-app1-1.internal.cloudapp.net.60134 > 10.70.0.4.14200: Flags [.], ack 225, win 501, options [nop,nop,TS val 3385207497 ecr 2019128616], length 0
07:23:34.274249 IP ngx-oss-app1-1.internal.cloudapp.net.60134 > 10.70.0.4.14200: Flags [F.], seq 81, ack 225, win 501, options [nop,nop,TS val 3385207497 ecr 2019128616], length 0
07:23:34.274277 IP ngx-plus-1.internal.cloudapp.net.14694 > 10.80.0.132.2775: Flags [F.], seq 81, ack 225, win 83, length 0
07:23:34.274325 IP 10.70.0.4.14200 > ngx-oss-app1-1.internal.cloudapp.net.60134: Flags [F.], seq 225, ack 82, win 85, options [nop,nop,TS val 2019128617 ecr 3385207497], length 0
07:23:34.274771 IP ngx-oss-app1-1.internal.cloudapp.net.60134 > 10.70.0.4.14200: Flags [.], ack 226, win 501, options [nop,nop,TS val 3385207498 ecr 2019128617], length 0
How can I make NGINX send the frontend IP of the load balancer as the source IP for the traffic it routes?
Solutions I’ve tried:
have tried with transparent mode, without it, with the geo module to assign variables, with NAT rules in iptables (iptables -t nat -A POSTROUTING -p tcp --dport 80 -d 10.80.0.132 -j SNAT --to 10.70.0.4 ), and changing the rp_filter .
My config:
Right now, we have a configuration with a single NGINX node. However, due to requirements, we need to set it up in HA (High Availability), but with the limitation of having to present ourselves with the same IP despite having two nodes. We cannot use NAT in the IPsec VPN tunnel, which would be the fastest and simplest solution.