I am blocking a large number of IP addresses (tens of millions) (with “deny”) and specifically allowing several hundred IP addresses to have access to my website (with “allow”). I know this is not ideal, but it is what I have to work with. Nginx is occasionally allowing some of the blocked IP addresses to reach my website, and it is sometimes blocking the allowed IP addresses. All of my allowed IP addresses are included in my Nginx configuration file before my denied IP addresses are included. Is there some setting that tells Nginx as it is scanning its lists of blocked and allowed IP addresses to give up if it doesn’t find a match within some pre-determined time limit? Is there some other explanation for this behavior? Can it be fixed, say by using a faster CPU?
NGINX will check the list sequentially and stop checking when it hits the first rule that matches. It should continue checking against the config until finds a match, but with a large enough configuration it could cause issues. If you are allowing only specific IP’s, would it be easier to use a deny all statement and just specify the IP’s you want to access it?
Thanks for the response. I can’t deny all and then use a white list. I need a white list and a black list. I have noticed that I see more IP addresses getting through my black list for very large blocks of IP addresses that have been blocked, for example 50.8.0.0/11. Would putting those first in my black list help to make sure they are scanned? Does no Nginx setting exist that tells Nginx to spend more time scanning through my black list before it serves the next web page? Would using a faster CPU help?
It should scan the entire list, but it does check the list sequentially, so listing the larger IP blocks at the top of the list should help nginx find the match sooner. You may also want to look at leveraging the geo module as it can work better with the larger sets of IP’s.