make nginx ignore site config when its upstream cannot be reached

Question

There are multiple site config in my nginx, and when I restart the machine, if one of the sites' upstream cannot be reached, nginx won't start at all, and those healthy site won't start as a result, how to let nginx ignore those invalid sites?

http {

##
# Basic Settings
##

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;

# server_names_hash_bucket_size 64;
# server_name_in_redirect off;

include /etc/nginx/mime.types;
default_type application/octet-stream;

##
# Logging Settings
##

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

##
# Gzip Settings
##

gzip on;
gzip_disable "msie6";

# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

##
# nginx-naxsi config
##
# Uncomment it if you installed nginx-naxsi
##

#include /etc/nginx/naxsi_core.rules;

##
# nginx-passenger config
##
# Uncomment it if you installed nginx-passenger
##

#passenger_root /usr;
#passenger_ruby /usr/bin/ruby;

##
# Virtual Host Configs
##

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

and sites-enabled/example1 is

upstream example1 {
    server example1.service.example.com;
}
server {
listen 80;
server_name example1.com;
location / {
    proxy_pass http://example1/;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}
}

and sites-enabled/example2 is

upstream example2 {
    server example2.service.example.com;
}
server {
listen 80;
server_name example2.com;
location / {
    proxy_pass http://example2/;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}
}

When I restart the machine and at that time example2.service.example.com is down, nginx will not start at all, i.e. even example1.service.example.com is available, nginx will not serve for example1

=====update Explanation of "is down": All the subdomains is automatically registered/deregistered on my own dns server, so if the server is down, the dns will respond no such domain when try to resolve it.

Answer 1

Finally I find out the walkaround, resolve the domain insdie location works!

example:

server {
    listen 9000;
    server_name example1.example.com;
    location / {
        set $target http://something.service.lab.mu;
        proxy_pass $target;
    }
}

And nginx won't try to resolve http://something.service.lab.mu at start time.

Answer 2

For anyone stumbling upon this issue, @cgcgbcbc is correct.

But you also need to add a

resolver 8.8.8.8;

directive above the

set $target http://something.service.lab.mu;

otherwise you'll get an error in nginx, like:

no resolver defined to resolve

Answer 3

There are few ways to avoid it:

1. Use static IP, nginx will return 503's if it doesn't respond.

2. Use the resolver directive to point to something that can resolve the host, regardless if it's currently up or not.

3. Use the resolver directive at the location level, if you can't do the above (this will allow Nginx to start):

    location /some_path {
      resolver    127.0.0.1   valid=30s;   
      # resolver    8.8.8.8     valid=30s;   # or some other DNS
      # resolver    127.0.0.11  valid=30s;   # or Docker's DNS server
   
      set         $dummy_var  http://some_domain:80;
      proxy_pass  $dummy_var;
    }