The only way I was able to get this was sending the upstream to another port on the receiving server.

ie

upstream backend {
    server unix:/home/www/api.to/app.sock weight=1;
    server 1.2.3.4:80000 weight=1;
}

And in the recieving nginx file put a

listen 8000;

next to the listen 80/443 (respectively one you use)

The only way I was able to get this was sending the upstream to another port on the receiving server.

ie

upstream backend {
    server unix:/home/www/api.to/app.sock weight=1;
    server 1.2.3.4:80000 weight=1;
}

And in the recieving nginx file put a

listen 8000;

next to the listen 80/443 (respectively one you use), or as I just realized... send the upstream to the 443. This was all caused because I was sending traffic via the upstream to the IP, but that sends it to port 80, my app was on ssl port, 443. Hence why it works when listening and sending to port 8000. 🤦