Nginx slow downloads even with low disk io

Question

I run a small video streaming website and i've been noticing some problems with videos on my site. I have multiple 1gbit servers and the servers are only going at 1/4 port speed. When I try to load some files downloads take a long time to start going. Disk IO is at 2% and I see disk read speeds between 3 and 10mb/s.

The disk read speeds are incredibly slow considering the amount of connection that the servers have. Each server has four drives set-up with raid 10. Here's the nginx configuration that I am testing right now. It's very frustrating because these are very powerful servers but i feel like something isn't configured correctly. Perhaps something not related to nginx but related to TCP connections? Thank you in advance for your suggestions.

worker_processes  8;
worker_rlimit_nofile 10240;
worker_rlimit_sigpending 32768;
error_log  logs/error.log  crit;


events {
    worker_connections  1240;
}


http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile on;
    tcp_nopush on;

    access_log  off;
    limit_conn_log_level info;

    #sendfile       on;
    #tcp_nopush     on;
    reset_timedout_connection on;

    server_tokens off;
    autoindex off;

    keepalive_timeout  0;
    #keepalive_timeout  65;

    limit_zone one $binary_remote_addr 10m;
    perl_modules  perl;
    perl_require  download.pm;

    server {
        listen       182;
        server_name  localhost;
        proxy_set_header X-Real-IP $remote_addr;

        location /nstatus {
            stub_status on;
            #allow 127.0.0.1;
            #deny all;
                        }

        location / {
            root   html;
            index  index.html index.htm;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

    }

}

Answer 1

You might be interested in activating asynchronous disk I/O through the use of the aio directive.

Its documentation is thorough and provides example configuration blocks.

Using asynchronous I/O will allow nginx to multiplex accesses to the disk and push the limit towards hardware ones by stacking up requests without waiting for the previous one to complete. The OS/driver/hardware layers will do the rest and answer ASAP.