Heroku request timeout vs Nodejs architecture

Question

At this link (https://blog.heroku.com/timeout-quickly#how-webservers-work), I read:

All webservers will work in a similar way. Any new request will go to a queue, and the server will process them one after the other.

This means if you have 30 requests in your queue, each taking 1 second to be processed, that will take 30 seconds for your server to empty the queue. If one of those requests is a file upload for example and takes 5 minutes to be processed, it means that any other request will be stuck for 5 minutes. That's 5 minutes during which no one else can visit your app.

This reasoning is used to justify having a 30 second timeout for every request on Heroku-hosted web servers (https://devcenter.heroku.com/articles/error-codes#h12-request-timeout).

However, is the above really true for Node, which is supposed to be async? Is there any real justification for having such a timeout for a Node server running on Heroku?

Answer 1

Yes and no. Many servers these days use non-blocking I/O, which allows the server to quickly switch between sockets as they provide more data. To be clear the concept of there being a queue of work is still accurate, but the server is notified when there is the next chunk of data to process.

When serving up static files, this works rather well, as the same server can handle thousands of connections at once. However, when you start dealing with server side processing, you have to wait until enough data comes in (typically at least the request headers) to begin handing logic over to your web application.

Usually, there is a worker queue here as well. The server has a certain number of threads available to actually do work. So let's take that queue of 30 requests again, with 1 second response time on average. If the work queue is 30 threads you would be working on the entire queue at once.

I'm not as familiar with Node's architecture specifically, but there are several ways to make asynchronous work to support hundreds of requests simultaneously. One such animal is the JavaScript Promise concept.

• You set up a Promise so that work can happen in the background
• Work is started while you are doing other things
• As soon as you wait on that promise, the task gets moved out of the processing queue and other work can happen in it's place
• Once the Promise is fulfilled, the wait is over and the task can come back and resume where it left off

I'm kind of glossing over a lot of details here, but the point I'm trying to make is that the description that the Heroku developer gave you is only one way of doing things. Node has a reputation to scale pretty well even on a modest machine, but it's not the only web server that does.