Error 403 (Forbidden) with many file requests

Question

I am writing a webpage which displays an svg image. A java script should load many small thumbnail images (about 400) in this svg by adding <image> tags. As the browsers debugging output shows, not all of these thumbnails are loaded. Many fail with 'Error 403'.

The permissions of the thumbnail files are correct. I can load them individually with the browser. The issue seems to appear consistently with several browsers.

I suspect that I am sending too many simultaneous file requests. I probably cannot have direct access to the server, but it is likely apache2 on linux. Are there parameters that control the maximum number of files per client or hard-coded limits (for instance there is a parameter called MaxKeepAliveRequests) or are there any other issues that can cause this behavior?

Is there a way to find the IP of the actual computer running the server (Of course I find the IP that the URL translates into, but as far as I see this computer is not running the actual server. The port might be forwarded)?

Even though this might be off-topic: Is there a recommended java script solution to address such issues when loading many files?

---

I found some apache configuration which could be the relevant one. It seems to load mod_evasive. Could this be it?

Relevant configuration is

<IfModule mod_evasive20.c>
    DOSHashTableSize    3097
    DOSPageCount        10
    DOSSiteCount        50
    DOSPageInterval     1
    DOSSiteInterval     3
    DOSBlockingPeriod   20
    DOSEmailNotify      ""
    DOSLogDir           "/var/log/apache2/mod_evasive.log"
</IfModule>

Answer 1

It sounds lime some kind of rate limiting. There are several apache modules that will do that, mod_qos being one of them. Usually they will return short body in addition to 403 forbidden code with little more details, like link to Acceptable Usage Policy or similar. See if your scraper script could show you that in addition to 403 error code. There could also be reverse-proxy before apache doing the limiting.

As a solution, keep your number of concurrent request to same site low (if not 1). And obey robots.txt ! Also make note of destination site Terms of Service and Acceptable Usage Policy .

UPDATE yes, mod_evasive will also do that. You can disable it (if you have control over site) or tune its parameters. Specifically in your case, it will block if you do more than DOSSiteCount requests in DOSSiteInterval time. So you need to increase allowed number of requests or throttle your fetching speed (by limiting download parallelism and/or inserting delay after each one)

Answer 2

Yes, evasive can do that. To verify this, just grep trough Apache's error log files looking for evasive and you should get all the info you need.