Unlike the other question, these outside requests are not binary requests trying to probe a possible weakness ( i dont think). I was staring blankly half asleep when my http.server (run using python -m http.server) shows up a strange requests for URLs. First Australia Telstra Telecom requested HTTP/1.1 and minutes later China TianJin Telecom requested http://www.baidu.com/ HTTP/1.1. Were they trying to scan me just in case I was illegally hosting things, or what?
Add a comment
|
1 Answer
There are a number of sites that scan IP addresses for web servers, or just scan for open ports. Port scanners may attempt to verify the protocol on the port. Most of these have no malicious intent. Scanning works effectively on IPv4, but is minimally effective on IPv6.
Baidu, Yahoo, Google and a whole host of other search sites are likely to scan your site. They are just trying to index your site for their search engine. Many of these sites provide a link staring with plus (+) sign in the user agent string, where you can find more information about their scanning.
Well behaved scanners will only use GET requests for links you expose (or have exposed in the past). (Don't allow GET requests to change the state of your site.) They will also follow any restrictions provided by a robots.txt file found in the root of your site (http://example.com/robots.txt). This file has a well documented format.
