Skip to main content
added 214 characters in body
Source Link
BillThor
  • 11.2k
  • 2
  • 27
  • 25

There are a number of sites that scan IP addresses for web servers, or just scan for open ports. Port scanners may attempt to verify the protocol on the port. Most of these have no malicious intent. Scanning works effectively on IPv4, but is minimally effective on IPv6.

Baidu, Yahoo, Google and a whole host of other search sites are likely to scan your site. They are just trying to index your site for their search engine. Many of these sites provide a link staring with plus (+) sign in the user agent string, where you can find more information about their scanning.

Well behaved scanners will only use GET requests for links you expose (or have exposed in the past). (Don't allow GET requests to change the state of your site.) They will also follow any restrictions provided by a robots.txt file found in the root of your site (http://example.com/robots.txt). This file has a well documented format.

Baidu, Yahoo, Google and a whole host of other search sites are likely to scan your site. They are just trying to index your site for their search engine. Many of these sites provide a link staring with plus (+) sign in the user agent string, where you can find more information about their scanning.

Well behaved scanners will only use GET requests for links you expose (or have exposed in the past). (Don't allow GET requests to change the state of your site.) They will also follow any restrictions provided by a robots.txt file found in the root of your site (http://example.com/robots.txt). This file has a well documented format.

There are a number of sites that scan IP addresses for web servers, or just scan for open ports. Port scanners may attempt to verify the protocol on the port. Most of these have no malicious intent. Scanning works effectively on IPv4, but is minimally effective on IPv6.

Baidu, Yahoo, Google and a whole host of other search sites are likely to scan your site. They are just trying to index your site for their search engine. Many of these sites provide a link staring with plus (+) sign in the user agent string, where you can find more information about their scanning.

Well behaved scanners will only use GET requests for links you expose (or have exposed in the past). (Don't allow GET requests to change the state of your site.) They will also follow any restrictions provided by a robots.txt file found in the root of your site (http://example.com/robots.txt). This file has a well documented format.

Source Link
BillThor
  • 11.2k
  • 2
  • 27
  • 25

Baidu, Yahoo, Google and a whole host of other search sites are likely to scan your site. They are just trying to index your site for their search engine. Many of these sites provide a link staring with plus (+) sign in the user agent string, where you can find more information about their scanning.

Well behaved scanners will only use GET requests for links you expose (or have exposed in the past). (Don't allow GET requests to change the state of your site.) They will also follow any restrictions provided by a robots.txt file found in the root of your site (http://example.com/robots.txt). This file has a well documented format.