0

I want to tell Baidu not to crawl. One of their crawlers is Baidu Spider.

Should that be: User-agent: Baidu%20Spider or User-agent: Baidu Spider

Improve this question
2
  • I can't find anything that says you can use %20, so I wouldn't. Checking my own robot.text it contains: User-agent: Baiduspider. Google says: Spaces are optional. In reality, I cannot find anything definite on how you should do it.
    – KIKO Software
    Commented Jul 8 at 20:13
  • Yeah information is woefully sparse even on supposedly authoritative sites on the space issue. FWIW there are sites I've seen that list known spiders and they include Baidu Spider, with a space. My concern is mostly having robots.txt not read by, say, Google because of a syntax error (it's happened before albeit with an obvious 'whoops' mistake.
    – SKidd
    Commented Jul 9 at 8:17

0

Reset to default

Browse other questions tagged or ask your own question.