wget won't ignore no-follow attributes

Question

I'm using the following command to download all files off of a webpage:

wget --recursive "http://example.com"

This gives me sometimes the following error:

no-follow attribute found in www.example.com. Will not follow any links on this page

According to gnu.org, I have to add -e robots=off --wait 0.25 to my command.

My final command looks like this (I don't want span-hosts):

wget --recursive -e robots=off --wait 0.5 "http://example.com"

However I am still getting the above error. What can I do to ignore those attributes?

Answer 1

The right way is there. You just have to try it

wget -r -erobots=off "your_url"

Answer 2

The message is a bug, wget is in fact following the links despite showing the message "Will not follow any links on this page".

This has been fixed on the master branch and should be fixed in the next version of wget.

See this for more details.

Answer 3

In my case I had a syntax error in --follow-tags. Removing the syntax error let wget continue despite the no-follow attribute.

See more info here: https://unix.stackexchange.com/a/258850