Recursive downloads following redirects

Question

My goals:

1. I want to download all *.zip files on a page where all relevant links are redirects done by a server script ( */download.php?standard=yes&file=*.zip ).
2. I want the original link to be used to name the files, since the name given by the server is in this case always "download.zip".

Specifically, I want to download the OsmAnd maps from here: http://download.osmand.net/rawindexes/

How do I do it?

What I tried:

I could use wget to do something similar with the maps for MAPS.ME:

wget -rl1 --accept="mwm" http://direct.mapswithme.com/direct/latest/

But in this case the links were direct, and there was no problem with the file names.

I also tried to use curl -L, but that's without recursion, and it doesn't seem to work for the kind of redirects that are used on that page.

Answer 1

Wget works fine but you need to quote the URL since it's not escaped.

i.e.

wget 'http://download.osmand.net/download.php?standard=yes&file=Afghanistan_asia_2.obf.zip' -O Afghanistan_asia_2.obf.zip

The -O option controls the output name. It should be fairly easy to create a shell script that will do that given a list of filenames.

Edit: You can get file names by downloading the page and doing a regex search for the file pattern:

wget -nv 'http://download.osmand.net/rawindexes/' -O - | grep -oE "file=[A-Za-z0-9_]*.obf.zip" | cut -c6-

Which could be combined with xargs and piped directly into wget to download each file:

xargs -I{} wget 'http://download.osmand.net/download.php?standard=yes&file={}' -O {}

Or your full, one line command (works in Cygwin, other *nix may require some playing with quotes and xargs):

wget -nv 'http://download.osmand.net/rawindexes/' -O - | grep -oE "file=[A-Za-z0-9_]*.obf.zip" | cut -c6- | xargs -I{} wget 'http://download.osmand.net/download.php?standard=yes&file={}' -O {}

Answer 2

Some filenames have "-" in them. So the script to get the list of files should be changed with one character. This works for me:

wget -nv 'http://download.osmand.net/rawindexes/' -O - | grep -oE "file=[A-Za-z0-9_-]*.obf.zip" | cut -c6-