328

Here is a simple wget command:

wget http://www.example.com/images/misc/pic.png

How to make wget skip download if pic.png is already available ?

Improve this question
1

5 Answers 5

Reset to default
414

Try the following parameter:

-nc, --no-clobber: skip downloads that would download to existing files.

Sample usage:

wget -nc http://example.com/pic.png
Improve this answer
5
  • 9
    As noted on the linked question, I disagree - If no-clobber is used and the filename exists it exits. No HEAD request even. Even if this wasn't the case, check if you have a file to begin with :-) [ ! -e "$(basename $URL)" ] && wget $URL
    – plundra
    Commented Oct 21, 2015 at 11:56
  • 4
    I think I may be getting different results because I'm using the --recursive option.
    – ma11hew28
    Commented Oct 22, 2015 at 1:48
  • 3
    Great answer! Going to disagree with ma11hew28. I just tested this on a list of 3,000 URL's with GNU Wget 1.14 and wget -nc -i list.txt. Don't think it's possible for a server to crawl 3k links in a tenth of a second!
    – HoldOffHunger
    Commented May 13, 2021 at 12:00
  • 2
    Additionally, -N, --timestamping says don't re-retrieve files unless newer than local if you are looking to sync, in-case some remote files might ACTUALLY be worth re-downloading (edit: I see another answer now that says the same).
    – bunkerdive
    Commented Aug 30, 2021 at 20:13
  • Note that wget exits with return code 1 if the file exists, which can be a problem when used in scripts. In that case, it seems you have to check if the file exists yourself (see e.g. the code snippet in plundra's comment above).
    – luator
    Commented May 28 at 7:13
289

The -nc, --no-clobber option isn't the best solution as newer files will not be downloaded. One should use -N instead which will download and overwrite the file only if the server has a newer version, so the correct answer is: 

wget -N http://www.example.com/images/misc/pic.png

Then running Wget with -N, with or without -r or -p, the decision as to whether or not to download a newer copy of a file depends on the local and remote timestamp and size of the file. -nc may not be specified at the same time as -N.

-N, --timestamping: Turn on time-stamping.

Improve this answer
9
  • 62
    When server is not configured properly -N may fail and wget will always redownload. So sometimes -nc is better solution.
    – user
    Commented Feb 23, 2014 at 18:43
  • 2
    what could be the applicable scenario where 'When server is not configured properly' would occur?
    – AjayKumarBasuthkar
    Commented Jul 17, 2015 at 13:56
  • 2
    when you are downloading from a location that was copied, changing all the timestamps.
    – Robert
    Commented Oct 28, 2016 at 16:22
  • 1
    Whether this is best depends on context. For example, I'm downloading ~1600 files from a list, and then updated the list to include some more files. The files don't change so I don't care about the latest version and I don't want it to check the server for new versions of the 1600 files that I already have.
    – JBentley
    Commented Oct 3, 2017 at 19:45
  • 4
    @AjayKumarBasuthkar: When the server doesn't support any way of checking for newer file, wget will complain Last-modified header missing; this is exactly the situation outlined.
    – Piskvor left the building
    Commented Feb 21, 2018 at 9:58
37

The answer I was looking for is at https://unix.stackexchange.com/a/9557/114862.

Using the -c flag when the local file is of greater or equal size to the server version will avoid re-downloading.

Improve this answer
3
  • 2
    This is especially great when you are downloading a bunch of files with the -i flag. wget -i filelist.txt -c will resume a failed download of a list of files.
    – Trevor
    Commented Sep 6, 2018 at 4:30
  • 2
    I am downloading from a server which provides neither the Length header nor the Last-modified header (mentioned elsewhere on this page). So, I'd like to check only if a file with the same name exists on the disk and skip the re-download if it does. Still looking for that solution.
    – daveloyall
    Commented Sep 11, 2020 at 22:27
  • 5
    -c means continue. If the file is was changed to a bigger file with different content you get will start download at the end of the local file and add the new file contents. You may end up garbage.
    – Soerendip
    Commented May 4, 2021 at 22:23
27

When running Wget with -r or -p, but without -N, -nd, or -nc, re-downloading a file will result in the new copy simply overwriting the old.

So adding -nc will prevent this behavior, instead causing the original version to be preserved and any newer copies on the server to be ignored.

See more info at GNU.

Improve this answer
1
  • The following doesn't work as it is not recursive. The already downloaded files should be parsed for links so that everything that's missing is downloaded: wget -w 10 -r -nc -l inf --no-remove-listing -H "<URL>"
    – Terje Oseberg
    Commented Jan 18, 2023 at 15:32
3

I had issues with -N as I wanted to save output to a different file name.

Timestamping, wget docs:

A file is considered new if one of these two conditions are met:

  1. A file of that name does not already exist locally.
  2. A file of that name does exist, but the remote file was modified more recently than the local file.

Using test:

test -f stackoverflow.html || wget -O stackoverflow.html https://stackoverflow.com/

If the file exists does not exist test will evaluate to FALSE so wget will be executed.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.