Wget downloaded more data than the actual file

Question

I have downloaded lubuntu image using wget with mentioned file size 694MB , but the downloaded file is of size 789MB. when i do wget -c it gives that the file is already downloaded. But the file size is 100MB larger than given.

Downloaded file size can be seen here :

-rw-rw-r-- 1 alex alex  98537904 Jun  1  2011 R284575.exe
-rw-rw-r-- 1 alex alex      4928 Sep 25 11:06 wget-log
-rw-rw-r-- 1 alex alex 827390116 Sep 25 11:06 lubuntu-14.04-desktop-amd64.iso
-rw-rw-r-- 1 alex alex    148598 Sep 25 11:44 lubuntu-14.04-desktop-amd64.iso.1

Here we can see that the lubuntu image size when a new download is started

wget--limit-rate=40k http://cdimage.ubuntu.com/lubuntu/releases/14.04/release/lubuntu-14.04-desktop-amd64.iso
wget--limit-rate=40k: command not found
alex@dracula:~/Downloads$ wget http://cdimage.ubuntu.com/lubuntu/releases/14.04/release/lubuntu-14.04-desktop-amd64.iso
--2014-09-25 11:44:45--  http://cdimage.ubuntu.com/lubuntu/releases/14.04/release/lubuntu-14.04-desktop-amd64.iso
Resolving cdimage.ubuntu.com (cdimage.ubuntu.com)... 91.189.92.174, 2001:67c:1360:8c01::20
Connecting to cdimage.ubuntu.com (cdimage.ubuntu.com)|91.189.92.174|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 727711744 (694M) [application/x-iso9660-image]
Saving to: ‘lubuntu-14.04-desktop-amd64.iso.1’

 0% [                                                                                                             ] 1,44,218    20.8KB/s  eta 9h 29m 
^C

How do i solve this problem. I am sure that i can retrieve the required file from the downloaded file. Can anyone point me in that direction please.

Answer 1

some remarks:

a) the webserver tells you (if it is a nice webserver) via a http-header-entry about the content-length that is about to come down the line. see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13 this might be the truth or it might be not.

b) if the webserver does not stop sending data after reaching the content-length than the webserver either lied about the content-length or something else.

c) some involved party between your wget and the webserver fiddled with the data.

so, i would at least use 'https' to at least make c) much harder.

the file-listing in http://cdimage.ubuntu.com/lubuntu/releases/14.04/release/ shows that the server itself thinks the content-length is about 694mb. the only valid way to check, if the downloaded file is exactly the one that is on the server (bitwise) is to check the sha256sum downloaded_file against the entry in http://cdimage.ubuntu.com/lubuntu/releases/14.04/release/SHA256SUMS

you might also try either the torrent versions there OR you might want to try the zsync-links (eg, http://cdimage.ubuntu.com/lubuntu/releases/14.04/release/lubuntu-14.04-desktop-amd64.iso.zsync ). what is zsync? well: http://zsync.moria.org.uk/

"zsync is a file transfer program. It allows you to download a file from a remote server, where you have a copy of an older version of the file on your computer already. zsync downloads only the new parts of the file."

in order to " I am sure that i can retrieve the required file from the downloaded file" you would have to be sure that at least "some blocks" of the data you have downloaded equals to some of the blocks of the original file. so, how to test this? you checksum blocks of your downloaded file and check against the checksums of the blocks of the original file. congratulations, you have reinvented the rsync-protocol and that is what zsync uses (to download only the missing blocks).