Python download a file

Question

I'm unsure how to do this. One way is:

import urllib.request;
urllib.request.urlretrieve('www.example.com/file.tar', 'file.tar')

Another way would be:

import urllib.request;

#Set as appropriate
userAgent = ....;

req = urllib.request.Request('www.example.com/file.tar', headers={'User-Agent' : userAgent});
response = urllib.request.urlopen(req);

#Save the file
f = open('file.tar', 'wb');
f.write(response.read());
f.close()

I'm not sure which method to use. I'll be downloading many files (with a pattern filename) in a loop. However, I would like to be able to set up a user-agent header. It's not critical but I'd like to.

EDIT: I forgot to mention that I prefer the first method but I don't know how to set the user-agent header with urlretrieve.

Answer 1

I am moving what started as comments, to an answer...

Your second example is pretty much doing what it needs to, in making a request object with a custom header and then reading the results into a local file.

urlretrieve is a higher level function so it only does exactly what the docs say: Downloads a network resource to a local file and tells you where the file is. If you don't like the slightly lower level approach of your second example and you want more higher-level functionality, you can look into using the Requests library

Answer 2

As @jdi said, you may use the requests library. This is also mentioned on https://docs.python.org/2/library/urllib2.html. You will need to pip the library, e.g.

pip install requests

My code looks like this:

import requests

def download_file(url):
    file = requests.get(url)
    return file.text

It can't be easier.