1182

I have a small utility that I use to download an MP3 file from a website on a schedule and then builds/updates a podcast XML file which I've added to iTunes.

The text processing that creates/updates the XML file is written in Python. However, I use wget inside a Windows .bat file to download the actual MP3 file. I would prefer to have the entire utility written in Python.

I struggled to find a way to actually download the file in Python, thus why I resorted to using wget.

So, how do I download the file using Python?

martineau
124k29 gold badges182 silver badges321 bronze badges
asked Aug 22, 2008 at 15:34
4
  • See also: How to save an image locally using Python whose URL address I already know? Commented Mar 14, 2016 at 11:24
  • 4
    Many of the answers below are not a satisfactory replacement for wget. Among other things, wget (1) preserves timestamps (2) auto-determines filename from url, appending .1 (etc.) if the file already exists (3) has many other options, some of which you may have put in your .wgetrc. If you want any of those, you have to implement them yourself in Python, but it's simpler to just invoke wget from Python. Commented Sep 27, 2016 at 17:22
  • 4
    Short solution for Python 3: import urllib.request; s = urllib.request.urlopen('http://example.com/').read().decode() Commented Nov 26, 2019 at 9:47
  • wget is still a better approach, if you need to automatically retrieve filename and timestamps and handling duplicating files as stackoverflow.com/users/4958/shreevatsar stated. If the urls are variables, one can still handle in python using subprocess. Commented Mar 2, 2023 at 8:19

31 Answers 31

1
2
1426

One more, using urlretrieve:

import urllib.request
urllib.request.urlretrieve("http://www.example.com/songs/mp3.mp3", "mp3.mp3")

(for Python 2 use import urllib and urllib.urlretrieve)

bitmask
35.3k17 gold badges108 silver badges175 bronze badges
answered Aug 22, 2008 at 16:19
Sign up to request clarification or add additional context in comments.

17 Comments

Oddly enough, this worked for me on Windows when the urllib2 method wouldn't. The urllib2 method worked on Mac, though.
Bug: file_size_dl += block_sz should be += len(buffer) since the last read is often not a full block_sz. Also on windows you need to open the output file as "wb" if it isn't a text file.
Me too urllib and urllib2 didn't work but urlretrieve worked well, was getting frustrated - thanks :)
Wrap the whole thing (except the definition of file_name) with if not os.path.isfile(file_name): to avoid overwriting podcasts! useful when running it as a cronjob with the urls found in a .html file
According to the documentation, urllib.request.urlretrieve is a "legacy interface" and "might become deprecated in the future. docs.python.org/3/library/urllib.request.html#legacy-interface
578

Use urllib.request.urlopen():

import urllib.request
with urllib.request.urlopen('http://www.example.com/') as f:
 html = f.read().decode('utf-8')

This is the most basic way to use the library, minus any error handling. You can also do more complex stuff such as changing headers.

On Python 2, the method is in urllib2:

import urllib2
response = urllib2.urlopen('http://www.example.com/')
html = response.read()
answered Aug 22, 2008 at 15:38

8 Comments

This won't work if there are spaces in the url you provide. In that case, you'll need to parse the url and urlencode the path.
Here is the Python 3 solution: stackoverflow.com/questions/7243750/…
Just for reference. The way to urlencode the path is urllib2.quote
@JasonSundram: If there are spaces in it, it isn't a URI.
This does not work on windows with larger files. You need to read all blocks!
438

In 2012, use the python requests library

>>> import requests
>>> 
>>> url = "http://download.thinkbroadband.com/10MB.zip"
>>> r = requests.get(url)
>>> print len(r.content)
10485760

You can run pip install requests to get it.

Requests has many advantages over the alternatives because the API is much simpler. This is especially true if you have to do authentication. urllib and urllib2 are pretty unintuitive and painful in this case.


2015年12月30日

People have expressed admiration for the progress bar. It's cool, sure. There are several off-the-shelf solutions now, including tqdm:

from tqdm import tqdm
import requests
url = "http://download.thinkbroadband.com/10MB.zip"
response = requests.get(url, stream=True)
with open("10MB", "wb") as handle:
 for data in tqdm(response.iter_content(chunk_size=1024), unit="kB"):
 handle.write(data)

This is essentially the implementation @kvance described 30 months ago.

ngoldbaum
5,6833 gold badges33 silver badges37 bronze badges
answered May 24, 2012 at 20:08

15 Comments

How does this handle large files, does everything get stored into memory or can this be written to a file without large memory requirement?
It is possible to stream large files by setting stream=True in the request. You can then call iter_content() on the response to read a chunk at a time.
Why would a url library need to have a file unzip facility? Read the file from the url, save it and then unzip it in whatever way floats your boat. Also a zip file is not a 'folder' like it shows in windows, Its a file.
@Ali: r.text: For text or unicode content. Returned as unicode. r.content: For binary content. Returned as bytes. Read about it here: docs.python-requests.org/en/latest/user/quickstart
I think a chunk_size argument is desirable along with stream=True. The default chunk_size is 1, which means, each chunk could be as small as 1 byte and so is very inefficient.
174
import urllib2
mp3file = urllib2.urlopen("http://www.example.com/songs/mp3.mp3")
with open('test.mp3','wb') as output:
 output.write(mp3file.read())

The wb in open('test.mp3','wb') opens a file (and erases any existing file) in binary mode so you can save data with it instead of just text.

Matthew Strawbridge
20.8k10 gold badges84 silver badges97 bronze badges
answered Aug 22, 2008 at 15:58

5 Comments

The disadvantage of this solution is, that the entire file is loaded into ram before saved to disk, just something to keep in mind if using this for large files on a small system like a router with limited ram.
@tripplet so how would we fix that?
To avoid reading the whole file into memory, try passing an argument to file.read that is the number of bytes to read. See: gist.github.com/hughdbrown/c145b8385a2afa6570e2
@hughdbrown I found your script useful, but have one question: can I use the file for post-processing? suppose I download a jpg file that I want to process with OpenCV, can I use the 'data' variable to keep working? or do I have to read it again from the downloaded file?
Use shutil.copyfileobj(mp3file, output) instead.
166

Python 3

  • urllib.request.urlopen

    import urllib.request
    response = urllib.request.urlopen('http://www.example.com/')
    html = response.read()
    
  • urllib.request.urlretrieve

    import urllib.request
    urllib.request.urlretrieve('http://www.example.com/songs/mp3.mp3', 'mp3.mp3')
    

    Note: According to the documentation, urllib.request.urlretrieve is a "legacy interface" and "might become deprecated in the future" (thanks gerrit)

Python 2

  • urllib2.urlopen (thanks Corey)

    import urllib2
    response = urllib2.urlopen('http://www.example.com/')
    html = response.read()
    
  • urllib.urlretrieve (thanks PabloG)

    import urllib
    urllib.urlretrieve('http://www.example.com/songs/mp3.mp3', 'mp3.mp3')
    
answered Aug 6, 2015 at 13:30

5 Comments

It sure took a while, but there, finally is the easy straightforward api I expect from a python stdlib :)
Very nice answer for python3, see also docs.python.org/3/library/…
@EdouardThiel If you click on urllib.request.urlretrieve above it'll bring you to that exact link. Cheers!
urllib.request.urlretrieve is documented as a "legacy interface" and "might become deprecated in the future".
You should mention that you are getting a bunch of bytes that need to be handled after that.
49
import os,requests
def download(url):
 get_response = requests.get(url,stream=True)
 file_name = url.split("/")[-1]
 with open(file_name, 'wb') as f:
 for chunk in get_response.iter_content(chunk_size=1024):
 if chunk: # filter out keep-alive new chunks
 f.write(chunk)
download("https://example.com/example.jpg")
answered Nov 5, 2018 at 11:28

1 Comment

Thanks, also, replace with open(file_name,... with with open('thisname'...) because it may throw an error
47

use wget module:

import wget
wget.download('url')
msanford
12.4k13 gold badges74 silver badges100 bronze badges
answered Mar 25, 2015 at 12:59

2 Comments

The repo seems to be removed.
project was moved to github, but then archived by its author
28

An improved version of the PabloG code for Python 2/3:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import ( division, absolute_import, print_function, unicode_literals )
import sys, os, tempfile, logging
if sys.version_info >= (3,):
 import urllib.request as urllib2
 import urllib.parse as urlparse
else:
 import urllib2
 import urlparse
def download_file(url, dest=None):
 """ 
 Download and save a file specified by url to dest directory,
 """
 u = urllib2.urlopen(url)
 scheme, netloc, path, query, fragment = urlparse.urlsplit(url)
 filename = os.path.basename(path)
 if not filename:
 filename = 'downloaded.file'
 if dest:
 filename = os.path.join(dest, filename)
 with open(filename, 'wb') as f:
 meta = u.info()
 meta_func = meta.getheaders if hasattr(meta, 'getheaders') else meta.get_all
 meta_length = meta_func("Content-Length")
 file_size = None
 if meta_length:
 file_size = int(meta_length[0])
 print("Downloading: {0} Bytes: {1}".format(url, file_size))
 file_size_dl = 0
 block_sz = 8192
 while True:
 buffer = u.read(block_sz)
 if not buffer:
 break
 file_size_dl += len(buffer)
 f.write(buffer)
 status = "{0:16}".format(file_size_dl)
 if file_size:
 status += " [{0:6.2f}%]".format(file_size_dl * 100 / file_size)
 status += chr(13)
 print(status, end="")
 print()
 return filename
if __name__ == "__main__": # Only run if this file is called directly
 print("Testing with 10MB download")
 url = "http://download.thinkbroadband.com/10MB.zip"
 filename = download_file(url)
 print(filename)
Steve Barnes
28.6k6 gold badges69 silver badges80 bronze badges
answered May 13, 2013 at 8:59

1 Comment

I would remove the parentheses from the first line, because it is not too old feature.
23

Simple yet Python 2 & Python 3 compatible way comes with six library:

from six.moves import urllib
urllib.request.urlretrieve("http://www.example.com/songs/mp3.mp3", "mp3.mp3")
answered Jun 22, 2017 at 7:59

1 Comment

This is the best way to do it for 2+3 compatibility.
21

Following are the most commonly used calls for downloading files in python:

  1. urllib.urlretrieve ('url_to_file', file_name)

  2. urllib2.urlopen('url_to_file')

  3. requests.get(url)

  4. wget.download('url', file_name)

Note: urlopen and urlretrieve are found to perform relatively bad with downloading large files (size> 500 MB). requests.get stores the file in-memory until download is complete.

answered Sep 19, 2016 at 12:45

Comments

21

In python3 you can use urllib3 and shutil libraires. Download them by using pip or pip3 (Depending whether python3 is default or not)

pip3 install urllib3 shutil

Then run this code

import urllib.request
import shutil
url = "http://www.somewebsite.com/something.pdf"
output_file = "save_this_name.pdf"
with urllib.request.urlopen(url) as response, open(output_file, 'wb') as out_file:
 shutil.copyfileobj(response, out_file)

Note that you download urllib3 but use urllib in code

answered Feb 8, 2018 at 17:30

Comments

20

Wrote wget library in pure Python just for this purpose. It is pumped up urlretrieve with these features as of version 2.0.

answered Sep 25, 2013 at 17:55

8 Comments

No option to save with custom filename ?
@Alex added -o FILENAME option to version 2.1
The progress bar does not appear when I use this module under Cygwin.
You should change from -o to -O to avoid confusion, as it is in GNU wget. Or at least both options should be valid.
Is the library been supported in 2023? There is no homepage available on pypi.python.org/pypi/wget.
16

I agree with Corey, urllib2 is more complete than urllib and should likely be the module used if you want to do more complex things, but to make the answers more complete, urllib is a simpler module if you want just the basics:

import urllib
response = urllib.urlopen('http://www.example.com/sound.mp3')
mp3 = response.read()

Will work fine. Or, if you don't want to deal with the "response" object you can call read() directly:

import urllib
mp3 = urllib.urlopen('http://www.example.com/sound.mp3').read()
answered Aug 22, 2008 at 15:58

Comments

12

If you have wget installed, you can use parallel_sync.

pip install parallel_sync

from parallel_sync import wget
urls = ['http://something.png', 'http://somthing.tar.gz', 'http://somthing.zip']
wget.download('/tmp', urls)
# or a single file:
wget.download('/tmp', urls[0], filenames='x.zip', extract=True)

Doc: https://pythonhosted.org/parallel_sync/pages/examples.html

This is pretty powerful. It can download files in parallel, retry upon failure , and it can even download files on a remote machine.

answered Nov 19, 2015 at 23:48

1 Comment

Note this is for Linux only
12

You can use python requests

import os
import requests
outfile = os.path.join(SAVE_DIR, file_name)
response = requests.get(URL, stream=True)
with open(outfile,'wb') as output:
 output.write(response.content)

You can use shutil

import os
import requests
import shutil
 
outfile = os.path.join(SAVE_DIR, file_name)
response = requests.get(url, stream = True)
with open(outfile, 'wb') as f:
 shutil.copyfileobj(response.content, f)
  • If you are downloading from restricted url, don't forget to include access token in headers
answered Dec 13, 2022 at 4:58

Comments

9

You can get the progress feedback with urlretrieve as well:

def report(blocknr, blocksize, size):
 current = blocknr*blocksize
 sys.stdout.write("\r{0:.2f}%".format(100.0*current/size))
def downloadFile(url):
 print "\n",url
 fname = url.split('/')[-1]
 print fname
 urllib.urlretrieve(url, fname, report)
answered Jan 26, 2014 at 13:12

Comments

8

Use Python Requests in 5 lines

import requests as req
remote_url = 'http://www.example.com/sound.mp3'
local_file_name = 'sound.mp3'
data = req.get(remote_url)
# Save file data to local copy
with open(local_file_name, 'wb')as file:
 file.write(data.content)

Now do something with the local copy of the remote file

Jieqin
5885 silver badges21 bronze badges
answered May 16, 2022 at 6:43

2 Comments

Why go through the hassle of importing requests "as req" for just a single call of the package?
data = req.get(remote_url) : This assumes that the entire response contents can fit in memory. Use streaming.
6

If speed matters to you, I made a small performance test for the modules urllib and wget, and regarding wget I tried once with status bar and once without. I took three different 500MB files to test with (different files- to eliminate the chance that there is some caching going on under the hood). Tested on debian machine, with python2.

First, these are the results (they are similar in different runs):

$ python wget_test.py 
urlretrive_test : starting
urlretrive_test : 6.56
==============
wget_no_bar_test : starting
wget_no_bar_test : 7.20
==============
wget_with_bar_test : starting
100% [......................................................................] 541335552 / 541335552
wget_with_bar_test : 50.49
==============

The way I performed the test is using "profile" decorator. This is the full code:

import wget
import urllib
import time
from functools import wraps
def profile(func):
 @wraps(func)
 def inner(*args):
 print func.__name__, ": starting"
 start = time.time()
 ret = func(*args)
 end = time.time()
 print func.__name__, ": {:.2f}".format(end - start)
 return ret
 return inner
url1 = 'http://host.com/500a.iso'
url2 = 'http://host.com/500b.iso'
url3 = 'http://host.com/500c.iso'
def do_nothing(*args):
 pass
@profile
def urlretrive_test(url):
 return urllib.urlretrieve(url)
@profile
def wget_no_bar_test(url):
 return wget.download(url, out='/tmp/', bar=do_nothing)
@profile
def wget_with_bar_test(url):
 return wget.download(url, out='/tmp/')
urlretrive_test(url1)
print '=============='
time.sleep(1)
wget_no_bar_test(url2)
print '=============='
time.sleep(1)
wget_with_bar_test(url3)
print '=============='
time.sleep(1)

urllib seems to be the fastest

answered Nov 3, 2017 at 14:25

1 Comment

There must be something completely horrible going on under the hood to make the bar increase the time so much.
6

Just for the sake of completeness, it is also possible to call any program for retrieving files using the subprocess package. Programs dedicated to retrieving files are more powerful than Python functions like urlretrieve. For example, wget can download directories recursively (-R), can deal with FTP, redirects, HTTP proxies, can avoid re-downloading existing files (-nc), and aria2 can do multi-connection downloads which can potentially speed up your downloads.

import subprocess
subprocess.check_output(['wget', '-O', 'example_output_file.html', 'https://example.com'])

In Jupyter Notebook, one can also call programs directly with the ! syntax:

!wget -O example_output_file.html https://example.com
answered Aug 29, 2018 at 12:24

Comments

5

I wrote the following, which works in vanilla Python 2 or Python 3.


import sys
try:
 import urllib.request
 python3 = True
except ImportError:
 import urllib2
 python3 = False
def progress_callback_simple(downloaded,total):
 sys.stdout.write(
 "\r" +
 (len(str(total))-len(str(downloaded)))*" " + str(downloaded) + "/%d"%total +
 " [%3.2f%%]"%(100.0*float(downloaded)/float(total))
 )
 sys.stdout.flush()
def download(srcurl, dstfilepath, progress_callback=None, block_size=8192):
 def _download_helper(response, out_file, file_size):
 if progress_callback!=None: progress_callback(0,file_size)
 if block_size == None:
 buffer = response.read()
 out_file.write(buffer)
 if progress_callback!=None: progress_callback(file_size,file_size)
 else:
 file_size_dl = 0
 while True:
 buffer = response.read(block_size)
 if not buffer: break
 file_size_dl += len(buffer)
 out_file.write(buffer)
 if progress_callback!=None: progress_callback(file_size_dl,file_size)
 with open(dstfilepath,"wb") as out_file:
 if python3:
 with urllib.request.urlopen(srcurl) as response:
 file_size = int(response.getheader("Content-Length"))
 _download_helper(response,out_file,file_size)
 else:
 response = urllib2.urlopen(srcurl)
 meta = response.info()
 file_size = int(meta.getheaders("Content-Length")[0])
 _download_helper(response,out_file,file_size)
import traceback
try:
 download(
 "https://geometrian.com/data/programming/projects/glLib/glLib%20Reloaded%200.5.9/0.5.9.zip",
 "output.zip",
 progress_callback_simple
 )
except:
 traceback.print_exc()
 input()

Notes:

  • Supports a "progress bar" callback.
  • Download is a 4 MB test .zip from my website.
answered May 13, 2017 at 21:33

1 Comment

works great, run it through jupyter got what i want :-)
4

Source code can be:

import urllib
sock = urllib.urlopen("http://diveintopython.org/")
htmlSource = sock.read() 
sock.close() 
print htmlSource 
Mailerdaimon
6,1503 gold badges38 silver badges47 bronze badges
answered Nov 26, 2013 at 13:21

Comments

4

You can use PycURL on Python 2 and 3.

import pycurl
FILE_DEST = 'pycurl.html'
FILE_SRC = 'http://pycurl.io/'
with open(FILE_DEST, 'wb') as f:
 c = pycurl.Curl()
 c.setopt(c.URL, FILE_SRC)
 c.setopt(c.WRITEDATA, f)
 c.perform()
 c.close()
Guillaume Jacquenot
11.8k6 gold badges46 silver badges51 bronze badges
answered Aug 8, 2018 at 3:51

Comments

4

urlretrieve and requests.get are simple, however the reality not. I have fetched data for couple sites, including text and images, the above two probably solve most of the tasks. but for a more universal solution I suggest the use of urlopen. As it is included in Python 3 standard library, your code could run on any machine that run Python 3 without pre-installing site-package

import urllib.request
url_request = urllib.request.Request(url, headers=headers)
url_connect = urllib.request.urlopen(url_request)
#remember to open file in bytes mode
with open(filename, 'wb') as f:
 while True:
 buffer = url_connect.read(buffer_size)
 if not buffer: break
 #an integer value of size of written data
 data_wrote = f.write(buffer)
#you could probably use with-open-as manner
url_connect.close()

This answer provides a solution to HTTP 403 Forbidden when downloading file over http using Python. I have tried only requests and urllib modules, the other module may provide something better, but this is the one I used to solve most of the problems.

answered Mar 13, 2017 at 13:12

Comments

4

Late answer, but for python>=3.6 you can use:

import dload
dload.save(url)

Install dload with:

pip3 install dload
answered Feb 24, 2020 at 7:12

2 Comments

Can I ask - where does the file save once the program runs? Also, is there a way to name it and save it in a specific location? This is the link I am working with - when you click the link it immediately downloads an excel file: ons.gov.uk/generator?format=xls&uri=/economy/…
You can supply the save location as second argument, e.g.: dload.save(url, "/home/user/test.xls")
3

This may be a little late, But I saw pabloG's code and couldn't help adding a os.system('cls') to make it look AWESOME! Check it out :

 import urllib2,os
 url = "http://download.thinkbroadband.com/10MB.zip"
 file_name = url.split('/')[-1]
 u = urllib2.urlopen(url)
 f = open(file_name, 'wb')
 meta = u.info()
 file_size = int(meta.getheaders("Content-Length")[0])
 print "Downloading: %s Bytes: %s" % (file_name, file_size)
 os.system('cls')
 file_size_dl = 0
 block_sz = 8192
 while True:
 buffer = u.read(block_sz)
 if not buffer:
 break
 file_size_dl += len(buffer)
 f.write(buffer)
 status = r"%10d [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)
 status = status + chr(8)*(len(status)+1)
 print status,
 f.close()

If running in an environment other than Windows, you will have to use something other then 'cls'. In MAC OS X and Linux it should be 'clear'.

answered Oct 14, 2013 at 2:54

3 Comments

cls doesn't do anything on my OS X or nor on an Ubuntu server of mine. Some clarification could be good.
I think you should use clear for linux, or even better replace the print line instead of clearing the whole command line output.
this answer just copies another answer and adds a call to a deprecated function (os.system()) that launches a subprocess to clear the screen using a platform specific command (cls). How does this have any upvotes?? Utterly worthless "answer" IMHO.
3

New Api urllib3 based implementation

>>> import urllib3
>>> http = urllib3.PoolManager()
>>> r = http.request('GET', 'your_url_goes_here')
>>> r.status
 200
>>> r.data
 *****Response Data****

More info: https://pypi.org/project/urllib3/

answered Jul 2, 2021 at 12:00

Comments

2

Another possibility is with built-in http.client:

from http import HTTPStatus, client
from shutil import copyfileobj
# using https
connection = client.HTTPSConnection("www.example.com")
with connection.request("GET", "/noise.mp3") as response:
 if response.status == HTTPStatus.OK:
 copyfileobj(response, open("noise.mp3")
 else:
 raise Exception("request needs work")

The HTTPConnection object is considered "low-level" in that it performs the desired request once and assumes the developer will subclass it or script in a way to handle the nuances of HTTP. Libraries such as requests tend to handle more special cases such as automatically following redirects and so on.

answered Oct 17, 2022 at 13:19

Comments

2

Options like:

  1. wget.download('url', file_name),
  2. urllib.urlretrieve ('url_to_file', file_name),

Have the flaw that it's not possible to setup User Agent. That may cause issues while downloading the files from external sites (they're blocking particular User Agents so you have to pass a custom one).

I found urllib3 much handy.

import urllib3
resp = urllib3.request('GET', 'http://www.example.com/sound.mp3', preload_content=False, headers={'User-Agent': 'Customer User Agent If Needed'})
with open('sound.mp3', 'wb') as f:
 for chunk in resp.stream(65536):
 f.write(chunk)
resp.release_conn()
answered Nov 30, 2023 at 10:12

Comments

1

I wanted do download all the files from a webpage. I tried wget but it was failing so I decided for the Python route and I found this thread.

After reading it, I have made a little command line application, soupget, expanding on the excellent answers of PabloG and Stan and adding some useful options.

It uses BeatifulSoup to collect all the URLs of the page and then download the ones with the desired extension(s). Finally it can download multiple files in parallel.

Here it is:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from __future__ import (division, absolute_import, print_function, unicode_literals)
import sys, os, argparse
from bs4 import BeautifulSoup
# --- insert Stan's script here ---
# if sys.version_info >= (3,): 
#...
#...
# def download_file(url, dest=None): 
#...
#...
# --- new stuff ---
def collect_all_url(page_url, extensions):
 """
 Recovers all links in page_url checking for all the desired extensions
 """
 conn = urllib2.urlopen(page_url)
 html = conn.read()
 soup = BeautifulSoup(html, 'lxml')
 links = soup.find_all('a')
 results = [] 
 for tag in links:
 link = tag.get('href', None)
 if link is not None: 
 for e in extensions:
 if e in link:
 # Fallback for badly defined links
 # checks for missing scheme or netloc
 if bool(urlparse.urlparse(link).scheme) and bool(urlparse.urlparse(link).netloc):
 results.append(link)
 else:
 new_url=urlparse.urljoin(page_url,link) 
 results.append(new_url)
 return results
if __name__ == "__main__": # Only run if this file is called directly
 # Command line arguments
 parser = argparse.ArgumentParser(
 description='Download all files from a webpage.')
 parser.add_argument(
 '-u', '--url', 
 help='Page url to request')
 parser.add_argument(
 '-e', '--ext', 
 nargs='+',
 help='Extension(s) to find') 
 parser.add_argument(
 '-d', '--dest', 
 default=None,
 help='Destination where to save the files')
 parser.add_argument(
 '-p', '--par', 
 action='store_true', default=False, 
 help="Turns on parallel download")
 args = parser.parse_args()
 # Recover files to download
 all_links = collect_all_url(args.url, args.ext)
 # Download
 if not args.par:
 for l in all_links:
 try:
 filename = download_file(l, args.dest)
 print(l)
 except Exception as e:
 print("Error while downloading: {}".format(e))
 else:
 from multiprocessing.pool import ThreadPool
 results = ThreadPool(10).imap_unordered(
 lambda x: download_file(x, args.dest), all_links)
 for p in results:
 print(p)

An example of its usage is:

python3 soupget.py -p -e <list of extensions> -d <destination_folder> -u <target_webpage>

And an actual example if you want to see it in action:

python3 soupget.py -p -e .xlsx .pdf .csv -u https://healthdata.gov/dataset/chemicals-cosmetics
answered Mar 6, 2020 at 0:17

Comments

-3

You can use keras.utils.get_file to do it:

from tensorflow import keras
path_to_downloaded_file = keras.utils.get_file(
 fname="file name",
 origin="https://www.linktofile.com/link/to/file",
 extract=True,
 archive_format="zip", # downloaded file format
 cache_dir="/", # cache and extract in current directory
)
answered Nov 15, 2022 at 12:33

Comments

1
2

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.