How do you download multiple files/videos, simultaneously, on a basic digitalocean droplet/VPS?

Question

How do you download multiple files/videos, simultaneously, on a basic digitalocean droplet?

Let's say I create a basic digitalocean droplet/VPS, that runs Ubuntu

I start the console

I do apt install python and wget https://youtube-dl.org/downloads/latest/youtube-dl and chmod 777 youtube-dl

There are some files that I want to download e.g. these two video files

https://www.youtube.com/watch?v=iXFvkJLqPqo (the johnny depp one, about 11min long)

and

https://www.youtube.com/watch?v=-QwtKBC6iQY (a ricky gervais one)

I've got a file with the two videos

user@basicdroplettest1:~# cat vidfilelist.txt 
https://www.youtube.com/watch?v=iXFvkJLqPqo
https://www.youtube.com/watch?v=-QwtKBC6iQY 

If I were to do while read in; do ./youtube-dl "$in"; done < vidfilelist

it runs youtube-dl on each line of the file so it does e.g. youtube-dl https://www.youtube.com/watch?v=iXFvkJLqPqo and then youtube-dl https://www.youtube.com/watch?v=-QwtKBC6iQY

So it downloads the first one, then when that one has completed, it starts downloading the second one.

So it's not simultaneous

If I put an ampersand at the end of each line

user@basicdroplettest1:~# cat ./vidfilelist.txt 
https://www.youtube.com/watch?v=iXFvkJLqPqo &
https://www.youtube.com/watch?v=-QwtKBC6iQY &

Then do

(this bash while command courtesy of https://stackoverflow.com/questions/13939038/how-do-you-run-a-command-eg-chmod-for-each-line-of-a-file )

user@basicdroplettest1:~# while read in; do ./youtube-dl "$in"; done < vidfilelist.txt 
[youtube] iXFvkJLqPqo: Downloading webpage
[download] Destination: Top Johnny Depp Comebacks & Reactions to Questioning While Testifying-iXFvkJLqPqo.mp4
[download]   2.5% of 20.67MiB at 50.65KiB/s ETA 06:47

[download]   3.0% of 20.67MiB at 50.90KiB/s ETA 06:43



[download]   3.2% of 20.67MiB at 50.95KiB/s ETA 06:41

[download]   5.5% of 20.67MiB at 51.04KiB/s ETA 06:31

See above, i'm hitting ENTER, but it's not going into the background and onto the next one..

So that hasn't worked.

Any ideas how I can get it to download each of the video files in the file, in the background?

(as opposed to downloading one, followed by the other).

Answer 1

You're on the right track with the &, but you don't want that as part of the file name. You want to run this:

youtube-dl "https://www.youtube.com/watch?v=iXFvkJLqPqo" &

But because you have the & in the file, you are actually running this:

youtube-dl "https://www.youtube.com/watch?v=iXFvkJLqPqo &"

Also, you probably want to redirect standard error so your terminal isn't polluted by the progress reports. Try this:

while read in; do ./youtube-dl "$in" 2>/dev/null & done < vidfilelist 

This should be fine for a list of URLs, but as a general rule when parsing text files in the shell, you also want to clear the input field separator with IFS= and ensure you don't expand \-escape sequences, so do this:

while IFS= read -r in; do ./youtube-dl "$in" 2>/dev/null & done < vidfilelist 

Answer 2

Besides Terdon's answer.

I'd say

a)use yt-dlp , because it's much faster than youtube-dl. It's a fork of youtube-dl.

b)use screen (the screen command). Because otherwise when you disconnect the session, the ssh session, by e.g. closing the window, then the download stops! Run screen hit ENTER. There are shortcuts like ctrl-a , ctrl-d, and a command screen -r, that cover to attach and detact from "screens", and screen -ls to list "screens". There are youtube videos about the screen command. It can also be used to create a kind of tiled view but you don't need that feature.

c) If you choose to not download them simultaneously, eg concern that it might risk getting blocked, yt-dlp can be passed a filename with a list of URLs. And there might be an option to pause between each one, or you could just have a list of commands to run(eg yt-dlp http://..../watch?v=adfsdf), with sleeps in between them.

d) If you have a list of youtube URLs, there could be some different file formats eg one might come out as .webm And if for some reason a file didn't download properly then one might come out as .part or something like that. So use commands like DIR or if *nix then ls and find, to see not just the total number of files downloaded, but what file types.

Answer 3

TLTR; This answer may help someone

From any bash shell you could use youtube-dl

Edit: yt-dlp is an enhanced fork of youtube-dl and could be a replacement in any script as @barlop mentioned (but it seems to be instailable and upgradable only from pip - not the system package manager)

---

youtube-dl installation for Ubuntu

Package manager (apt / yum / ... repo)

sudo apt install youtube-dl

only youtube-dl : this method isn't applicable for yt-dlp

python package (python scriptable) (mostly updated)

# you will need pip installed
sudo apt install python3 python3-pip

# always use latest version or upgrade
# you may use sudo for system-wide installation (not recommended)
# if the instlation is successful and still command-not-found
# - add python's `dist-packages` dir to path on your system
pip install -U youtube-dl

keep in mind: most of time only latest version is the working one

---

Download one link videos (file/playlist/search)

# replace `mp4` with extention needed
# replace `https://link/to/download` with desired download link
youtube-dl -f mp4 "https://link/to/download"

---

Download multible files

From file containing links

while IFS= read -r line; do ( youtube-dl "$line" ); done < file.txt 

refer to @terdon answer on this question for silent interpreting

From a search link or playlist you could just let youtube-dl download it as files

or refer to this answer of my past question for a more structured download https://stackoverflow.com/a/67144906/9907243