Bash: execute in parallel some processes but wait the slowest

Question

Hello I'm writing a bash script where I have to restore around 8+ big sets of ip with ipset-restore command.

Actually my shell script basically has a list like

ipset-restore < list1.txt ipset-restore < list2.txt ipset-restore < list3.txt ...

doing it in this way is pretty slow and because machine has resources and ipset is multhitreaded and can perform various operations in parallel I would like to cast many ipset-restore at once.

I tought of using nohup however doing it this way would make me being unable to see if the processes have finished before carry out new instructions.

I need that every process have finished before proceeding next.

(basically I need to wait for the slowest before proceeding).

Bonus: it exists a linux package to do so with also monitoring the resources so that spawn of processes is handled automatically without hugging all the resources?

Answer 1

To have all the jobs running in parallel, you should have run them as background jobs, while the main script continues on creating them all. This is done with a command like:

process &

For more information see Running Linux Commands in Background and Foreground.

Once all the commands were issued and are running in the background, you may wait for all of them to finish by using the the wait command, which is described as:

If the wait utility is invoked with no operands, it shall wait until all process IDs known to the invoking shell have terminated and exit with a zero exit status.

If you run into problems with it waiting for too many processes, you could use a solution that keeps the PIDs of the started processes and waits for them only. See the post How to wait in bash for several subprocesses to finish, and return exit code !=0 when any subprocess ends with code !=0?.