Run a bash command when a process is done

Question

I have two scripts that use GPU and train ML models. I want to start them before I go to sleep so they work at the night and I expect to see some results in the morning.

But because of the GPU memory is limited, I want to run them in serial instead of parallel.

I can do it with python train_v1.py && python train_v2.py; but let's say I started to train the train_v1. In the mean time, because the training takes long time, I started and finished the implementation of the second script, train_v2.py, and I want to run it automatically when python train_v1.py is finished.

How can I achieve that? Thank you.

Answer 1

Here's an approach that doesn't involve looping and checking if the other process is still alive, or calling train_v1.py in a manner different from what you'd normally do:

$ python train_v1.py
^Z
[1]+  Stopped                 python train_v1.py
$ % && python train_v2.py

The ^Z is me pressing Ctrl+Z while the process is running to sleep train_v1.py through sending it a SIGTSTP signal. Then, I tell the shell to wake it with %, using it as a command to which I can add the && python train_v2.py at the end. This makes it behave just as if you'd done python train_v1.py && python train_v2.py from the very beginning.

Instead of %, you can also use fg. It's the same thing. If you want to learn more about these types of features of the shell, you can read about them in the "JOB CONTROL" section of bash's manpage.

EDIT: How to keep adding to the queue

As pointed out by jamesdlin in a comment, if you try to continue the pattern to add train_v3.py for example before v2 starts, you'll find that you can't:

$ % && python train_v2.py
^Z
[1]+  Stopped                 python train_v1.py

Only train_v1.py gets stopped because train_v2.py hasn't started, and you can't stop/suspend/sleep something that hasn't even started.

$ % && python train_v3.py

would result in the same as

python train_v1.py && python train_v3.py

because % corresponds to the last suspended process. Instead of trying to add v3 like that, one should instead use history:

$ !! && python train_v3.py
% && python train_v2.py && python train_v3.py

One can do history expansion like above, or recall the last command with a keybinding (like up) and add v3 to the end.

$ % && python train_v2.py && python train_v3.py

That's something that can be repeated to add more to the pipeline.

$ !! && python train_v3.py
% && python train_v2.py && python train_v3.py
^Z
[1]+  Stopped                 python train_v1.py
$ !! && python train_v4.py
% && python train_v2.py && python train_v3.py && python train_v4.py

Answer 2

If you have already started python train_v1.py, you could possibly use pgrep to poll that process until it disappears, and then run your second Python script:

while pgrep -u "$USER" -fx 'python train_v1.py' >/dev/null
do
    # sleep for a minute
    sleep 60
done
python train_v2.py

By using -f and -x you match against the exact command line that was used to launch the first Python script. On some systems, pgrep implements a -q option, which makes it quiet (just like grep -q), which means that the redirection to /dev/null wouldn't be needed.

The -u option restricts the match to commands that you are running (and not a friend or other person on the same system).

If you haven't started the first script yet:

As mentioned in comments, you could just launch the second script straight after the first script. The fact that the second script does not exist, or isn't quite ready to run yet, does not matter (as long as it ready to run when the first script finishes):

python train_v1.py; python train_v2.py

Doing it this way will launch the second script regardless of the exit status of the first script. Using && instead of ;, as you show in the question, will also work, but will require the first script to finish successfully for the second script to start.

Answer 3

You can launch the first script with

python train_v1.py; touch finished

Then simply make a loop that checks regularly if finished exists:

while [ ! -f finished ] ; do     
    sleep 5
done
python train_v2.py
rm finished

Answer 4

You can always simply wait for a running program by passing the program ID as a parameter. The PID can be obtained from a ps call. To make it robust in a script is difficult, which is why programs which anticipate that somebody wants to wait for (or kill) them usually write their own PID into a known location. But for the interactive situation you describe looking the PID up and copying it is easy enough.

Answer 5

If you don't need to know the exit status of the first script, then I recommend something like what Kusalananda wrote.

If you do need to know the exit status (which you probably don't in this case, but someone else may come along looking for a solution that does this), it's more complicated. I've written a small Linux utility pwait that lets you wait for a process to finish and find out its exit status.

Answer 6

If you are working in Linux (with a bash avaliable) it would be as easy as:

TL;DR: use this:

((python train_v1.py; kill $(cat ./pid) && rm ./pid) &

python train_v2.py & echo $! > pid

The longer version:

You create a sub-shell where your training V1 is running - when V1 finished (regardless of the return code - if you want to check for a return code of 0, use && instead of ;) the output of a file named pid is provided to the kill application and if kill returns successful the pid file is removed.

Afterwards you start the second training and write the process ID of the second training to a file named pid and there the cycle is closed. now both training programs run in the background until python train_v1.py is finished.