Can a process that was started in a tmux session fall asleep? If yes, what is the cause(s), how to prevent it?
Example reason for the question: I started a process on a server yesterday (training neural networks, it prints the current training epoch to stdout). I had a split window, and in the one with the process running, I had activated scroll mode before detaching from the session.
Today I come back, and it has made no progress at all.
More specifically, the epoch is the same. After quitting scroll mode, it now happily continued.
The log reads something like
...
Epoch 40: 1h few mins
Epoch 41: 12h few mins
Epoch 42: 12h few more mins
...
Epoch 73: 13h
Meaning, the time it took to get from epoch 0 to 49 was definitely less than two hours; from epoch 40 to 41 it took around 11 hours (!), from epoch 41 to 76 average time per epoch was around 1.7 minutes. The epochs are in a loop, and there shouldn't be a reason why one takes around 400 times longer than the others.
Additional information: This 'sleeping' doesn't happen every time I detach while being in scroll mode. But it already happened before. The scroll mode might not have anything to do with it at all.
The program is a python script, including tensorflow code running on a GPU; the command to run it was :
python train_script.py 2>&1 | tee train_log.txt.
For tmux I use tmux attach
to re-attach, the standard key mapping and ctrl-b + d
to detach, ctrl-b + up(number block)
to start scrolling, q
to quit scroll mode.