How to prevent currently running Linux task to be killed after logout from SSH?

Question

I have a computational task running for a few days on a linux server via SSH connection. It is not on the background, so the SSH connection is under hold of the task.

I want to restart the local machine (not the server) from which I have opened SSH session, but want to keep the task running.

Is that any possible?

Answer 1

If your task is already launched, it is too late¹ to consider alternative solutions that insert an additional layer between your ssh session and the shell running the command, like screen, tmux, byobu, nohup and the likes.

If your process support to be placed in the background and particularly doesn't hang when stdout and stderr are unwritable/closed, you can put it in the background before logging out with ControlZ and bg then detach it from your shell with the disown builtin.

eg:

$ ssh localhost
You have new mail.
Last login: Fri Jun  6 11:26:56 2014

$ /bin/sleep 3600    


^Z[1] + Stopped                  /bin/sleep 3600
$ bg
[1] /bin/sleep 3600&
$ jobs
[1] +  Running                 /bin/sleep 3600
$ disown %1
$ exit
Connection to localhost closed.
$ ps -ef|grep sleep
jlliagre 12864     1  0 21:12 ?        00:00:00 /bin/sleep 3600
jlliagre 13056 12477  0 21:13 pts/18   00:00:00 grep sleep
$ 

¹ As Bob commented, there are actually several hackish ways to reparent a tty session under Linux. repty, retty, injcode and neercs. The most advanced looks to be reptyr but you might need root privileges to enable ptrace to hack your process.

Answer 2

One solution is to use GNU screen. You could start up screen, run your command, then detach with C-a d. Later, to reconnect, do screen -r, and you are back in your previous session.

Other benefits of screen are window management (so you can switch to other shells while your command is running, without needing a new SSH connection), and it allows your command to remain in the foreground, whether in the current session or a later one.

Edit: As noted in the comments, this will only work if you remember to start screen before running the command. If the command is already running, then you will need @jlliagre's solution.

Answer 3

One of the "standard" ways to do so, is to use the nohup command, included in coreutils, like this:

nohup COMMAND [ARGS] &

But the command will redirect the output (STDOUT & STDERR AFAIK) of the program into a file nohup.out, making it somehow annoying sometimes (like generating a huge log file), so you may want to make your own redirection, or redirect it to /dev/null if you want.

Answer 4

In addition to nohup you can launch the process in background using "&" and subshell:

Suffix the command with & and wrap it in parentheses

$ (thecommand args &)

Lets say your process gains the pid 1922:

$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
usr      11473  2643  0 15:07 pts/1    00:00:00 bash
usr      11922     1  0 15:11 pts/1    00:00:00 thecommand

Look that it's unattached from the shell process 11473 which was its original parent. So, if you exit or kill the current shell (11473), the process 11922 keep running, and it will be disconnected from the pts.

Try to exit the shell, and enter in a new shell. Even if, this shell is connected to the same pts, you can see the process now without a pts:

$ ps -fp 11922
UID        PID  PPID  C STIME TTY          TIME CMD
usr      11922     1  0 15:11 ?        00:00:00 thecommand

I don't how it's called or documented in Posix, but I use this way since 1990 in bsh, ksh and now in bash.

Lastly, you can use the bg builtin shell command:

Just launch your program and if you decide to pause it, or plan to keep it in background, type CTRL+Z:

$ thecommand
^Z
[1] Stopped          thecommand

Now let it continue to run in background:

$ bg
[1]+ thecommand &

If you look to the process infos, it has yet a parent process:

$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
usr      12046  2643  0 15:18 pts/6    00:00:00 bash
usr      12571 12046  0 16:00 pts/6    00:00:00 thecommand
usr      12601 12046  0 16:04 pts/6    00:00:00 ps -f

So, exit from the current process. The running process in background is unattached from its original parent and keep running in background:

$ exit

Login again and look at process info:

$ ps -f 12571
UID        PID  PPID  C STIME TTY          TIME CMD
usr      12571     1  0 16:00 ?        00:00:00 thecommand