keep server running on EC2 instance after ssh is terminated

Question

Currently, I have two servers running on an EC2 instance (MongoDB and bottlepy). Everything works when I SSHed to the instance and started those two servers. However, when I closed the SSH session (the instance is still running), I lost those two servers. Is there a way to keep the server running after logging out? I am using Bitvise Tunnelier on Windows 7. The instance I am using is Ubuntu Server 12.04.3 LTS.

Answer 1

For those landing here from a google search, I would like to add tmux as another option. tmux is widely used for this purpose, and is preinstalled on new Ubuntu EC2 instances.

Managing a single session

Here is a great answer by Hamish Downer given to a similar question over at askubuntu.com:

I would use a terminal multiplexer - screen being the best known, and tmux being a more recent implementation of the idea. I use tmux, and would recommend you do to.

Basically tmux will run a terminal (or set of terminals) on a computer. If you run it on a remote server, you can disconnect from it without the terminal dying. Then when you login in again later you can reconnect, and see all the output you missed.

To start it the first time, just type

tmux

Then, when you want to disconnect, you do Ctrl+B, D (ie press Ctrl+B, then release both keys, and then press d)

When you login again, you can run

tmux attach

and you will reconnect to tmux and see all the output that happened. Note that if you accidentally lose the ssh connection (say your network goes down), tmux will still be running, though it may think it is still attached to a connection. You can tell tmux to detach from the last connection and attach to your new connection by running

tmux attach -d

In fact, you can use the -d option all the time. On servers, I have this in my >.bashrc

alias tt='tmux attach -d'

So when I login I can just type tt and reattach. You can go one step further >if you want and integrate the command into an alias for ssh. I run a mail client >inside tmux on a server, and I have a local alias:

alias maileo='ssh -t mail.example.org tmux attach -d'

This does ssh to the server and runs the command at the end - tmux attach -d The -t option ensures that a terminal is started - if a command is supplied then it is not run in a terminal by default. So now I can run maileo on a local command line and connect to the server, and the tmux session. When I disconnect from tmux, the ssh connection is also killed.

This shows how to use tmux for your specific use case, but tmux can do much more than this. This tmux tutorial will teach you a bit more, and there is plenty more out there.

Managing multiple sessions

This can be useful if you need to run several processes in the background simultaneously. To do this effectively, each session will be given a name.

Start (and connect to) a new named session:

tmux new-session -s session_name

Detach from any session as described above: Ctrl+B, D.

List all active sessions:

tmux list-sessions

Connect to a named session:

tmux attach-session -t session_name

To kill/stop a session, you have two options. One option is to enter the exit command while connected to the session you want to kill. Another option is by using the command:

tmux kill-session -t session_name

Answer 2

If you don't want to run some process as a service (or via an apache module) you can (like I do for using IRC) use gnome-screen Install screen http://hostmar.co/software-small.

screen keeps running on your server even if you close the connection - and thus every process you started within will keep running too.

Answer 3

It would be nice if you provided more info about your environment but assuming it's Ubuntu Linux you can start the services in the background or as daemons.

sudo service mongodb start
nohup python yourbottlepyapp.py &

(Use nohup if you want are in a ssh session and want to prevent it from closing file descriptors)

You can also run your bottle.py app using Apache mod_wsgi. (Running under the apache service) More info here: http://bottlepy.org/docs/dev/deployment.html

Hope this helps.

Addition: (your process still runs after you exit the ssh session)

Take this example time.py

 import time
 time.sleep(3600)

Then run:

 $ python3 time.py &
 [1] 3027
 $ ps -Af | grep -v grep | grep  time.py
 ubuntu    3027  2986  0 18:50 pts/3    00:00:00 python3 time.py
 $ exit

Then ssh back to the server

 $ ps -Af | grep -v grep | grep  time.py
 ubuntu    3027     1  0 18:50 ?        00:00:00 python3 time.py

Process still running (notice with no tty)

Answer 4

You can use pm2 to keep your Node.js application running on an EC2 instance after disconnecting from SSH.

PM2 is a process manager for Node.js applications that provides built-in support for keeping applications running in the background and automatically restarting them if they crash. You can start your Node.js application with PM2 and it will continue running even after you disconnect from the EC2 instance.

• https://www.npmjs.com/package/pm2
• https://github.com/Unitech/pm2
• https://pm2.keymetrics.io/

Here are the basic steps to use PM2:

Install PM2: npm install pm2 -g

Start your Node.js application with PM2: pm2 start <your-app.js>

ubuntu@ip-172-31-4-234:~$ pm2 start index.js
[PM2] Applying action restartProcessId on app [index](ids: [ 0 ])
[PM2] [index](0) ✓
[PM2] Process successfully started
┌────┬──────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ index    │ default     │ N/A     │ fork    │ 4559     │ 0s     │ 16   │ online    │ 0%       │ 7.2mb    │ ubuntu   │ disabled │

Verify that your application is running: pm2 list

ubuntu@ip-172-31-4-234:~$ pm2 list
┌────┬──────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ index    │ default     │ N/A     │ fork    │ 4040     │ 110s   │ 15   │ online    │ 0%       │ 58.5mb   │ ubuntu   │ disabled │
└────┴──────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴────────���──┴──────────┴──────────┴──────────┴──────────┘

To stop your application: pm2 stop <app-name-or-id>

ubuntu@ip-172-31-4-234:~$ pm2 stop index
[PM2] Applying action stopProcessId on app [index](ids: [ 0 ])
[PM2] [index](0) ✓
┌────┬──────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ index    │ default     │ N/A     │ fork    │ 0        │ 0      │ 15   │ stopped   │ 0%       │ 0b       │ ubuntu   │ disabled │

To restart your application: pm2 restart <app-name-or-id>

ubuntu@ip-172-31-4-234:~$ pm2 restart index
Use --update-env to update environment variables
[PM2] Applying action restartProcessId on app [index](ids: [ 0 ])
[PM2] [index](0) ✓
┌────┬──────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ index    │ default     │ N/A     │ fork    │ 4528     │ 0s     │ 15   │ online    │ 0%       │ 15.4mb   │ ubuntu   │ disabled │
└────┴──────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴────────���──┴──────────┴──────────┴──────────┴──────────┘

To view logs for your application: pm2 logs <app-name-or-id>

ubuntu@ip-172-31-4-234:~$ pm2 logs index
[TAILING] Tailing last 15 lines for [index] process (change the value with --lines option)
/home/ubuntu/.pm2/logs/index-error.log last 15 lines:
0|index    |     at listenInCluster (node:net:1788:12)
0|index    |     at Server.listen (node:net:1876:7)
0|index    |     at Function.listen (webpack://zen-technical-interview-1/./node_modules/express/lib/application.js?:635:24)
0|index    |     at eval (webpack://zen-technical-interview-1/./src/index.ts?:70:5)
0|index    |     at ./src/index.ts (/home/ubuntu/index.js:1315:1)
0|index    |     at __webpack_require__ (/home/ubuntu/index.js:1690:42)
0|index    |     at /home/ubuntu/index.js:1719:37
0|index    |     at Object.<anonymous> (/home/ubuntu/index.js:1721:12)
0|index    |     at Module._compile (node:internal/modules/cjs/loader:1254:14) {
0|index    |   code: 'EADDRINUSE',
0|index    |   errno: -98,
0|index    |   syscall: 'listen',
0|index    |   address: '::',
0|index    |   port: 3000
0|index    | }

/home/ubuntu/.pm2/logs/index-out.log last 15 lines:
0|index    | --- [
0|index    |   { scenario: 1, provider: { id: 2, name: 'Provider 2' }, price: 250 },
0|index    |   { scenario: 2, provider: { id: 1, name: 'Provider 1' }, price: 300 },
0|index    |   { scenario: 3, provider: { id: 3, name: 'Provider 3' }, price: 200 },

To start PM2 automatically when the server starts, you can use the pm2 startup command, which will generate a startup script and add it to the system startup configuration.

By using PM2, you can ensure that your Node.js application remains running in the background even if you disconnect from your EC2 instance, and can easily manage your application with PM2's built-in commands.

Answer 5

You will want the started services to disconnect from the controlling terminal. I would suggest you use nohup to do that, e.g.

ssh my.server "/bin/sh -c nohup /path/to/service"

you may need to put an & in there (in the quotes) to run it in the background.

As others have commented, if you run proper init scripts to start/stop services (or ubuntu's service command), you should not see this issue.

Answer 6

If on Linux based instances putting the job in the background followed by disown seems to do the job.

$ ./script & $ disown