Is there a program that can send me a notification e-mail when a process finishes?

Question

I am a computational scientist, and I run a lot of lengthy calculations on Linux. Specifically, I run molecular dynamics (MD) simulations using the GROMACS package. These simulations can take days or weeks, running on (for example) 8 to 24 cores. I have access to several nodes of a cluster, which means that at any given time, I am running approximately 4 or 5 jobs (each on a different node, and each on 8-24 cores).

The problem is that the simulation take a variable amount of time. I like to keep all nodes working on simulations around the clock, but to start a new simulation, I need to do log in with a terminal and do some manual work. But I always forget how much time is left in a simulation, so I always end up constantly checking them.

Is there any way that I can receive an e-mail when a Linux process finishes? Could there be a Linux program that does this? That way I would know when to log in with a terminal and prepare the next simulation.

I am using Ubuntu Linux. Thanks for your time.

Answer 1

Yeah, there is

command; echo "Process done" | mail -s "Process done" [email protected]

Where -s "text" is the subject, the echo gives mail some Text to send to you.

Answer 2

Jobs submitted to the at daemon will send any output to you from stderr and stdout upon completion. It can also be configured to send mail even if the job has no output. It also has the benefit of running without a controlling terminal, so you don't have to worry about the effect that closing your terminal will have on the job.

example:

echo "/opt/product/bin/job.sh data123"|at -m NOW

When this job completes, the user who submitted the job will receive an email, and if there is any output at all you will receive it. You can change the email recipient by changing the LOGNAME environment variable.

at has a batch mode where you can queue jobs to run when the system is not busy. This is not a very good queueing system when multiple users are competing for resources, but nonetheless, if you wanted to run jobs with it:

echo "/opt/product/bin/job.sh dataA"|batch
echo "/opt/product/bin/job.sh dataB"|batch
echo "/opt/product/bin/job.sh dataC"|batch

By default the jobs will not start unless the system load is under 1.5, but that load figure can be adjusted (and with 24 cores I'd say you should). They can run in parallel if they don't bump the loadavg over the load limit (1.5 default again), or if they individually bump the loadavg over 1.5, they will run in serial.

You can view the job queue with atq, and delete jobs with atrm

Answer dependencies:

1. Running atd daemon ( ps -ef|grep atd )
2. You are allowed to submit jobs to atd (not denied by /etc/at.deny//etc/at.allow configurations)
3. Functional sendmail MTA

Most systems have no problem with these requirements, but it's worthwhile to check.

Answer 3

I would recommend setting a python script to send an email, they are very easy to write and configure the correct mailing servers for whatever service you use. They look something like this:

#!/usr/bin/python

import smtplib

sender = '[email protected]'
receivers = ['[email protected]']

message = """From: From Person <[email protected]>
To: To Person <[email protected]>
Subject: SMTP e-mail test

This is a test e-mail message.
"""

try:
   smtpObj = smtplib.SMTP('localhost')
   smtpObj.sendmail(sender, receivers, message)         
   print "Successfully sent email"
except SMTPException:
   print "Error: unable to send email"

Use this in conjunction with the pipe operator suggested in the other answers.

Another great solution I have found to this problem is using Pushover. Pushover - "Pushover makes it easy to send real-time notifications to your Android and iOS devices." I setup a simple script which ultised the easy API to send a message to my phone when my builds are done.

curl -s \
  -F "token=APP_TOKEN" \
  -F "user=USER_KEY" \
  -F "message=The build is done." \
  https://api.pushover.net/1/messages.json

Answer 4

You can also use trap command for sending mail on exit of process ,or at the interupt,hangup or at termination of process .Here is the code that you have to place at the top of your script.

    trap 'mail -s "Process done" [email protected]' exit # It will mail on normal exit
    trap 'mail -s "Process interrupted" [email protected]' INT HUP# it will mail on interrupt or hangup  of the process

Arun

Answer 5

I wrote process_watcher.py

process_watcher --pid 1234 --to [email protected]

Currently, email body looks like:

PID 18851: /usr/lib/libreoffice/program/soffice.bin --writer --splash-pipe=5
Started: Thu, Mar 10 18:33:37 Ended: Thu, Mar 10 18:34:26 (duration 0:00:49)
Memory (current/peak) - Resident: 155,280 / 155,304 kB Virtual: 1,166,968 / 1,188,216 kB

[+] indicates the argument may be specified multiple times, for example:
 process-watcher -p 1234 -p 4258 -c myapp -c "exec\d+" --to [email protected] --to [email protected]

optional arguments:
  -h, --help            show this help message and exit
  -p PID, --pid PID     process ID(s) to watch [+]
  -c COMMAND_PATTERN, --command COMMAND_PATTERN
                        watch all processes matching the command name. (RegEx pattern) [+]
  -w, --watch-new       watch for new processes that match --command. (run forever)
  --to EMAIL_ADDRESS    email address to send to [+]
  -n, --notify          send DBUS Desktop notification
  -i SECONDS, --interval SECONDS
                        how often to check on processes. (default: 15.0 seconds)
  -q, --quiet           don't print anything to stdout

Ceate a GitHub issue if you need any improvements to it.

Answer 6

Assuming your program is already running, use bash+pgrep to watch it.

# Check that "pgrep $PROGRAM" returns the rid PID.
PROGRAM=foo
[email protected]
echo Watching $PROGRAM; while pgrep $PROGRAM &>/dev/null ; do sleep 1; echo -ne "."; done ; echo "$PROGRAM process done" | /bin/mail -s "$PROGRAM process done" $RECIPIENTS;

Answer 7

I have a home-grown script that works for me. Usage:

mail-after <your command>

It will send an e-mail with meaningful information in the subject, and with the process's output in the body.

Installation instructions are on the main page of the GitHub repo -- currently for all scripts a symlink is created in ~/bin.

Answer 8

Ages ago, I created this alias (in csh, I now use it in tcsh):

alias shunt "( \!* | Mail -s 'Output of: \!*' $user) >& /dev/null &"

It requires that Mail be installed and working on your system, but has the advantage that your command runs exactly as it would from the command line (including aliases, tilde expansions, etc)-- when you run a job using at or cron it runs under a default environment that may be sufficiently different to cause your script to fail or provide unusual output (I've been bitten by that from cron many times)