Cron job not working consistently

Question

I have a cron job:

1 0 * * * /usr/bin/wget -q -O /dev/null 'http://123.456.78.90/index/parsedata?&today=1'

It should gather a bunch of data from the DB at 12:01am each day, organize it for use by our app in graphs and displays and then be done.

Sometimes, certain graphs seem to have not been processed as they show no results for some period of time. Manually running the command will cause them to have the correct data.

So it seems cron is part of the issue. Either cron is not firing this, not true as I see most entries display expected graph data, or perhaps cron is not finishing? I do not know where to look as I felt that since manually triggering the above line of script works, why wouldn't it work in the same manner when I set cron to do it? Any ideas?

There is another cron job that runs every 5 minutes on the same server, using the same code:

*/5 * * * * /usr/bin/wget -q -O /dev/null http://123.456.78.90/index/parsedata

Is it possible that the first run takes too long and is still being processed when this script, which runs every 5 minutes, kicks in and this is messing things up? The first script can take longer than 5 minutes to run so this second instance can get run while the first is still grinding way. They both use the same action, parsedata(), in our framework and write to the same database. Maybe? Any ideas appreciated.

Answer 1

Jobs failing in cron but working from a terminal are usually a sign that the job is trying to produce output to stdout or stderr and falling over because it can't.

It sounds like you need some logging. Try redirecting stdout and stderr to a file to get a better picture of what's going on:

/usr/bin/wget -q -O /dev/null http://123.456.78.90/index/parsedata >> /tmp/somelogfile.txt 2>&1

If some output is generated then perhaps it will give you an idea of what the root cause of the problem might be. If the logfile is still empty after a job has failed then at least you know to look somewhere else.

Answer 2

If your two processes are getting into a deadlock situation at the database then that could be the cause of the longer job failing (it being chosen as the task that gets killed to resolve the deadlock). If that is the case, or if some other error is causing the process to fail, you might find your parsedata script is returning a useful error message (or other clue) but you are never seeing it because you are discarding its output with -O /dev/null. I suggest you log the output instead of sending it to /dev/null, for instance:

-O /var/log/dailyparsedata/`date +%Y%m%d_%H%M`

(note: those are back-ticks, not single quotes) will create a new file each day in /var/log/dailyparsedata/. You can set another cron job (or configure logwatch) to remove files past a given age so you don't fill up the partition with these files over time.

Also, there may be error conditions that stop wget even seeing the output of the script (perhaps for some reason it can't even see the web server for some reason occasionally: quite possible if the server it is calling is remote) in which case there will be nothing for it to output to the file specified by -O, so as well as logging the script outout log wget's stdout and stderr as suggested by Stephen's answer. As your cron lines currently stand most cron implementations will be sending that some of that output to someone as an email, adding >> /var/log/dailyparsedata/wget.output 2>&1 will keep that info in a file instead. I woudl avoid sending this sort of output to /tmp as there are two risks there: filling /tmp with logs so its main use breaks, and it being cleared down by default when the server reboots meaning you lose logs you might later want to check.