Check what process is spiking the average load with atop

Question

In trying to find the culprit of a high load average on a system during night (which does not seem to be related to logrotate) I installed atop to write a raw file with a specific interval. While reading the file, it seems the processlist stands still, can I somehow go back and forth between the samples to see what sticks out, and further sort by any column (like cpu usage)?

Answer 1

Regarding your second question, the list printed by atop is sorted by cpu usage by default. If you want to sort it by other parameter (like resident memory consumption) you can do that (man atop tells you how for both the interactive and the raw file modes).

Regarding your first question, this small AWK script may help:

BEGIN {
  printline = "false"
}
{
   if (printline == "true") { print($0); printline = "false" }
   if ($1 == "PID") { printline = "true" }
}

Run it as awk -f myScript.awk logFromAtop.log and it will give you the top line of atop for every interval; it will probably be easy to see the few lines that stand out. (The script just looks for the lines that start with PID and prints the next ones.)

Depending on your version of atop, it may give you an ASCII or binary raw file depending on the flags when you run it. In the latter case, you can get an ASCII version with atop itself, e.g. atop -r binary.raw > logFromAtop.log.

Answer 2

You will never discover what process is causing high average loads looking at the CPU % usage. Load average depends on whenever a process is running (R) or waiting for I/O (D). So the actual option you should use is s:

s Show scheduling characteristics.

Per process the following fields are shown in case of a window-width of 80 positions: process-id, number of threads in state 'running' (R), number of threads in state 'interruptible sleeping' (S), number of threads in state 'uninterruptible sleeping' (D), scheduling policy (normal timesharing, realtime round-robin, realtime fifo), nice value, priority, realtime priority, current processor, status, exit code, state, the occupation percentage for the choosen resource and the process name.

When more than 80 positions are available, other information is added.

Just change your configuration to show scheduling characteristics and you will find the culprit.

Answer 3

Run atop with the -r argument followed by your log file :

Then, while atop is running you can use ctrl+F to see the next page, or ctrl+B to see the previous one.