Piping tail -f into awk

Question

I am trying to pipe tail -f into awk to monitor a logfile in realtime, but according to examples, there should be not problem but I can't get it to work.

here is the command I'm running

tail -f logfile.log | awk -F" " '{print $1, $2, $7, $8}'

But when I edit my file using nano add a line, it is not printed in real time, If I run the awk command directly, my new line appear in the result.

Answer 1

You don't see it in real time because, for purposes of efficiency, pipes are buffered. tail -f has to fill up the buffer, typically 4 kB, before the output is passed to awk.

A fix is to use the unbuffer command which is part of the expect package:

unbuffer tail -f logfile.log | awk -F" " '{print $1, $2, $7, $8}'

This tricks tail into thinking it is writing to an interactive terminal. As a result, it doesn't buffer.

For more, see https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe

Alternatively, if you have GNU coreutils 7.5 or better, you can disable output buffering with the stdbuf command:

stdbuf -o0 tail -f logfile.log | awk -F" " '{print $1, $2, $7, $8}'

Answer 2

@John1024's alternative answer is not quite correct, as the stdbuf -o0 prefix is misplaced. It belongs as the prefix to the awk command, not the tail -f command, so the correct command should be:

tail -f logfile.log | stdbuf -o0 awk -F" " '{print $1, $2, $7, $8}'

This is what worked for me, BUT only after I located an awk version that would cooperate. So note that not all versions of awk allow this! I had to try nawk, gawk, mawk, etc until I found one that worked. If you need that particular command string to function as written, keep trying various awk/gawk/mawk versions by being explicit with the path of the awk binary and even downloading others until you hit one that works.

Answer 3

On Ubuntu 20.10 where awk links to mawk version 1.3.4,

Add -W interactive. E.g. tail -f $file | awk -W interactive '{print}'

(Thanks to the other folks whose answers here caused me to read the man page looking for buffer.)

Answer 4

You could un-buffer it by adding a while loop as given in the example below. I have tested this on a running log file and it worked for me.

tail -f input.log | while read a; do echo "$a" | awk -F" " '{print $1, $2, $7, $8}' >> output.log; done

Answer 5

Neither stdbuf nor unbuffer seem to work for me (Ubuntu 18.04), but using a named pipe does. I created a shell script containing:

mkfifo /tmp/pipe
awk -F" " '{print $1, $2, $7, $8}' < /tmp/pipe &
tail -f input.log > /tmp/pipe

Answer 6

In my experience with this problem, it appears that buffering issues can affect both the read and write sides of each command. In my case, I had a command like the following:

$ parallel ... | ggrep ...

This worked fine, printing output as it was produced by parallel. When I tried to process further with awk, though:

$ parallel ... | ggrep ... | awk ...

I didn't see output until parallel was done. I naturally thought the issue was with awk, hence finding this post, so I tried this:

$ parallel ... | ggrep ... | stdbuf -o0 awk ...

But what actually fixed it was this:

$ parallel ... | stdbuf -o0 ggrep ... | awk ...

So it appears that ggrep can tell when it's printing vs. hooked up to another command, and buffers its output in the latter case. (Incidentally, ggrep actually has a --line-buffered flag which fixes my problem as well.)

Answer 7

This works for me. Not sure what all the fuss is about:

tail -f mylog.txt | awk -f ~/mylog.awk

Your thinking too hard. Check your fire.