Return to Answer

This is because of buffering in the pipe, which in general doesn't care about lines and can accumulate data.

I think tail -f uses line buffering by itself; and the last grep writes to the tty, so it also uses line buffering. Therefore your first example works.

But grep in the middle is different and you need to adjust its behavior by forcing line buffering or by disabling buffering. The bellow commands will work as you expected.

• If your grep supports --line-buffered (it does in Ubuntu):

    tail -f foo.log | grep --line-buffered aaa | grep bbb
  

• More generic solutions (they will work with many filters other than grep):

    tail -f foo.log | unbuffer -p grep aaa | grep bbb
    tail -f foo.log | stdbuf -oL grep aaa | grep bbb
    tail -f foo.log | stdbuf -o0 grep aaa | grep bbb
  

See man 1 grep, man 1 unbuffer and man 1 stdbuf for details and quirks.

Notes:

• Neither solution is portable (grep --line-buffered, unbuffer and stdbuf are not specified by POSIX).
• If you can do this with grep --line-buffered then it should be your choice. There's no point in using extra tools.
• Related question on Unix & Linux SE: Turn off buffering in pipe.
• unbuffer and stdbuf work in completely different ways.
• With stdbuf, line buffering (-oL) should be preferred over no buffering (-o0) here, because
• it most likely performs better,
• and other parts of your pipe use line buffering anyway.
• If your last grep wrote to yet another file, it would behave like the other grep. In such case if you want lines to appear in the final file immediately, then you should also modify the behavior of the last grep.
• In fish, if grep is a wrapper function, you may not get the desired behavior with --line-buffered. Use command grep --line-buffered. See this question: Output pipe waits for EOF in fish.

Side note: tail foo.log | grep aaa | grep bbb (i.e. tail without -f) does not cause the issue because tail exits. When tail exits, the first grep detects EOF, flushes its buffer and exits, then the second grep does the same.

This is because of buffering in the pipe, which in general doesn't care about lines and can accumulate data.

I think tail -f uses line buffering by itself; and the last grep writes to the tty, so it also uses line buffering. Therefore your first example works.

But grep in the middle is different and you need to adjust its behavior by forcing line buffering or by disabling buffering. The bellow commands will work as you expected.

• If your grep supports --line-buffered (it does in Ubuntu):

    tail -f foo.log | grep --line-buffered aaa | grep bbb
  

• More generic solutions (they will work with many filters other than grep):

    tail -f foo.log | unbuffer -p grep aaa | grep bbb
    tail -f foo.log | stdbuf -oL grep aaa | grep bbb
    tail -f foo.log | stdbuf -o0 grep aaa | grep bbb
  

See man 1 grep, man 1 unbuffer and man 1 stdbuf for details and quirks.

Notes:

• Neither solution is portable (grep --line-buffered, unbuffer and stdbuf are not specified by POSIX).
• If you can do this with grep --line-buffered then it should be your choice. There's no point in using extra tools.
• Related question on Unix & Linux SE: Turn off buffering in pipe.
• unbuffer and stdbuf work in completely different ways.
• With stdbuf, line buffering (-oL) should be preferred over no buffering (-o0) here, because
  • it most likely performs better,
  • and other parts of your pipe use line buffering anyway.
• If your last grep wrote to yet another file, it would behave like the other grep. In such case if you want lines to appear in the final file immediately, then you should also modify the behavior of the last grep.
• In fish, if grep is a wrapper function, you may not get the desired behavior with --line-buffered. Use command grep --line-buffered. See this question: Output pipe waits for EOF in fish.

Side note: tail foo.log | grep aaa | grep bbb (i.e. tail without -f) does not cause the issue because tail exits. When tail exits, the first grep detects EOF, flushes its buffer and exits, then the second grep does the same.

This is because of buffering in the pipe, which in general doesn't care about lines and can accumulate data.

I think tail -f uses line buffering by itself; and the last grep writes to the tty, so it also uses line buffering. Therefore your first example works.

But grep in the middle is different and you need to adjust its behavior by forcing line buffering or by disabling buffering. The bellow commands will work as you expected.

• If your grep supports --line-buffered (it does in Ubuntu):

    tail -f foo.log | grep --line-buffered aaa | grep bbb
  

• More generic solutions (they will work with many filters other than grep):

    tail -f foo.log | unbuffer -p grep aaa | grep bbb
    tail -f foo.log | stdbuf -oL grep aaa | grep bbb
    tail -f foo.log | stdbuf -o0 grep aaa | grep bbb
  

See man 1 grep, man 1 unbuffer and man 1 stdbuf for details and quirks.

Notes:

• Neither solution is portable (grep --line-buffered, unbuffer and stdbuf are not specified by POSIX).
• If you can do this with grep --line-buffered then it should be your choice. There's no point in using extra tools.
• Related question on Unix & Linux SE: Turn off buffering in pipe.
• unbuffer and stdbuf work in completely different ways.
• With stdbuf, line buffering (-oL) should be preferred over no buffering (-o0) here, because
• it most likely performs better,
• and other parts of your pipe use line buffering anyway.
• If your last grep wrote to yet another file, it would behave like the other grep. In such case if you want lines to appear in the final file immediately, then you should also modify the behavior of the last grep.
• In fish, if grep is a wrapper function, you may not get the desired behavior with --line-buffered. Use command grep --line-buffered. See this question: Output pipe waits for EOF in fish.

This is because of buffering in the pipe, which in general doesn't care about lines and can accumulate data.

I think tail -f uses line buffering by itself; and the last grep writes to the tty, so it also uses line buffering. Therefore your first example works.

But grep in the middle is different and you need to adjust its behavior by forcing line buffering or by disabling buffering. The bellow commands will work as you expected.

• If your grep supports --line-buffered (it does in Ubuntu):

    tail -f foo.log | grep --line-buffered aaa | grep bbb
  

• More generic solutions (they will work with many filters other than grep):

    tail -f foo.log | unbuffer -p grep aaa | grep bbb
    tail -f foo.log | stdbuf -oL grep aaa | grep bbb
    tail -f foo.log | stdbuf -o0 grep aaa | grep bbb
  

See man 1 grep, man 1 unbuffer and man 1 stdbuf for details and quirks.

Notes:

• Neither solution is portable (grep --line-buffered, unbuffer and stdbuf are not specified by POSIX).
• If you can do this with grep --line-buffered then it should be your choice. There's no point in using extra tools.
• Related question on Unix & Linux SE: Turn off buffering in pipe.
• unbuffer and stdbuf work in completely different ways.
• With stdbuf, line buffering (-oL) should be preferred over no buffering (-o0) here, because
• it most likely performs better,
• and other parts of your pipe use line buffering anyway.
• If your last grep wrote to yet another file, it would behave like the other grep. In such case if you want lines to appear in the final file immediately, then you should also modify the behavior of the last grep.
• In fish, if grep is a wrapper function, you may not get the desired behavior with --line-buffered. Use command grep --line-buffered. See this question: Output pipe waits for EOF in fish.

Side note: tail foo.log | grep aaa | grep bbb (i.e. tail without -f) does not cause the issue because tail exits. When tail exits, the first grep detects EOF, flushes its buffer and exits, then the second grep does the same.

This is because of buffering in the pipe, which in general doesn't care about lines and can accumulate data.

I think tail -f uses line buffering by itself; and the last grep writes to the tty, so it also uses line buffering. Therefore your first example works.

But grep in the middle is different and you need to adjust its behavior by forcing line buffering or by disabling buffering. The bellow commands will work as you expected.

• If your grep supports --line-buffered (it does in Ubuntu):

    tail -f foo.log | grep --line-buffered aaa | grep bbb
  

• More generic solutions (they will work with many filters other than grep):

    tail -f foo.log | unbuffer -p grep aaa | grep bbb
    tail -f foo.log | stdbuf -oL grep aaa | grep bbb
    tail -f foo.log | stdbuf -o0 grep aaa | grep bbb
  

See man 1 grep, man 1 unbuffer and man 1 stdbuf for details and quirks.

Notes:

• Neither solution is portable (grep --line-buffered, unbuffer and stdbuf are not specified by POSIX).
• If you can do this with grep --line-buffered then it should be your choice. There's no point in using extra tools.
• Related question on Unix & Linux SE: Turn off buffering in pipe.
• unbuffer and stdbuf work in completely different ways.
• With stdbuf, line buffering (-oL) should be preferred over no buffering (-o0) here, because
• it most likely performs better,
• and other parts of your pipe use line buffering anyway.
• If your last grep wrote to yet another file, it would behave like the other grep. In such case if you want lines to appear in the final file immediately, then you should also modify the behavior of the last grep.

This is because of buffering in the pipe, which in general doesn't care about lines and can accumulate data.

I think tail -f uses line buffering by itself; and the last grep writes to the tty, so it also uses line buffering. Therefore your first example works.

But grep in the middle is different and you need to adjust its behavior by forcing line buffering or by disabling buffering. The bellow commands will work as you expected.

• If your grep supports --line-buffered (it does in Ubuntu):

    tail -f foo.log | grep --line-buffered aaa | grep bbb
  

• More generic solutions (they will work with many filters other than grep):

    tail -f foo.log | unbuffer -p grep aaa | grep bbb
    tail -f foo.log | stdbuf -oL grep aaa | grep bbb
    tail -f foo.log | stdbuf -o0 grep aaa | grep bbb
  

See man 1 grep, man 1 unbuffer and man 1 stdbuf for details and quirks.

Notes:

• Neither solution is portable (grep --line-buffered, unbuffer and stdbuf are not specified by POSIX).
• If you can do this with grep --line-buffered then it should be your choice. There's no point in using extra tools.
• Related question on Unix & Linux SE: Turn off buffering in pipe.
• unbuffer and stdbuf work in completely different ways.
• With stdbuf, line buffering (-oL) should be preferred over no buffering (-o0) here, because
• it most likely performs better,
• and other parts of your pipe use line buffering anyway.
• If your last grep wrote to yet another file, it would behave like the other grep. In such case if you want lines to appear in the final file immediately, then you should also modify the behavior of the last grep.
• In fish, if grep is a wrapper function, you may not get the desired behavior with --line-buffered. Use command grep --line-buffered. See this question: Output pipe waits for EOF in fish.