How to "tail" multiple pipe files or streams?

Question

I need to monitor a lot of web servers for runtime errors and I have an idea of running a script like

#!/bin/bash -e

# Example set of hosts
HOSTS=(
host1
host2
)

for host in ${HOSTS[@]}
do
        [[ ! -e ${host}.pipe ]] && mkfifo ${host}.pipe
        (ssh -n $host "tail -n0 -F /tmp/test" >> ${host}.pipe) &
done

tail -F -c +0 *.pipe

and it would supposedly give me such output:

==> host1.pipe <==
event 1
event 2

==> host2.pipe <==
event 3
event 4

==> host1.pipe <==
event 5

You get the point. I see log lines as they appear, with their origin being obvious from how tail -F shows it usually when you run on more than one file.

Tail options -c +0 supposedly makes it print stream contents starting from 0th byte.

But this doesn't work! What I see is like

==> host1.pipe <==
event 1
event 2
event 5

That is, no stream except first is shown.

Here's simpler variation of above script, still reproducing the issue:

tail -F -c +0 \
        <(ssh host1 "tail -n1 -F /tmp/test") \
        <(ssh host2 "tail -n1 -F /tmp/test") \
        ;

Even simpler, still reproducing the issue:

tail -F -c +0 \
    <(while true; do date;     sleep 1; done) \
    <(while true; do date +%s; sleep 1; done)

I'm using tail from coreutils 8.27 and Linux kernel 4.9.14.

Thanks in advance for any hint!

Note that I won't like to be suggested to use "multitail". I've tried it, splitting screen in panes won't scale to dozens of servers which we have now.

Answer 1

Note that I won't like to be suggested to use multitail. I've tried it, splitting screen in panes won't scale to dozens of servers which we have now.

How about -L option? It doesn't split screen.

multitail \
  --mark-change \
  -L "while sleep 1; do date;     done" \
  -L "while sleep 5; do date +%s; done" \
  -L "while sleep 7; do date +%Y; done"

If you want to use named pipes you should use -I instead of -L. Unfortunatelly multitail requires separate -I for every file, this makes the wildcard (*) approach more difficult. Bad luck, its -Iw option doesn't seem to work with named pipes; and I couldn't make the tool cooperate with xargs (at least on my Debian).

Still there is a way:

multitail --mark-change --mergeall *.pipe

The output will be very similar to what you expected from tail.