Pipe partial output of background process and store it in variable

Question

Background:

I'm working on a project in which a server is started at a random available port. The server is started by a command:

node server.js

Once the server is ready and starts serving on an available port, it outputs (along with a bunch of other logs) the complete server URL (including port number) on stdout in the following form:

Server started! Listening on URL: http://localhost:12927

This server process keeps running until you manually kill it (and keeps printing some other logs).

Actual problem:

I'm writing a script to start the server and then listen to its output. As soon as I get the message something like "Server started! Listening on URL: [SERVER_URL]", I want to store this SERVER_URL in a variable and process later in the script (e.g. open a browser for automation testing).

What I've tried:

Having only a very basic understanding of Bash, I've tried the following options (of course, after researching on SE):

1. I researched a bit about using grep with regex. This works pretty well if I do it manually (i.e. not storing it to variable and just piping from an echo command.

   SERVER_URL=$(node server.js | grep -m 1 -ohP 'Server started! Listening on URL: \K([^\s$]*)')
   

However, it kinda blocks on this line, probably it's waiting for the node server to get killed. My requirement is clear: I want to populate SERVER_URL from the server's output without waiting for it beyond the point it spits out that URL on stdout.

2. I also fiddled around a bit with redirection and process substitution. I learnt that the pipe actually waits for the source process to get completed. I've tried to do the following with redirection, but it's not working:

   SERVER_URL=$(node server.js >&3 | grep -m 1 -ohP 'Server started! Listening on URL: \K([^\s$]*)')
   

It is throwing some weird "Unhandled 'error' event" exception (Error: write EPIPE).

Any help is appreciated. Thanks!

Answer 1

Yes, in:

var=$(cmd | grep -m1 pattern)

cmd's stdout is a pipe to grep, and grep's stdout is a pipe to the shell.

After grep has found the first match, it outputs it on its stdout and exits.

The shell will read grep's output, see end-of-file when grep exits, but in the case of bash, the shell will also wait for cmd to exit.

Now, because grep has exited, cmd's stdout is now a broken pipe. So the next time cmd writes to stdout, it will receive a SIGPIPE signal and die.

Here, the question is, what do you want to do with the rest of cmd's output once you've read up to the first line that matches the pattern?

If you're not interested in it you could run a background cat command that reads it and discards it:

{
   url=$(grep -m1 -oP 'Server started! Listening on URL: \K\S+')
   cat <&0 > /dev/null & # <&0 to work around the fact that bash redirects
                         # background commands stdin from /dev/null
} < <(node server.js)

If you want to keep it, you could redirect it to some file, and use tail -f for instance in the shell to wait for the URL message:

node server.js > server.log &
{
  IFS= read -r pid
  url=$(grep -m1 -oP 'Server started! Listening on URL: \K\S+')
  kill -- "$pid"
} < <(echo "$BASHPID"; exec tail -n +1 -f server.log)

Here passing the pid of tail so as to kill it after the message is found rather than waiting for tail to be killed by a SIGPIPE which may never happen if tail never outputs anything else.