bash: a simple FIFO (first-in first-out) queue

Question

I am looking for a portable bash solution for a simple FIFO queue. My queue input will always be a single line of text.I came up with a simple solution, which works well.

fifo_queue="fifo.txt";
fifo_tmp="fifo.tmp"

# put a new line into FIFO
echo "append to FIFO a line" >> $fifo_stack;

#get line back from FIFO
echo $(head -n1 "$fifo_queue")
tail -n +2 "$fifo_queue" > "$fifo_tmp" && mv "$fifo_tmp" "$fifo_queue"

Now the question: Is there a more elegant (i.e. easy understandable) way to retrieve the FIFO line. For example with a simple one-liner instead of two lines?

Answer 1

If you want to edit a file, use an editor; don't cobble something together with tail.

put () {
    printf '%s\n' "$1" >> "$2"
}

get () {   
    printf '1p\n1d\nwq\n' | ed -s "$1"
}

Then

$ put a queue.txt
$ put b queue.txt
$ get queue.txt
a
$ put c queue.txt
$ get queue.txt
b

Answer 2

One option for retrieving the next FIFO line is to use the POSIX standard ed utility:

printf '%s\n' 1p 1d w | ed -s -- "$fifo_stack"

This invokes ed on the FIFO file and issues 3 commands to it. 1p prints the first line. 1d deletes the first line. w saves the change back to the file.

You may run into problems with this approach if the FIFO file is very large, but you should be OK up to tens of megabytes on a modern machine.

Answer 3

you can use a real fifo. Example

#!/bin/bash

#--- setup
FIFO="fifo.tmp"
rm -f "$FIFO"
mkfifo "$FIFO"

#--- sending data in background
for (( i = 0; i < 10; i++ )); do echo $i; sleep 1; done >"$FIFO" &

#--- receiving data
echo "start"
while read line
do
    echo "> $line"
done < "$FIFO"
echo "stop"

#--- clean
rm -f "$FIFO"

Answer 4

For a fifo stack, the modified version of your logic would be something like the below script. Naturally, you would need to modify that to allow passing the desired line as a parameter on the command line (or as redirected input).

As for removing the first/last line from the stack/buffer file from the file in-place, that would be a task for sed.

I was able to identify the logic for deleting the first/last line depending on fifo/lifo mode using very simple sed directives. That is implemented in the below script.

#!/bin/bash

Push=0
Pop=0
fifo=1 ; mode="fifo"    ### DEFAULT
lifo=0

while [ $# -gt 0 ]
do
    case $1 in
        --push ) Push=1 ; Pop=0 ; shift ;;
        --pop ) Pop=1 ; Push=0 ; shift ;;
        --fifo ) fifo=1 ; lifo=0 ; mode="fifo" ; shift ;;   ### Buffer MODE
        --lifo ) lifo=1 ; fifo=0 ; mode="lifo" ; shift ;;   ### Stack MODE
        * ) echo -e "\n Invalid parameter used on command line.  Only valid options: [ --push | --pop ]\n Bye!\n" ; exit 1 ;;
    esac
done

if [ ${Push} -eq 0 -a ${Pop} -eq 0 ]
then
    echo -e "\n ERROR: must specify one of the two stack operations: --push or --pop \n Bye!\n" ; exit 1
fi

stack="${mode}.txt";
tmp="${mode}.tmp"
if [ ! -f "${stack}" ] ; then touch "${stack}" ; fi

i=$(( $(wc -l "${stack}" | awk '{ print $1 }' ) + 1 ))

if [ ${Push} -eq 1 ]
then
    # put a new line into FIFO/LIFO
    echo "append to FIFO - line ${i}" >> "${stack}"
    wc -l "${stack}"
fi

if [ ${Pop} -eq 1 ]
then
    if [ -s "${stack}" ]
    then
        #get line back from FIFO
        if [ ${fifo} -eq 1 ]
        then
            head -n1 "${stack}"
            sed --in-place '1d' "${stack}"
            wc -l "${stack}"
        else
            tail -n1 "${stack}"
            sed --in-place '$d' "${stack}"
            wc -l "${stack}"
        fi
    else
        echo -e "\t ERROR: attempt to pop line from EMPTY stack.\n" >&2
        exit 1
    fi
fi

Session log is as follows:

me@OasisMega1:/0__WORK$ ./test_129.sh --push -fifo
1 fifo.txt
me@OasisMega1:/0__WORK$ ./test_129.sh --push -fifo
2 fifo.txt
me@OasisMega1:/0__WORK$ ./test_129.sh --push -fifo
3 fifo.txt
me@OasisMega1:/0__WORK$ ./test_129.sh --push -fifo
4 fifo.txt
me@OasisMega1:/0__WORK$ ./test_129.sh --pop -fifo
append to FIFO - line 1
3 fifo.txt
me@OasisMega1:/0__WORK$ ./test_129.sh --pop -fifo
append to FIFO - line 2
2 fifo.txt
me@OasisMega1:/0__WORK$ ./test_129.sh --pop -fifo
append to FIFO - line 3
1 fifo.txt
me@OasisMega1:/0__WORK$ ./test_129.sh --pop -fifo
append to FIFO - line 4
0 fifo.txt
me@OasisMega1:/0__WORK$ ./test_129.sh --pop -fifo
     ERROR: attempt to pop line from EMPTY stack.
me@OasisMega1:/0__WORK$

Session log for LIFO mode is as follows:

me@OasisMega1:0__WORK$ ./test_129.sh --push --lifo
1 lifo.txt
me@OasisMega1:0__WORK$ ./test_129.sh --push --lifo
2 lifo.txt
me@OasisMega1:0__WORK$ ./test_129.sh --push --lifo
3 lifo.txt
me@OasisMega1:0__WORK$ ./test_129.sh --push --lifo
4 lifo.txt
me@OasisMega1:0__WORK$ ./test_129.sh --pop --lifo
append to FIFO - line 4
3 lifo.txt
me@OasisMega1:0__WORK$ ./test_129.sh --pop --lifo
append to FIFO - line 3
2 lifo.txt
me@OasisMega1:0__WORK$ ./test_129.sh --pop --lifo
append to FIFO - line 2
1 lifo.txt
me@OasisMega1:0__WORK$ ./test_129.sh --pop --lifo
append to FIFO - line 1
0 lifo.txt
me@OasisMega1:0__WORK$ ./test_129.sh --pop --lifo
     ERROR: attempt to pop line from EMPTY stack.

me@OasisMega1:0__WORK$ 

This version of the script incorporates providing input for buffer/stack on command line, and option for display of current buffer/stack contents. Also provides return code which might be used for decision on whether contents requiring "processing" are still in buffer or not.

#!/bin/bash
    
Push=0
Pop=0
Display=0
fifo=1 ; mode="FIFO"    ### DEFAULT
lifo=0
dataline=""

while [ $# -gt 0 ]
do
    case $1 in
        --push ) Push=1 ; Pop=0 ; shift ;;
        --pop ) Pop=1 ; Push=0 ; shift ;;
        --show ) Push=0; Pop=0 ; Display=1 ; shift ;;
        --fifo ) fifo=1 ; lifo=0 ; mode="FIFO" ; shift ;;   ### Buffer MODE
        --lifo ) lifo=1 ; fifo=0 ; mode="LIFO" ; shift ;;   ### Stack MODE
        -- ) shift ; dataline=${@} ; break ;;
        * ) echo -e "\n Invalid parameter used on command line.  Only valid options: [ --push | --pop ]\n Bye!\n" ; exit 1 ;;
    esac
done

if [ ${Display} -eq 1 ]
then
    if [ ${fifo} -eq 1 ]
    then
        if [ -f FIFO.txt ]
        then
            if [ -s FIFO.txt ]
            then
                wc -l FIFO.txt >&2
                echo -e "Hit return to view contents ...\c" >&2 ; read k <&2
                more FIFO.txt
                RC=98
            else
                echo -e "\n FIFO buffer is empty at this time.\n" >&2
                RC=0
            fi
        else
            echo -e "\n FIFO buffer does not exist.\n" >&2
            RC=99
        fi
    fi
    if [ ${lifo} -eq 1 ]
    then
        if [ -f LIFO.txt ]
        then
            if [ -s LIFO.txt ]
            then
                wc -l LIFO.txt >&2
                echo -e "Hit return to view contents ...\c" >&2 ; read k <&2
                more LIFO.txt
                RC=98
            else
                echo -e "\n LIFO buffer is empty at this time.\n" >&2
                RC=0
            fi
        else
            echo -e "\n LIFO buffer does not exist.\n" >&2
            RC=99
        fi
    fi
    exit ${RC}
fi

if [ ${Push} -eq 1 -a -z "${dataline}" ]
then
    echo -e "\n ERROR: did not provide data for capture in BUFFER/STACK\n Bye!\n" ; exit 1
fi

if [ ${Push} -eq 0 -a ${Pop} -eq 0 ]
then
    echo -e "\n ERROR: must specify one of the two stack operations: --push or --pop \n Bye!\n" >&2 ; exit 1
fi

stack="${mode}.txt";
tmp="${mode}.tmp"
if [ ! -f "${stack}" ] ; then touch "${stack}" ; fi

if [ ${Push} -eq 1 ]
then
    # put a new line into FIFO/LIFO
    echo "${dataline}" >> "${stack}"
    wc -l "${stack}" >&2
fi

if [ ${Pop} -eq 1 ]
then
    if [ -s "${stack}" ]
    then
        #get line back from FIFO
        if [ ${fifo} -eq 1 ]
        then
            head -n1 "${stack}"
            sed --in-place '1d' "${stack}"
            wc -l "${stack}" >&2
        else
            tail -n1 "${stack}"
            sed --in-place '$d' "${stack}"
            wc -l "${stack}" >&2
        fi
    else
        echo -e "\t ERROR: attempt to pop line from EMPTY stack.\n" >&2
        exit 1
    fi
fi

Answer 5

A directory-based version with simple usage, error handling for common problems, and O(1) performance characteristics (meaning it doesn't get slower as the queue gets longer):

$ source fifo-utils.bash # assuming this file has the below code
$ get fifo.d
ERROR: fifo directory 'fifo.d' not initialized
$ put fifo.d 13
$ get fifo.d
13
$ get fifo.d
ERROR: fifo directory 'fifo.d' is empty
$ put fifo.d fourteen
$ put fifo.d fifteen
$ get fifo.d
fourteen
$ get fifo.d
fifteen
$ get fifo.d
ERROR: fifo directory 'fifo.d' is empty

Implementation (be sure to note the "future enhancements" list):

# Format:
# "head" contains the name of the next item ready for read
# "tail" contains the next place where newly-added data would be written
#
# If "head" matches "tail", the FIFO is empty.
#
# Possible future enhancements:
# - Use create-and-rename pattern to update counters atomically
# - If head or tail is unexpectedly empty, but numbered files exist, recover from
#   highest/lowest numbered file
# - Lock head and tail files with flock to make concurrent use safe
# - Add "get" support for blocking if queue is empty (can do this by using inotifywait
#   to block until tail is updated before proceeding should head == tail)

get() {
  local fifo_dir=$1 head tail
  [[ -s $fifo_dir/head && -e $fifo_dir/tail ]] || {
    echo "ERROR: fifo directory ${fifo_dir@Q} not initialized" >&2
    return 1
  }
  head=$(<"$fifo_dir/head") || return
  tail=$(<"$fifo_dir/tail") || return
  (( head >= tail )) && {
    echo "ERROR: fifo directory ${fifo_dir@Q} is empty" >&2
    return 2
  }
  [[ -e "$fifo_dir/$head" ]] || {
    echo "ERROR: fifo directory ${fifo_dir@Q} has head at $head, but data does not exist" >&2
  }
  cat -- "$fifo_dir/$head" || return
  printf '%s\n' "$(( head + 1 ))" >"$fifo_dir/head" || return
}

put() {
  local fifo_dir=$1 data=$2 head tail
  [[ -d $fifo_dir ]] || { mkdir -p -- "$fifo_dir" || exit; }
  [[ -s "$fifo_dir/head" ]] || { echo 0 >"$fifo_dir/head" || return; }
  [[ -s "$fifo_dir/tail" ]] || { echo 0 >"$fifo_dir/tail" || return; }
  head=$(<"$fifo_dir/head") || return
  tail=$(<"$fifo_dir/tail") || return
  printf '%s\n' "$data" >"$fifo_dir/$tail" || return
  echo "$(( tail + 1 ))" >"$fifo_dir/tail" || return
}