Getting text from last marker to EOF in POSIX.2

Question

I have a text with marker lines like:

aaa
---
bbb
---
ccc

I need to get a text from the last marker (not inclusive) to EOF. In this case it will be

ccc

Is there an elegant way within POSIX.2? Right now I use two runs: first with nl and grep for the last occurrence with respective line number. Then I extract the line number and use sed to extract the chunk in question.

The text segments may be quite large, so I'm afraid to use some text-adding method like we add the text to a buffer, if we encounter the marker we empty the buffer, so that at EOF we have our last chunk in the buffer.

Answer 1

Unless your segments are really huge (as in: you really can't spare that much RAM, presumably because this is a tiny embedded system controlling a large filesystem), a single pass is really the better approach. Not just because it'll be faster, but most importantly because it allows the source to be a stream, from which any data read and not saved is lost. This is really a job for awk, though sed can do it too.

sed -n -e 's/^---$//' -e 't a' \
       -e 'H' -e '$g' -e '$s/^\n//' -e '$p' -e 'b' \
       -e ':a' -e 'h'              # you are not expected to understand this
awk '{if (/^---$/) {chunk=""}      # separator ==> start new chunk
      else {chunk=chunk $0 RS}}    # append line to chunk
     END {printf "%s", chunk}'     # print last chunk (without adding a newline)

If you must use a two-pass approach, determine the line offset of the last separator and print from that. Or determine the byte offset and print from that.

</input/file tail -n +$((1 + $(</input/file         # print the last N lines, where N=…
                               grep -n -e '---' |   # list separator line numbers
                               tail -n 1 |          # take the last one
                               cut -d ':' -f 1) ))  # retain only line number
</input/file tail -n +$(</input/file awk '/^---$/ {n=NR+1} END {print n}')
</input/file tail -c +$(</input/file LC_CTYPE=C awk '
    {pos+=length($0 RS)}        # pos contains the current byte offset in the file
    /^---$/ {last=pos}          # last contains the byte offset after the last separator
    END {print last+1}          # print characters from last (+1 because tail counts from 1)
')

Addendum: If you have more than POSIX, here's a simple one-pass version that relies on a common extension to awk that allows the record separator RS to be a regular expression (POSIX only allows a single character). It's not completely correct: if the file ends with a record separator, it prints the chunk before the last record separator instead of an empty record. The second version using RT avoids that defect, but RT is specific to GNU awk.

awk -vRS='(^|\n)---+($|\n)' 'END{printf $0}'
gawk -vRS='(^|\n)---+($|\n)' 'END{if (RT == "") printf $0}'

Answer 2

lnum=$(($(sed -n '/^---$/=' file | sed '$!d') +1)); sed -n "${lnum},$ p" file 

The first sed ouputs line numbers of the "---" lines...
The second sed extracts the last number from the first sed's output...
Add 1 to that number to get the start of your "ccc" block...
The third 'sed' outputs from the start of the "ccc" block to EOF

Update (with ammended info re Gilles methods)

Well I was wondereing about how glenn jackman's tac would perform, so I time-tested the three answers (at the time of writing)... The test file(s) each contained 1 million lines (of their own line numbers).
All answers did what was expected...

Here are the times..

---

Gilles sed (single pass)

# real    0m0.470s
# user    0m0.448s
# sys     0m0.020s

---

Gilles awk (single pass)

# very slow, but my data had a very large data block which awk needed to cache.

---

Gilles 'two-pass' (first method)

# real    0m0.048s
# user    0m0.052s
# sys     0m0.008s

---

Gilles 'two-pass' (second method) ... very fast

# real    0m0.204s
# user    0m0.196s
# sys     0m0.008s

---

Gilles 'two-pass' (third method)

# real    0m0.774s
# user    0m0.688s
# sys     0m0.012s

---

Gilles 'gawk' (RT method) ... very fast, but is not POSIX.

# real    0m0.221s
# user    0m0.200s
# sys     0m0.020s

---

glenn jackman ... very fast, but is not POSIX.

# real    0m0.022s
# user    0m0.000s
# sys     0m0.036s

---

fred.bear

# real    0m0.464s
# user    0m0.432s
# sys     0m0.052s

---

Mackie Messer

# real    0m0.856s
# user    0m0.832s
# sys     0m0.028s

Answer 3

A two pass strategy seems to be the right thing. Instead of sed I would use awk(1). The two passes could look like this:

$ LINE=`awk '/^---$/{n=NR}END{print n}' file`

to get the line number. And then echo all text starting from that line number with:

$ awk "NR>$LINE" file

This should not require excessive buffering.

Answer 4

Use "tac" which outputs a file's lines from end to beginning:

tac afile | awk '/---/ {exit} {print}' | tac

Answer 5

You could just use ed

ed -s infile <<\IN
.t.
1,?===?d
$d
,p
q
IN

How it works: t duplicates the current (.) line - which is always the last line when ed starts (just in case the delimiter is present on the last line), 1,?===?d deletes all lines up to and including the previous match (ed is still on the last line) then $d deletes the (duplicate) last line, ,p prints the text buffer (replace with w to edit the file in place) and finally q quits ed.

---

If you know there's at least one delimiter in the input (and don't care if it's also printed) then

sed 'H;/===/h;$!d;x' infile

would be the shorter.
How it works: it appends all lines to the Hold buffer, it overwrites the hold buffer when encountering a match, it deletes all lines except the la$t one when it exchanges buffers (and autoprints).