Move first N lines of output to end without using temporary file

Question

Imagine an output of a command like

44444
55555
11111
22222
33333

how can I yank the first N lines (first two in the example above) and append them at the end, but without using temp file (so only using pipes)?

11111
22222
33333
44444
55555

Something along the lines of | sed -n '3,5p;1,2p' (which obviously doesn't work as sed doesn't care about the order of the commands).

Answer 1

Just copy those lines to the hold buffer (then delete them) and when on last line append the content of hold buffer to pattern space:

some command | sed '1,NUMBER{           # in this range
H                                       # append line to hold space and
1h                                      # overwrite if it's the 1st line
d                                       # then delete the line
}
$G'                                     # on last line append hold buffer content

With gnu sed you could write it as

some command | sed '1,NUMBER{H;1h;d;};$G'

---

Here's another way with ol' ed (it reads the output of some command into the text buffer and then moves lines 1,NUMBER after the la$t one):

ed -s <<IN
r ! some command
1,NUMBERm$
,p
q
IN

Note that - as pointed out - these will both fail if the output has less than NUMBER+1 lines. A more solid approach would be (gnu sed syntax):

some command | sed '1,NUMBER{H;1h;$!d;${g;q;};};$G'

this one only deletes lines in that range as long as they're not the last line ($!d) - else it overwrites pattern space with hold buffer content (g) and then quits (after printing the current pattern space).

Answer 2

An awk approach:

cmd | awk -v n=3 '
  NR <= n {head = head $0 "\n"; next}
  {print}
  END {printf "%s", head}'

One benefit over @don_crissti's sed approach is that it still works (outputs the lines) if the output has n lines or fewer.

Answer 3

I have xclip and with it this can be done in this way:

./a_command | xclip -in && xclip -o | tail -n +3 && xclip -o | head -n 2

Here is its description:

xclip - command line interface to X selections (clipboard)

NAME
       xclip - command line interface to X selections (clipboard)

SYNOPSIS
       xclip [OPTION] [FILE]...

DESCRIPTION
       Reads from standard in, or from one or more files, and makes the data available as an X selection for pasting into X applications. Prints current X selection to standard out.

       -i, -in
              read text into X selection from standard input or files (default)

       -o, -out
              prints the selection to standard out (generally for piping to a file or program)

Answer 4

A Perl way:

perl -ne '$.<3?($f.=$_):print;}{print $f'

Or, the same thing written less cryptically:

perl -ne 'if($.<3){ $f.=$_ } else{ print } END{print $f}'

For example:

$ cat file
44444
55555
11111
22222
33333

$ cat file | perl -ne '$.<3?($f.=$_):print;}{print $f'
11111
22222
33333
44444
55555

Explanation

• -ne : read the input file/stream line by line and apply the script given by -e to each line.
• $.<3 : $. is the current line number, so change 3 to the number of lines you want to shift.
• $.<3?($f.=$_):print; : this is the conditional operator, the general format is condition ? case1 : case2, it will run case1 if condition is true and case2 if it is false. Here, if the current line number is less than 3, it appends the current line ($_) to the variable $f and, if the line number is greater than 3, it prints.
• }{ print $f : the }{ is perl shorthand for END{}. It will run after all input lines have been processed. At this point, we will have collected all the lines we want to shift and will have printed all the ones we want to leave alone, so print the lines saved as $f.

Answer 5

Use POSIX ex. Yes, it's intended for file editing, but it will work in a pipeline.

printf %s\\n 111 222 333 444 555 | ex -sc '1,2m$|%p|q!' /dev/stdin

This can have any arbitrary commands added at the beginning or end of the pipeline and will work the same. Better yet, given the presence of /dev/stdin, it's POSIX compliant.

(I don't know if /dev/stdin is specified in POSIX or not, but I see it's present both in Linux and Mac OS X.)

This has a readability advantage over using sed's hold space—you just tell ex "move these lines to the end" and it does so. (The rest of the commands mean "print the buffer" and "exit," which are fairly readable as well.)

NB: The ex command given above will fail if given less than 2 lines as input.

Further reading:

• How to edit files non-interactively (e.g. in pipeline)?

Answer 6

A short python snippet:

#!/usr/bin/env python3
import sys
file_ = sys.argv[1]
lines = int(sys.argv[2])
with open(file_) as f:
    f = f.readlines()
    out = f[lines:] + f[:lines]
    print(''.join(out), end='')

Pass the filename as first argument and number of lines to move as second argument.

Example:

$ cat input.txt
44444
55555
11111
22222
33333

$ ./sw_lines.py input.txt 2
11111
22222
33333
44444
55555

$ ./sw_lines.py input.txt 4
33333
44444
55555
11111
22222

Answer 7

If you can store the entire output in memory:

data=$(some command)
n=42                  # or whatever
{ tail -n +$((n+1)) <<<"$data"; head -n $n <<<"$data"; } > outputfile

Answer 8

Here is a another option which requires GNU sed:

(x=$(gsed -u 3q);cat;printf %s\\n "$x")

-u makes GNU sed unbuffered so that the sed command does not consume more than 3 lines of STDIN. The command substitution removes empty lines, so that the command does not include empty lines at the end of the output if the third, third and second, or third, second, and first lines are empty.

You can also do something like this:

tee >(sponge /dev/stdout|sed -u 3q)|sed 1,3d

Without sponge /dev/stdout| the command would fail with long inputs. sponge /dev/stdout can also be replaced with tac|tac, even though that prints for example a\ncb if the input is a\nb\nc where \n is a linefeed, or with (x=$(cat);printf %s\\n "$x"), even though that removes empty lines from the end of the input. The command above deletes the first line when the number of lines of the input is one or two.