How would I mix patterns and numeric ranges in sed (or any similar tool - awk for example)? What I want to do is match certain lines in a file, and delete the next n lines before proceeding, and I want to do that as part of a pipeline.
7 Answers
I'll have a go at this.
To delete 5 lines after a pattern (including the line with the pattern):
sed -e '/pattern/,+5d' file.txt
To delete 5 lines after a pattern (excluding the line with the pattern):
sed -e '/pattern/{n;N;N;N;N;d}' file.txt
-
21Note that the
+N
pattern is a GNU extension. Change the firstn
to anN
in your second example to make it include the line with the pattern. Commented Dec 9, 2010 at 17:25 -
2how to delete all lines after the pattern is matched ? I am using sed -e '/<!--#content end--></div>/,$d' out.txt but it gives error saying : sed: -e expression #1, char 24: extra characters after command Thanks in advance.– N molCommented Aug 24, 2013 at 2:37
-
10What's happening is similar but slightly different in each case. In the first recipe,
/pattern/,+5
defines a range, which starts with a line containing "pattern" (/pattern/
) and ends 5 lines later (+5
). The last characterd
is a command to run on each line in that range, which is "delete". In the second recipe, instead of matching a range, it matches just at the line containing the pattern (/pattern/
) and then runs a series of commands:{n;N;N;N;N;d}
, which basically prints the next line (n
) and then reads and finally discards the next 4 lines (N;N;N;N;d
).– pimlottcCommented Oct 2, 2013 at 1:38 -
22On Mac/OS X systems you need to add a semicolon before the closing bracket:
sed -e '/pattern/{n;N;N;N;N;d;}' file.txt
– AvLCommented Nov 18, 2013 at 9:37 -
3For completeness: To delete all lines following a certain pattern
something
do:sed -E '/^something$/,$d'
, where-E
is the POSIX portability extended regex. Commented Nov 17, 2018 at 15:17
Without GNU extensions (e.g. on macOS):
To delete 5 lines after a pattern (including the line with the pattern)
sed -e '/pattern/{N;N;N;N;d;}' file.txt
Add -i ''
to edit in-place.
-
Any ideas on how to delete several lines before and after the match altogether? Can't find any suitable POSIX variant.– t7eCommented May 21, 2022 at 13:37
Simple awk
solutions:
Assume that the regular expression to use for finding matching lines is stored in shell variable $regex
, and the count of lines to skip in $count
.
If the matching line should also be skipped ($count + 1
lines are skipped):
... | awk -v regex="$regex" -v count="$count" \
'$0 ~ regex { skip=count; next } --skip >= 0 { next } 1'
If the matching line should not be skipped ($count
lines after the match are skipped):
... | awk -v regex="$regex" -v count="$count" \
'$0 ~ regex { skip=count; print; next } --skip >= 0 { next } 1'
Explanation:
-v regex="$regex" -v count="$count"
definesawk
variables based on shell variables of the same name.$0 ~ regex
matches the line of interest{ skip=count; next }
initializes the skip count and proceeds to the next line, effectively skipping the matching line; in the 2nd solution, theprint
beforenext
ensures that it is not skipped.--skip >= 0
decrements the skip count and takes action if it is (still) >= 0, implying that the line at hand should be skipped.{ next }
proceeds to the next line, effectively skipping the current line
1
is a commonly used shorthand for{ print }
; that is, the current line is simply printed- Only non-matching and non-skipped lines reach this command.
- The reason that
1
is equivalent to{ print }
is that1
is interpreted as a Boolean pattern that by definition always evaluates to true, which means that its associated action (block) is unconditionally executed. Since there is no associated action in this case,awk
defaults to printing the line.
Using Perl
$ cat delete_5lines.txt
1
2
3
4
5 hello
6
7
8
9
10
11 hai
$ perl -ne ' BEGIN{$y=1} $y=$. if /hello/ ; print if $y==1 or $.-$y > 5 ' delete_5lines.txt
1
2
3
4
11 hai
$
This solution allows you to pass "n" as a parameter and it will read your patterns from a file:
awk -v n=5 '
NR == FNR {pattern[$0]; next}
{
for (patt in pattern) {
if ($0 ~ patt) {
print # remove if you want to exclude a matched line
for (i=0; i<n; i++) getline
next
}
}
print
}
' file.with.patterns -
The file named "-" means stdin for awk, so this is suitable for your pipeline
-
2awk is capable of being a lot more perl-like than i realised! Commented Dec 10, 2010 at 15:55
This might work for you:
cat <<! >pattern_number.txt
> 5 3
> 10 1
> 15 5
> !
sed 's|\(\S*\) \(\S*\)|/\1/,+\2{//!d}|' pattern_number.txt |
sed -f - <(seq 21)
1
2
3
4
5
9
10
12
13
14
15
21
-
4A clever (albeit GNU-Sed-specific) solution, but few people will benefit from it, unless you add an explanation.
pattern_number.txt
is a 2-column file containing the pattern to match in the 1st column, and in the 2nd the number of lines to skip. The firstsed
command transforms the file into ased
script that performs the corresponding matching and skipping; that script is provided via-f
and stdin (-
) to the 2ndsed
command. The 2ndsed
command operates on a sample ad-hoc input file formed from the output ofseq 21
to demonstrate that it works. Commented May 20, 2015 at 18:28 -
Also, the solution comes with one caveat: the method it uses not to skip the first line (the one matching the pattern) has the side effect of also not skipping duplicate lines in the range. Commented May 20, 2015 at 19:04
-
-
Patterns may show up within the next 5 lines, so a straight-up delete would inadvertently skip past criteria matches that would extend the boundary of "next 5 lines". Using awk
, just set __
to # of lines skipped, and FS
to the matching criteria's pattern (i.e. properly-escaped ERE
regex
).
By utilizing FS
, checking whether pattern has matched or not is straight up NF > 1
. Otherwise, every line would be matched twice - first by FS
to split the input row into unnecessary fields, then once more by the desired pattern itself.
jot 30 |
mawk '(_ < NR) * (NF < 2 || _ = NR + __)' FS='(.1|9)$' __=5
1
2
3
4
5
6
7
8
9 <— 9 bypassed 10-14, and 11 bypassed 12-16
17
18
19 <— 19 bypassed 20-24, and 21 bypassed 22-26
27
28
29 <— 29 bypassed 30
( Before _
gets explicitly assigned, depending on context, it's either empty string ""
or numeric 0
, so short of fudging NR
itself, all lines prior to the 1st pattern match would get auto printed, including blank lines. )