249

Trying to debug an issue with a server and my only log file is a 20GB log file (with no timestamps even! Why do people use System.out.println() as logging? In production?!)

Using grep, I've found an area of the file that I'd like to take a look at, line 347340107.

Other than doing something like

head -<$LINENUM + 10> filename | tail -20 

... which would require head to read through the first 347 million lines of the log file, is there a quick and easy command that would dump lines 347340100 - 347340200 (for example) to the console?

update I totally forgot that grep can print the context around a match ... this works well. Thanks!

2

19 Answers 19

455

I found two other solutions if you know the line number but nothing else (no grep possible):

Assuming you need lines 20 to 40,

sed -n '20,40p;41q' file_name

or

awk 'FNR>=20 && FNR<=40' file_name

When using sed it is more efficient to quit processing after having printed the last line than continue processing until the end of the file. This is especially important in the case of large files and printing lines at the beginning. In order to do so, the sed command above introduces the instruction 41q in order to stop processing after line 41 because in the example we are interested in lines 20-40 only. You will need to change the 41 to whatever the last line you are interested in is, plus one.

0
140
# print line number 52
sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3,  efficient on large files 

method 3 efficient on large files

fastest way to display specific lines

4
  • I'm trying to figure out how to adapt method 3 to use a range instead of a single line, but I'm afraid my sed-foo isn't up to the task. Commented Jul 7, 2013 at 17:44
  • 11
    @XiongChiamiov How about sed -n '1,500p;501q' for printing 1-500 ?
    – Sam
    Commented Aug 12, 2014 at 1:17
  • 8
    The reason the first two lines/methods are less efficient, is that they continue processing all lines after Line 52, till the end, whereas #3 stops after printing Line 52.
    – flow2k
    Commented May 28, 2019 at 0:59
  • 5
    This answer would benefit from explaining what all argument do. Commented Feb 3, 2020 at 13:02
70

with GNU-grep you could just say

grep --context=10 ...
5
  • 8
    Or more specifically 10 lines before: grep -B 10 ... Or 10 lines after: grep -A 10 ... Commented May 21, 2012 at 11:14
  • 22
    This command not working, below sed -n '<start>,<end>p' is working
    – Basav
    Commented Jun 21, 2013 at 5:40
  • 5
    This is actually not what you want because it will process the whole file even if the match is in the top bit. At this point a head/tail or tail/head combo is much more effective.
    – Sklivvz
    Commented May 22, 2015 at 13:24
  • 3
    This doesn't satisfy the asked question at all as this doesn't offer a way to output a specific line, as asked. Commented Nov 17, 2016 at 16:21
  • 2
    NOT WORKING! Commented Mar 25, 2020 at 20:56
26

No there isn't, files are not line-addressable.

There is no constant-time way to find the start of line n in a text file. You must stream through the file and count newlines.

Use the simplest/fastest tool you have to do the job. To me, using head makes much more sense than grep, since the latter is way more complicated. I'm not saying "grep is slow", it really isn't, but I would be surprised if it's faster than head for this case. That'd be a bug in head, basically.

1
  • 2
    Unless lines are fixed width in bytes, you don't know where to move the file pointer without counting new line characters from the start of the file. Commented May 4, 2013 at 18:49
23

What about:

tail -n +347340107 filename | head -n 100

I didn't test it, but I think that would work.

1
  • No, usually tail has a limit of 256 last kilobytes or similar, depending on version and OS. Commented Jun 8, 2015 at 12:48
18

I prefer just going into less and

  • typing 50% to goto halfway the file,
  • 43210G to go to line 43210
  • :43210 to do the same

and stuff like that.

Even better: hit v to start editing (in vim, of course!), at that location. Now, note that vim has the same key bindings!

14

You can use the ex command, a standard Unix editor (part of Vim now), e.g.

  • display a single line (e.g. 2nd one):

    ex +2p -scq file.txt
    

    corresponding sed syntax: sed -n '2p' file.txt

  • range of lines (e.g. 2-5 lines):

    ex +2,5p -scq file.txt
    

    sed syntax: sed -n '2,5p' file.txt

  • from the given line till the end (e.g. 5th to the end of the file):

    ex +5,p -scq file.txt
    

    sed syntax: sed -n '2,$p' file.txt

  • multiple line ranges (e.g. 2-4 and 6-8 lines):

    ex +2,4p +6,8p -scq file.txt
    

    sed syntax: sed -n '2,4p;6,8p' file.txt

Above commands can be tested with the following test file:

seq 1 20 > file.txt

Explanation:

  • + or -c followed by the command - execute the (vi/vim) command after file has been read,
  • -s - silent mode, also uses current terminal as a default output,
  • q followed by -c is the command to quit editor (add ! to do force quit, e.g. -scq!).
1
  • As indicated above, don't forget to quit processing entire file with sed after last line of interest is displayed.
    – AnrDaemon
    Commented Mar 2, 2022 at 14:24
13

If your line number is 100 to read

head -100 filename | tail -1
12

I'd first split the file into few smaller ones like this

$ split --lines=50000 /path/to/large/file /path/to/output/file/prefix

and then grep on the resulting files.

1
  • agreed, break that log up and create a cron job to do that properly. use logrotate or something similar to keep them from getting so huge.
    – Tanj
    Commented Oct 10, 2008 at 19:52
9

Get ack

Ubuntu/Debian install:

$ sudo apt-get install ack-grep

Then run:

$ ack --lines=$START-$END filename

Example:

$ ack --lines=10-20 filename

From $ man ack:

--lines=NUM
    Only print line NUM of each file. Multiple lines can be given with multiple --lines options or as a comma separated list (--lines=3,5,7). --lines=4-7 also works. 
    The lines are always output in ascending order, no matter the order given on the command line.
2
  • 1
    This, to me seems like the command with the most intuitive syntax out of all the answers here.
    – nzn
    Commented Apr 26, 2018 at 13:25
  • From version 2.999_06 on Jan 10, 2019 the --lines parameter has been removed.
    – burny
    Commented Jan 7, 2020 at 10:17
4

sed will need to read the data too to count the lines. The only way a shortcut would be possible would there to be context/order in the file to operate on. For example if there were log lines prepended with a fixed width time/date etc. you could use the look unix utility to binary search through the files for particular dates/times

4

Use

x=`cat -n <file> | grep <match> | awk '{print $1}'`

Here you will get the line number where the match occurred.

Now you can use the following command to print 100 lines

awk -v var="$x" 'NR>=var && NR<=var+100{print}' <file>

or you can use "sed" as well

sed -n "${x},${x+100}p" <file>
1
  • If you have more than one match, use : "awk 'NR==1{print $1}" for first match and so on Commented Jul 30, 2015 at 12:05
2

With sed -e '1,N d; M q' you'll print lines N+1 through M. This is probably a bit better then grep -C as it doesn't try to match lines to a pattern.

1
  • 1
    -e is optional here.
    – flow2k
    Commented May 28, 2019 at 0:57
2

Building on Sklivvz' answer, here's a nice function one can put in a .bash_aliases file. It is efficient on huge files when printing stuff from the front of the file.

function middle()
{
    startidx=$1
    len=$2
    endidx=$(($startidx+$len))
    filename=$3

    awk "FNR>=${startidx} && FNR<=${endidx} { print NR\" \"\$0 }; FNR>${endidx} { print \"END HERE\"; exit }" $filename
}
1

To display a line from a <textfile> by its <line#>, just do this:

perl -wne 'print if $. == <line#>' <textfile>

If you want a more powerful way to show a range of lines with regular expressions -- I won't say why grep is a bad idea for doing this, it should be fairly obvious -- this simple expression will show you your range in a single pass which is what you want when dealing with ~20GB text files:

perl -wne 'print if m/<regex1>/ .. m/<regex2>/' <filename>

(tip: if your regex has / in it, use something like m!<regex>! instead)

This would print out <filename> starting with the line that matches <regex1> up until (and including) the line that matches <regex2>.

It doesn't take a wizard to see how a few tweaks can make it even more powerful.

Last thing: perl, since it is a mature language, has many hidden enhancements to favor speed and performance. With this in mind, it makes it the obvious choice for such an operation since it was originally developed for handling large log files, text, databases, etc.

2
  • really, it doesn't seem that way to me, since when is running one perl command more complicated than say, running 2+ programs piped together (further down the page), and, I think you are actually saying because I typed more of an explanation that required you to READ, since there are equally complex (or more) down the page that did not get blown out of the water... sheesh Commented Apr 7, 2015 at 11:39
  • Note that the user asked for a range of lines -- your example can be trivially adapted though.
    – Sklivvz
    Commented May 22, 2015 at 13:31
0

You could try this command:

egrep -n "*" <filename> | egrep "<line number>"
0

Easy with perl! If you want to get line 1, 3 and 5 from a file, say /etc/passwd:

perl -e 'while(<>){if(++$l~~[1,3,5]){print}}' < /etc/passwd
0
0

I am surprised only one other answer (by Ramana Reddy) suggested to add line numbers to the output. The following searches for the required line number and colours the output.

file=FILE
lineno=LINENO
wb="107"; bf="30;1"; rb="101"; yb="103"
cat -n ${file} | { GREP_COLORS="se=${wb};${bf}:cx=${wb};${bf}:ms=${rb};${bf}:sl=${yb};${bf}" grep --color -C 10 "^[[:space:]]\\+${lineno}[[:space:]]"; }
0
0

print line 5

sed -n '5p' file.txt
sed '5q' file.txt

print everything else than line 5

`sed '5d' file.txt

and my creation using google

#!/bin/bash
#removeline.sh
#remove deleting it comes move line xD

usage() {                                 # Function: Print a help message.
  echo "Usage: $0 -l LINENUMBER -i INPUTFILE [ -o OUTPUTFILE ]"
  echo "line is removed from INPUTFILE"
  echo "line is appended to OUTPUTFILE"
}
exit_abnormal() {                         # Function: Exit with error.
  usage
  exit 1
}

while getopts l:i:o:b flag
do
    case "${flag}" in
        l) line=${OPTARG};;
        i) input=${OPTARG};;
        o) output=${OPTARG};;
    esac
done

if [ -f tmp ]; then
echo "Temp file:tmp exist. delete it yourself :)"
exit
fi

if [ -f "$input" ]; then
   re_isanum='^[0-9]+$'
   if ! [[ $line =~ $re_isanum ]] ; then
      echo "Error: LINENUMBER must be a positive, whole number."
      exit 1
   elif [ $line -eq "0" ]; then
      echo "Error: LINENUMBER must be greater than zero."
      exit_abnormal
   fi
   if [ ! -z $output ]; then
      sed -n "${line}p" $input >> $output
   fi
   if [ ! -z $input ]; then
      # remove this sed command and this comes move line to other file
      sed "${line}d" $input > tmp && cp tmp $input
   fi
fi

if [ -f tmp ]; then
rm tmp
fi

Not the answer you're looking for? Browse other questions tagged or ask your own question.