More efficient way to loop through lines in shell

Question

I've come to learn that looping through lines in bash by

while read line; do stuff; done <file

Is not the most efficient way to do it. https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice

What is a more time/resource efficient method?

Answer 1

Here's a time'd example using Bash and awk. I have 1 million records in a file:

$ wc -l 1M
1000000 1M

Counting it's records with bash, using while read:

$ time while read -r line ; do ((i++)) ; done < 1M ; echo $i

real    0m12.440s
user    0m11.548s
sys     0m0.884s
1000000

Using let "i++" took 15.627 secs (real) and NOPing with do : ; 10.466. Using awk:

$ time awk '{i++}END{print i}' 1M
1000000

real    0m0.128s
user    0m0.128s
sys     0m0.000s

Answer 2

As others have said, it depends on what you're doing.

The reason it's inefficient is that everything runs in its own process. Depending on what you are doing, that may or may not be a big deal.

If what you want to do in the loop is run another shell process, you won't get any gain from eliminating the loop. If you can do what you need without the need for a loop, you could get a gain.

Answer 3

awk? Perl? C(++)? Of course it depends on if you're interested in CPU time or programmer time, and the latter depends on what the programmer is used to using.

The top answer to the question you linked to pretty much explains that the biggest problem is spawning external processes for simple text processing tasks. E.g. running an instance of awk or a pipeline of sed and cut for each single line just to get a part of the string is silly.

If you want to stay in shell, use the string processing parameter expansions (${var#word}, ${var:n:m}, ${var/search/replace} etc.) and other shell features as much as you can. If you see yourself running a set of commands for each input line, it's time to think the structure of the script again. Most of the text processing commands can process a whole file with one execution, so use that.

A trivial/silly example:

while read -r line; do
    x=$(echo "$line" | awk '{print $2}')
    somecmd "$x"
done < file

would be better as

awk < file '{print $2}' | while read -r x ; do somecmd "$x" ; done

Answer 4

Choose between awk or perl both are efficient