find, xarg, Awk Problem

Question

I have the command:

awk 'BEGIN{print "Name, Number"}/value/{print FILENAME, "," $8}' *.txt >> out.csv

Which works perfectly to go through my txt files in the directory, parse the value(s) and write the final csv file with the header (Name, Number).

My issue is that I have "too many" and so I modify it with find and xarg:

find ./ -maxdepth 1 -type f -name '*.txt' | xargs awk 'BEGIN{print "Name, Number"}/value/{print FILENAME, "," $8}' | sed 's/\.\///g' >> out.csv

This has worked in the past, but now I find that -- on occasion -- the header is written more than once to the final csv file. I don't know why. It does ssem to be related to the total number of txt files in the directory such that if I hit a certain number, this happens, but I am not really sure.

thanks.

Answer 1

The find will call awk in batches of files so the BEGIN will be executed once per batch instead of once for all files as you want. Instead of having awk called with all the files as arguments and having the shell fail with a "too many arguments" error, you can have awk read all the files as input and populate it's internal array of files to read (ARGV[]) from that:

find ./ -maxdepth 1 -type f -name '*.txt' |
awk '
    BEGIN { OFS=","; print "Name", "Number" }
    NR==FNR { ARGV[ARGC++]=$0; next }
    /value/ { print substr(FILENAME,3), $8 }
' - > out.csv

I also tidied up a couple of things in the awk script and got rid of the pipe to sed as you never need sed when you're using awk. I changed >> to > as I assume you want to create the output file from scratch whenever the above command is called rather than appending to it.

The above assumes none of your file names contain newlines. If they do then use GNU tools and add -print0 to the end of the find command and RS="\0"; to the BEGIN section of the awk command. It also assumes your file names don't contain " as then the output wouldn't be valid CSV but your first script that you said works perfectly apart from the "too many arguments" issue would fail if your file names contained any of those so they must not.

Answer 2

Run find and awk in a group command (i.e. wrapped in { and }) or in a sub-shell (i.e. wrapped in ( and )) and print the header before running find. Redirect the output from the entire group command or sub-shell to your output file.

For example:

{
  echo "Name,Number"
  find ./ -maxdepth 1 -type f -name '*.txt' -exec \
    awk -v OFS=, '
      FNR==1 { fn = FILENAME; sub(/^\.\//, "", fn };
     /value/ {print fn, $8}' {} +
} >> out.csv

NOTES:

1. See man bash and search for Compound Commands
2. You don't need xargs here - use find's -exec option instead, e.g. find ... -exec awk ... {} +.
3. You don't need sed, either - awk's built-in sub() function can be used to remove the leading ./ from the filenames coming from find. BTW, gsub() can be used for global search and replace, like the /g modifier in sed.