GNU parallel does not split work evenly

Question

My understanding is that the -X option should distribute arguments evenly among the jobs. Yet, I get a very skewed distribution:

user@host:/tmp/ptest$ count() {
>   echo $#
> }
user@host:/tmp/ptest$ export -f count

user@host:/tmp/ptest$ count *.jpg
5825
user@host:/tmp/ptest$ parallel -X count ::: *.jpg
5039
197
197
197
195

Interestingly enough, using only a subset of the files leads to an even distribution of parameters:

user@host:/tmp/ptest$ count p129*.jpg
975
user@host:/tmp/ptest$ parallel -X count ::: p129*.jpg
244
244
244
243

user@host:/tmp/ptest$ count p12*.jpg
4007
user@host:/tmp/ptest$ parallel -X count ::: p12*.jpg
1002
1002
1002
1001

user@host:/tmp/ptest$ count p13*.jpg
1818
user@host:/tmp/ptest$ parallel -X count ::: p13*.jpg
455
455
455
453

Why is wrong in the first case and how can I fix it?

Answer 1

-X distributes evenly when it hits EOF.

So in your case it fills up the buffer for a full command line (5039 names) and starts that. It then reads another 800 names until it reaches EOF. That is not enough to start a full job, so these are distributed amongst the jobslots.

See page 37 https://zenodo.org/record/1146014

It is done this way to avoid having to read all jobs in advance, as all jobs may not be available (think tail -f file.names | parallel ...).

It might be a better idea, if GNU Parallel read enough names to fill up all jobslots before starting the next job in -Xmode, so it could detect EOF earlier. This, however, has not been implemented. Patch welcome.

A workaround is to use:

ls *.jpg |
  parallel --round --pipe -N1 parallel -Xj1 count

It cannot output after each job, but it can do linebuffered output:

ls *.jpg |
  parallel --lb --round --pipe -N1 parallel --lb -Xj1 count

If the filenames contain \n:

printf '%s\0' *.jpg |
  parallel --recend '\0' --round --lb --pipe -N1 parallel -0Xj1 count