Questions tagged [gnu-parallel]
GNU Parallel is a command-line tool that allows one to run multiple commands in parallel
50
questions
0
votes
0
answers
13
views
GNU parallel: use `--pipe` to process one line at a time
(This is a follow-up to GNU "parallel --pipe" doesn't process stdin by lines as I cannot get the behavior I want even after reading that question and applying the answer that was given.)
...
- 184
1
vote
1
answer
76
views
GNU "parallel" how to achieve tee behaviour
Is it possible to use gnu-parallel to run multiple commands which all read from stdin?
For example:
echo 'hello world' | parallel --keep-order --tagstring 'Part {=$_=uc($_)=}:' --pipe "cat" :...
- 416
1
vote
1
answer
41
views
Gnu Parallel is ignoring the (--sshlogin) attribute order when used with (--keep-order ) option
Here is my example where I request that the job runs on LNXVP1262 first before starting on LNXVP1251.
date ; echo 'hostname;date;sleep 10;date' | parallel --no-run-if-empty --will-cite --onall --keep-...
- 11
2
votes
1
answer
407
views
split file and pass split parts as stdin
Tutorial explains:
The input source can be a file:
parallel -a abc-file echo
but how this actually work? I'd like to take input file: seq 1000 and process it through pipe, but I cannot find way ...
- 583
1
vote
1
answer
76
views
split file and pass split parts as file, not pipe to stdin
How to take file (seq 1000000), split it into multiple smaller ones, which will be then passed into process as a file, not a pipe? I.e.
parallel -a <(seq 1000000) -j 6 -l 10000 <what here?> /...
- 583
-1
votes
1
answer
85
views
Script to Find and Resample Audio via FFMPEG Fails on Files with double quotes
Why does this script explode when file names contain double quotation marks?
for i in **/*.flac; do
echo "$i";
done | parallel -j+0 --eta --joblog /tmp/parallel.log --progress --bar --...
- 101
2
votes
1
answer
316
views
GNU parallel & RSync via intermediate server
I'm trying to spawn multiple rsync instances with gnu parallel to download via an intermediate server. This is the command I use -
cat ~/filelist | parallel --will-cite -j 5 --verbose --progress --...
- 61
1
vote
1
answer
216
views
After a command ran in GNU parallel exits, how can I automatically send SIGINT to another command ran by the same instance of GNU parallel?
This is the command I'm running:
parallel -u ::: 'watch -n 0.1 /home/cringeposter/zfs-mount.sh' 'watch -n 0.1 drill graph.microsoft.com' 'rclone bisync onedrive: /home/cringeposter/zfs/onedrive --...
- 13
0
votes
0
answers
289
views
Passing a piped variable to a sub-command called within GNU parallel
In bash, I am trying to find a list of files and pipe them to parallel.
find *.fasta | parallel "Gblocks {} -b2=$(( "$(cat {} | grep \> | wc -l)" / 2 + 1 ))"
Where Gblocks is a ...
1
vote
1
answer
468
views
GNU Parallel is behaving differently on SIGTERM in two cygwin installations
Consider this small bash function:
saySomethingToSomebody() {
sleep 1
echo "$1 $2."
sleep 2
}
When I call saySomethingToSomebody Hello Adam, it says Hello Adam. and waits a bit ...
- 1,537
0
votes
1
answer
366
views
GNU parallel is not processing all the files
I am using GNU parallel to process BAM files for marking duplicates using java-based picard tool. The reason of using parallel utility is that the picard tool is not multi-thread and the dataset is ...
- 111
1
vote
1
answer
495
views
GNU Parallel usage - how get currently passed string to parallel?
I am using GNU parallel and want to understand - how can I get the individual string passed to each parallel command?
As an example, GNU Parallel documentation shows how to move files from the current ...
- 155
0
votes
1
answer
202
views
Copy interrupted and difficulties to take it over by relaunch with cp -u or rsync
I have a huge data to transfer between 2 macOS (1.3 TB).
Unfortunately, my copy from mac1 to mac2 (by mounting the concerned directory of mac1 on mac2 as network directory), the copy with 'rsync' (...
user342714
2
votes
1
answer
287
views
parallel current datetime in tagstring
I'm using parallel to run multiple test suites and I want to prefix each output line from parallel with datetime to ease debuging.
This how I run parallel
parallel -j12 --linebuffer --colsep ';' --...
- 153
1
vote
1
answer
77
views
gnu parallel error : "Expected a command, got perl"
I have been gnu parallel before but for some reason, it is not running anymore.
I have tried to uninstall and reinstall it without success.
Trying to run the parallel command gives the following error ...
- 175