Questions tagged [parallelism]
Performing tasks in parallel, to make use of multiple processors
277
questions
0
votes
1
answer
51
views
How to verify size of pigz (parallel gzip) archive contents? [duplicate]
I created some pigz (parallel gzip) - home page - compressed archives of my SSD disk drives. (compiled version 2.8)
I called one of them 4TB-SATA-disk--Windows10--2024-Jan-21.img.gz which says the ...
- 29k
1
vote
1
answer
33
views
Curious lscpu Output Related to CPU Calculation
I have been trying to understand lscpu's output and came across several threads dedicated to concepts of CPUs, physical cores, and threads. Based on those threads, to get the total number of CPUs (...
- 13
0
votes
1
answer
51
views
Starting another OS to run parallel with another system
So at work we have a few systems running on their own hardware that is identical to all of the systems. If one of the hardware fails, one of the running system will take over and start the system that ...
0
votes
3
answers
103
views
Problem in writing a parallel version of my Bash code
I am trying to parallelise my sample Bash script and have tried commands like & and wait. Please let me know what is an effective way to make it parallel
My current code is working fine for ...
- 11
1
vote
2
answers
589
views
How can I run `$(nproc)`× parallel instances of `openssl dhparam` & when first instance exits with `0`, kill the other instances?
I'm running openssl dhparam -out dhparam4096.pem 4096 and it pegs a single core at 100% for the duration of the task (which can be considerable on some processors). I have 1 or more additional cores ...
- 131
3
votes
2
answers
856
views
Race condition not seen while two scripts write to a same file
I have two scripts running parallelly and they are echoing to the same file. One script is echoing +++++++++++++++ to the file while the other script is echoing =========== to the file.
Below is the ...
- 101
1
vote
1
answer
105
views
optipng in parallel using shell
I would like to run optipng in parallel with my 8-thread CPU using the shell.
I know the program itself is not multicore optimized, the only way will be to run 8 files with it.
I have 500+ PNG images ...
- 29k
2
votes
0
answers
479
views
How do you fetch a large file over http in parallel?
Question:
Since HTTP supports resuming at an offset, are there any tools (or existing options for commands like wget or curl) that will launch multiple threads to fetch the file in parallel with ...
- 495
0
votes
0
answers
553
views
Terminal closes itself when running many python processes
I was trying to run multiple machine learning experiments simultaneously to save time. My OS is Fedora 36, and I use emacs. To automate the experiment I used runs.py
import subprocess
subprocess.run(&...
- 103
4
votes
4
answers
390
views
Parallelize filling of a variable in Bash
I am trying to speed up a find command which accesses files on multiple harddrives by utilizing parallelization. Unfortunately, either the parallelization is ignored or the variable is not filled.
...
- 278
1
vote
1
answer
216
views
Does parallel limit cause some items to not be checked?
I have near 400 git repos on my machine.
And this is a script that I use to find their status collectively:
function Check()
{
gitFolder="$1"
parent=$(dirname $gitFolder);
...
- 721
1
vote
1
answer
234
views
How to parallelize a bash for loop and use the iteration variable for file IO
I have a bash script that more or less looks like this:
N=32
for i in $(seq -f "%06g" 0 ${LAST_NUM}) # LAST_NUM is an env variable
do
((j=j%N)); ((j++==0)) && wait # Wait for all ...
- 11
2
votes
1
answer
1k
views
GNU Parallel with -j -N still uses one CPU
How can I get reasonable parallelisation on multi-core nodes without saturating resources? As in many other similar questions, the question is really how to learn to tweak GNU Parallel to get ...
- 1,490
1
vote
1
answer
133
views
Bash: how to check if any subshell failed?
Consider this code:
job()
{
local id=$1
sleep $id
}
do_job_in_parallel()
{
local pids=()
# run subshells
for id in $(seq 4)
do
job $id &
pids=("${pids[@]}" $!)
...
- 619
0
votes
0
answers
555
views
Increasing max_parallel_downloads is not effective in yum commands
To install all the updates in a Red Hat or CentOS machine, we use the following command
yum -y update --skip-broken --nobest
We are looking for ways to improve total running time of installing the ...
