Questions tagged [doparallel]
R package that is a “parallel backend” for the foreach package. It provides a mechanism needed to execute foreach loops in parallel.
doparallel
478
questions
0
votes
0
answers
33
views
How can I handle dead workers in R-future parallel processing
I want to fit multiple models to each participant in my data. I want to parallelize this by applying the fitting function in parallel to the data of multiple participants. For one of the models, the ...
0
votes
0
answers
27
views
Error in {task 1 failed - [readValues] cannot read values in R doParallel "foreach"
I am trying to speed up an intervisibility calculation between landscape viewpoints (X, Y, height), using a rasterprofile and line of sight functions over a digital elevation model. I ran a large-...
- 36
1
vote
1
answer
67
views
How would I bootstrap the lmer model? [closed]
I am trying to bootstrap (non-parametric) my lmer model, but I am struggling as to how to specify the formula when the response variables are in columns 18:1543 ( 5:10 for example data). I've tried ...
- 45
0
votes
0
answers
18
views
Error with parallelize='variables' using "missForest" in R
I've started using missForest to potentially replace rfImpute and while doing some testing with both synthetic and real data and the different flavours of parallelization strategies offered by ...
- 320
3
votes
1
answer
122
views
Utilizing multiple nodes for parallel computing in R
I have access to a HPC system. Let's say I have three nodes/system available. Details of each node is as follows:
scontrol show node
Arch=x86_64 CoresPerSocket=10
CPUAlloc=20 CPUTot=20 CPULoad=...
- 444
0
votes
0
answers
29
views
how do I resolve this system singular error?
This is a question about futuremice.
I am trying to run the mice MI in parallel. This is my code.
imp_df <- futuremice(df,
pred = pred_matrix,
meth = ...
3
votes
2
answers
104
views
Shared memory in parallel foreach using set.seed
As seen in this question, in Windows it is not possible to run parallel processes with shared memory in R. Therefore, I have devised the following methodology, using a series of set.seed(), to ...
- 43
0
votes
1
answer
24
views
R crashes with segfault in doing keras::unserialize_model() in foreach loop from doParallel
I'm having a problem where R crashes when calling keras::unserialize_model() in a doParallel foreach loop.
I have to sanitize this code, so hopefully I don't munge anything. And I'm not an R developer;...
- 1,528
2
votes
2
answers
81
views
How to pass once the full dataset to one worker and specific subsets to the other workers in foreach loop using isplit()
I am currently fitting a set of models on a subset of data for each level of a factor variable. As the models take a long time to run, I use the foreach and doParallel package to estimate the set of ...
- 199
0
votes
0
answers
23
views
doParallel foreach with many for-loops within the foreach
I am stuck with an issue when trying to parallelize a function that I have coded.
The function contains several for loops within for loops (which is required to fill multiple arrays with data).
I have ...
- 33
1
vote
0
answers
78
views
R foreach do parallel %dopar% performance problems (and possibly affecting entire computer)
Note:
I recognize this is a slightly more amorphous/non-replicable problem than is ideal, but I feel it is worthwhile given the other instances we've seen on stackoverflow and potential general ...
- 173
0
votes
2
answers
100
views
StopIteration error in nested foreach/for loop
I've got a nasty error which only shows up when you ask R with geterrmessage() to actually display it. It also is no problem as long as you run the code in an interactive console session but when you ...
- 320
1
vote
1
answer
31
views
Parallel processing in R using multiple cores and using a function
I would like to run the apply function (my_func2) more efficiently by using parallelization in R across multiple imputed datasets by using all 8 cores on my computer. Each imputed dataset is about 1.7 ...
- 11
0
votes
0
answers
20
views
uses a function that calls another in a foreach loop dopar
I have two functions myfunct1 and myfunct2. myfunct1 uses the myfunct2 function
I want to use these two functions in a foreach()%dopar%{} loop but i get a error:
Error in {:
task 1 failed - "...
- 173
0
votes
1
answer
42
views
Continue to interact when there is a long calculate
I have code that is taking a long time to execute and I would like to know if there is a way to continue interacting with other elements of the application?
When you press the "go" button, a ...
- 173
0
votes
1
answer
80
views
memory usage when splitting large dataframe using lapply
I have a large dataframe 54k rows, 38k columns. My goal is to run a simple function, test_func, on this dataframe. test_func requires the use of 8 index columns that contain generic information, such ...
- 2,813
1
vote
0
answers
57
views
When is furrr::future_pmap() faster than pmap()?
I implemented a function that is quite expansive.
start<-Sys.time()
plan(multicore, workers = 4)
grid <- expand.grid(v,x,y,z)
mapping<-pmap(list(grid[,1],grid[,2],grid[,3],grid[,4]),.f=...
- 11
1
vote
1
answer
53
views
doPar does not initialize when run through Rscript
I want to create an Rscript that uses dopar to count the lines of many files using many cores, then outputs a TSV of the file names, plus the number of lines (divided by 4).
I am using Ubuntu 20.04. ...
- 159
1
vote
0
answers
64
views
Parallel tasks not removed after stopping cluster
After running and stopping a parallel process, workers still appear to be running on my system and taking up resources. I know processes belong to parallel because they are the same number as the ...
- 51
2
votes
1
answer
54
views
do.call() function within %do.par%, function arguments not found
I am implementing a Monte Carlo simulation to test several methods (many of them).
The methods are implementented in a methods.R script. To illustrate, let's say that I have only implemented 2 methods....
1
vote
2
answers
69
views
Ordering issues in foreach in R
To my knowledge, the results obtained from parallel computations are expected to be unordered. Does foreach perform any operations by default to ensure that the order of results matches the order of a ...
- 649
2
votes
0
answers
21
views
Acceleration with DoParallel in R not working in my code
# approximation of PI using random numbers and large iterations
# the surface of a square of one by one = 1
# the program will generate random numbers between 0 and 1
# these are x and y as ...
- 21
1
vote
1
answer
61
views
Limit iterations in foreach and doParallel
I'm trying to implement a nested for-loop using foreach and doParallel, but I don't want to loop over all combinations of values. Basically, I've got a square dataset and I want to run a function over ...
- 13
0
votes
1
answer
67
views
Error in parallel calculation on R function
I am mastering parallel programming in R simple examples tried to implement the function given in the code, it loads the processor, but the results of calculations are not. Could you please tell me ...
0
votes
0
answers
39
views
What causes "subscript out of bounds" when using doParallel
I am using the MethylMix package, which requires equal rowlength for KIRP.meth, KIRP.mrna, and normal.meth. My code raised "subscript out of bounds" error.
library(MethylMix)
cl <- ...
- 1,505
1
vote
1
answer
58
views
Getting foreach to store its return value in a 3d array
I have a function which returns a (numVals x N) array/matrix. This function needs to be evaluated K times. My goal is to store all results in a multidimensional array containing doubles with shape c(...
- 25
5
votes
1
answer
276
views
Difference between the working process of `mclapply` and `foreach()` loop
This is a general question out of curiosity. I am using the doParallel package for parallel computing. I use these packages for the simulation purposes.
What I observed is that when I was using the ...
- 89
0
votes
0
answers
47
views
Parallel collapse + reduction not working
I am trying to parallelize a two nested loop and the collapse clause fails.
Hey there, I am trying to parallelize these two nested loops in order to calculate two integrals (int_coulomb and ...
0
votes
0
answers
22
views
Subgraph count from a large graph (undirected) in a efficient way. How to increase performace of the R code? (Rcpp, doParallel)
I am working with a very large graph (The number of vertices in the graph is 1000+, which means I working with a (1000+ x 1000+) adjacency matrix). My laptop has an 8-core CPU and 8 gigs of RAM (...
- 89
2
votes
0
answers
38
views
R zombie processes
Is there a way to safely remove/stop/shut down those R processes? Using doParallel package
I'm always calling stopImplicitCluster() but apparantly it doesn't affect.
Thanks in advance!
- 79
0
votes
0
answers
54
views
Assigning a Vector and Outputting Dataframe from foreach loop
I would like to assign a value to a vector, then add that vector to a data frame within a foreach loop. It looks like I will need a combine function, but I'm not sure which one.
My specific example ...
- 159
0
votes
0
answers
22
views
Parallelizing tree creation in R using the packages data.tree and foreach
I'm trying to parallelize the creation of a tree in R using the following code :
# Creation of the root
root <- Node$new("Root")
# Function to create the tree recursively
create.tree....
2
votes
1
answer
79
views
Using doParallel to start multiple system calls from R within a Slurm job
I am using an R script that basically pastes together commandline commands to execute through system2(). The commands run some Java application.
Now, I want to spawn multiple processes of that Java ...
- 1,131
0
votes
0
answers
294
views
Error in summary.connection(connection): invalid connection
I am using the foreach %dopar% and I get that error. I read that I should register a backend before. I am not sure how to do this in Windows. Can you help me, please?
A chunk of the code below:
`...
- 31
1
vote
1
answer
141
views
Parallel ODE solver using deSolve in R?
I am trying to find the numerical solution for an ODE at time points (t_1,...,t_n) with parameters (theta_1,...,theta_n), where n is in the order of tens of thousands. In fact, this can be seen as n &...
- 381
1
vote
0
answers
31
views
How can I use libraries within a parallelized function with ply?
Novice programmer here. I am trying to use plyr's ldply function and doParallel to speed up a process, but my function uses the libraries terra and sp. My limited understanding of parallel processing ...
- 11
1
vote
1
answer
59
views
Parallelization in R
I am new to parallelization and I want to improve the current version of parallelization of my code to see if I can improve the time. I have 4 rasters and one shapefile with 9000 features. I want for ...
- 437
3
votes
0
answers
100
views
How to call system() within %dopar% iterations in R
How should I call external programs from sub-instances of parallelized R? The problem could occur also on other contexts, but I am using library(foreach) and library(doFuture) on slurm-based HPC. As ...
- 79
0
votes
0
answers
102
views
Error in checkForRemoteErrors(lapply(cl, recvResult)) : 7 nodes produced errors; first error: object '.doSnowGlobals' not found
I'm attempting to run an r script across multiple cores and everything seems to work, however when I change where I install my packages (reset the libpath to a project folder instead my home directory)...
0
votes
1
answer
211
views
R in RStudio with doParallel - CPU usage of each core reduces after initial ~10 secods
I'm using RStudio 2023.03.0 on my machine with Windows 10 Pro 22H2.
My machine has 8 cores and I'm running my code on a cluster of 7 cores with doParallel:
cores=detectCores()
cl <- makeCluster(...
- 793
0
votes
0
answers
95
views
Nested parLapply : Error in checkForRemoteErrors(val) : 7 nodes produced errors; first error: invalid connexion
I'm trying to execute a code with nested parlapply. My code is as the following:
cl <- (snow::makeCluster(detectCores() - 1))
clusterEvalQ(cl, c(library(dplyr), library(parallel), library(tidyr),...
- 385
0
votes
0
answers
54
views
Troubleshooting foreach/dopar in R with error in larger model function but not in foreach loop
I am working on parallelizing a meta-population (disease) model using foreach and doParallel. In short, I'm trying to run the same process for each meta-population in parallel (there are ~ 2000 ...
- 11
0
votes
1
answer
62
views
Parallelizing a function using ´foreach´ and ´doParallel´
Im trying to parallelize a function, but Im running into problems.
Basically, Im running the function manysimulations over a desig_grid
This is my main function (it works if I don't do any ...
- 141
1
vote
0
answers
55
views
Parallelise a simulation using 'foreach' and 'doParallel'; "error in evaluating the argument 'x' in selecting a method for function 't'"
I want to parallelise a simulation.
I used this example of a simple parallelization, which works just fine.
#install.packages("foreach")
#install.packages("doParallel")
library(...
- 141
0
votes
0
answers
38
views
About try() function in parallel computing
I am using pbapply::pblapply(.., FUN = myfun, ..) to do some parallel computing. Inside the myfun, I use try(my_algo(), silent = T) to catch errors for the algorithm. This algorithm sometimes meets ...
- 103
1
vote
1
answer
106
views
How do I loop through a list of formulas in GEE model in R?
I am trying to run a set of GEE models. However, each model takes about an hour to an hour and a half to run. This wouldnt normally be an issue, but given the amount of models i am trying to run, I am ...
- 75
-2
votes
1
answer
80
views
Append CSV file inside foreach loop
I have a parallelized foreach loop code in R that generates a list of dataframes. I want to save and append a .csv file in each iteration, in order to get a final .csv files containing the complete ...
0
votes
1
answer
24
views
RMySql doParallel failing
I am using parallel querying with RMySQL. Is it possible to share the connection between workers?
library(foreach)
library(doParallel)
library(RMySQL)
con <- dbConnect(MySQL(),
...
- 823
0
votes
2
answers
142
views
Fast Sampling with Replacement in R without using a loop or Apply
I need a fast and efficient way of sampling with replacement for my Bootstrapping exercise.
I found a similar question here but the solution doesn't offer enough of a speed up
similar question
Here is ...
- 938
0
votes
1
answer
142
views
%dopar% is very slow when using iterators compared to loading the data from text file
%dopar% is phenomenally slower when using iterators, compared to loading the data from text file.
I've compared 3 use cases
using %dopar% without any iterators dopar1
using %dopar% with iterators ...
- 97