Questions tagged [doparallel]
R package that is a “parallel backend” for the foreach package. It provides a mechanism needed to execute foreach loops in parallel.
doparallel
478
questions
0
votes
1
answer
80
views
memory usage when splitting large dataframe using lapply
I have a large dataframe 54k rows, 38k columns. My goal is to run a simple function, test_func, on this dataframe. test_func requires the use of 8 index columns that contain generic information, such ...
- 2,813
1
vote
0
answers
57
views
When is furrr::future_pmap() faster than pmap()?
I implemented a function that is quite expansive.
start<-Sys.time()
plan(multicore, workers = 4)
grid <- expand.grid(v,x,y,z)
mapping<-pmap(list(grid[,1],grid[,2],grid[,3],grid[,4]),.f=...
- 11
1
vote
1
answer
53
views
doPar does not initialize when run through Rscript
I want to create an Rscript that uses dopar to count the lines of many files using many cores, then outputs a TSV of the file names, plus the number of lines (divided by 4).
I am using Ubuntu 20.04. ...
- 159
1
vote
0
answers
64
views
Parallel tasks not removed after stopping cluster
After running and stopping a parallel process, workers still appear to be running on my system and taking up resources. I know processes belong to parallel because they are the same number as the ...
- 51
2
votes
1
answer
54
views
do.call() function within %do.par%, function arguments not found
I am implementing a Monte Carlo simulation to test several methods (many of them).
The methods are implementented in a methods.R script. To illustrate, let's say that I have only implemented 2 methods....
1
vote
2
answers
69
views
Ordering issues in foreach in R
To my knowledge, the results obtained from parallel computations are expected to be unordered. Does foreach perform any operations by default to ensure that the order of results matches the order of a ...
- 649
2
votes
0
answers
21
views
Acceleration with DoParallel in R not working in my code
# approximation of PI using random numbers and large iterations
# the surface of a square of one by one = 1
# the program will generate random numbers between 0 and 1
# these are x and y as ...
- 21
1
vote
1
answer
61
views
Limit iterations in foreach and doParallel
I'm trying to implement a nested for-loop using foreach and doParallel, but I don't want to loop over all combinations of values. Basically, I've got a square dataset and I want to run a function over ...
- 13
0
votes
1
answer
68
views
Error in parallel calculation on R function
I am mastering parallel programming in R simple examples tried to implement the function given in the code, it loads the processor, but the results of calculations are not. Could you please tell me ...
0
votes
0
answers
39
views
What causes "subscript out of bounds" when using doParallel
I am using the MethylMix package, which requires equal rowlength for KIRP.meth, KIRP.mrna, and normal.meth. My code raised "subscript out of bounds" error.
library(MethylMix)
cl <- ...
- 1,505
1
vote
1
answer
58
views
Getting foreach to store its return value in a 3d array
I have a function which returns a (numVals x N) array/matrix. This function needs to be evaluated K times. My goal is to store all results in a multidimensional array containing doubles with shape c(...
- 25
5
votes
1
answer
276
views
Difference between the working process of `mclapply` and `foreach()` loop
This is a general question out of curiosity. I am using the doParallel package for parallel computing. I use these packages for the simulation purposes.
What I observed is that when I was using the ...
- 89
0
votes
0
answers
47
views
Parallel collapse + reduction not working
I am trying to parallelize a two nested loop and the collapse clause fails.
Hey there, I am trying to parallelize these two nested loops in order to calculate two integrals (int_coulomb and ...
0
votes
0
answers
22
views
Subgraph count from a large graph (undirected) in a efficient way. How to increase performace of the R code? (Rcpp, doParallel)
I am working with a very large graph (The number of vertices in the graph is 1000+, which means I working with a (1000+ x 1000+) adjacency matrix). My laptop has an 8-core CPU and 8 gigs of RAM (...
- 89
2
votes
0
answers
38
views
R zombie processes
Is there a way to safely remove/stop/shut down those R processes? Using doParallel package
I'm always calling stopImplicitCluster() but apparantly it doesn't affect.
Thanks in advance!
- 79