Questions tagged [doparallel]
R package that is a “parallel backend” for the foreach package. It provides a mechanism needed to execute foreach loops in parallel.
doparallel
479
questions
1
vote
1
answer
106
views
How do I loop through a list of formulas in GEE model in R?
I am trying to run a set of GEE models. However, each model takes about an hour to an hour and a half to run. This wouldnt normally be an issue, but given the amount of models i am trying to run, I am ...
- 75
-2
votes
1
answer
80
views
Append CSV file inside foreach loop
I have a parallelized foreach loop code in R that generates a list of dataframes. I want to save and append a .csv file in each iteration, in order to get a final .csv files containing the complete ...
0
votes
1
answer
24
views
RMySql doParallel failing
I am using parallel querying with RMySQL. Is it possible to share the connection between workers?
library(foreach)
library(doParallel)
library(RMySQL)
con <- dbConnect(MySQL(),
...
- 823
0
votes
2
answers
142
views
Fast Sampling with Replacement in R without using a loop or Apply
I need a fast and efficient way of sampling with replacement for my Bootstrapping exercise.
I found a similar question here but the solution doesn't offer enough of a speed up
similar question
Here is ...
- 938
0
votes
1
answer
142
views
%dopar% is very slow when using iterators compared to loading the data from text file
%dopar% is phenomenally slower when using iterators, compared to loading the data from text file.
I've compared 3 use cases
using %dopar% without any iterators dopar1
using %dopar% with iterators ...
- 97
4
votes
3
answers
1k
views
How can I read multiple large CSVs in chunks in parallel using R?
I have 10 very large CSV files (which may or may not have the same headers) that I am reading in and processing consecutively in chunks using "readr" package read_csv_chunked(). Currently, I ...
- 95
0
votes
2
answers
92
views
parallel computing in R
I am familiar with Matlab, but recently started to use R. I encounterd a problem when using parallel computing in R.
I want to use a matrix or a 3-d array as an output after parellel computing. In ...
- 125
0
votes
1
answer
60
views
foreach %dopar% do not produce all results
I was trying to using parallel computing by using foreach %dopar%. In short, when I try set nCores > 1, I received incomplete results. When I set nCores = 1, the results are fine. Here are the ...
1
vote
1
answer
210
views
Writing to database in parallel in R
I try to write a table which is a processed subset of a global data variable, in a normal for loop this piece of code works fine but when I try to do it in parallel it raises an error.
Here is my ...
- 2,641
1
vote
2
answers
140
views
How to get simple dopar in R example working
I'm trying to get a simple dopar example working so I can parallelize some very expensive cross validation calculations. When I run the following code, my result array r is filled with NA:
library(...
- 2,080
1
vote
1
answer
164
views
Problem with parallel computing over an index using foreach and %dopar% in R
I am trying to efficiently code the computation of the KDN complexity measure, which involves a loop over all of the rows of a distance matrix and making some computations out of it.
I am trying to ...
0
votes
0
answers
140
views
Using different libraries in clusterCall and doParallel
I am trying to use foreach and dopar for my loops. I will try to be brief.
My code looks like:
cl = makeCluster (10)
registerDoParallel(cl)
clusterCall(cl,function()library(dplyr))
But now i created ...
- 71
1
vote
0
answers
56
views
foreach not scaling in bootstrapping when number of selected observations is large
I have an interesting problem with bootstrapping in R using the foreach package. I have an algorithm that works as follows
Given R, the number of repetitions in the bootstrap and N_B, the number of ...
- 305
0
votes
0
answers
40
views
dimension problem on replacing for loops with foreach in R
I have the following (very simplified) loop setting:
res <- array(0, c(4, 3, 2))
for (i in 1:4) {
for (j in 1:3) {
res[i,j,] <- c(i,j) # this is not the vector I have, it should ...
1
vote
1
answer
326
views
How to optimize simple parallel processing with tasks taking different amounts of time to complete?
I have some R code using the foreach and doParallel to parallelize (?) an lapply() call, aka parLapply(). It is 100 tasks and I'm splitting across my laptop's 4 cores. The function automatically ...
- 765