All Questions
Tagged with doparallel dplyr
9
questions
0
votes
0
answers
95
views
Nested parLapply : Error in checkForRemoteErrors(val) : 7 nodes produced errors; first error: invalid connexion
I'm trying to execute a code with nested parlapply. My code is as the following:
cl <- (snow::makeCluster(detectCores() - 1))
clusterEvalQ(cl, c(library(dplyr), library(parallel), library(tidyr),...
- 385
0
votes
0
answers
140
views
Using different libraries in clusterCall and doParallel
I am trying to use foreach and dopar for my loops. I will try to be brief.
My code looks like:
cl = makeCluster (10)
registerDoParallel(cl)
clusterCall(cl,function()library(dplyr))
But now i created ...
- 71
1
vote
0
answers
191
views
R foreach: modify .combine to create equivalent of dplyr::bind_rows and include progress bar
Take the following MWE:
df <- data.frame(id =seq(1,100,1),random = seq(2,200,2))
cl <- parallel::makeForkCluster(5) # do not change
doParallel::registerDoParallel(cl) # do not change
results &...
- 634
6
votes
1
answer
468
views
What triggers "Ancestor must be an environment" error?
I am running a parallelized calculation using foreach to work on a lot of time series simultaneously. Among those calculations (within a function called compute_slope() I do something like this
lBd &...
- 448
0
votes
0
answers
713
views
R claims packages that doParallel depends on are not available even though they are installed and can be loaded
I am trying to install the doParallel package on a shared cluster (Platform: x86_64-redhat-linux-gnu) where I do not have root privileges. The cluster is running R version 3.5.0 (I have asked if this ...
- 33
1
vote
1
answer
106
views
Improve nested for loop speed R creating adjacency matrix
I want to create an adjacency matrix. The matrix should show how many identical Value-strings there are per keyword.
My current approach with two for loops takes a while when working with more data. ...
- 426
2
votes
1
answer
138
views
Processing Large Data Sets in R
I have a data set of ~5mm rows of businesses with contact information (ID(int), Email(text), BusinessPhone(text), WorkPhone(text), CellPhone(text)) - over 3 million of these rows contain duplicate ...
0
votes
0
answers
132
views
Nested foreach with changing index size
I'm trying to obtain the return of daily prices for each stock I have. The data is cross-sectionnal and very large. Thus I use doParallel and nested foreach.
Here is the code I've been using so far. (...
- 69
2
votes
1
answer
386
views
Run breakpoint (lm) detection in parallel in R
I am doing about 80000 time series breakpoint detection calculations in R. I have all these extremely different time series where I cannot apply ARIMA models so I am calculating a linear model per ...
- 158