StopIteration error in nested foreach/for loop

Question

I've got a nasty error which only shows up when you ask R with geterrmessage() to actually display it. It also is no problem as long as you run the code in an interactive console session but when you call R non-interactively (in this case through a 3rd party software) this error results in the script being aborted.

Does anyone have an idea why the below code generates a (hidden) StopIteration error?

# Load the required packages
library(foreach)
library(parallel)
library(doParallel)

# ----- set-up cluster for parallel processing -----
ncl <- max(2,floor(detectCores()*0.75)) #number of cores
clst <- makePSOCKcluster(n=ncl) #create cluster for R
registerDoParallel(cl = clst) #register cluster

# ----- display latest error  -----
print(geterrmessage())

# ----- run nested foreach/for loop and stop cluster afterwards -----
t <- foreach (j=1:8) %dopar% {
        for (i in j:8) {
            print(paste("j",j,"i",i))
        }
    }
stopCluster(clst)

# ----- display latest error  -----
print(geterrmessage())

Thanks, Mark

Answer 1

try this black magic : .Internal(seterrmessage("")) idea sourced from stat.ethz.ch/pipermail/r-help/2006-February/087931.html

Answer 2

There are many ways of using foreach, and proceeding despite errors, and logging them; here is an example of one possible approach

library(foreach)
library(parallel)
library(doParallel)
library(purrr) # for safely()

# ----- set-up cluster for parallel processing -----
ncl <- max(2,floor(detectCores()*0.75)) #number of cores
clst <- makePSOCKcluster(n=ncl) #create cluster for R
registerDoParallel(cl = clst) #register cluster


do_stuff <- function(j) {
  
  # bomb ; break on j = 6 
  if(j==6) stop("Bad News")
  make_string <- ""
  for (i in j:8) {
    make_string <- paste0(make_string,paste("j",j,"i",i))
  }
  make_string
}

do_stuff(8) # fine
do_stuff(6) #  error 

do_stuff_safely <- purrr::safely(do_stuff,"")

do_stuff_safely(8) # fine
do_stuff_safely(6) #  error 


# ----- run nested foreach/for loop and stop cluster afterwards -----
t <- foreach (j=1:8) %dopar% {
  do_stuff_safely(j)
}
stopCluster(clst)

# here you have a list of 8 coupled lists, result and error

(results <- lapply(t,\(x)x$result))  # <-- see how 6 is blank
(errors <- lapply(t,\(x)x$error))  # <-- see how 6  has an error recorded

# 
names(errors) <- seq_along(errors)
(any_error <- Filter(\(x)!is.null(x),x = errors) ) 

One can throw a try around the foreach itself to look for errors in foreach i.e. outside any single safe function call example

t <- try(expr = {foreach (j=1:8) %dopar% {
  stop("Error in foreach itself")
  do_stuff_safely(j)
}})

inherits(t,"try-error")

try the above having having removed the stop() and t should no longer inherit from try-error