9  mclapply() - is it really magic?

9.1 Output is at best fake from mclapply()

library(parallel)

y <- mclapply(1:3, print)

The output produced by print(1), print(2), and print(3) on the parallel workers, may or may not be visible. If you call the above in in the RStudio Console, or in a RMarkdown document, there will be no output visible. If you run R in a Linux terminal, you will probably see something like:

[1] 1
[1] 3
[1] 2

That is more luck than skill by R - it is the Linux terminal that saves us by relaying the output. Note also that the output is in whatever parallel worker calls print() first. There is also a risk that the different output interweave each other, e.g.

[1[1]
] 1
 3
[1] 2

We can confirm that the output never reaches the main R session by testing with capture.output(). If we do this using a regular lapply() call, then we get:

output <- capture.output(y <- lapply(1:3, print))
output

[1] "[1] 1" "[1] 2" "[1] 3"

However, when we use mclapply(), we get nothing:

output <- capture.output(y <- mclapply(1:3, print))
output

character(0)

This is actually not that surprising. capture.output() sets up a sink, which internally writes output to a textConnection() connection. When used in a forked parallel workers, this connection ends up writing to its locally cloned text connection. That captured output is not available in the parent R session. In other words, all “captured” output happening in parallel workers are lost.

In contrast, all functions in the Futureverse relays output in parallel workers to the main R session. It is also done such that the “natural” order is respected. For example,

library(future.apply)
1plan(multicore)

y <- future_lapply(1:3, print)

1

multicore uses forked parallelization based on the same code as mclapply().

[1] 1
[1] 2
[1] 3

and

output <- capture.output(y <- future_lapply(1:3, print))
output

[1] "[1] 1" "[1] 2" "[1] 3"

9.2 Warnings are lost by mclapply()

What happens with warning? Consider:

y <- lapply(-1:1, sqrt)

Warning in FUN(X[[i]], ...): NaNs produced

which produces a warning from sqrt(-1). If we try the same with mclapply(), that warning is lost:

library(parallel)

y <- mclapply(-1:1, sqrt)

In contrast, all functions in the Futureverse relays warnings, and any other type of condition, in parallel workers to the main R session. It is also done such that the “natural” order is respected. For example,

library(future.apply)
plan(multicore)

y <- future_lapply(-1:1, sqrt)

Warning in ...future.FUN(...future.X_jj, ...): NaNs produced

9.3 Errors are mangled

What happens with errors? Consider:

y <- lapply(list(1, 2, "a"), sqrt)

Error in FUN(X[[i]], ...): non-numeric argument to mathematical function

which produces an error because of sqrt("a"). If we try the same with mclapply(), that error is turned into an obscure warning:

library(parallel)

y <- mclapply(list(1, 2, "a"), sqrt)

Warning in mclapply(list(1, 2, "a"), sqrt): scheduled core 1 encountered error
in user code, all values of the job will be affected

Because it is just a warning, it means that your code keeps running as nothing really happened!

This is one example how mistakes in scientific pipelines can go by unnoticed. If we inspect y, we can see there is information about the error:

str(y)

List of 3
 $ : 'try-error' chr "Error in FUN(X[[i]], ...) : non-numeric argument to mathematical function\n"
  ..- attr(*, "condition")=List of 2
  .. ..$ message: chr "non-numeric argument to mathematical function"
  .. ..$ call   : language FUN(X[[i]], ...)
  .. ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"
 $ : num 1.41
 $ : 'try-error' chr "Error in FUN(X[[i]], ...) : non-numeric argument to mathematical function\n"
  ..- attr(*, "condition")=List of 2
  .. ..$ message: chr "non-numeric argument to mathematical function"
  .. ..$ call   : language FUN(X[[i]], ...)
  .. ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"

To make sure that errors are not slipping by unnoticed, we need to do something like:

is_error <- vapply(y, inherits, "try-error", FUN.VALUE = NA)
if (any(is_error)) {
  ## error objects are stored in attributes
  first_error <- attr(y[is_error][[1]], "condition")
  stop("Detected one or more errors: ", conditionMessage(first_error))
}

Error in eval(expr, envir, enclos): Detected one or more errors: non-numeric argument to mathematical function

In contrast, all functions in the Futureverse relays errors. For example,

library(future.apply)
plan(multicore)

y <- future_lapply(list(1, 2, "a"), sqrt)

Error in ...future.FUN(...future.X_jj, ...): non-numeric argument to mathematical function

9.4 What happens when a parallel crashes?

Sometime a parallel workers crashes. This can happen if there are too many processes running on the same machine and all memory gets consumed. Then the operating systems, particularly on Linux, decides to kill process in order for the machine not to go down. Another reason could be that the R code calls some incorrect C code, and it terminates with a segfault because of that, e.g.

 *** caught illegal operation ***
address 0x2b3a8b234ccd, cause 'illegal operand'

We can emulate terminating the current R process by calling tools::pskill(Sys.getpid()). For example,

y <- lapply(1:3, function(idx) {
  if (idx == 2) tools::pskill(Sys.getpid())
  idx
})

This will result in R terminating abruptly:

Terminated

Now, what happens if we terminate a parallel worker? If we use mclapply(), this is what happens:

library(parallel)

y <- mclapply(1:3, function(idx) {
  if (idx == 2) tools::pskill(Sys.getpid())
  idx
})

Warning in mclapply(1:3, function(idx) {: scheduled core 2 did not deliver a
result, all values of the job will be affected

Again, just a warning! Danger!

In contrast, the Futureverse detects when a parallel workers crashes and gives an informative error message:

library(future.apply)
plan(multicore)

y <- future_lapply(1:3, function(idx) {
  if (idx == 2) tools::pskill(Sys.getpid())
  idx
})

Warning in mccollect(jobs = jobs, wait = TRUE): 1 parallel job did not deliver
a result

Error: Failed to retrieve the result of MulticoreFuture (future_lapply-2) from the forked worker (on localhost; PID 22393). Post-mortem diagnostic: No process exists with this PID, i.e. the forked localhost worker is no longer alive. The total size of the 5 globals exported is 6.30 KiB. The three largest globals are '...future.FUN' (6.20 KiB of class 'function'), '...future.elements_ii' (112 bytes of class 'list') and 'future.call.arguments' (0 bytes of class 'list')