Part 5 Reporting on progress updates

5.1 Basic progress updates

Three strokes writing three in Chinese

The progressr package can be used to report on progress when using the future framework for parallelization.

slow_sqrt <- function(X) {
  p <- progressr::progressor(along = X)
  lapply(X, function(x) {
    Sys.sleep(0.1)
    p()
    sqrt(x)
  })
}

Note how we do not specify how progress is reported or rendered. We just report on where we do progress.

If we call the above as-is, no progress is reported:

X <- 1:50
y <- slow_sqrt(X)

With progress reporting:

progressr::handlers(global = TRUE)

X <- 1:50
y <- slow_sqrt(X)
#  |====================                               |  40%

5.2 Progress updates in parallel

Update to use future_lapply() instead of lapply():

slow_sqrt <- function(X) {
  p <- progressr::progressor(along = X)
  future.apply::future_lapply(X, function(x) {
    Sys.sleep(0.1)
    p(message = sprintf("x=%g", x)) # <= passing along a message
    sqrt(x)
  })
}

This works the same way when running sequentially:

progressr::handlers(global = TRUE)
plan(sequential)

X <- 1:50
y <- slow_sqrt(X)
#  |====================                               |  40%

and when running in parallel:

progressr::handlers(global = TRUE)
plan(multisession)

X <- 1:50
y <- slow_sqrt(X)
#  |====================                               |  40%

as well as running remotely:

progressr::handlers(global = TRUE)
hostnames <- c("pi", "remote.server.org")
plan(cluster, workers = hostnames)

X <- 1:50
y <- slow_sqrt(X)
#  |====================                               |  40%

“Near-live” progress updates: Note that, even if there are only two parallel workers, and therefore two futures here, we will still receive progress updates whilst these futures are busy processing all elements. This works because the progress updates, signalled by p() above, are sent back to our main R session using background communication channels. These channels are frequently polled (by resolved() and value()) until the futures are completed, which is how we can relay and handle these progress updates in parallel.

5.3 Customizing how progress is reported

You might have noticed I’m using the term “progress updates” and “progress reporting”, rather than “progress bars”. The reason is that progress can be reported in many other ways than via visual progress bars, e.g. by audio, via the operating system’s built-in notification framework, via email, etc.

Below is an example where we use the beepr package to play sounds as we progress. There are no visual cues of progress - just sound.

progressr::handlers(global = TRUE)
progressr::handlers("beepr")

X <- 1:50
y <- slow_sqrt(X)
# 𝅘𝅥𝅮 ♫ . . . . . . . . . ♫

We can also combine multiple ways of reporting on progress, e.g. by combining audio from beepr and progress bars from the progress package:

progressr::handlers(c("beepr", "progress"))

and also configure them in detail:

library(progressr)
handlers(list(
  handler_progress(
    format   = ":spin :current/:total (:message) [:bar] :percent in :elapsed ETA: :eta",
    width    = 60,
    complete = "+"
  ),
  handler_beepr(
    finish   = "wilhelm"
  )
))

which gives:

X <- 1:50
y <- slow_sqrt(X)
# \ 4/10 (x=32) [+++++++++>-------------]  40% in  1s ETA:  1s

If you use RStudio, you can report on progress via the RStudio Job interface:

progressr::handlers("rstudio")

5.4 Demo: Mandelbrot sets

library(future)
plan(sequential)
options(future.demo.mandelbrot.region = 1L)

demo("mandelbrot", package = "progressr", ask = FALSE)

library(future)
plan(multisession, workers = 4)
options(future.demo.mandelbrot.region = 1L)

demo("mandelbrot", package = "progressr", ask = FALSE)

library(future)
plan(multisession, workers = 4)
options(future.demo.mandelbrot.region = 2L)

demo("mandelbrot", package = "progressr", ask = FALSE)

library(future)
plan(multisession, workers = 4)
options(future.demo.mandelbrot.region = 2L)
options(future.demo.mandelbrot.delay = FALSE)

demo("mandelbrot", package = "progressr", ask = FALSE)

library(future)
plan(multisession, workers = 8)
options(future.demo.mandelbrot.region = 3L)
options(future.demo.mandelbrot.delay = FALSE)

demo("mandelbrot", package = "progressr", ask = FALSE)

Source: https://github.com/HenrikBengtsson/progressr/blob/develop/demo/mandelbrot.R