Skip to content

Get/set global R session pool capacity (DEPRECATED)

Description

Deprecated. Use polars_options() to get, and pl$set_options() to set.

Usage

pl_get_global_rpool_cap()

pl_set_global_rpool_cap(n)

Arguments

n Integer, the capacity limit R sessions to process R code.

Details

Background R sessions communicate via polars arrow IPC (series/vectors) or R serialize + shared memory buffers via the rust crate ipc-channel. Multi-process communication has overhead because all data must be serialized/de-serialized and sent via buffers. Using multiple R sessions will likely only give a speed-up in a low io - high cpu scenario. Native polars query syntax runs in threads and have no overhead. Polars has as default double as many thread workers as cores. If any worker are queuing for or using R sessions, other workers can still continue any native polars parts as much as possible.

Value

polars_options()$rpool_cap returns the capacity ("limit") of co-running external R sessions / processes. polars_options()$rpool_active is the number of R sessions are already spawned in the pool. rpool_cap is the limit of new R sessions to spawn. Anytime a polars thread worker needs a background R session specifically to run R code embedded in a query via $map_batches(…, in_background = TRUE) or $map_elements(…, in_background = TRUE), it will obtain any R session idling in rpool, or spawn a new R session (process) if capacity is not already reached. If capacity is already reached, the thread worker will sleep and in a R job queue until an R session is idle.

Examples

library("polars")

default = polars_options()$rpool_cap |> print()
#> [1] 4
options(polars.rpool_cap = 8)
polars_options()$rpool_cap
#> [1] 8
options(polars.rpool_cap = default)
polars_options()$rpool_cap
#> [1] 4