Skip to content

Collect multiple LazyFrames at the same time

Source code

Description

This can run all the computation graphs in parallel or combined. Common Subplan Elimination is applied on the combined plan, meaning that diverging queries will run only once.

Usage

pl$collect_all(
  lazy_frames,
  ...,
  engine = c("auto", "in-memory", "streaming"),
  optimizations = pl\$QueryOptFlags()
)

Arguments

lazy_frames A list of LazyFrames to collect.
… These dots are for future extensions and must be empty.
engine The engine name to use for processing the query. One of the followings:
  • “auto” (default): Select the engine automatically. In Polars 1.16, the “in-memory” engine is selected for most cases. Starting with Polars 2.0, the streaming engine is selected instead. Use “in-memory” to keep the current execution engine explicitly.
  • “in-memory”: Use the in-memory engine. This is an escape hatch for queries whose incidental row order differs under streaming.
  • “streaming”: Use the streaming engine.
optimizations [Experimental] A QueryOptFlags object to indicate optimization passes done during query optimization.

Value

A list containing all the collected DataFrames, in the same order as the input LazyFrames.

Examples

library("polars")

lf <- as_polars_lf(mtcars)$with_columns(sqrt_mpg = pl$col("mpg")$sqrt())

cyl_4 <- lf$filter(pl$col("cyl") == 4)
cyl_6 <- lf$filter(pl$col("cyl") == 6)

# We could do `cyl_4$collect()` and `cyl_6$collect()`, but this would be
# wasteful because `sqrt_mpg` would be computed twice.
# `pl$collect_all()` executes only once the parts of the query that are
# identical across LazyFrames.
pl$collect_all(list(cyl_4, cyl_6))

#> [[1]]
#> shape: (11, 12)
#> ┌──────┬─────┬───────┬───────┬───┬─────┬──────┬──────┬──────────┐
#> │ mpg  ┆ cyl ┆ disp  ┆ hp    ┆ … ┆ am  ┆ gear ┆ carb ┆ sqrt_mpg │
#> │ ---  ┆ --- ┆ ---   ┆ ---   ┆   ┆ --- ┆ ---  ┆ ---  ┆ ---      │
#> │ f64  ┆ f64 ┆ f64   ┆ f64   ┆   ┆ f64 ┆ f64  ┆ f64  ┆ f64      │
#> ╞══════╪═════╪═══════╪═══════╪═══╪═════╪══════╪══════╪══════════╡
#> │ 22.8 ┆ 4.0 ┆ 108.0 ┆ 93.0  ┆ … ┆ 1.0 ┆ 4.0  ┆ 1.0  ┆ 4.774935 │
#> │ 24.4 ┆ 4.0 ┆ 146.7 ┆ 62.0  ┆ … ┆ 0.0 ┆ 4.0  ┆ 2.0  ┆ 4.939636 │
#> │ 22.8 ┆ 4.0 ┆ 140.8 ┆ 95.0  ┆ … ┆ 0.0 ┆ 4.0  ┆ 2.0  ┆ 4.774935 │
#> │ 32.4 ┆ 4.0 ┆ 78.7  ┆ 66.0  ┆ … ┆ 1.0 ┆ 4.0  ┆ 1.0  ┆ 5.6921   │
#> │ 30.4 ┆ 4.0 ┆ 75.7  ┆ 52.0  ┆ … ┆ 1.0 ┆ 4.0  ┆ 2.0  ┆ 5.51362  │
#> │ …    ┆ …   ┆ …     ┆ …     ┆ … ┆ …   ┆ …    ┆ …    ┆ …        │
#> │ 21.5 ┆ 4.0 ┆ 120.1 ┆ 97.0  ┆ … ┆ 0.0 ┆ 3.0  ┆ 1.0  ┆ 4.636809 │
#> │ 27.3 ┆ 4.0 ┆ 79.0  ┆ 66.0  ┆ … ┆ 1.0 ┆ 4.0  ┆ 1.0  ┆ 5.22494  │
#> │ 26.0 ┆ 4.0 ┆ 120.3 ┆ 91.0  ┆ … ┆ 1.0 ┆ 5.0  ┆ 2.0  ┆ 5.09902  │
#> │ 30.4 ┆ 4.0 ┆ 95.1  ┆ 113.0 ┆ … ┆ 1.0 ┆ 5.0  ┆ 2.0  ┆ 5.51362  │
#> │ 21.4 ┆ 4.0 ┆ 121.0 ┆ 109.0 ┆ … ┆ 1.0 ┆ 4.0  ┆ 2.0  ┆ 4.626013 │
#> └──────┴─────┴───────┴───────┴───┴─────┴──────┴──────┴──────────┘
#> 
#> [[2]]
#> shape: (7, 12)
#> ┌──────┬─────┬───────┬───────┬───┬─────┬──────┬──────┬──────────┐
#> │ mpg  ┆ cyl ┆ disp  ┆ hp    ┆ … ┆ am  ┆ gear ┆ carb ┆ sqrt_mpg │
#> │ ---  ┆ --- ┆ ---   ┆ ---   ┆   ┆ --- ┆ ---  ┆ ---  ┆ ---      │
#> │ f64  ┆ f64 ┆ f64   ┆ f64   ┆   ┆ f64 ┆ f64  ┆ f64  ┆ f64      │
#> ╞══════╪═════╪═══════╪═══════╪═══╪═════╪══════╪══════╪══════════╡
#> │ 21.0 ┆ 6.0 ┆ 160.0 ┆ 110.0 ┆ … ┆ 1.0 ┆ 4.0  ┆ 4.0  ┆ 4.582576 │
#> │ 21.0 ┆ 6.0 ┆ 160.0 ┆ 110.0 ┆ … ┆ 1.0 ┆ 4.0  ┆ 4.0  ┆ 4.582576 │
#> │ 21.4 ┆ 6.0 ┆ 258.0 ┆ 110.0 ┆ … ┆ 0.0 ┆ 3.0  ┆ 1.0  ┆ 4.626013 │
#> │ 18.1 ┆ 6.0 ┆ 225.0 ┆ 105.0 ┆ … ┆ 0.0 ┆ 3.0  ┆ 1.0  ┆ 4.254409 │
#> │ 19.2 ┆ 6.0 ┆ 167.6 ┆ 123.0 ┆ … ┆ 0.0 ┆ 4.0  ┆ 4.0  ┆ 4.38178  │
#> │ 17.8 ┆ 6.0 ┆ 167.6 ┆ 123.0 ┆ … ┆ 0.0 ┆ 4.0  ┆ 4.0  ┆ 4.219005 │
#> │ 19.7 ┆ 6.0 ┆ 145.0 ┆ 175.0 ┆ … ┆ 1.0 ┆ 5.0  ┆ 6.0  ┆ 4.438468 │
#> └──────┴─────┴───────┴───────┴───┴─────┴──────┴──────┴──────────┘