Skip to content

Collect multiple LazyFrames at the same time

Source code

Description

This can run all the computation graphs in parallel or combined. Common Subplan Elimination is applied on the combined plan, meaning that diverging queries will run only once.

Usage

pl$collect_all(lazy_frames, ..., engine = c("auto", "in-memory", "streaming"))

Arguments

lazy_frames A list of LazyFrames to collect.
… These dots are for future extensions and must be empty.
engine The engine name to use for processing the query. One of the followings:
  • “auto” (default): Select the engine automatically. The “in-memory” engine will be selected for most cases.
  • “in-memory”: Use the in-memory engine.
  • “streaming”: [Experimental] Use the (new) streaming engine.

Value

A list containing all the collected DataFrames, in the same order as the input LazyFrames.

Examples

library("polars")

lf <- as_polars_lf(mtcars)$with_columns(sqrt_mpg = pl$col("mpg")$sqrt())

cyl_4 <- lf$filter(pl$col("cyl") == 4)
cyl_6 <- lf$filter(pl$col("cyl") == 6)

# We could do `cyl_4$collect()` and `cyl_6$collect()`, but this would be
# wasteful because `sqrt_mpg` would be computed twice.
# `pl$collect_all()` executes only once the parts of the query that are
# identical across LazyFrames.
pl$collect_all(list(cyl_4, cyl_6))

#> [[1]]
#> shape: (11, 12)
#> ┌──────┬─────┬───────┬───────┬───┬─────┬──────┬──────┬──────────┐
#> │ mpg  ┆ cyl ┆ disp  ┆ hp    ┆ … ┆ am  ┆ gear ┆ carb ┆ sqrt_mpg │
#> │ ---  ┆ --- ┆ ---   ┆ ---   ┆   ┆ --- ┆ ---  ┆ ---  ┆ ---      │
#> │ f64  ┆ f64 ┆ f64   ┆ f64   ┆   ┆ f64 ┆ f64  ┆ f64  ┆ f64      │
#> ╞══════╪═════╪═══════╪═══════╪═══╪═════╪══════╪══════╪══════════╡
#> │ 22.8 ┆ 4.0 ┆ 108.0 ┆ 93.0  ┆ … ┆ 1.0 ┆ 4.0  ┆ 1.0  ┆ 4.774935 │
#> │ 24.4 ┆ 4.0 ┆ 146.7 ┆ 62.0  ┆ … ┆ 0.0 ┆ 4.0  ┆ 2.0  ┆ 4.939636 │
#> │ 22.8 ┆ 4.0 ┆ 140.8 ┆ 95.0  ┆ … ┆ 0.0 ┆ 4.0  ┆ 2.0  ┆ 4.774935 │
#> │ 32.4 ┆ 4.0 ┆ 78.7  ┆ 66.0  ┆ … ┆ 1.0 ┆ 4.0  ┆ 1.0  ┆ 5.6921   │
#> │ 30.4 ┆ 4.0 ┆ 75.7  ┆ 52.0  ┆ … ┆ 1.0 ┆ 4.0  ┆ 2.0  ┆ 5.51362  │
#> │ …    ┆ …   ┆ …     ┆ …     ┆ … ┆ …   ┆ …    ┆ …    ┆ …        │
#> │ 21.5 ┆ 4.0 ┆ 120.1 ┆ 97.0  ┆ … ┆ 0.0 ┆ 3.0  ┆ 1.0  ┆ 4.636809 │
#> │ 27.3 ┆ 4.0 ┆ 79.0  ┆ 66.0  ┆ … ┆ 1.0 ┆ 4.0  ┆ 1.0  ┆ 5.22494  │
#> │ 26.0 ┆ 4.0 ┆ 120.3 ┆ 91.0  ┆ … ┆ 1.0 ┆ 5.0  ┆ 2.0  ┆ 5.09902  │
#> │ 30.4 ┆ 4.0 ┆ 95.1  ┆ 113.0 ┆ … ┆ 1.0 ┆ 5.0  ┆ 2.0  ┆ 5.51362  │
#> │ 21.4 ┆ 4.0 ┆ 121.0 ┆ 109.0 ┆ … ┆ 1.0 ┆ 4.0  ┆ 2.0  ┆ 4.626013 │
#> └──────┴─────┴───────┴───────┴───┴─────┴──────┴──────┴──────────┘
#> 
#> [[2]]
#> shape: (7, 12)
#> ┌──────┬─────┬───────┬───────┬───┬─────┬──────┬──────┬──────────┐
#> │ mpg  ┆ cyl ┆ disp  ┆ hp    ┆ … ┆ am  ┆ gear ┆ carb ┆ sqrt_mpg │
#> │ ---  ┆ --- ┆ ---   ┆ ---   ┆   ┆ --- ┆ ---  ┆ ---  ┆ ---      │
#> │ f64  ┆ f64 ┆ f64   ┆ f64   ┆   ┆ f64 ┆ f64  ┆ f64  ┆ f64      │
#> ╞══════╪═════╪═══════╪═══════╪═══╪═════╪══════╪══════╪══════════╡
#> │ 21.0 ┆ 6.0 ┆ 160.0 ┆ 110.0 ┆ … ┆ 1.0 ┆ 4.0  ┆ 4.0  ┆ 4.582576 │
#> │ 21.0 ┆ 6.0 ┆ 160.0 ┆ 110.0 ┆ … ┆ 1.0 ┆ 4.0  ┆ 4.0  ┆ 4.582576 │
#> │ 21.4 ┆ 6.0 ┆ 258.0 ┆ 110.0 ┆ … ┆ 0.0 ┆ 3.0  ┆ 1.0  ┆ 4.626013 │
#> │ 18.1 ┆ 6.0 ┆ 225.0 ┆ 105.0 ┆ … ┆ 0.0 ┆ 3.0  ┆ 1.0  ┆ 4.254409 │
#> │ 19.2 ┆ 6.0 ┆ 167.6 ┆ 123.0 ┆ … ┆ 0.0 ┆ 4.0  ┆ 4.0  ┆ 4.38178  │
#> │ 17.8 ┆ 6.0 ┆ 167.6 ┆ 123.0 ┆ … ┆ 0.0 ┆ 4.0  ┆ 4.0  ┆ 4.219005 │
#> │ 19.7 ┆ 6.0 ┆ 145.0 ┆ 175.0 ┆ … ┆ 1.0 ┆ 5.0  ┆ 6.0  ┆ 4.438468 │
#> └──────┴─────┴───────┴───────┴───┴─────┴──────┴──────┴──────────┘