NEWS
polars 1.1.0
This is an update that corresponds to Python Polars 1.32.0, which includes significant internal changes.
Deprecations
pl$Categorical()
's first argumentordering
is deprecated (pola-rs/polars#23016, #1452, #1468). In this version, global categories are always used, and the behavior matches the previousordering = "lexical"
.- The experimental feature "auto structify" is deprecated
(pola-rs/polars#23351, #1452, #1468).
Since this feature could previously be used in two ways, both are now deprecated:
as_polars_expr()
's argumentstructify
.- Setting the
POLARS_AUTO_STRUCTIFY
environment variable to1
.
<lazyframe>$unique()
and<dataframe>$unique()
's first argument is replaced fromsubset
to...
(dynamic dots) (#1463). Because of this change, it is also deprecated to pass the following objects as the first argument of these functions:NULL
: Usecs$all()
or pass nothing to select all columns.- A list of column names or selectors: Use
!!!
to expand the list. e.g.!!!list("col1", "col2")
.
New features
- New experimental polars selectors have been added
(pola-rs/polars#23351, #1452).
cs$empty()
to avoid matching any column.cs$enum()
for Enum data types.cs$list()
for List data types.cs$array()
for Array data types.cs$struct()
for Struct data types.cs$nested()
for List, Array, or Struct data types.
- polars selectors can now be used in place of column names in more locations (#1452).
- The
...
argument (dynamic dots) of<dataframe>$to_dummies()
. - The
...
argument (dynamic dots) of<dataframe>$partition_by()
. - The
...
argument (dynamic dots) of<lazyframe>$drop()
and<dataframe>$drop()
. - The
...
argument (dynamic dots) of<lazyframe>$drop_nulls()
and<dataframe>$drop_nulls()
. - The
...
argument (dynamic dots) of<lazyframe>$drop_nans()
and<dataframe>$drop_nans()
. - The
...
argument (dynamic dots) of<lazyframe>$unnest()
and<dataframe>$unnest()
. - The
...
argument (dynamic dots) of<lazyframe>$explode()
and<dataframe>$explode()
. - The
...
argument (dynamic dots) of<lazyframe>$unique()
and<dataframe>$unique()
(#1463). - The
on
,index
, andvalues
arguments of<dataframe>$pivot()
. - The
on
andindex
arguments of<lazyframe>$unpivot()
and<dataframe>$unpivot()
.
- The
pl$nth()
gains thestrict
argument (#1452).<expr>$str$pad_end()
and<expr>$str$pad_start()
'slength
argument accepts a polars expression (#1452).<expr>$str$to_integer()
gains thedtype
argument to specify the output data type (#1452).<lazyframe>$sink_csv()
and<dataframe>$write_csv()
gains thedecimal_commna
argument (#1452).
polars 1.0.1
This is a small patch release that includes minor improvements discovered right after the 1.0.0 release.
Performance
- The performance of creating polars expressions has been significantly improved (#1444).
Other improvements
- To improve interoperability with other
data.frame
-like objects, the[[
operator can now be used to extract a column from a polars DataFrame as a Series (#1442).
polars 1.0.0
This is a completely rewritten new version of the polars R package. It improves the internal structure of the package and catches up with Python Polars' API. This version of R Polars matches Python Polars 1.31.0.
Therefore it contains many breaking changes compared to the previous R Polars implementation. Some of those breaking changes are explained below, but many others are due to modifications of function names, argument names, or argument positions. There are too many to list here, so you should refer to the Python Polars API docs.
For compatibility, the old version (polars 0.22.4) is now available as a separate package named "polars0". We can install both polars and polars0 at the same time. See the polars0 documentation for details.
Breaking changes
-
The class names of polars objects have changed:
RPolarsLazyFrame
->polars_lazy_frame
RPolarsDataFrame
->polars_data_frame
RPolarsSeries
->polars_series
RPolarsExpr
->polars_expr
-
Conversion from unknown classes to Polars objects now fails. Developers can specify how those objects should be handled by polars by creating a method for
as_polars_series.my_class
.
-
Conversion from polars objects to R vectors has been revamped:
<series>$to_r()
,<series>$to_list()
and<dataframe>$to_data_frame()
no longer exist. Instead, you must useas.data.frame(<dataframe>)
,as.list(<dataframe>)
,as.vector(<series>)
, or<series>$to_r_vector()
.as.vector(<series>)
will remove attributes that might be useful, for instance to convert Int64 values using the bit64 package or to convert Time values using the hms package. It is therefore recommended to use<series>$to_r_vector()
instead for usual conversions.s_time <- as_polars_series(c("00:00", "12:00"))$str$to_time() as.vector(s_time) #> ℹ `as.vector()` on a Polars Series of type time may drop some useful attributes. #> ℹ Use `$to_r_vector()` instead for finer control of the conversion from Polars to R. #> [1] 0 43200 s_time$to_r_vector() #> 00:00:00 #> 12:00:00
-
In general, polars now uses dots (
...
) in two scenarios:-
to pass an unlimited number of inputs (for instance in
<lazyframe>$select()
,<lazyframe>$cast()
, or<lazyframe>$group_by()
), using dynamic-dots.For example, if you used to pass a vector of column names or a list of expressions, you now need to expand it with
!!!
:### OLD dat <- as_polars_df(head(mtcars, 3)) my_exprs <- list(pl$col("drat") + 1, "mpg", "cyl") dat$select(my_exprs) #> shape: (6, 3) #> ┌──────┬──────┬─────┐ #> │ drat ┆ mpg ┆ cyl │ #> │ --- ┆ --- ┆ --- │ #> │ f64 ┆ f64 ┆ f64 │ #> ╞══════╪══════╪═════╡ #> │ 4.9 ┆ 21.0 ┆ 6.0 │ #> │ 4.9 ┆ 21.0 ┆ 6.0 │ #> │ 4.85 ┆ 22.8 ┆ 4.0 │ #> └──────┴──────┴─────┘
### NEW dat <- as_polars_df(head(mtcars, 3)) my_exprs <- list(pl$col("drat") + 1, "mpg", "cyl") dat$select(!!!my_exprs) #> shape: (3, 3) #> ┌──────┬──────┬─────┐ #> │ drat ┆ mpg ┆ cyl │ #> │ --- ┆ --- ┆ --- │ #> │ f64 ┆ f64 ┆ f64 │ #> ╞══════╪══════╪═════╡ #> │ 4.9 ┆ 21.0 ┆ 6.0 │ #> │ 4.9 ┆ 21.0 ┆ 6.0 │ #> │ 4.85 ┆ 22.8 ┆ 4.0 │ #> └──────┴──────┴─────┘
This also affects
pl$col()
:### NEW pl$col(c("foo", "bar"), "baz") #> Error in `pl$col()`: #> ! Evaluation failed in `$col()`. #> Caused by error in `pl$col()`: #> ! Invalid input for `pl$col()`. #> • `pl$col()` accepts either single strings or Polars data types. pl$col(!!!c("foo", "bar"), "baz") #> cols(["foo", "bar", "baz"])
Another important change in functions that accept dynamic dots is that additional arguments are prefixed with
.
. For example,<lazyframe>$group_by()
now takes dynamic dots, meaning that the argumentmaintain_order
is renamed.maintain_order
(for now, we add a warning if we detect an argument namedmaintain_order
in the dots). -
to force some arguments to be named. We now throw an error if an argument is not named while it should be, for example:
df <- pl$DataFrame(a = 1:4) df$with_columns(pl$col("a")$shift(1, 3)) #> Error in `df$with_columns()`: #> ! Evaluation failed in `$with_columns()`. #> Caused by error: #> ! Evaluation failed in `$with_columns()`. #> Caused by error: #> ! Evaluation failed in `$shift()`. #> Caused by error: #> ! `...` must be empty. #> ✖ Problematic argument: #> • ..1 = 3 #> ℹ Did you forget to name an argument? df$with_columns(pl$col("a")$shift(1, fill_value = 3)) #> shape: (4, 1) #> ┌─────┐ #> │ a │ #> │ --- │ #> │ f64 │ #> ╞═════╡ #> │ 3.0 │ #> │ 1.0 │ #> │ 2.0 │ #> │ 3.0 │ #> └─────┘
-
-
Related to the extended use of dynamic dots,
pl$DataFrame()
andpl$LazyFrame()
more accurately convert input to the correct datatype, for instance when the input is an Rdata.frame
:### OLD pl$DataFrame(data.frame(x = 1, y = "a")) #> shape: (1, 2) #> ┌─────┬─────┐ #> │ x ┆ y │ #> │ --- ┆ --- │ #> │ f64 ┆ str │ #> ╞═════╪═════╡ #> │ 1.0 ┆ a │ #> └─────┴─────┘
### NEW pl$DataFrame(data.frame(x = 1, y = "a")) #> shape: (1, 1) #> ┌───────────┐ #> │ │ #> │ --- │ #> │ struct[2] │ #> ╞═══════════╡ #> │ {1.0,"a"} │ #> └───────────┘ pl$DataFrame(!!!data.frame(x = 1, y = "a")) #> shape: (1, 2) #> ┌─────┬─────┐ #> │ x ┆ y │ #> │ --- ┆ --- │ #> │ f64 ┆ str │ #> ╞═════╪═════╡ #> │ 1.0 ┆ a │ #> └─────┴─────┘
Use
as_polars_df()
andas_polars_lf()
to convert existing Rdata.frame
s to their polars equivalents.
-
The class names
PTime
andrpolars_raw_list
(used to handle time and binary variables) are removed. One should use the classes provided in packages hms and blob instead.### OLD r_df <- tibble::tibble( time = hms::as_hms(c("12:00:00", NA, "14:00:00")), binary = blob::as_blob(c(1L, NA, 2L)), ) # R to Polars pl_df <- as_polars_df(r_df) pl_df #> shape: (3, 2) #> ┌─────────┬──────────────┐ #> │ time ┆ binary │ #> │ --- ┆ --- │ #> │ f64 ┆ list[binary] │ #> ╞═════════╪══════════════╡ #> │ 43200.0 ┆ [b"\x01"] │ #> │ null ┆ [] │ #> │ 50400.0 ┆ [b"\x02"] │ #> └─────────┴──────────────┘ # Polars to R tibble::as_tibble(pl_df) #> # A tibble: 3 × 2 #> time binary #> <dbl> <list> #> 1 43200 <rplrs_r_ [1]> #> 2 NA <rplrs_r_ [0]> #> 3 50400 <rplrs_r_ [1]>
### NEW r_df <- tibble::tibble( time = hms::as_hms(c("12:00:00", NA, "14:00:00")), binary = blob::as_blob(c(1L, NA, 2L)), ) ## R to Polars pl_df <- as_polars_df(r_df) pl_df #> shape: (3, 2) #> ┌──────────┬─────────┐ #> │ time ┆ binary │ #> │ --- ┆ --- │ #> │ time ┆ binary │ #> ╞══════════╪═════════╡ #> │ 12:00:00 ┆ b"\x01" │ #> │ null ┆ null │ #> │ 14:00:00 ┆ b"\x02" │ #> └──────────┴─────────┘ ## Polars to R tibble::as_tibble(pl_df) #> # A tibble: 3 × 2 #> time binary #> <time> <blob> #> 1 12:00 <raw 1 B> #> 2 NA NA #> 3 14:00 <raw 1 B>
Other changes
-
R objects that convert to a Series of length 1 are now treated like scalar values when converting to polars expressions:
### OLD series <- pl$Series("foo", 1) pl$DataFrame(bar = 1:2)$with_columns(series) #> [...truncated...] #> Encountered the following error in Rust-Polars: #> Series foo, length 1 doesn't match the DataFrame height of 2 #> #> If you want expression: Series[foo] to be broadcasted, ensure it is a #> scalar (for instance by adding '.first()').
<expr>$map_batches()
still exists but its usage is discouraged. This function is not guaranteed to interact correctly with the streaming engine. To apply functions from external packages or custom functions that cannot be translated to polars syntax, we now recommend converting the data to adata.frame
and using purrr (note that as of 1.1.0, purrr enables parallel computation). The vignette "Using custom functions" contains more details about this.