Function readJSONStream

  • readJSONStream(
        stream: Readable,
        options?: Partial<ReadJsonOptions>,
    ): Promise<pl.DataFrame>

  • Read a newline delimited JSON stream into a DataFrame.

    Parameters

    • stream: Readable

      readable stream containing json data

    • Optionaloptions: Partial<ReadJsonOptions>
      • inferSchemaLength

        Maximum number of lines to read to infer schema. If set to 0, all columns will be read as pl.Utf8. If set to null, a full table scan will be done (slow). Note: this is done per batch

      • batchSize

        Number of lines to read into the buffer at once. Modify this to change performance.

    Returns Promise<pl.DataFrame>

    Example

    >>> const readStream = new Stream.Readable({read(){}});
    >>> readStream.push(`${JSON.stringify({a: 1, b: 2})} \n`);
    >>> readStream.push(`${JSON.stringify({a: 2, b: 2})} \n`);
    >>> readStream.push(`${JSON.stringify({a: 3, b: 2})} \n`);
    >>> readStream.push(`${JSON.stringify({a: 4, b: 2})} \n`);
    >>> readStream.push(null);

    >>> pl.readJSONStream(readStream, { format: "lines" }).then(df => console.log(df));
    shape: (4, 2)
    ┌─────┬─────┐
    │ a   ┆ b   │
    │ --- ┆ --- │
    │ i64 ┆ i64 │
    ╞═════╪═════╡
    │ 1   ┆ 2   │
    ├╌╌╌╌╌┼╌╌╌╌╌┤
    │ 2   ┆ 2   │
    ├╌╌╌╌╌┼╌╌╌╌╌┤
    │ 3   ┆ 2   │
    ├╌╌╌╌╌┼╌╌╌╌╌┤
    │ 4   ┆ 2   │
    └─────┴─────┘

Settings

Member Visibility