toArrow

Returns the contents of this DataFrame as PyArrow pyarrow.Table.

Added in Databricks Runtime 15.3

Syntax

toArrow()

Returns

pyarrow.Table

Notes

This method should only be used if the resulting PyArrow pyarrow.Table is expected to be small, as all the data is loaded into the driver's memory.

This is only available if PyArrow is installed and available.

This API is a developer API.

Examples

df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], schema=["age", "name"])
df.coalesce(1).toArrow()
# pyarrow.Table
# age: int64
# name: string
# ----
# age: [[2,5]]
# name: [["Alice","Bob"]]