eqNullSafe

Equality test that is safe for null values.

Added in Databricks Runtime 11.0

Changed in Databricks Runtime 13.0: Supports Spark Connect.

Syntax

eqNullSafe(other)

Parameters

Returns

Column

Notes

Unlike Pandas, PySpark doesn't consider NaN values to be NULL. See the NaN Semantics for details.

Examples

from pyspark.sql import Row
df1 = spark.createDataFrame([
    Row(id=1, value='foo'),
    Row(id=2, value=None)
])
df1.select(
    df1['value'] == 'foo',
    df1['value'].eqNullSafe('foo'),
    df1['value'].eqNullSafe(None)
).show()

# +-------------+---------------+----------------+
# |(value = foo)|(value <=> foo)|(value <=> NULL)|
# +-------------+---------------+----------------+
# |         true|           true|           false|
# |         NULL|          false|            true|
# +-------------+---------------+----------------+