pyspark.sql.tvf.TableValuedFunction.variant_explode

TableValuedFunction.variant_explode(input)

Separates a variant object/array into multiple rows containing its fields/elements.

Its result schema is struct<pos int, key string, value variant>. pos is the position of the field/element in its parent object/array, and value is the field/element value. key is the field name when exploding a variant object, or is NULL when exploding a variant array. It ignores any input that is not a variant array/object, including SQL NULL, variant null, and any other variant values.

New in version 4.0.0.

Parameters

inputColumn

input column of values to explode.

Returns

DataFrame

Examples

Example 1: Using variant_explode with a variant array

>>> import pyspark.sql.functions as sf
>>> spark.tvf.variant_explode(sf.parse_json(sf.lit('["hello", "world"]'))).show()
+---+----+-------+
|pos| key|  value|
+---+----+-------+
|  0|NULL|"hello"|
|  1|NULL|"world"|
+---+----+-------+

Example 2: Using variant_explode with a variant object

>>> import pyspark.sql.functions as sf
>>> spark.tvf.variant_explode(sf.parse_json(sf.lit('{"a": true, "b": 3.14}'))).show()
+---+---+-----+
|pos|key|value|
+---+---+-----+
|  0|  a| true|
|  1|  b| 3.14|
+---+---+-----+

Example 3: Using variant_explode with an empty variant array

>>> import pyspark.sql.functions as sf
>>> spark.tvf.variant_explode(sf.parse_json(sf.lit('[]'))).show()
+---+---+-----+
|pos|key|value|
+---+---+-----+
+---+---+-----+

Example 4: Using variant_explode with an empty variant object

>>> import pyspark.sql.functions as sf
>>> spark.tvf.variant_explode(sf.parse_json(sf.lit('{}'))).show()
+---+---+-----+
|pos|key|value|
+---+---+-----+
+---+---+-----+