pyspark.sql.tvf.TableValuedFunction.variant_explode#

TableValuedFunction.variant_explode(input, recursive=False)[source]#

Separates a variant object/array into multiple rows containing its fields/elements.

Its result schema is struct<pos int, key string, value variant>. pos is the position of the field/element in its parent object/array, and value is the field/element value. key is the field name when exploding a variant object, or is NULL when exploding a variant array. When recursive is true, it also emits all nested descendants and adds a leading path column containing each field/element’s JSONPath. It ignores any input that is not a variant array/object, including SQL NULL, variant null, and any other variant values.

Added in version 4.0.0.

Changed in version 4.4.0: Added the recursive parameter.

Parameters:

inputColumn: input column of values to explode.
recursivebool, optional: whether to recursively explode nested objects and arrays, by default False.

Returns:

DataFrame

Examples

Example 1: Using variant_explode with a variant array

>>> import pyspark.sql.functions as sf
>>> spark.tvf.variant_explode(sf.parse_json(sf.lit('["hello", "world"]'))).show()
+---+----+-------+
|pos| key|  value|
+---+----+-------+
|  0|NULL|"hello"|
|  1|NULL|"world"|
+---+----+-------+

Example 2: Using variant_explode with a variant object

>>> import pyspark.sql.functions as sf
>>> spark.tvf.variant_explode(sf.parse_json(sf.lit('{"a": true, "b": 3.14}'))).show()
+---+---+-----+
|pos|key|value|
+---+---+-----+
|  0|  a| true|
|  1|  b| 3.14|
+---+---+-----+

Example 3: Using variant_explode with an empty variant array

>>> import pyspark.sql.functions as sf
>>> spark.tvf.variant_explode(sf.parse_json(sf.lit('[]'))).show()
+---+---+-----+
|pos|key|value|
+---+---+-----+
+---+---+-----+

Example 4: Using variant_explode with an empty variant object

>>> import pyspark.sql.functions as sf
>>> spark.tvf.variant_explode(sf.parse_json(sf.lit('{}'))).show()
+---+---+-----+
|pos|key|value|
+---+---+-----+
+---+---+-----+

Example 5: Recursively exploding a nested variant

>>> value = sf.parse_json(sf.lit('{"a": [1, 2]}'))
>>> spark.tvf.variant_explode(value, recursive=True).show()
+------+---+----+-----+
|  path|pos| key|value|
+------+---+----+-----+
|   $.a|  0|   a|[1,2]|
|$.a[0]|  0|NULL|    1|
|$.a[1]|  1|NULL|    2|
+------+---+----+-----+