pyspark.sql.table_arg.TableArg.orderBy#

TableArg.orderBy(*cols)[source]#

Orders the data within each partition by the specified columns.

This method orders the data within partitions. It must be called after partitionBy() or withSinglePartition() has been called.

Parameters:

colsstr, Column, or list: Column names or Column objects to order by. Columns can be ordered in ascending or descending order using Column.asc() or Column.desc().

Returns:

TableArg: A new TableArg instance with ordering applied.

Examples

>>> from pyspark.sql.functions import udtf
>>>
>>> @udtf(returnType="key: int, value: string")
... class ProcessUDTF:
...     def eval(self, row):
...         yield row["key"], row["value"]
...
>>> df = spark.createDataFrame(
...     [(1, "b"), (1, "a"), (2, "d"), (2, "c")], ["key", "value"]
... )
>>>
>>> # Order by a single column within partitions
>>> result = ProcessUDTF(df.asTable().partitionBy("key").orderBy("value"))
>>> result.show()
+---+-----+
|key|value|
+---+-----+
|  1|    a|
|  1|    b|
|  2|    c|
|  2|    d|
+---+-----+
>>>
>>> # Order by multiple columns
>>> df2 = spark.createDataFrame(
...     [(1, "a", 2), (1, "a", 1), (1, "b", 3)], ["key", "value", "num"]
... )
>>> result2 = ProcessUDTF(df2.asTable().partitionBy("key").orderBy("value", "num"))
>>> result2.show()
+---+-----+
|key|value|
+---+-----+
|  1|    a|
|  1|    a|
|  1|    b|
+---+-----+
>>>
>>> # Order by descending order
>>> result3 = ProcessUDTF(df.asTable().partitionBy("key").orderBy(df.value.desc()))
>>> result3.show()
+---+-----+
|key|value|
+---+-----+
|  1|    b|
|  1|    a|
|  2|    d|
|  2|    c|
+---+-----+