pyspark.sql.functions.substring#

pyspark.sql.functions.substring(str, pos, len)[source]#

Substring starts at pos and is of length len when str is String type or returns the slice of byte array that starts at pos in byte and is of length len when str is Binary type.

Added in version 1.5.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters:

strColumn or column name: target column to work on. A column that evaluates to a string or binary.
posColumn or column name or int: starting position in str. A column that evaluates to an integer.

Changed in version 4.0.0: pos now accepts column and column name.
lenColumn or column name or int: length of chars. A column that evaluates to an integer.

Changed in version 4.0.0: len now accepts column and column name.

Returns:

Column: substring of given value. Returns a column of the same type as the input.

See also

pyspark.sql.functions.instr()
pyspark.sql.functions.locate()
pyspark.sql.functions.substr()
pyspark.sql.functions.substring_index()
pyspark.sql.Column.substr()

Notes

The position is not zero based, but 1 based index.

Examples

Example 1: Using literal integers as arguments

>>> import pyspark.sql.functions as sf
>>> df = spark.createDataFrame([('abcd',)], ['s',])
>>> df.select('*', sf.substring(df.s, 1, 2)).show()
+----+------------------+
|   s|substring(s, 1, 2)|
+----+------------------+
|abcd|                ab|
+----+------------------+

Example 2: Using columns as arguments

>>> import pyspark.sql.functions as sf
>>> df = spark.createDataFrame([('Spark', 2, 3)], ['s', 'p', 'l'])
>>> df.select('*', sf.substring(df.s, 2, df.l)).show()
+-----+---+---+------------------+
|    s|  p|  l|substring(s, 2, l)|
+-----+---+---+------------------+
|Spark|  2|  3|               par|
+-----+---+---+------------------+

>>> df.select('*', sf.substring(df.s, df.p, 3)).show()
+-----+---+---+------------------+
|    s|  p|  l|substring(s, p, 3)|
+-----+---+---+------------------+
|Spark|  2|  3|               par|
+-----+---+---+------------------+

>>> df.select('*', sf.substring(df.s, df.p, df.l)).show()
+-----+---+---+------------------+
|    s|  p|  l|substring(s, p, l)|
+-----+---+---+------------------+
|Spark|  2|  3|               par|
+-----+---+---+------------------+

Example 3: Using column names as arguments

>>> import pyspark.sql.functions as sf
>>> df = spark.createDataFrame([('Spark', 2, 3)], ['s', 'p', 'l'])
>>> df.select('*', sf.substring(df.s, 2, 'l')).show()
+-----+---+---+------------------+
|    s|  p|  l|substring(s, 2, l)|
+-----+---+---+------------------+
|Spark|  2|  3|               par|
+-----+---+---+------------------+

>>> df.select('*', sf.substring('s', 'p', 'l')).show()
+-----+---+---+------------------+
|    s|  p|  l|substring(s, p, l)|
+-----+---+---+------------------+
|Spark|  2|  3|               par|
+-----+---+---+------------------+