Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports 246 functions and partially supports 28 functions.
Gluten also fully supports 2 additional functions introduced in Spark 4.0.
Array Functions
Spark Functions
Spark Expressions
Status
Restrictions
array
CreateArray
S
array_append
ArrayAppend
S
array_compact
ArrayCompact
S
array_contains
ArrayContains
S
array_distinct
ArrayDistinct
S
array_except
ArrayExcept
S
array_insert
ArrayInsert
S
array_intersect
ArrayIntersect
S
array_join
ArrayJoin
S
array_max
ArrayMax
S
array_min
ArrayMin
S
array_position
ArrayPosition
S
array_prepend
ArrayPrepend
S
array_remove
ArrayRemove
S
array_repeat
ArrayRepeat
S
array_union
ArrayUnion
S
arrays_overlap
ArraysOverlap
S
arrays_zip
ArraysZip
S
flatten
Flatten
S
get
Get
S
sequence
Sequence
shuffle
Shuffle
S
slice
Slice
S
sort_array
SortArray
S
Bitwise Functions
Spark Functions
Spark Expressions
Status
Restrictions
&
BitwiseAnd
S
^
BitwiseXor
S
bit_count
BitwiseCount
S
bit_get
BitwiseGet
S
getbit
BitwiseGet
S
shiftright
ShiftRight
S
shiftrightunsigned
ShiftRightUnsigned
|
BitwiseOr
S
~
BitwiseNot
S
Collection Functions
Spark Functions
Spark Expressions
Status
Restrictions
array_size
ArraySize
S
cardinality
Size
S
concat
Concat
S
reverse
Reverse
S
size
Size
S
Conditional Functions
Spark Functions
Spark Expressions
Status
Restrictions
coalesce
Coalesce
S
if
If
S
ifnull
Nvl
S
nanvl
NaNvl
S
nullif
NullIf
S
nvl
Nvl
S
nvl2
Nvl2
S
when
CaseWhen
S
Conversion Functions
Spark Functions
Spark Expressions
Status
Restrictions
bigint
S
binary
S
boolean
S
cast
Cast
S
date
S
decimal
S
double
S
float
S
int
S
smallint
S
string
S
timestamp
S
tinyint
S
Csv Functions
Spark Functions
Spark Expressions
Status
Restrictions
from_csv
CsvToStructs
schema_of_csv
SchemaOfCsv
to_csv
StructsToCsv
Date and Timestamp Functions
Spark Functions
Spark Expressions
Status
Restrictions
add_months
AddMonths
S
convert_timezone
ConvertTimezone
curdate
CurDateExpressionBuilder
current_date
CurrentDate
current_timestamp
CurrentTimestamp
current_timezone
CurrentTimeZone
date_add
DateAdd
S
date_diff
DateDiff
S
date_format
DateFormatClass
S
date_from_unix_date
DateFromUnixDate
S
date_part
DatePartExpressionBuilder
date_sub
DateSub
S
date_trunc
TruncTimestamp
S
dateadd
DateAdd
S
datediff
DateDiff
S
datepart
DatePartExpressionBuilder
day
DayOfMonth
S
dayname
DayName
S
Spark 4.0+
dayofmonth
DayOfMonth
S
dayofweek
DayOfWeek
S
dayofyear
DayOfYear
S
extract
Extract
S
from_unixtime
FromUnixTime
S
from_utc_timestamp
FromUTCTimestamp
S
hour
Hour
S
last_day
LastDay
S
localtimestamp
LocalTimestamp
make_date
MakeDate
S
make_dt_interval
MakeDTInterval
make_interval
MakeInterval
make_timestamp
MakeTimestamp
S
make_timestamp_ltz
MakeTimestampLTZExpressionBuilder
make_timestamp_ntz
MakeTimestampNTZExpressionBuilder
make_ym_interval
MakeYMInterval
S
minute
Minute
S
month
Month
S
monthname
MonthName
S
Spark 4.0+
months_between
MonthsBetween
S
next_day
NextDay
S
now
Now
quarter
Quarter
S
second
Second
S
session_window
SessionWindow
timestamp_micros
MicrosToTimestamp
S
timestamp_millis
MillisToTimestamp
S
timestamp_seconds
SecondsToTimestamp
PS
to_date
ParseToDate
to_timestamp
ParseToTimestamp
to_timestamp_ltz
ParseToTimestampLTZExpressionBuilder
to_timestamp_ntz
ParseToTimestampNTZExpressionBuilder
to_unix_timestamp
ToUnixTimestamp
PS
to_utc_timestamp
ToUTCTimestamp
S
trunc
TruncDate
S
try_to_timestamp
TryToTimestampExpressionBuilder
unix_date
UnixDate
S
unix_micros
UnixMicros
S
unix_millis
UnixMillis
S
unix_seconds
UnixSeconds
S
unix_timestamp
UnixTimestamp
S
weekday
WeekDay
S
weekofyear
WeekOfYear
S
window
TimeWindow
window_time
WindowTime
year
Year
S
Hash Functions
Spark Functions
Spark Expressions
Status
Restrictions
crc32
Crc32
S
hash
Murmur3Hash
S
md5
Md5
S
sha
Sha1
S
sha1
Sha1
S
sha2
Sha2
S
xxhash64
XxHash64
S
JSON Functions
Spark Functions
Spark Expressions
Status
Restrictions
from_json
JsonToStructs
PS
from_json with ‘spark.sql.caseSensitive = true’ is not supported in Velox from_json with ‘spark.sql.json.enablePartialResults = false’ is not supported in Velox from_json with column corrupt record is not supported in Velox from_json with duplicate keys is not supported in Velox from_json with options is not supported in Velox
get_json_object
GetJsonObject
S
json_array_length
LengthOfJsonArray
S
json_object_keys
JsonObjectKeys
S
json_tuple
JsonTuple
S
schema_of_json
SchemaOfJson
to_json
StructsToJson
PS
When ‘spark.sql.caseSensitive = false’, to_json produces unexpected result for struct field with uppercase name to_json with options is not supported in Velox
Lambda Functions
Spark Functions
Spark Expressions
Status
Restrictions
aggregate
ArrayAggregate
S
array_sort
ArraySort
S
exists
ArrayExists
S
filter
ArrayFilter
S
forall
ArrayForAll
S
map_filter
MapFilter
S
map_zip_with
MapZipWith
S
reduce
ArrayAggregate
S
transform
ArrayTransform
S
transform_keys
TransformKeys
S
transform_values
TransformValues
S
zip_with
ZipWith
S
Map Functions
Spark Functions
Spark Expressions
Status
Restrictions
element_at
ElementAt
S
map
CreateMap
PS
map_concat
MapConcat
PS
map_contains_key
MapContainsKey
S
map_entries
MapEntries
S
map_from_arrays
MapFromArrays
map_from_entries
MapFromEntries
S
map_keys
MapKeys
S
map_values
MapValues
S
str_to_map
StringToMap
PS
Only spark.sql.mapKeyDedupPolicy = EXCEPTION is supported for Velox backend
try_element_at
TryElementAt
Mathematical Functions
Spark Functions
Spark Expressions
Status
Restrictions
%
Remainder
S
*
Multiply
S
+
Add
S
-
Subtract
S
/
Divide
S
abs
Abs
S
acos
Acos
S
acosh
Acosh
S
asin
Asin
S
asinh
Asinh
S
atan
Atan
S
atan2
Atan2
S
atanh
Atanh
S
bin
Bin
S
bround
BRound
cbrt
Cbrt
S
ceil
CeilExpressionBuilder
PS
ceiling
CeilExpressionBuilder
PS
conv
Conv
S
cos
Cos
S
cosh
Cosh
S
cot
Cot
S
csc
Csc
S
degrees
ToDegrees
S
div
IntegralDivide
S
e
EulerNumber
S
exp
Exp
S
expm1
Expm1
S
factorial
Factorial
S
floor
FloorExpressionBuilder
PS
greatest
Greatest
S
hex
Hex
S
hypot
Hypot
S
least
Least
S
ln
Log
log
Logarithm
S
log10
Log10
S
log1p
Log1p
S
log2
Log2
S
mod
Remainder
S
negative
UnaryMinus
S
pi
Pi
S
pmod
Pmod
S
positive
UnaryPositive
S
pow
Pow
S
power
Pow
S
radians
ToRadians
rand
Rand
S
randn
Randn
S
random
Rand
S
rint
Rint
S
round
Round
S
sec
Sec
S
shiftleft
ShiftLeft
S
sign
Signum
S
signum
Signum
S
sin
Sin
sinh
Sinh
S
sqrt
Sqrt
S
tan
Tan
S
tanh
Tanh
S
try_add
TryAdd
PS
try_divide
TryDivide
try_multiply
TryMultiply
try_subtract
TrySubtract
unhex
Unhex
S
width_bucket
WidthBucket
S
Misc Functions
Spark Functions
Spark Expressions
Status
Restrictions
aes_decrypt
AesDecrypt
aes_encrypt
AesEncrypt
assert_true
AssertTrue
S
bitmap_bit_position
BitmapBitPosition
bitmap_bucket_number
BitmapBucketNumber
bitmap_count
BitmapCount
current_catalog
CurrentCatalog
current_database
CurrentDatabase
current_schema
CurrentDatabase
current_user
CurrentUser
equal_null
EqualNull
S
hll_sketch_estimate
HllSketchEstimate
hll_union
HllUnion
input_file_block_length
InputFileBlockLength
input_file_block_start
InputFileBlockStart
input_file_name
InputFileName
java_method
CallMethodViaReflection
monotonically_increasing_id
MonotonicallyIncreasingID
reflect
CallMethodViaReflection
spark_partition_id
SparkPartitionID
S
try_aes_decrypt
TryAesDecrypt
typeof
TypeOf
user
CurrentUser
uuid
Uuid
S
version
SparkVersion
S
||
S
Predicate Functions
Spark Functions
Spark Expressions
Status
Restrictions
!
Not
S
!=
S
<
LessThan
S
<=
LessThanOrEqual
S
<=>
EqualNullSafe
S
<>
S
=
EqualTo
S
==
EqualTo
S
>
GreaterThan
S
>=
GreaterThanOrEqual
S
and
And
S
between
S
case
S
ilike
ILike
S
in
In
PS
isnan
IsNaN
S
isnotnull
IsNotNull
S
isnull
IsNull
S
like
Like
S
not
Not
S
or
Or
S
regexp
RLike
PS
Lookaround unsupported
regexp_like
RLike
PS
Lookaround unsupported
rlike
RLike
PS
Lookaround unsupported
String Functions
Spark Functions
Spark Expressions
Status
Restrictions
ascii
Ascii
S
base64
Base64
PS
base64 with chunkBase64String disabled is not supported
bit_length
BitLength
S
btrim
StringTrimBoth
S
char
Chr
S
char_length
Length
S
character_length
Length
S
chr
Chr
S
concat_ws
ConcatWs
S
contains
ContainsExpressionBuilder
PS
BinaryType unsupported
decode
Decode
elt
Elt
encode
Encode
endswith
EndsWithExpressionBuilder
PS
BinaryType unsupported
find_in_set
FindInSet
S
format_number
FormatNumber
PS
format_number only supports tinyint, smallint, integer, bigint, float and double input; DecimalType input is not supported in Velox format_number with a string format argument (e.g. ‘#,###.##’) is not supported in Velox; only an integer number of decimal places is supported