Package org.apache.spark.sql
Class functions
Object
org.apache.spark.sql.functions
Commonly used functions available for DataFrame operations. Using functions defined here
provides a little bit more compile-time safety to make sure the function exists.
You can call the functions defined here by two ways: _FUNC_(...) and
functions.expr("_FUNC_(...)").
As an example, regr_count is a function that is defined here. You can use
regr_count(col("yCol", col("xCol"))) to invoke the regr_count function. This way the
programming language's compiler ensures regr_count exists and is of the proper form. You can
also use expr("regr_count(yCol, xCol)") function to invoke the same function. In this case,
Spark itself will ensure regr_count exists when it analyzes the query.
You can find the entire list of functions at SQL API documentation of your Spark version, see also the latest list
This function APIs usually have methods with Column signature only because it can support not
only Column but also other types such as a native string. The other variants currently exist
for historical reasons.
- Since:
- 1.3.0
-
Nested Class Summary
Nested Classes -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionstatic ColumnComputes the absolute value of a numeric value.static Columnstatic Columnstatic Columnstatic Columnstatic Columnadd_months(Column startDate, int numMonths) Returns the date that isnumMonthsafterstartDate.static Columnadd_months(Column startDate, Column numMonths) Returns the date that isnumMonthsafterstartDate.static Columnaes_decrypt(Column input, Column key) Returns a decrypted value ofinput.static Columnaes_decrypt(Column input, Column key, Column mode) Returns a decrypted value ofinput.static Columnaes_decrypt(Column input, Column key, Column mode, Column padding) Returns a decrypted value ofinput.static ColumnReturns a decrypted value ofinputusing AES inmodewithpadding.static Columnaes_encrypt(Column input, Column key) Returns an encrypted value ofinput.static Columnaes_encrypt(Column input, Column key, Column mode) Returns an encrypted value ofinput.static Columnaes_encrypt(Column input, Column key, Column mode, Column padding) Returns an encrypted value ofinput.static ColumnReturns an encrypted value ofinput.static ColumnReturns an encrypted value ofinputusing AES in givenmodewith the specifiedpadding.static ColumnApplies a binary operator to an initial state and all elements in the array, and reduces this to a single state.static Columnaggregate(Column expr, Column initialValue, scala.Function2<Column, Column, Column> merge, scala.Function1<Column, Column> finish) Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.static ColumnAggregate function: returns true if at least one value ofeis true.static ColumnAggregate function: returns some value ofefor a group of rows.static ColumnAggregate function: returns some value ofefor a group of rows.static Columnapprox_count_distinct(String columnName) Aggregate function: returns the approximate number of distinct items in a group.static Columnapprox_count_distinct(String columnName, double rsd) Aggregate function: returns the approximate number of distinct items in a group.static ColumnAggregate function: returns the approximate number of distinct items in a group.static Columnapprox_count_distinct(Column e, double rsd) Aggregate function: returns the approximate number of distinct items in a group.static Columnapprox_percentile(Column e, Column percentage, Column accuracy) Aggregate function: returns the approximatepercentileof the numeric columncolwhich is the smallest value in the orderedcolvalues (sorted from least to greatest) such that no more thanpercentageofcolvalues is less than the value or equal to that value.static ColumnapproxCountDistinct(String columnName) Deprecated.Use approx_count_distinct.static ColumnapproxCountDistinct(String columnName, double rsd) Deprecated.Use approx_count_distinct.static ColumnDeprecated.Use approx_count_distinct.static ColumnapproxCountDistinct(Column e, double rsd) Deprecated.Use approx_count_distinct.static ColumnCreates a new array column.static ColumnCreates a new array column.static ColumnCreates a new array column.static ColumnCreates a new array column.static ColumnAggregate function: returns a list of objects with duplicates.static Columnarray_append(Column column, Object element) Returns an ARRAY containing all elements from the source ARRAY as well as the new element.static Columnarray_compact(Column column) Remove all null elements from the given array.static Columnarray_contains(Column column, Object value) Returns true if the array containsvalue, false if not.static ColumnRemoves duplicate values from the array.static Columnarray_except(Column col1, Column col2) Returns an array of the elements in the first array but not in the second array, without duplicates.static Columnarray_insert(Column arr, Column pos, Column value) Adds an item into a given array at a specified positionstatic Columnarray_intersect(Column col1, Column col2) Returns an array of the elements in the intersection of the given two arrays, without duplicates.static Columnarray_join(Column column, String delimiter) Concatenates the elements ofcolumnusing thedelimiter.static Columnarray_join(Column column, String delimiter, String nullReplacement) Concatenates the elements ofcolumnusing thedelimiter.static ColumnReturns the maximum value in the array.static ColumnReturns the minimum value in the array.static Columnarray_position(Column column, Object value) Locates the position of the first occurrence of the value in the given array as long.static Columnarray_prepend(Column column, Object element) Returns an array containing value as well as all elements from array.static Columnarray_remove(Column column, Object element) Remove all elements that equal to element from the given array.static Columnarray_repeat(Column e, int count) Creates an array containing the left argument repeated the number of times given by the right argument.static Columnarray_repeat(Column left, Column right) Creates an array containing the left argument repeated the number of times given by the right argument.static Columnarray_size(Column e) Returns the total number of elements in the array.static Columnarray_sort(Column e) Sorts the input array in ascending order.static Columnarray_sort(Column e, scala.Function2<Column, Column, Column> comparator) Sorts the input array based on the given comparator function.static Columnarray_union(Column col1, Column col2) Returns an array of the elements in the union of the given two arrays, without duplicates.static Columnarrays_overlap(Column a1, Column a2) Returnstrueifa1anda2have at least one non-null element in common.static Columnarrays_zip(Column... e) Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.static Columnarrays_zip(scala.collection.immutable.Seq<Column> e) Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.static ColumnReturns a sort expression based on ascending order of the column.static Columnasc_nulls_first(String columnName) Returns a sort expression based on ascending order of the column, and null values return before non-null values.static Columnasc_nulls_last(String columnName) Returns a sort expression based on ascending order of the column, and null values appear after non-null values.static ColumnComputes the numeric value of the first character of the string column, and returns the result as an int column.static Columnstatic Columnstatic Columnstatic Columnstatic ColumnReturns null if the condition is true, and throws an exception otherwise.static Columnassert_true(Column c, Column e) Returns null if the condition is true; throws an exception with the error message otherwise.static Columnstatic Columnstatic Columnstatic Columnstatic Columnstatic Columnstatic Columnstatic Columnstatic Columnstatic Columnstatic Columnstatic Columnstatic ColumnAggregate function: returns the average of the values in a group.static ColumnAggregate function: returns the average of the values in a group.static ColumnComputes the BASE64 encoding of a binary column and returns it as a string column.static ColumnAn expression that returns the string representation of the binary value of the given long column.static ColumnAn expression that returns the string representation of the binary value of the given long column.static ColumnAggregate function: returns the bitwise AND of all non-null input values, or null if none.static ColumnReturns the number of bits that are set in the argument expr as an unsigned 64-bit integer, or NULL if the argument is NULL.static ColumnReturns the value of the bit (0 or 1) at the specified position.static Columnbit_length(Column e) Calculates the bit length for the specified string column.static ColumnAggregate function: returns the bitwise OR of all non-null input values, or null if none.static ColumnAggregate function: returns the bitwise XOR of all non-null input values, or null if none.static Columnbitmap_and_agg(Column col) Returns a bitmap that is the bitwise AND of all of the bitmaps from the input column.static ColumnReturns the bit position for the given input column.static ColumnReturns the bucket number for the given input column.static ColumnReturns a bitmap with the positions of the bits set from all the values from the input column.static Columnbitmap_count(Column col) Returns the number of set bits in the input bitmap.static Columnbitmap_or_agg(Column col) Returns a bitmap that is the bitwise OR of all of the bitmaps from the input column.static ColumnComputes bitwise NOT (~) of a number.static ColumnbitwiseNOT(Column e) Deprecated.Use bitwise_not.static ColumnAggregate function: returns true if all values ofeare true.static ColumnAggregate function: returns true if at least one value ofeis true.static <U> DatasetMarks a DataFrame as small enough for use in broadcast joins.static ColumnReturns the value of the columnerounded to 0 decimal places with HALF_EVEN round mode.static ColumnRound the value ofetoscaledecimal places with HALF_EVEN round mode ifscaleis greater than or equal to 0 or at integral part whenscaleis less than 0.static ColumnRound the value ofetoscaledecimal places with HALF_EVEN round mode ifscaleis greater than or equal to 0 or at integral part whenscaleis less than 0.static ColumnRemoves the leading and trailing space characters fromstr.static ColumnRemove the leading and trailingtrimcharacters fromstr.static Column(Java-specific) A transform for any type that partitions by a hash of the input column.static Column(Java-specific) A transform for any type that partitions by a hash of the input column.static Columncall_function(String funcName, Column... cols) Call a SQL function.static Columncall_function(String funcName, scala.collection.immutable.Seq<Column> cols) Call a SQL function.static ColumnCall an user-defined function.static ColumnCall an user-defined function.static ColumnCall an user-defined function.static ColumnDeprecated.Use call_udf.static ColumnReturns length of array or map.static ColumnComputes the cube-root of the given column.static ColumnComputes the cube-root of the given value.static ColumnComputes the ceiling of the given value ofcolumnNameto 0 decimal places.static ColumnComputes the ceiling of the given value ofeto 0 decimal places.static ColumnComputes the ceiling of the given value ofetoscaledecimal places.static ColumnComputes the ceiling of the given value ofeto 0 decimal places.static ColumnComputes the ceiling of the given value ofetoscaledecimal places.static Columnchar_length(Column str) Returns the character length of string data or number of bytes of binary data.static Columncharacter_length(Column str) Returns the character length of string data or number of bytes of binary data.static ColumnReturns the ASCII character having the binary equivalent ton.static ColumnReturns the first column that is not null, or null if all inputs are null.static ColumnReturns the first column that is not null, or null if all inputs are null.static ColumnReturns aColumnbased on the given column name.static ColumnMarks a given column with specified collation.static ColumnReturns the collation name of a given column.static Columncollect_list(String columnName) Aggregate function: returns a list of objects with duplicates.static ColumnAggregate function: returns a list of objects with duplicates.static Columncollect_set(String columnName) Aggregate function: returns a set of objects with duplicate elements eliminated.static ColumnAggregate function: returns a set of objects with duplicate elements eliminated.static Columncollect_union(String columnName) Aggregate function: returns the distinct union of the elements of an array-typed column across rows.static ColumnAggregate function: returns the distinct union of the elements of an array-typed column across rows.static ColumnReturns aColumnbased on the given column name.static ColumnConcatenates multiple input columns together into a single column.static ColumnConcatenates multiple input columns together into a single column.static ColumnConcatenates multiple input string columns together into a single string column, using the given separator.static ColumnConcatenates multiple input string columns together into a single string column, using the given separator.static ColumnReturns a boolean.static ColumnConvert a number in a string column from one base to another.static Columnconvert_timezone(Column targetTz, Column sourceTs) Converts the timestamp without time zonesourceTsfrom the current time zone totargetTz.static Columnconvert_timezone(Column sourceTz, Column targetTz, Column sourceTs) Converts the timestamp without time zonesourceTsfrom thesourceTztime zone totargetTz.static ColumnAggregate function: returns the Pearson Correlation Coefficient for two columns.static ColumnAggregate function: returns the Pearson Correlation Coefficient for two columns.static Columnstatic Columnstatic Columnstatic Columnstatic Columnstatic TypedColumn<Object,Object> Aggregate function: returns the number of items in a group.static ColumnAggregate function: returns the number of items in a group.static Columncount_distinct(Column expr, Column... exprs) Aggregate function: returns the number of distinct items in a group.static Columncount_distinct(Column expr, scala.collection.immutable.Seq<Column> exprs) Aggregate function: returns the number of distinct items in a group.static ColumnAggregate function: returns the number ofTRUEvalues for the expression.static Columncount_min_sketch(Column e, Column eps, Column confidence) Returns a count-min sketch of a column with the given esp, confidence and seed.static Columncount_min_sketch(Column e, Column eps, Column confidence, Column seed) Returns a count-min sketch of a column with the given esp, confidence and seed.static ColumncountDistinct(String columnName, String... columnNames) Aggregate function: returns the number of distinct items in a group.static ColumncountDistinct(String columnName, scala.collection.immutable.Seq<String> columnNames) Aggregate function: returns the number of distinct items in a group.static ColumncountDistinct(Column expr, Column... exprs) Aggregate function: returns the number of distinct items in a group.static ColumncountDistinct(Column expr, scala.collection.immutable.Seq<Column> exprs) Aggregate function: returns the number of distinct items in a group.static Columncounter_diff(Column value) Window function: computes the differences between consecutive cumulative counter values in a time series, thereby converting the counter from the cumulative to the delta format.static Columncounter_diff(Column value, Column startTime) Window function: computes the differences between consecutive cumulative counter values in a time series, thereby converting the counter from the cumulative to the delta format.static ColumnAggregate function: returns the population covariance for two columns.static ColumnAggregate function: returns the population covariance for two columns.static Columncovar_samp(String columnName1, String columnName2) Aggregate function: returns the sample covariance for two columns.static Columncovar_samp(Column column1, Column column2) Aggregate function: returns the sample covariance for two columns.static ColumnCalculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint.static Columnstatic ColumnWindow function: returns the cumulative distribution of values within a window partition, i.e.static Columncurdate()Returns the current date at the start of query evaluation as a date column.static ColumnReturns the current catalog.static ColumnReturns the current database.static ColumnReturns the current date at the start of query evaluation as a date column.static ColumnReturns the current SQL path as a comma-separated list of qualified schema names.static ColumnReturns the current schema.static ColumnReturns the current time at the start of query evaluation.static Columncurrent_time(int precision) Returns the current time at the start of query evaluation.static ColumnReturns the current timestamp at the start of query evaluation as a timestamp column.static ColumnReturns the current session local timezone.static ColumnReturns the user name of current execution context.static ColumnReturns the date that isdaysdays afterstartstatic ColumnReturns the date that isdaysdays afterstartstatic ColumnReturns the number of days fromstarttoend.static Columndate_format(Column dateExpr, String format) Converts a date/timestamp/string to a value of string in the format specified by the date format given by the second argument.static Columndate_from_unix_date(Column days) Create date from the number ofdayssince 1970-01-01.static ColumnExtracts a part of the date/timestamp or interval source.static ColumnReturns the date that isdaysdays beforestartstatic ColumnReturns the date that isdaysdays beforestartstatic Columndate_trunc(String format, Column timestamp) Returns timestamp truncated to the unit specified by the format.static ColumnReturns the date that isdaysdays afterstartstatic ColumnReturns the number of days fromstarttoend.static ColumnExtracts a part of the date/timestamp or interval source.static ColumnExtracts the day of the month as an integer from a given date/timestamp/string.static ColumnExtracts the three-letter abbreviated day name from a given date/timestamp/string.static Columndayofmonth(Column e) Extracts the day of the month as an integer from a given date/timestamp/string.static ColumnExtracts the day of the week as an integer from a given date/timestamp/string.static ColumnExtracts the day of the year as an integer from a given date/timestamp/string.static Column(Java-specific) A transform for timestamps and dates to partition data into days.static ColumnComputes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32').static ColumnConverts an angle measured in radians to an approximately equivalent angle measured in degrees.static ColumnConverts an angle measured in radians to an approximately equivalent angle measured in degrees.static ColumnWindow function: returns the rank of rows within a window partition, without any gaps.static ColumnReturns a sort expression based on the descending order of the column.static Columndesc_nulls_first(String columnName) Returns a sort expression based on the descending order of the column, and null values appear before non-null values.static Columndesc_nulls_last(String columnName) Returns a sort expression based on the descending order of the column, and null values appear after non-null values.static Columne()Returns Euler's number.static Columnelement_at(Column column, Object value) Returns element of array at given index in value if column is array.static ColumnReturns then-th input, e.g., returnsinput2whennis 2.static ColumnReturns then-th input, e.g., returnsinput2whennis 2.static ColumnComputes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32').static ColumnReturns a boolean.static Columnequal_null(Column col1, Column col2) Returns same result as the EQUAL(=) operator for non-null operands, but returns true if both are null, false if one of the them is null.static ColumnAggregate function: returns true if all values ofeare true.static ColumnReturns whether a predicate holds for one or more elements in the array.static ColumnComputes the exponential of the given column.static ColumnComputes the exponential of the given value.static ColumnCreates a new row for each element in the given array or map column.static ColumnCreates a new row for each element in the given array or map column.static ColumnComputes the exponential of the given column minus one.static ColumnComputes the exponential of the given value minus one.static ColumnParses the expression string into the column that it represents, similar toDataset.selectExpr(java.lang.String...).static ColumnExtracts a part of the date/timestamp or interval source.static ColumnComputes the factorial of the given value.static ColumnReturns an array of elements for which a predicate holds in a given array.static ColumnReturns an array of elements for which a predicate holds in a given array.static Columnfind_in_set(Column str, Column strArray) Returns the index (1-based) of the given string (str) in the comma-delimited list (strArray).static ColumnAggregate function: returns the first value of a column in a group.static ColumnAggregate function: returns the first value of a column in a group.static ColumnAggregate function: returns the first value in a group.static ColumnAggregate function: returns the first value in a group.static ColumnAggregate function: returns the first value in a group.static Columnfirst_value(Column e, Column ignoreNulls) Aggregate function: returns the first value in a group.static ColumnCreates a single array from an array of arrays.static ColumnComputes the floor of the given column value to 0 decimal places.static ColumnComputes the floor of the given value ofeto 0 decimal places.static ColumnComputes the floor of the given value ofetoscaledecimal places.static ColumnReturns whether a predicate holds for every element in the array.static Columnformat_number(Column x, int d) Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with HALF_EVEN round mode, and returns the result as a string column.static Columnformat_string(String format, Column... arguments) Formats the arguments in printf-style and returns the result as a string column.static Columnformat_string(String format, scala.collection.immutable.Seq<Column> arguments) Formats the arguments in printf-style and returns the result as a string column.static Column(Java-specific) Parses a column containing a CSV string into aStructTypewith the specified schema.static Columnfrom_csv(Column e, StructType schema, scala.collection.immutable.Map<String, String> options) Parses a column containing a CSV string into aStructTypewith the specified schema.static Column(Java-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema.static Column(Scala-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema.static Column(Scala-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypeofStructTypes with the specified schema.static Column(Java-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypeofStructTypes with the specified schema.static ColumnParses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema.static Column(Java-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema.static Column(Scala-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema.static Columnfrom_json(Column e, StructType schema) Parses a column containing a JSON string into aStructTypewith the specified schema.static Column(Java-specific) Parses a column containing a JSON string into aStructTypewith the specified schema.static Columnfrom_json(Column e, StructType schema, scala.collection.immutable.Map<String, String> options) (Scala-specific) Parses a column containing a JSON string into aStructTypewith the specified schema.static Columnfrom_unixtime(Column ut) Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the yyyy-MM-dd HH:mm:ss format.static Columnfrom_unixtime(Column ut, String f) Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.static Columnfrom_utc_timestamp(Column ts, String tz) Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone.static Columnfrom_utc_timestamp(Column ts, Column tz) Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone.static Column(Java-specific) Parses a column containing a XML string into aStructTypewith the specified schema.static Column(Java-specific) Parses a column containing a XML string into aStructTypewith the specified schema.static Column(Java-specific) Parses a column containing a XML string into aStructTypewith the specified schema.static Columnfrom_xml(Column e, StructType schema) Parses a column containing a XML string into the data type corresponding to the specified schema.static ColumnParses a column containing a XML string into the data type corresponding to the specified schema.static ColumnReturns element of array at given (0-based) index.static Columnget_json_object(Column e, String path) Extracts json object from a json string based on json path specified, and returns json string of the extracted json object.static ColumnReturns the value of the bit (0 or 1) at the specified position.static ColumnReturns the greatest value of the list of column names, skipping null values.static ColumnReturns the greatest value of the list of column names, skipping null values.static ColumnReturns the greatest value of the list of values, skipping null values.static ColumnReturns the greatest value of the list of values, skipping null values.static ColumnAggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set.static ColumnAggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set.static Columngrouping_id(String colName, String... colNames) Aggregate function: returns the level of grouping, equals tostatic Columngrouping_id(String colName, scala.collection.immutable.Seq<String> colNames) Aggregate function: returns the level of grouping, equals tostatic Columngrouping_id(Column... cols) Aggregate function: returns the level of grouping, equals tostatic Columngrouping_id(scala.collection.immutable.Seq<Column> cols) Aggregate function: returns the level of grouping, equals tostatic ColumnCalculates the hash code of given columns, and returns the result as an int column.static ColumnCalculates the hash code of given columns, and returns the result as an int column.static ColumnComputes hex value of the given column.static Columnhistogram_numeric(Column e, Column nBins) Aggregate function: computes a histogram on numeric 'expr' using nb bins.static Columnhll_sketch_agg(String columnName) Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.static Columnhll_sketch_agg(String columnName, int lgConfigK) Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.static ColumnAggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.static Columnhll_sketch_agg(Column e, int lgConfigK) Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.static Columnhll_sketch_agg(Column e, Column lgConfigK) Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.static Columnhll_sketch_estimate(String columnName) Returns the estimated number of unique values given the binary representation of a Datasketches HllSketch.static ColumnReturns the estimated number of unique values given the binary representation of a Datasketches HllSketch.static ColumnMerges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.static ColumnMerges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.static ColumnMerges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.static ColumnMerges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.static Columnhll_union_agg(String columnName) Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.static Columnhll_union_agg(String columnName, boolean allowDifferentLgConfigK) Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.static ColumnAggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.static Columnhll_union_agg(Column e, boolean allowDifferentLgConfigK) Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.static Columnhll_union_agg(Column e, Column allowDifferentLgConfigK) Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.static ColumnReturns the keyed-hash message authentication code (HMAC) ofmessageusingkeyand SHA-256.static ColumnReturns the keyed-hash message authentication code (HMAC) ofmessageusingkeyand the given hashalgorithm.static ColumnExtracts the hours as an integer from a given date/time/timestamp/string.static Column(Java-specific) A transform for timestamps to partition data into hours.static ColumnComputessqrt(a^2^ + b^2^)without intermediate overflow or underflow.static ColumnComputessqrt(a^2^ + b^2^)without intermediate overflow or underflow.static ColumnComputessqrt(a^2^ + b^2^)without intermediate overflow or underflow.static ColumnComputessqrt(a^2^ + b^2^)without intermediate overflow or underflow.static ColumnComputessqrt(a^2^ + b^2^)without intermediate overflow or underflow.static ColumnComputessqrt(a^2^ + b^2^)without intermediate overflow or underflow.static ColumnComputessqrt(a^2^ + b^2^)without intermediate overflow or underflow.static ColumnComputessqrt(a^2^ + b^2^)without intermediate overflow or underflow.static ColumnReturnscol2ifcol1is null, orcol1otherwise.static ColumnReturns true if str matchespatternwithescapeChar('\') case-insensitively, null if any arguments are null, false otherwise.static ColumnReturns true if str matchespatternwithescapeCharcase-insensitively, null if any arguments are null, false otherwise.static ColumnReturns a new string column by converting the first letter of each word to uppercase.static ColumnCreates a new row for each element in the given array of structs.static ColumnCreates a new row for each element in the given array of structs.static ColumnReturns the length of the block being read, or -1 if not available.static ColumnReturns the start offset of the block being read, or -1 if not available.static ColumnCreates a string column for the file name of the current Spark task.static ColumnLocate the position of the first occurrence of substr column in the given string.static ColumnLocate the position of the first occurrence of substr column in the given string.static ColumnLocate the position of the first occurrence ofsubstringinstr, starting the search from positionstart.static ColumnLocate the position of theoccurrence-th occurrence ofsubstringinstr, starting the search from positionstart.static ColumnLocate the position of the first occurrence ofsubstringinstr, starting the search from positionstart.static ColumnLocate the position of theoccurrence-th occurrence ofsubstringinstr, starting the search from positionstart.static Columnis_valid_utf8(Column str) Returns true if the input is a valid UTF-8 string, otherwise returns false.static ColumnCheck if a variant value is valid.static ColumnCheck if a variant value is a variant null.static ColumnReturn true iff the column is NaN.static ColumnReturns true ifcolis not null, or false otherwise.static ColumnReturn true iff the column is null.static ColumnComputes the Jaro-Winkler similarity between the two given string columns.static Columnjava_method(Column... cols) Calls a method with reflection.static Columnjava_method(scala.collection.immutable.Seq<Column> cols) Calls a method with reflection.static ColumnReturns the number of elements in the outermost JSON array.static ColumnReturns all the keys of the outermost JSON object as an array.static Columnjson_tuple(Column json, String... fields) Creates a new row for a json column according to the given field names.static Columnjson_tuple(Column json, scala.collection.immutable.Seq<String> fields) Creates a new row for a json column according to the given field names.static Columnkll_merge_agg_bigint(String columnName) Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.static Columnkll_merge_agg_bigint(String columnName, int k) Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.static ColumnAggregate function: merges binary KllLongsSketch representations and returns the merged sketch.static Columnkll_merge_agg_bigint(Column e, int k) Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.static Columnkll_merge_agg_bigint(Column e, Column k) Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.static Columnkll_merge_agg_double(String columnName) Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.static Columnkll_merge_agg_double(String columnName, int k) Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.static ColumnAggregate function: merges binary KllDoublesSketch representations and returns merged sketch.static Columnkll_merge_agg_double(Column e, int k) Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.static Columnkll_merge_agg_double(Column e, Column k) Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.static Columnkll_merge_agg_float(String columnName) Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.static Columnkll_merge_agg_float(String columnName, int k) Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.static ColumnAggregate function: merges binary KllFloatsSketch representations and returns merged sketch.static Columnkll_merge_agg_float(Column e, int k) Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.static Columnkll_merge_agg_float(Column e, Column k) Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.static Columnkll_sketch_agg_bigint(String columnName) Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column with default k value of 200.static Columnkll_sketch_agg_bigint(String columnName, int k) Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column.static ColumnAggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column with default k value of 200.static Columnkll_sketch_agg_bigint(Column e, int k) Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column.static ColumnAggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column.static Columnkll_sketch_agg_double(String columnName) Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column with default k value of 200.static Columnkll_sketch_agg_double(String columnName, int k) Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column.static ColumnAggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column with default k value of 200.static Columnkll_sketch_agg_double(Column e, int k) Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column.static ColumnAggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column.static Columnkll_sketch_agg_float(String columnName) Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column with default k value of 200.static Columnkll_sketch_agg_float(String columnName, int k) Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column.static ColumnAggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column with default k value of 200.static Columnkll_sketch_agg_float(Column e, int k) Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column.static Columnkll_sketch_agg_float(Column e, Column k) Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column.static ColumnReturns the number of items collected in the KLL bigint sketch.static ColumnReturns the number of items collected in the KLL double sketch.static ColumnReturns the number of items collected in the KLL float sketch.static Columnkll_sketch_get_quantile_bigint(Column sketch, Column rank) Extracts a quantile value from a KLL bigint sketch given an input rank value.static Columnkll_sketch_get_quantile_double(Column sketch, Column rank) Extracts a quantile value from a KLL double sketch given an input rank value.static Columnkll_sketch_get_quantile_float(Column sketch, Column rank) Extracts a quantile value from a KLL float sketch given an input rank value.static Columnkll_sketch_get_rank_bigint(Column sketch, Column quantile) Extracts a rank value from a KLL bigint sketch given an input quantile value.static Columnkll_sketch_get_rank_double(Column sketch, Column quantile) Extracts a rank value from a KLL double sketch given an input quantile value.static Columnkll_sketch_get_rank_float(Column sketch, Column quantile) Extracts a rank value from a KLL float sketch given an input quantile value.static Columnkll_sketch_merge_bigint(Column left, Column right) Merges two KLL bigint sketch buffers together into one.static Columnkll_sketch_merge_double(Column left, Column right) Merges two KLL double sketch buffers together into one.static Columnkll_sketch_merge_float(Column left, Column right) Merges two KLL float sketch buffers together into one.static ColumnReturns a string with human readable summary information about the KLL bigint sketch.static ColumnReturns a string with human readable summary information about the KLL double sketch.static ColumnReturns a string with human readable summary information about the KLL float sketch.static ColumnAggregate function: returns the kurtosis of the values in a group.static ColumnAggregate function: returns the kurtosis of the values in a group.static ColumnWindow function: returns the value that isoffsetrows before the current row, andnullif there is less thanoffsetrows before the current row.static ColumnWindow function: returns the value that isoffsetrows before the current row, anddefaultValueif there is less thanoffsetrows before the current row.static ColumnWindow function: returns the value that isoffsetrows before the current row, andnullif there is less thanoffsetrows before the current row.static ColumnWindow function: returns the value that isoffsetrows before the current row, anddefaultValueif there is less thanoffsetrows before the current row.static ColumnWindow function: returns the value that isoffsetrows before the current row, anddefaultValueif there is less thanoffsetrows before the current row.static ColumnAggregate function: returns the last value of the column in a group.static ColumnAggregate function: returns the last value of the column in a group.static ColumnAggregate function: returns the last value in a group.static ColumnAggregate function: returns the last value in a group.static ColumnReturns the last day of the month which the given date belongs to.static Columnlast_value(Column e) Aggregate function: returns the last value in a group.static Columnlast_value(Column e, Column ignoreNulls) Aggregate function: returns the last value in a group.static ColumnReturnsstrwith all characters changed to lowercase.static ColumnWindow function: returns the value that isoffsetrows after the current row, andnullif there is less thanoffsetrows after the current row.static ColumnWindow function: returns the value that isoffsetrows after the current row, anddefaultValueif there is less thanoffsetrows after the current row.static ColumnWindow function: returns the value that isoffsetrows after the current row, andnullif there is less thanoffsetrows after the current row.static ColumnWindow function: returns the value that isoffsetrows after the current row, anddefaultValueif there is less thanoffsetrows after the current row.static ColumnWindow function: returns the value that isoffsetrows after the current row, anddefaultValueif there is less thanoffsetrows after the current row.static ColumnReturns the least value of the list of column names, skipping null values.static ColumnReturns the least value of the list of column names, skipping null values.static ColumnReturns the least value of the list of values, skipping null values.static ColumnReturns the least value of the list of values, skipping null values.static ColumnReturns the leftmostlen(lencan be string type) characters from the stringstr, iflenis less or equal than 0 the result is an empty string.static ColumnComputes the character length of a given string or number of bytes of a binary string.static ColumnComputes the character length of a given string or number of bytes of a binary string.static Columnlevenshtein(Column l, Column r) Computes the Levenshtein distance of the two given string columns.static Columnlevenshtein(Column l, Column r, int threshold) Computes the Levenshtein distance of the two given string columns if it's less than or equal to a given threshold.static ColumnReturns true if str matchespatternwithescapeChar('\'), null if any arguments are null, false otherwise.static ColumnReturns true if str matchespatternwithescapeChar, null if any arguments are null, false otherwise.static ColumnAggregate function: returns the concatenation of non-null input values.static ColumnAggregate function: returns the concatenation of non-null input values, separated by the delimiter.static ColumnAggregate function: returns the concatenation of distinct non-null input values.static Columnlistagg_distinct(Column e, Column delimiter) Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter.static ColumnCreates aColumnof literal value.static ColumnComputes the natural logarithm of the given value.static ColumnReturns the current timestamp without time zone at the start of query evaluation as a timestamp without time zone column.static ColumnLocate the position of the first occurrence of substr.static ColumnLocate the position of the first occurrence of substr in a string column, after position pos.static ColumnReturns the first argument-base logarithm of the second argument.static ColumnReturns the first argument-base logarithm of the second argument.static ColumnComputes the natural logarithm of the given column.static ColumnComputes the natural logarithm of the given value.static ColumnComputes the logarithm of the given value in base 10.static ColumnComputes the logarithm of the given value in base 10.static ColumnComputes the natural logarithm of the given column plus one.static ColumnComputes the natural logarithm of the given value plus one.static ColumnComputes the logarithm of the given value in base 2.static ColumnComputes the logarithm of the given column in base 2.static ColumnConverts a string column to lower case.static ColumnLeft-pad the binary column with pad to a byte length of len.static ColumnLeft-pad the string column with pad to a length of len.static ColumnLeft-pad the string column with pad to a length of len.static ColumnTrim the spaces from left end for the specified string value.static ColumnTrim the specified character string from left end for the specified string column.static ColumnTrim the specified character string from left end for the specified string column.static Columnstatic ColumnMake DayTimeIntervalType duration.static Columnmake_dt_interval(Column days) Make DayTimeIntervalType duration from days.static Columnmake_dt_interval(Column days, Column hours) Make DayTimeIntervalType duration from days and hours.static Columnmake_dt_interval(Column days, Column hours, Column mins) Make DayTimeIntervalType duration from days, hours and mins.static Columnmake_dt_interval(Column days, Column hours, Column mins, Column secs) Make DayTimeIntervalType duration from days, hours, mins and secs.static ColumnMake interval.static Columnmake_interval(Column years) Make interval from years.static Columnmake_interval(Column years, Column months) Make interval from years and months.static Columnmake_interval(Column years, Column months, Column weeks) Make interval from years, months and weeks.static Columnmake_interval(Column years, Column months, Column weeks, Column days) Make interval from years, months, weeks and days.static ColumnMake interval from years, months, weeks, days and hours.static ColumnMake interval from years, months, weeks, days, hours and mins.static Columnmake_interval(Column years, Column months, Column weeks, Column days, Column hours, Column mins, Column secs) Make interval from years, months, weeks, days, hours, mins and secs.static ColumnCreate time from hour, minute and second fields.static Columnmake_timestamp(Column date, Column time) Create a local date-time from date and time fields.static Columnmake_timestamp(Column date, Column time, Column timezone) Create a local date-time from date, time, and timezone fields.static ColumnCreate timestamp from years, months, days, hours, mins and secs fields.static Columnmake_timestamp(Column years, Column months, Column days, Column hours, Column mins, Column secs, Column timezone) Create timestamp from years, months, days, hours, mins, secs and timezone fields.static Columnmake_timestamp_ltz(Column years, Column months, Column days, Column hours, Column mins, Column secs) Create the current timestamp with local time zone from years, months, days, hours, mins and secs fields.static Columnmake_timestamp_ltz(Column years, Column months, Column days, Column hours, Column mins, Column secs, Column timezone) Create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields.static Columnmake_timestamp_ntz(Column date, Column time) Create a local date-time from date and time fields.static Columnmake_timestamp_ntz(Column years, Column months, Column days, Column hours, Column mins, Column secs) Create local date-time from years, months, days, hours, mins, secs fields.static Columnmake_valid_utf8(Column str) Returns a new string in which all invalid UTF-8 byte sequences, if any, are replaced by the Unicode replacement character (U+FFFD).static ColumnMake year-month interval.static Columnmake_ym_interval(Column years) Make year-month interval from years.static Columnmake_ym_interval(Column years, Column months) Make year-month interval from years, months.static ColumnCreates a new map column.static ColumnCreates a new map column.static Columnmap_concat(Column... cols) Returns the union of all the given maps.static Columnmap_concat(scala.collection.immutable.Seq<Column> cols) Returns the union of all the given maps.static Columnmap_contains_key(Column column, Object key) Returns true if the map contains the key.static ColumnReturns an unordered array of all entries in the given map.static Columnmap_filter(Column expr, scala.Function2<Column, Column, Column> f) Returns a map whose key-value pairs satisfy a predicate.static Columnmap_from_arrays(Column keys, Column values) Creates a new map column.static ColumnReturns a map created from the given array of entries.static ColumnReturns an unordered array containing the keys of the map.static Columnmap_values(Column e) Returns an unordered array containing the values of the map.static ColumnMerge two given maps, key-wise into a single map using a function.static ColumnMasks the given string value.static ColumnMasks the given string value.static ColumnMasks the given string value.static ColumnMasks the given string value.static ColumnMasks the given string value.static ColumnAggregate function: returns the maximum value of the column in a group.static ColumnAggregate function: returns the maximum value of the expression in a group.static ColumnAggregate function: returns the value associated with the maximum value of ord.static ColumnAggregate function: returns an array of values associated with the topkvalues oford.static ColumnAggregate function: returns an array of values associated with the topkvalues oford.static ColumnCalculates the MD5 digest of a binary column and returns the value as a 32 character hex string.static ColumnAggregate function: returns the average of the values in a group.static ColumnAggregate function: returns the average of the values in a group.static ColumnAggregate function: returns the median of the values in a group.static ColumnAggregate function: returns the minimum value of the column in a group.static ColumnAggregate function: returns the minimum value of the expression in a group.static ColumnAggregate function: returns the value associated with the minimum value of ord.static ColumnAggregate function: returns an array of values associated with the bottomkvalues oford.static ColumnAggregate function: returns an array of values associated with the bottomkvalues oford.static ColumnExtracts the minutes as an integer from a given date/time/timestamp/string.static ColumnAggregate function: returns the most frequent value in a group.static ColumnAggregate function: returns the most frequent value in a group.static ColumnA column expression that generates monotonically increasing 64-bit integers.static ColumnDeprecated.Use monotonically_increasing_id().static ColumnExtracts the month as an integer from a given date/timestamp/string.static ColumnExtracts the three-letter abbreviated month name from a given date/timestamp/string.static Column(Java-specific) A transform for timestamps and dates to partition data into months.static Columnmonths_between(Column end, Column start) Returns number of months between datesstartandend.static Columnmonths_between(Column end, Column start, boolean roundOff) Returns number of months between datesendandstart.static Columnnamed_struct(Column... cols) Creates a struct with the given field names and values.static Columnnamed_struct(scala.collection.immutable.Seq<Column> cols) Creates a struct with the given field names and values.static ColumnReturns col1 if it is not NaN, or col2 if col1 is NaN.static ColumnUnary minus, i.e.static ColumnReturns the negated value.static ColumnReturns the first date which is later than the value of thedatecolumn that is on the specified day of the week.static ColumnReturns the first date which is later than the value of thedatecolumn that is on the specified day of the week.static ColumnInversion of boolean expression, i.e.static Columnnow()Returns the current timestamp at the start of query evaluation.static ColumnWindow function: returns the value that is theoffsetth row of the window frame (counting from 1), andnullif the size of window frame is less thanoffsetrows.static ColumnWindow function: returns the value that is theoffsetth row of the window frame (counting from 1), andnullif the size of window frame is less thanoffsetrows.static Columnntile(int n) Window function: returns the ntile group id (from 1 toninclusive) in an ordered window partition.static ColumnReturns null ifcol1equals tocol2, orcol1otherwise.static Columnnullifzero(Column col) Returns null ifcolis equal to zero, orcolotherwise.static ColumnReturnscol2ifcol1is null, orcol1otherwise.static ColumnReturnscol2ifcol1is not null, orcol3otherwise.static ColumnCalculates the byte length for the specified string column.static ColumnOverlay the specified portion ofsrcwithreplace, starting from byte positionposofsrc.static ColumnOverlay the specified portion ofsrcwithreplace, starting from byte positionposofsrcand proceeding forlenbytes.static Columnparse_json(Column json) Parses a JSON string and constructs a Variant value.static ColumnExtracts a part from a URL.static ColumnExtracts a part from a URL.static ColumnWindow function: returns the relative rank (i.e.static Columnpercentile(Column e, Column percentage) Aggregate function: returns the exact percentile(s) of numeric columnexprat the given percentage(s) with value range in [0.0, 1.0].static Columnpercentile(Column e, Column percentage, Column frequency) Aggregate function: returns the exact percentile(s) of numeric columnexprat the given percentage(s) with value range in [0.0, 1.0].static Columnpercentile_approx(Column e, Column percentage, Column accuracy) Aggregate function: returns the approximatepercentileof the numeric columncolwhich is the smallest value in the orderedcolvalues (sorted from least to greatest) such that no more thanpercentageofcolvalues is less than the value or equal to that value.static Columnpi()Returns Pi.static ColumnReturns the positive value of dividend mod divisor.static Columnposexplode(Column e) Creates a new row for each element with position in the given array or map column.static ColumnCreates a new row for each element with position in the given array or map column.static ColumnReturns the position of the first occurrence ofsubstrinstrafter position1.static ColumnReturns the position of the first occurrence ofsubstrinstrafter positionstart.static ColumnReturns the value.static ColumnReturns the value of the first argument raised to the power of the second argument.static ColumnReturns the value of the first argument raised to the power of the second argument.static ColumnReturns the value of the first argument raised to the power of the second argument.static ColumnReturns the value of the first argument raised to the power of the second argument.static ColumnReturns the value of the first argument raised to the power of the second argument.static ColumnReturns the value of the first argument raised to the power of the second argument.static ColumnReturns the value of the first argument raised to the power of the second argument.static ColumnReturns the value of the first argument raised to the power of the second argument.static ColumnReturns the value of the first argument raised to the power of the second argument.static ColumnFormats the arguments in printf-style and returns the result as a string column.static ColumnFormats the arguments in printf-style and returns the result as a string column.static ColumnAggregate function: returns the product of all numerical elements in a group.static ColumnExtracts the quarter as an integer from a given date/timestamp/string.static ColumnReturnsstrenclosed by single quotes and each instance of single quote in it is preceded by a backslash.static ColumnConverts an angle measured in degrees to an approximately equivalent angle measured in radians.static ColumnConverts an angle measured in degrees to an approximately equivalent angle measured in radians.static ColumnThrows an exception with the provided error message.static Columnrand()Generate a random column with independent and identically distributed (i.i.d.) samples uniformly distributed in [0.0, 1.0).static Columnrand(long seed) Generate a random column with independent and identically distributed (i.i.d.) samples uniformly distributed in [0.0, 1.0).static Columnrandn()Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.static Columnrandn(long seed) Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.static Columnrandom()Returns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).static ColumnReturns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).static ColumnReturns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z.static ColumnReturns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z, with the chosen random seed.static Columnrank()Window function: returns the rank of rows within a window partition.static ColumnApplies a binary operator to an initial state and all elements in the array, and reduces this to a single state.static Columnreduce(Column expr, Column initialValue, scala.Function2<Column, Column, Column> merge, scala.Function1<Column, Column> finish) Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.static ColumnCalls a method with reflection.static ColumnCalls a method with reflection.static ColumnReturns true ifstrmatchesregexp, or false otherwise.static Columnregexp_count(Column str, Column regexp) Returns a count of the number of times that the regular expression patternregexpis matched in the stringstr.static Columnregexp_extract(Column e, String exp, int groupIdx) Extract a specific group matched by a Java regex, from the specified string column.static Columnregexp_extract_all(Column str, Column regexp) Extract all strings in thestrthat match theregexpexpression and corresponding to the first regex group index.static Columnregexp_extract_all(Column str, Column regexp, Column idx) Extract all strings in thestrthat match theregexpexpression and corresponding to the regex group index.static Columnregexp_instr(Column str, Column regexp) Searches a string for a regular expression and returns an integer that indicates the beginning position of the matched substring.static Columnregexp_instr(Column str, Column regexp, Column idx) Searches a string for a regular expression and returns an integer that indicates the beginning position of the matched substring.static Columnregexp_like(Column str, Column regexp) Returns true ifstrmatchesregexp, or false otherwise.static Columnregexp_replace(Column e, String pattern, String replacement) Replace all substrings of the specified string value that match regexp with rep.static Columnregexp_replace(Column e, String pattern, String replacement, int pos) Replace all substrings of the specified string value that match regexp with rep, starting at the specified positionpos.static Columnregexp_replace(Column e, Column pattern, Column replacement) Replace all substrings of the specified string value that match regexp with rep.static Columnregexp_replace(Column e, Column pattern, Column replacement, Column pos) Replace all substrings of the specified string value that match regexp with rep, starting at the specified positionpos.static Columnregexp_substr(Column str, Column regexp) Returns the substring that matches the regular expressionregexpwithin the stringstr.static ColumnAggregate function: returns the average of the independent variable for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.static ColumnAggregate function: returns the average of the dependent variable for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.static Columnregr_count(Column y, Column x) Aggregate function: returns the number of non-null number pairs in a group, whereyis the dependent variable andxis the independent variable.static Columnregr_intercept(Column y, Column x) Aggregate function: returns the intercept of the univariate linear regression line for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.static ColumnAggregate function: returns the coefficient of determination for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.static Columnregr_slope(Column y, Column x) Aggregate function: returns the slope of the linear regression line for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.static ColumnAggregate function: returns REGR_COUNT(y, x) * VAR_POP(x) for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.static ColumnAggregate function: returns REGR_COUNT(y, x) * COVAR_POP(y, x) for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.static ColumnAggregate function: returns REGR_COUNT(y, x) * VAR_POP(y) for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.static ColumnRepeats a string column n times, and returns it as a new string column.static ColumnRepeats a string column n times, and returns it as a new string column.static ColumnReplaces all occurrences ofsearchwithreplace.static ColumnReplaces all occurrences ofsearchwithreplace.static ColumnReturns a reversed string or an array with reverse order of elements.static ColumnReturns the rightmostlen(lencan be string type) characters from the stringstr, iflenis less or equal than 0 the result is an empty string.static ColumnReturns the double value that is closest in value to the argument and is equal to a mathematical integer.static ColumnReturns the double value that is closest in value to the argument and is equal to a mathematical integer.static ColumnReturns true ifstrmatchesregexp, or false otherwise.static ColumnReturns the value of the columnerounded to 0 decimal places with HALF_UP round mode.static ColumnRound the value ofetoscaledecimal places with HALF_UP round mode ifscaleis greater than or equal to 0 or at integral part whenscaleis less than 0.static ColumnRound the value ofetoscaledecimal places with HALF_UP round mode ifscaleis greater than or equal to 0 or at integral part whenscaleis less than 0.static ColumnWindow function: returns a sequential number starting at 1 within a window partition.static ColumnRight-pad the binary column with pad to a byte length of len.static ColumnRight-pad the string column with pad to a length of len.static ColumnRight-pad the string column with pad to a length of len.static ColumnTrim the spaces from right end for the specified string value.static ColumnTrim the specified character string from right end for the specified string column.static ColumnTrim the specified character string from right end for the specified string column.static Columnschema_of_csv(String csv) Parses a CSV string and infers its schema in DDL format.static Columnschema_of_csv(Column csv) Parses a CSV string and infers its schema in DDL format.static Columnschema_of_csv(Column csv, Map<String, String> options) Parses a CSV string and infers its schema in DDL format using options.static Columnschema_of_json(String json) Parses a JSON string and infers its schema in DDL format.static Columnschema_of_json(Column json) Parses a JSON string and infers its schema in DDL format.static Columnschema_of_json(Column json, Map<String, String> options) Parses a JSON string and infers its schema in DDL format using options.static ColumnReturns schema in the SQL format of a variant.static ColumnReturns the merged schema in the SQL format of a variant column.static Columnschema_of_xml(String xml) Parses a XML string and infers its schema in DDL format.static Columnschema_of_xml(Column xml) Parses a XML string and infers its schema in DDL format.static Columnschema_of_xml(Column xml, Map<String, String> options) Parses a XML string and infers its schema in DDL format using options.static Columnstatic ColumnExtracts the seconds as an integer from a given date/time/timestamp/string.static ColumnSplits a string into arrays of sentences, where each sentence is an array of words.static ColumnSplits a string into arrays of sentences, where each sentence is an array of words.static ColumnSplits a string into arrays of sentences, where each sentence is an array of words.static ColumnGenerate a sequence of integers from start to stop, incrementing by 1 if start is less than or equal to stop, otherwise -1.static ColumnGenerate a sequence of integers from start to stop, incrementing by step.static ColumnReturns the user name of current execution context.static Columnsession_window(Column timeColumn, String gapDuration) Generates session window given a timestamp specifying column.static Columnsession_window(Column timeColumn, Column gapDuration) Generates session window given a timestamp specifying column.static ColumnReturns a sha1 hash value as a hex string of thecol.static ColumnCalculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.static ColumnCalculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.static ColumnShift the given value numBits left.static ColumnDeprecated.Use shiftleft.static Columnshiftright(Column e, int numBits) (Signed) shift the given value numBits right.static ColumnshiftRight(Column e, int numBits) Deprecated.Use shiftright.static Columnshiftrightunsigned(Column e, int numBits) Unsigned shift the given value numBits right.static ColumnshiftRightUnsigned(Column e, int numBits) Deprecated.Use shiftrightunsigned.static ColumnReturns a random permutation of the given array.static ColumnReturns a random permutation of the given array.static ColumnComputes the signum of the given value.static ColumnComputes the signum of the given column.static ColumnComputes the signum of the given value.static Columnstatic Columnstatic Columnstatic Columnstatic ColumnReturns length of array or map.static ColumnAggregate function: returns the skewness of the values in a group.static ColumnAggregate function: returns the skewness of the values in a group.static ColumnReturns an array containing all the elements inxfrom indexstart(or starting from the end ifstartis negative) with the specifiedlength.static ColumnReturns an array containing all the elements inxfrom indexstart(or starting from the end ifstartis negative) with the specifiedlength.static ColumnAggregate function: returns true if at least one value ofeis true.static Columnsort_array(Column e) Sorts the input array for the given column in ascending order, according to the natural ordering of the array elements.static Columnsort_array(Column e, boolean asc) Sorts the input array for the given column in ascending or descending order, according to the natural ordering of the array elements.static ColumnReturns the soundex code for the specified expression.static ColumnPartition ID.static ColumnSplits str around matches of the given pattern.static ColumnSplits str around matches of the given pattern.static ColumnSplits str around matches of the given pattern.static ColumnSplits str around matches of the given pattern.static Columnsplit_part(Column str, Column delimiter, Column partNum) Splitsstrby delimiter and return requested part of the split (1-based).static ColumnComputes the square root of the specified float value.static ColumnComputes the square root of the specified float value.static Columnst_asbinary(Column geo) Returns the input GEOGRAPHY or GEOMETRY value in WKB format.static Columnst_asbinary(Column geo, String endianness) Returns the input GEOGRAPHY or GEOMETRY value in WKB format using the specified endianness.static Columnst_asbinary(Column geo, Column endianness) Returns the input GEOGRAPHY or GEOMETRY value in WKB format using the specified endianness.static Columnst_geogfromwkb(Column wkb) Parses the WKB description of a geography and returns the corresponding GEOGRAPHY value.static Columnst_geomfromwkb(Column wkb) Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.static Columnst_geomfromwkb(Column wkb, int srid) Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.static Columnst_geomfromwkb(Column wkb, Column srid) Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.static Columnst_setsrid(Column geo, int srid) Returns a new GEOGRAPHY or GEOMETRY value whose SRID is the specified SRID value.static Columnst_setsrid(Column geo, Column srid) Returns a new GEOGRAPHY or GEOMETRY value whose SRID is the specified SRID value.static ColumnReturns the SRID of the input GEOGRAPHY or GEOMETRY value.static ColumnSeparatescol1, ...,colkintonrows.static ColumnSeparatescol1, ...,colkintonrows.static Columnstartswith(Column str, Column prefix) Returns a boolean.static ColumnAggregate function: alias forstddev_samp.static ColumnAggregate function: alias forstddev_samp.static ColumnAggregate function: alias forstddev_samp.static Columnstddev_pop(String columnName) Aggregate function: returns the population standard deviation of the expression in a group.static Columnstddev_pop(Column e) Aggregate function: returns the population standard deviation of the expression in a group.static Columnstddev_samp(String columnName) Aggregate function: returns the sample standard deviation of the expression in a group.static ColumnAggregate function: returns the sample standard deviation of the expression in a group.static Columnstr_to_map(Column text) Creates a map after splitting the text into key/value pairs using delimiters.static Columnstr_to_map(Column text, Column pairDelim) Creates a map after splitting the text into key/value pairs using delimiters.static Columnstr_to_map(Column text, Column pairDelim, Column keyValueDelim) Creates a map after splitting the text into key/value pairs using delimiters.static Columnstring_agg(Column e) Aggregate function: returns the concatenation of non-null input values.static Columnstring_agg(Column e, Column delimiter) Aggregate function: returns the concatenation of non-null input values, separated by the delimiter.static ColumnAggregate function: returns the concatenation of distinct non-null input values.static Columnstring_agg_distinct(Column e, Column delimiter) Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter.static ColumnCreates a new struct column that composes multiple input columns.static ColumnCreates a new struct column that composes multiple input columns.static ColumnCreates a new struct column.static ColumnCreates a new struct column.static ColumnReturns the substring ofstrthat starts atpos, or the slice of byte array that starts atpos.static ColumnReturns the substring ofstrthat starts atposand is of lengthlen, or the slice of byte array that starts atposand is of lengthlen.static ColumnSubstring starts atposand is of lengthlenwhen str is String type or returns the slice of byte array that starts atposin byte and is of lengthlenwhen str is Binary typestatic ColumnSubstring starts atposand is of lengthlenwhen str is String type or returns the slice of byte array that starts atposin byte and is of lengthlenwhen str is Binary typestatic Columnsubstring_index(Column str, String delim, int count) Returns the substring from string str before count occurrences of the delimiter delim.static ColumnAggregate function: returns the sum of all values in the given column.static ColumnAggregate function: returns the sum of all values in the expression.static ColumnAggregate function: returns the sum of distinct values in the expression.static ColumnsumDistinct(String columnName) Deprecated.Use sum_distinct.static ColumnDeprecated.Use sum_distinct.static Columnstatic Columnstatic Columnstatic Columnstatic Columntheta_difference(String columnName1, String columnName2) Subtracts two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches AnotB objectstatic Columntheta_difference(Column c1, Column c2) Subtracts two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches AnotB objectstatic Columntheta_intersection(String columnName1, String columnName2) Intersects two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Intersection objectstatic Columntheta_intersection(Column c1, Column c2) Intersects two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Intersection objectstatic Columntheta_intersection_agg(String columnName) Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by intersecting the Datasketches ThetaSketch instances in the input volumn via a Datasketches Intersection instance.static ColumnAggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by intersecting the Datasketches ThetaSketch instances in the input column via a Datasketches Intersection instance.static Columntheta_sketch_agg(String columnName) Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the default value of 12 forlgNomEntries.static Columntheta_sketch_agg(String columnName, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with thelgNomEntriesnominal entries.static ColumnAggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the default value of 12 forlgNomEntries.static Columntheta_sketch_agg(Column e, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with thelgNomEntriesnominal entries.static Columntheta_sketch_agg(Column e, Column lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with thelgNomEntriesnominal entries.static Columntheta_sketch_estimate(String columnName) Returns the estimated number of unique values given the binary representation of a Datasketches ThetaSketch.static ColumnReturns the estimated number of unique values given the binary representation of a Datasketches ThetaSketch.static Columntheta_union(String columnName1, String columnName2) Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.static Columntheta_union(String columnName1, String columnName2, int lgNomEntries) Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.static Columntheta_union(Column c1, Column c2) Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.static Columntheta_union(Column c1, Column c2, int lgNomEntries) Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.static Columntheta_union(Column c1, Column c2, Column lgNomEntries) Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.static Columntheta_union_agg(String columnName) Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.static Columntheta_union_agg(String columnName, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.static ColumnAggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.static Columntheta_union_agg(Column e, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.static Columntheta_union_agg(Column e, Column lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.static Columntime_bucket(Column bucketSize, Column ts) Returns the start of the fixed-size bucket ofbucketSizethat containsts, with buckets aligned to the default origin (1970-01-01 00:00:00).static Columntime_bucket(Column bucketSize, Column ts, Column origin) Returns the start of the fixed-size bucket ofbucketSizethat containsts, with buckets aligned toorigin.static ColumnReturns the difference between two times, measured in specified units.static ColumnCreates a TIME from the number of microseconds since midnight.static ColumnCreates a TIME from the number of milliseconds since midnight.static ColumnCreates a TIME from the number of seconds since midnight.static ColumnExtracts the number of microseconds since midnight from a TIME value.static ColumnExtracts the number of milliseconds since midnight from a TIME value.static ColumnExtracts the number of seconds (including fractional seconds) from a TIME value.static Columntime_trunc(Column unit, Column time) Returnstimetruncated to theunit.static Columntimestamp_add(String unit, Column quantity, Column ts) Adds the specified number of units to the given timestamp.static Columntimestamp_diff(String unit, Column start, Column end) Gets the difference between the timestamps in the specified units by truncating the fraction part.static ColumnCreates timestamp from the number of microseconds since UTC epoch.static ColumnCreates timestamp from the number of milliseconds since UTC epoch.static ColumnCreates a timestamp with the local time zone and nanosecond precision (TIMESTAMP_LTZ(9)) from the number of nanoseconds since UTC epoch.static ColumnConverts the number of seconds from the Unix epoch (1970-01-01T00:00:00Z) to a timestamp.static ColumnConverts the inputeto a binary value based on the default format "hex".static ColumnConverts the inputeto a binary value based on the suppliedformat.static ColumnConverteto a string based on theformat.static ColumnConverts a column containing aStructTypeinto a CSV string with the specified schema.static Column(Java-specific) Converts a column containing aStructTypeinto a CSV string with the specified schema.static ColumnConverts the column intoDateTypeby casting rules toDateType.static ColumnConverts the column into aDateTypewith a specified formatstatic ColumnConverts a column containing aStructType,ArrayTypeor aMapTypeinto a JSON string with the specified schema.static Column(Java-specific) Converts a column containing aStructType,ArrayTypeor aMapTypeinto a JSON string with the specified schema.static Column(Scala-specific) Converts a column containing aStructType,ArrayTypeor aMapTypeinto a JSON string with the specified schema.static ColumnConvert string 'e' to a number based on the string format 'format'.static ColumnParses a string value to a time value.static ColumnParses a string value to a time value.static ColumnConverts to a timestamp by casting rules toTimestampType.static Columnto_timestamp(Column s, String fmt) Converts time string with the given pattern to timestamp.static Columnto_timestamp_ltz(Column timestamp) Parses thetimestampexpression with the default format to a timestamp with local time zone.static Columnto_timestamp_ltz(Column timestamp, Column format) Parses thetimestampexpression with theformatexpression to a timestamp with local time zone.static Columnto_timestamp_ntz(Column timestamp) Parses thetimestampexpression with the default format to a timestamp without time zone.static Columnto_timestamp_ntz(Column timestamp, Column format) Parses thetimestamp_strexpression with theformatexpression to a timestamp without time zone.static Columnto_unix_timestamp(Column timeExp) Returns the UNIX timestamp of the given time.static Columnto_unix_timestamp(Column timeExp, Column format) Returns the UNIX timestamp of the given time.static Columnto_utc_timestamp(Column ts, String tz) Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC.static Columnto_utc_timestamp(Column ts, Column tz) Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC.static Columnto_varchar(Column e, Column format) Converteto a string based on theformat.static Columnto_variant_object(Column col) Converts a column containing nested inputs (array/map/struct) into a variants where maps and structs are converted to variant objects which are unordered unlike SQL structs.static ColumnConverts a column containing aStructTypeinto a XML string with the specified schema.static Column(Java-specific) Converts a column containing aStructTypeinto a XML string with the specified schema.static ColumnDeprecated.Use degrees.static ColumnDeprecated.Use degrees.static ColumnDeprecated.Use radians.static ColumnDeprecated.Use radians.static ColumnReturns an array of elements after applying a transformation to each element in the input array.static ColumnReturns an array of elements after applying a transformation to each element in the input array.static Columntransform_keys(Column expr, scala.Function2<Column, Column, Column> f) Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new keys for the pairs.static Columntransform_values(Column expr, scala.Function2<Column, Column, Column> f) Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new values for the pairs.static ColumnTranslate any character in the src by a character in replaceString.static ColumnTrim the spaces from both ends for the specified string column.static ColumnTrim the specified character from both ends for the specified string column.static ColumnTrim the specified character from both ends for the specified string column.static ColumnReturns date truncated to the unit specified by the format.static ColumnReturns the sum ofleftandrightand the result is null on overflow.static Columntry_aes_decrypt(Column input, Column key) Returns a decrypted value ofinput.static Columntry_aes_decrypt(Column input, Column key, Column mode) Returns a decrypted value ofinput.static Columntry_aes_decrypt(Column input, Column key, Column mode, Column padding) Returns a decrypted value ofinput.static ColumnThis is a special version ofaes_decryptthat performs the same operation, but returns a NULL value instead of raising an error if the decryption cannot be performed.static ColumnReturns the mean calculated from values of a group and the result is null on overflow.static Columntry_divide(Column left, Column right) Returnsdividend/divisor.static Columntry_element_at(Column column, Column value) (array, index) - Returns element of array at given (1-based) index.static Columntry_make_interval(Column years) This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.static Columntry_make_interval(Column years, Column months) This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.static Columntry_make_interval(Column years, Column months, Column weeks) This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.static Columntry_make_interval(Column years, Column months, Column weeks, Column days) This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.static ColumnThis is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.static Columntry_make_interval(Column years, Column months, Column weeks, Column days, Column hours, Column mins) This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.static Columntry_make_interval(Column years, Column months, Column weeks, Column days, Column hours, Column mins, Column secs) This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.static Columntry_make_timestamp(Column date, Column time) Try to create a local date-time from date and time fields.static Columntry_make_timestamp(Column date, Column time, Column timezone) Try to create a local date-time from date, time, and timezone fields.static Columntry_make_timestamp(Column years, Column months, Column days, Column hours, Column mins, Column secs) Try to create a timestamp from years, months, days, hours, mins, and secs fields.static Columntry_make_timestamp(Column years, Column months, Column days, Column hours, Column mins, Column secs, Column timezone) Try to create a timestamp from years, months, days, hours, mins, secs and timezone fields.static Columntry_make_timestamp_ltz(Column years, Column months, Column days, Column hours, Column mins, Column secs) Try to create the current timestamp with local time zone from years, months, days, hours, mins and secs fields.static Columntry_make_timestamp_ltz(Column years, Column months, Column days, Column hours, Column mins, Column secs, Column timezone) Try to create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields.static Columntry_make_timestamp_ntz(Column date, Column time) Try to create a local date-time from date and time fields.static Columntry_make_timestamp_ntz(Column years, Column months, Column days, Column hours, Column mins, Column secs) Try to create a local date-time from years, months, days, hours, mins, secs fields.static ColumnReturns the remainder ofdividend/divisor.static Columntry_multiply(Column left, Column right) Returnsleft*rightand the result is null on overflow.static Columntry_parse_json(Column json) Parses a JSON string and constructs a Variant value.static Columntry_parse_url(Column url, Column partToExtract) Extracts a part from a URL.static Columntry_parse_url(Column url, Column partToExtract, Column key) Extracts a part from a URL.static Columntry_reflect(Column... cols) This is a special version ofreflectthat performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.static Columntry_reflect(scala.collection.immutable.Seq<Column> cols) This is a special version ofreflectthat performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.static Columntry_subtract(Column left, Column right) Returnsleft-rightand the result is null on overflow.static ColumnReturns the sum calculated from values of a group and the result is null on overflow.static ColumnThis is a special version ofto_binarythat performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.static Columntry_to_binary(Column e, Column f) This is a special version ofto_binarythat performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.static ColumnThis is a special version ofto_datethat performs the same operation, but returns a NULL value instead of raising an error if date cannot be created.static Columntry_to_date(Column e, String fmt) This is a special version ofto_datethat performs the same operation, but returns a NULL value instead of raising an error if date cannot be created.static Columntry_to_number(Column e, Column format) Convert stringeto a number based on the string formatformat.static Columntry_to_time(Column str) Parses a string value to a time value.static Columntry_to_time(Column str, Column format) Parses a string value to a time value.static ColumnParses thesto a timestamp.static Columntry_to_timestamp(Column s, Column format) Parses theswith theformatto a timestamp.static Columntry_url_decode(Column str) This is a special version ofurl_decodethat performs the same operation, but returns a NULL value instead of raising an error if the decoding cannot be performed.static Columntry_validate_utf8(Column str) Returns the input value if it corresponds to a valid UTF-8 string, or NULL otherwise.static Columntry_variant_array_append(Column v, String path, Column value) Appends a value to the array in a variant at the given JSONPath location.static Columntry_variant_array_append(Column v, Column path, Column value) Appends a value to the array in a variant at the given JSONPath location.static Columntry_variant_get(Column v, String path, String targetType) Extracts a sub-variant fromvaccording topathstring, and then cast the sub-variant totargetType.static Columntry_variant_get(Column v, Column path, String targetType) Extracts a sub-variant fromvaccording topathcolumn, and then cast the sub-variant totargetType.static Columntry_variant_insert(Column v, String path, Column value) Inserts a value into a variant at the given JSONPath location.static Columntry_variant_insert(Column v, Column path, Column value) Inserts a value into a variant at the given JSONPath location.static Columntry_variant_set(Column v, String path, Column value) Sets or upserts a value in a variant at the given JSONPath location.static Columntry_variant_set(Column v, String path, Column value, boolean createIfMissing) Sets or upserts a value in a variant at the given JSONPath location.static Columntry_variant_set(Column v, Column path, Column value) Sets or upserts a value in a variant at the given JSONPath location.static Columntry_variant_set(Column v, Column path, Column value, boolean createIfMissing) Sets or upserts a value in a variant at the given JSONPath location.static Columntuple_difference_double(String columnName1, String columnName2) Subtracts two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches AnotB object.static Columntuple_difference_double(Column c1, Column c2) Subtracts two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches AnotB object.static Columntuple_difference_integer(String columnName1, String columnName2) Subtracts two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches AnotB object.static Columntuple_difference_integer(Column c1, Column c2) Subtracts two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches AnotB object.static Columntuple_difference_theta_double(String columnName1, String columnName2) Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with double summary data type in the input columns using a Datasketches AnotB object.static ColumnSubtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with double summary data type in the input columns using a Datasketches AnotB object.static Columntuple_difference_theta_integer(String columnName1, String columnName2) Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with integer summary data type in the input columns using a Datasketches AnotB object.static ColumnSubtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with integer summary data type in the input columns using a Datasketches AnotB object.static Columntuple_intersection_agg_double(String columnName) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static Columntuple_intersection_agg_double(String columnName, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static ColumnAggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static Columntuple_intersection_agg_double(Column e, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static Columntuple_intersection_agg_double(Column e, Column mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static Columntuple_intersection_agg_integer(String columnName) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static Columntuple_intersection_agg_integer(String columnName, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static ColumnAggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static Columntuple_intersection_agg_integer(Column e, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static Columntuple_intersection_agg_integer(Column e, Column mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.static Columntuple_intersection_double(String columnName1, String columnName2) Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_double(String columnName1, String columnName2, String mode) Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_double(Column c1, Column c2) Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_double(Column c1, Column c2, String mode) Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_double(Column c1, Column c2, Column mode) Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_integer(String columnName1, String columnName2) Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_integer(String columnName1, String columnName2, String mode) Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_integer(Column c1, Column c2) Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_integer(Column c1, Column c2, String mode) Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_integer(Column c1, Column c2, Column mode) Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.static Columntuple_intersection_theta_double(String columnName1, String columnName2) Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static Columntuple_intersection_theta_double(String columnName1, String columnName2, String mode) Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static ColumnIntersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static Columntuple_intersection_theta_double(Column c1, Column c2, String mode) Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static Columntuple_intersection_theta_double(Column c1, Column c2, Column mode) Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static Columntuple_intersection_theta_integer(String columnName1, String columnName2) Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static Columntuple_intersection_theta_integer(String columnName1, String columnName2, String mode) Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static ColumnIntersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static Columntuple_intersection_theta_integer(Column c1, Column c2, String mode) Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static Columntuple_intersection_theta_integer(Column c1, Column c2, Column mode) Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.static Columntuple_sketch_agg_double(String keyColumnName, String summaryColumnName) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns.static Columntuple_sketch_agg_double(String keyColumnName, String summaryColumnName, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries.static Columntuple_sketch_agg_double(String keyColumnName, String summaryColumnName, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode.static Columntuple_sketch_agg_double(Column key, Column summary) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns.static Columntuple_sketch_agg_double(Column key, Column summary, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries.static Columntuple_sketch_agg_double(Column key, Column summary, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode.static Columntuple_sketch_agg_double(Column key, Column summary, Column lgNomEntries, Column mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode.static Columntuple_sketch_agg_integer(String keyColumnName, String summaryColumnName) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns.static Columntuple_sketch_agg_integer(String keyColumnName, String summaryColumnName, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries.static Columntuple_sketch_agg_integer(String keyColumnName, String summaryColumnName, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode.static Columntuple_sketch_agg_integer(Column key, Column summary) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns.static Columntuple_sketch_agg_integer(Column key, Column summary, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries.static Columntuple_sketch_agg_integer(Column key, Column summary, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode.static Columntuple_sketch_agg_integer(Column key, Column summary, Column lgNomEntries, Column mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode.static Columntuple_sketch_estimate_double(String columnName) Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with double summary data type.static ColumnReturns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with double summary data type.static Columntuple_sketch_estimate_integer(String columnName) Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with integer summary data type.static ColumnReturns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with integer summary data type.static Columntuple_sketch_summary_double(String columnName) Aggregates the summary values from a Datasketches TupleSketch with double summary data type.static Columntuple_sketch_summary_double(String columnName, String mode) Aggregates the summary values from a Datasketches TupleSketch with double summary data type.static ColumnAggregates the summary values from a Datasketches TupleSketch with double summary data type.static Columntuple_sketch_summary_double(Column c, String mode) Aggregates the summary values from a Datasketches TupleSketch with double summary data type.static Columntuple_sketch_summary_double(Column c, Column mode) Aggregates the summary values from a Datasketches TupleSketch with double summary data type.static Columntuple_sketch_summary_integer(String columnName) Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.static Columntuple_sketch_summary_integer(String columnName, String mode) Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.static ColumnAggregates the summary values from a Datasketches TupleSketch with integer summary data type.static Columntuple_sketch_summary_integer(Column c, String mode) Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.static Columntuple_sketch_summary_integer(Column c, Column mode) Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.static Columntuple_sketch_theta_double(String columnName) Returns the theta value (sampling rate) from a Datasketches TupleSketch with double summary data type.static ColumnReturns the theta value (sampling rate) from a Datasketches TupleSketch with double summary data type.static Columntuple_sketch_theta_integer(String columnName) Returns the theta value (sampling rate) from a Datasketches TupleSketch with integer summary data type.static ColumnReturns the theta value (sampling rate) from a Datasketches TupleSketch with integer summary data type.static Columntuple_union_agg_double(String columnName) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_double(String columnName, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_double(String columnName, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static ColumnAggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_double(Column e, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_double(Column e, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_double(Column e, Column lgNomEntries, Column mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_integer(String columnName) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_integer(String columnName, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_integer(String columnName, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static ColumnAggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_integer(Column e, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_integer(Column e, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_agg_integer(Column e, Column lgNomEntries, Column mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.static Columntuple_union_double(String columnName1, String columnName2) Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.static Columntuple_union_double(String columnName1, String columnName2, int lgNomEntries) Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.static Columntuple_union_double(String columnName1, String columnName2, int lgNomEntries, String mode) Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.static Columntuple_union_double(Column c1, Column c2) Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.static Columntuple_union_double(Column c1, Column c2, int lgNomEntries) Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.static Columntuple_union_double(Column c1, Column c2, int lgNomEntries, String mode) Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.static Columntuple_union_double(Column c1, Column c2, Column lgNomEntries, Column mode) Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.static Columntuple_union_integer(String columnName1, String columnName2) Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.static Columntuple_union_integer(String columnName1, String columnName2, int lgNomEntries) Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.static Columntuple_union_integer(String columnName1, String columnName2, int lgNomEntries, String mode) Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.static Columntuple_union_integer(Column c1, Column c2) Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.static Columntuple_union_integer(Column c1, Column c2, int lgNomEntries) Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.static Columntuple_union_integer(Column c1, Column c2, int lgNomEntries, String mode) Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.static Columntuple_union_integer(Column c1, Column c2, Column lgNomEntries, Column mode) Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.static Columntuple_union_theta_double(String columnName1, String columnName2) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_double(String columnName1, String columnName2, int lgNomEntries) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_double(String columnName1, String columnName2, int lgNomEntries, String mode) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_double(Column c1, Column c2) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_double(Column c1, Column c2, int lgNomEntries) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_double(Column c1, Column c2, int lgNomEntries, String mode) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_double(Column c1, Column c2, Column lgNomEntries, Column mode) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_integer(String columnName1, String columnName2) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_integer(String columnName1, String columnName2, int lgNomEntries) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_integer(String columnName1, String columnName2, int lgNomEntries, String mode) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_integer(Column c1, Column c2) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_integer(Column c1, Column c2, int lgNomEntries) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_integer(Column c1, Column c2, int lgNomEntries, String mode) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static Columntuple_union_theta_integer(Column c1, Column c2, Column lgNomEntries, Column mode) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.static <T> Columntypedlit(T literal, scala.reflect.api.TypeTags.TypeTag<T> evidence$2) Creates aColumnof literal value.static <T> ColumntypedLit(T literal, scala.reflect.api.TypeTags.TypeTag<T> evidence$1) Creates aColumnof literal value.static ColumnReturn DDL-formatted type string for the data type of the input.static ColumnReturnsstrwith all characters changed to uppercase.static <IN,BUF, OUT>
UserDefinedFunctionudaf(Aggregator<IN, BUF, OUT> agg, Encoder<IN> inputEncoder) Obtains aUserDefinedFunctionthat wraps the givenAggregatorso that it may be used with untyped Data Frames.static <IN,BUF, OUT>
UserDefinedFunctionudaf(Aggregator<IN, BUF, OUT> agg, scala.reflect.api.TypeTags.TypeTag<IN> evidence$3) Obtains aUserDefinedFunctionthat wraps the givenAggregatorso that it may be used with untyped Data Frames.static UserDefinedFunctionDeprecated.Scala `udf` method with return type parameter is deprecated.static UserDefinedFunctionDefines a Java UDF0 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF1 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF10 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF2 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF3 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF4 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF5 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF6 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF7 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF8 instance as user-defined function (UDF).static UserDefinedFunctionDefines a Java UDF9 instance as user-defined function (UDF).static <RT> UserDefinedFunctionudf(scala.Function0<RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$4) Defines a Scala closure of 0 arguments as user-defined function (UDF).static <RT,A1> UserDefinedFunction udf(scala.Function1<A1, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$5, scala.reflect.api.TypeTags.TypeTag<A1> evidence$6) Defines a Scala closure of 1 arguments as user-defined function (UDF).static <RT,A1, A2, A3, A4, A5, A6, A7, A8, A9, A10>
UserDefinedFunctionudf(scala.Function10<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$59, scala.reflect.api.TypeTags.TypeTag<A1> evidence$60, scala.reflect.api.TypeTags.TypeTag<A2> evidence$61, scala.reflect.api.TypeTags.TypeTag<A3> evidence$62, scala.reflect.api.TypeTags.TypeTag<A4> evidence$63, scala.reflect.api.TypeTags.TypeTag<A5> evidence$64, scala.reflect.api.TypeTags.TypeTag<A6> evidence$65, scala.reflect.api.TypeTags.TypeTag<A7> evidence$66, scala.reflect.api.TypeTags.TypeTag<A8> evidence$67, scala.reflect.api.TypeTags.TypeTag<A9> evidence$68, scala.reflect.api.TypeTags.TypeTag<A10> evidence$69) Defines a Scala closure of 10 arguments as user-defined function (UDF).static <RT,A1, A2> UserDefinedFunction udf(scala.Function2<A1, A2, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$7, scala.reflect.api.TypeTags.TypeTag<A1> evidence$8, scala.reflect.api.TypeTags.TypeTag<A2> evidence$9) Defines a Scala closure of 2 arguments as user-defined function (UDF).static <RT,A1, A2, A3>
UserDefinedFunctionudf(scala.Function3<A1, A2, A3, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$10, scala.reflect.api.TypeTags.TypeTag<A1> evidence$11, scala.reflect.api.TypeTags.TypeTag<A2> evidence$12, scala.reflect.api.TypeTags.TypeTag<A3> evidence$13) Defines a Scala closure of 3 arguments as user-defined function (UDF).static <RT,A1, A2, A3, A4>
UserDefinedFunctionudf(scala.Function4<A1, A2, A3, A4, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$14, scala.reflect.api.TypeTags.TypeTag<A1> evidence$15, scala.reflect.api.TypeTags.TypeTag<A2> evidence$16, scala.reflect.api.TypeTags.TypeTag<A3> evidence$17, scala.reflect.api.TypeTags.TypeTag<A4> evidence$18) Defines a Scala closure of 4 arguments as user-defined function (UDF).static <RT,A1, A2, A3, A4, A5>
UserDefinedFunctionudf(scala.Function5<A1, A2, A3, A4, A5, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$19, scala.reflect.api.TypeTags.TypeTag<A1> evidence$20, scala.reflect.api.TypeTags.TypeTag<A2> evidence$21, scala.reflect.api.TypeTags.TypeTag<A3> evidence$22, scala.reflect.api.TypeTags.TypeTag<A4> evidence$23, scala.reflect.api.TypeTags.TypeTag<A5> evidence$24) Defines a Scala closure of 5 arguments as user-defined function (UDF).static <RT,A1, A2, A3, A4, A5, A6>
UserDefinedFunctionudf(scala.Function6<A1, A2, A3, A4, A5, A6, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$25, scala.reflect.api.TypeTags.TypeTag<A1> evidence$26, scala.reflect.api.TypeTags.TypeTag<A2> evidence$27, scala.reflect.api.TypeTags.TypeTag<A3> evidence$28, scala.reflect.api.TypeTags.TypeTag<A4> evidence$29, scala.reflect.api.TypeTags.TypeTag<A5> evidence$30, scala.reflect.api.TypeTags.TypeTag<A6> evidence$31) Defines a Scala closure of 6 arguments as user-defined function (UDF).static <RT,A1, A2, A3, A4, A5, A6, A7>
UserDefinedFunctionudf(scala.Function7<A1, A2, A3, A4, A5, A6, A7, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$32, scala.reflect.api.TypeTags.TypeTag<A1> evidence$33, scala.reflect.api.TypeTags.TypeTag<A2> evidence$34, scala.reflect.api.TypeTags.TypeTag<A3> evidence$35, scala.reflect.api.TypeTags.TypeTag<A4> evidence$36, scala.reflect.api.TypeTags.TypeTag<A5> evidence$37, scala.reflect.api.TypeTags.TypeTag<A6> evidence$38, scala.reflect.api.TypeTags.TypeTag<A7> evidence$39) Defines a Scala closure of 7 arguments as user-defined function (UDF).static <RT,A1, A2, A3, A4, A5, A6, A7, A8>
UserDefinedFunctionudf(scala.Function8<A1, A2, A3, A4, A5, A6, A7, A8, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$40, scala.reflect.api.TypeTags.TypeTag<A1> evidence$41, scala.reflect.api.TypeTags.TypeTag<A2> evidence$42, scala.reflect.api.TypeTags.TypeTag<A3> evidence$43, scala.reflect.api.TypeTags.TypeTag<A4> evidence$44, scala.reflect.api.TypeTags.TypeTag<A5> evidence$45, scala.reflect.api.TypeTags.TypeTag<A6> evidence$46, scala.reflect.api.TypeTags.TypeTag<A7> evidence$47, scala.reflect.api.TypeTags.TypeTag<A8> evidence$48) Defines a Scala closure of 8 arguments as user-defined function (UDF).static <RT,A1, A2, A3, A4, A5, A6, A7, A8, A9>
UserDefinedFunctionudf(scala.Function9<A1, A2, A3, A4, A5, A6, A7, A8, A9, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$49, scala.reflect.api.TypeTags.TypeTag<A1> evidence$50, scala.reflect.api.TypeTags.TypeTag<A2> evidence$51, scala.reflect.api.TypeTags.TypeTag<A3> evidence$52, scala.reflect.api.TypeTags.TypeTag<A4> evidence$53, scala.reflect.api.TypeTags.TypeTag<A5> evidence$54, scala.reflect.api.TypeTags.TypeTag<A6> evidence$55, scala.reflect.api.TypeTags.TypeTag<A7> evidence$56, scala.reflect.api.TypeTags.TypeTag<A8> evidence$57, scala.reflect.api.TypeTags.TypeTag<A9> evidence$58) Defines a Scala closure of 9 arguments as user-defined function (UDF).static ColumnDecodes a BASE64 encoded string column and returns it as a binary column.static ColumnInverse of hex.static ColumnReturns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers.static ColumnReturns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers, with the chosen random seed.static ColumnReturns the number of days since 1970-01-01.static ColumnReturns the number of microseconds since 1970-01-01 00:00:00 UTC.static ColumnReturns the number of milliseconds since 1970-01-01 00:00:00 UTC.static Columnunix_nanos(Column e) Returns the number of nanoseconds since 1970-01-01 00:00:00 UTC for a nanosecond-precision timestamp (TIMESTAMP_LTZ(p)/TIMESTAMP_NTZ(p),pin[7, 9]).static ColumnReturns the number of seconds since 1970-01-01 00:00:00 UTC.static ColumnReturns the current Unix timestamp (in seconds) as a long.static ColumnConverts time string in format yyyy-MM-dd HH:mm:ss to Unix timestamp (in seconds), using the default timezone and the default locale.static Columnunix_timestamp(Column s, String p) Converts time string with given pattern to Unix timestamp (in seconds).static Columnunwrap_udt(Column column) Unwrap UDT data type column into its underlying type.static ColumnConverts a string column to upper case.static Columnurl_decode(Column str) Decodes astrin 'application/x-www-form-urlencoded' format using a specific encoding scheme.static Columnurl_encode(Column str) Translates a string into 'application/x-www-form-urlencoded' format using a specific encoding scheme.static Columnuser()Returns the user name of current execution context.static Columnuuid()Returns an universally unique identifier (UUID) string.static ColumnReturns an universally unique identifier (UUID) string.static Columnvalidate_utf8(Column str) Returns the input value if it corresponds to a valid UTF-8 string, or emits a SparkIllegalArgumentException exception otherwise.static ColumnAggregate function: returns the population variance of the values in a group.static ColumnAggregate function: returns the population variance of the values in a group.static ColumnAggregate function: returns the unbiased variance of the values in a group.static ColumnAggregate function: returns the unbiased variance of the values in a group.static ColumnAggregate function: alias forvar_samp.static ColumnAggregate function: alias forvar_samp.static Columnvariant_array_append(Column v, String path, Column value) Appends a value to the array in a variant at the given JSONPath location.static Columnvariant_array_append(Column v, Column path, Column value) Appends a value to the array in a variant at the given JSONPath location.static Columnvariant_delete(Column v, String path, String... paths) Removes fields or array elements from a variant at the given JSONPath locations.static Columnvariant_delete(Column v, String path, scala.collection.immutable.Seq<String> paths) Removes fields or array elements from a variant at the given JSONPath locations.static Columnvariant_delete(Column v, Column path, Column... paths) Removes fields or array elements from a variant at the given JSONPath locations.static Columnvariant_delete(Column v, Column path, scala.collection.immutable.Seq<Column> paths) Removes fields or array elements from a variant at the given JSONPath locations.static Columnvariant_get(Column v, String path, String targetType) Extracts a sub-variant fromvaccording topathstring, and then cast the sub-variant totargetType.static Columnvariant_get(Column v, Column path, String targetType) Extracts a sub-variant fromvaccording topathcolumn, and then cast the sub-variant totargetType.static Columnvariant_insert(Column v, String path, Column value) Inserts a value into a variant at the given JSONPath location.static Columnvariant_insert(Column v, Column path, Column value) Inserts a value into a variant at the given JSONPath location.static Columnvariant_set(Column v, String path, Column value) Sets or upserts a value in a variant at the given JSONPath location.static Columnvariant_set(Column v, String path, Column value, boolean createIfMissing) Sets or upserts a value in a variant at the given JSONPath location.static Columnvariant_set(Column v, Column path, Column value) Sets or upserts a value in a variant at the given JSONPath location.static Columnvariant_set(Column v, Column path, Column value, boolean createIfMissing) Sets or upserts a value in a variant at the given JSONPath location.static Columnvector_avg(Column col) Aggregate function: returns the element-wise mean of float vectors in a group.static Columnvector_cosine_similarity(Column left, Column right) Returns the cosine similarity between two float vectors.static Columnvector_inner_product(Column left, Column right) Returns the inner product (dot product) between two float vectors.static Columnvector_l2_distance(Column left, Column right) Returns the Euclidean (L2) distance between two float vectors.static Columnvector_norm(Column vector) Returns the Lp norm of a float vector using degree 2.0 (Euclidean norm).static Columnvector_norm(Column vector, Column degree) Returns the Lp norm of a float vector.static Columnvector_normalize(Column vector) Normalizes a float vector to unit length using degree 2.0 (Euclidean norm).static Columnvector_normalize(Column vector, Column degree) Normalizes a float vector to unit length.static Columnvector_sum(Column col) Aggregate function: returns the element-wise sum of float vectors in a group.static Columnversion()Returns the Spark version.static ColumnReturns the day of the week for date/timestamp (0 = Monday, 1 = Tuesday, ..., 6 = Sunday).static Columnweekofyear(Column e) Extracts the week number as an integer from a given date/timestamp/string.static ColumnEvaluates a list of conditions and returns one of multiple possible result expressions.static Columnwidth_bucket(Column v, Column min, Column max, Column numBucket) Returns the bucket number into which the value of this expression would fall after being evaluated.static ColumnGenerates tumbling time windows given a timestamp specifying column.static ColumnBucketize rows into one or more time windows given a timestamp specifying column.static ColumnBucketize rows into one or more time windows given a timestamp specifying column.static Columnwindow_time(Column windowColumn) Extracts the event time from the window column.static ColumnReturns a string array of values within the nodes of xml that match the XPath expression.static Columnxpath_boolean(Column xml, Column path) Returns true if the XPath expression evaluates to true, or if a matching node is found.static Columnxpath_double(Column xml, Column path) Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.static Columnxpath_float(Column xml, Column path) Returns a float value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.static ColumnReturns an integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.static Columnxpath_long(Column xml, Column path) Returns a long integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.static Columnxpath_number(Column xml, Column path) Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.static Columnxpath_short(Column xml, Column path) Returns a short integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.static Columnxpath_string(Column xml, Column path) Returns the text contents of the first xml node that matches the XPath expression.static ColumnCalculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column.static ColumnCalculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column.static ColumnExtracts the year as an integer from a given date/timestamp/string.static Column(Java-specific) A transform for timestamps and dates to partition data into years.static Columnzeroifnull(Column col) Returns zero ifcolis null, orcolotherwise.static ColumnMerge two given arrays, element-wise, into a single array using a function.
-
Constructor Details
-
functions
public functions()
-
-
Method Details
-
countDistinct
Aggregate function: returns the number of distinct items in a group.An alias of
count_distinct, and it is encouraged to usecount_distinctdirectly.- Parameters:
expr- The first column. A column of any type.exprs- Additional columns. A column of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
countDistinct
Aggregate function: returns the number of distinct items in a group.An alias of
count_distinct, and it is encouraged to usecount_distinctdirectly.- Parameters:
columnName- first column to compute on. A column of any type.columnNames- additional columns to compute on. Columns of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
count_distinct
Aggregate function: returns the number of distinct items in a group.- Parameters:
expr- first column to compute on. A column of any type.exprs- additional columns to compute on. Columns of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.2.0
-
grouping_id
Aggregate function: returns the level of grouping, equals to(grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)- Parameters:
cols- columns to check for. Columns of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 2.0.0
- Note:
- The list of columns should match with grouping columns exactly, or empty (means all the grouping columns).
-
grouping_id
Aggregate function: returns the level of grouping, equals to(grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)- Parameters:
colName- the name of the first grouping column. A column of any type.colNames- the names of the remaining grouping columns. Columns of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 2.0.0
- Note:
- The list of columns should match with grouping columns exactly.
-
array
Creates a new array column. The input columns must all have the same data type.- Parameters:
cols- The columns to combine into an array. Each is a column of any type, and all must share the same data type.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.4.0
-
array
Creates a new array column. The input columns must all have the same data type.- Parameters:
colName- (undocumented)colNames- (undocumented)- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.4.0
-
map
Creates a new map column. The input columns must be grouped as key-value pairs, e.g. (key1, value1, key2, value2, ...). The key columns must all have the same data type, and can't be null. The value columns must all have the same data type.- Parameters:
cols- The columns grouped as key-value pairs (key1, value1, key2, value2, ...). Each is a column of any type; key columns must share a type and value columns must share a type.- Returns:
- Returns a column that evaluates to a map.
- Since:
- 2.0
-
named_struct
Creates a struct with the given field names and values.- Parameters:
cols- The field names and values grouped as pairs (name1, value1, name2, value2, ...). Names are columns that evaluate to a string; values are columns of any type.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 3.5.0
-
coalesce
Returns the first column that is not null, or null if all inputs are null.For example,
coalesce(a, b, c)will return a if a is not null, or b if a is null and b is not null, or c if both a and b are null but c is not null.- Parameters:
e- the columns to work on. A column that evaluates to any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
-
struct
Creates a new struct column. If the input column is a column in aDataFrame, or a derived column expression that is named (i.e. aliased), its name would be retained as the StructField's name, otherwise, the newly generated StructField's name would be auto generated ascolwith a suffixindex + 1, i.e. col1, col2, col3, ...- Parameters:
cols- the columns to contain in the output struct. A column of any type.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 1.4.0
-
struct
Creates a new struct column that composes multiple input columns.- Parameters:
colName- the name of the first column to contain in the output struct.colNames- the names of the remaining columns to contain in the output struct.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 1.4.0
-
greatest
Returns the greatest value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.- Parameters:
exprs- columns to check for greatest value. A column that evaluates to any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
greatest
Returns the greatest value of the list of column names, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.- Parameters:
columnName- the first column name to check for greatest value. A column of a comparable type.columnNames- the remaining column names to check for greatest value. Columns of a comparable type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
least
Returns the least value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.- Parameters:
exprs- The values to be compared. Columns that evaluate to a comparable type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
least
Returns the least value of the list of column names, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.- Parameters:
columnName- The name of the first column to be compared. A column of a comparable type.columnNames- The names of the remaining columns to be compared. Columns of a comparable type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
hash
Calculates the hash code of given columns, and returns the result as an int column.- Parameters:
cols- one or more columns to compute on. A column of any type.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 2.0.0
-
xxhash64
Calculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column. The hash computation uses an initial seed of 42.- Parameters:
cols- one or more columns to compute on. A column of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.0.0
-
reflect
Calls a method with reflection.- Parameters:
cols- (undocumented)- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
java_method
Calls a method with reflection.- Parameters:
cols- (undocumented)- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
try_reflect
This is a special version ofreflectthat performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.- Parameters:
cols- (undocumented)- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
stack
Separatescol1, ...,colkintonrows. Uses column names col0, col1, etc. by default unless specified otherwise.- Parameters:
cols- The first column must be a constant integer for the number of rows, and the remaining columns are the input elements to be separated into rows.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
concat_ws
Concatenates multiple input string columns together into a single string column, using the given separator.- Parameters:
sep- The words separator. A column that evaluates to a string. Must be a constant.exprs- The list of columns to work on. Each a column that evaluates to a string or an array of strings.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
- Note:
- Input strings which are null are skipped.
-
format_string
Formats the arguments in printf-style and returns the result as a string column.- Parameters:
format- the format string that can contain embedded format tags. A column that evaluates to a string. Must be a constant.arguments- the values to be used in formatting. Columns that evaluate to any type.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
printf
Formats the arguments in printf-style and returns the result as a string column.- Parameters:
format- A format string that can contain embedded format tags. A column that evaluates to a string.arguments- The values to be used in formatting. Columns that evaluate to any type.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
elt
Returns then-th input, e.g., returnsinput2whennis 2. The function returns NULL if the index exceeds the length of the array andspark.sql.ansi.enabledis set to false. Ifspark.sql.ansi.enabledis set to true, it throws ArrayIndexOutOfBoundsException for invalid indices.- Parameters:
inputs- The index followed by the inputs to select from. Columns where the first evaluates to an integral and the rest evaluate to strings or binaries.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
concat
Concatenates multiple input columns together into a single column. The function works with strings, binary and compatible array columns.- Parameters:
exprs- Input columns to concatenate. A column that evaluates to a string, binary or an array.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
- Note:
- Returns null if any of the input columns are null.
-
json_tuple
Creates a new row for a json column according to the given field names.- Parameters:
json- the JSON string column. A column that evaluates to a string.fields- the field names to extract. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.6.0
-
variant_delete
Removes fields or array elements from a variant at the given JSONPath locations. Multiple paths are applied left to right. Returns NULL ifvis NULL; NULL paths are skipped.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the column containing the first JSONPath string. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A column that evaluates to a string.paths- additional JSONPath arguments, applied afterpathin order. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a variant.
- Since:
- 5.0.0
-
variant_delete
Removes fields or array elements from a variant at the given JSONPath locations. Multiple paths are applied left to right. Returns NULL ifvis NULL; NULL paths are skipped.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the first JSONPath identifying a deletion target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A string. Must be a constant.paths- additional JSONPath strings, applied afterpathin order. A string. Must be a constant.- Returns:
- Returns a column that evaluates to a variant.
- Since:
- 5.0.0
-
arrays_zip
Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.- Parameters:
e- the columns of arrays to be merged. Each is a column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
map_concat
Returns the union of all the given maps.- Parameters:
cols- the maps to merge. Each is a column that evaluates to a map.- Returns:
- Returns a column that evaluates to a map.
- Since:
- 2.4.0
-
callUDF
Call an user-defined function.- Parameters:
udfName- (undocumented)cols- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.5.0
-
call_udf
Call an user-defined function. Example:import org.apache.spark.sql._ val df = Seq(("id1", 1), ("id2", 4), ("id3", 5)).toDF("id", "value") val spark = df.sparkSession spark.udf.register("simpleUDF", (v: Int) => v * v) df.select($"id", call_udf("simpleUDF", $"value"))- Parameters:
udfName- (undocumented)cols- (undocumented)- Returns:
- (undocumented)
- Since:
- 3.2.0
-
call_function
Call a SQL function.- Parameters:
funcName- function name that follows the SQL identifier syntax (can be quoted, can be qualified)cols- the expression parameters of function- Returns:
- (undocumented)
- Since:
- 3.5.0
-
col
Returns aColumnbased on the given column name.- Parameters:
colName- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
column
Returns aColumnbased on the given column name. Alias ofcol(java.lang.String).- Parameters:
colName- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
lit
Creates aColumnof literal value.The passed in object is returned directly if it is already a
Column. If the object is a Scala Symbol, it is converted into aColumnalso. Otherwise, a newColumnis created to represent the literal value.- Parameters:
literal- (undocumented)- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
-
typedLit
Creates aColumnof literal value.An alias of
typedlit, and it is encouraged to usetypedlitdirectly.- Parameters:
literal- (undocumented)evidence$1- (undocumented)- Returns:
- Returns a column of the same type as the input.
- Since:
- 2.2.0
-
typedlit
Creates aColumnof literal value.The passed in object is returned directly if it is already a
Column. If the object is a Scala Symbol, it is converted into aColumnalso. Otherwise, a newColumnis created to represent the literal value. The difference between this function andlit(java.lang.Object)is that this function can handle parameterized scala types e.g.: List, Seq and Map.- Parameters:
literal- (undocumented)evidence$2- (undocumented)- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.2.0
- Note:
typedlitwill call expensive Scala reflection APIs.litis preferred if parameterized Scala types are not used.
-
asc
Returns a sort expression based on ascending order of the column.df.sort(asc("dept"), desc("age"))- Parameters:
columnName- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
asc_nulls_first
Returns a sort expression based on ascending order of the column, and null values return before non-null values.df.sort(asc_nulls_first("dept"), desc("age"))- Parameters:
columnName- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.1.0
-
asc_nulls_last
Returns a sort expression based on ascending order of the column, and null values appear after non-null values.df.sort(asc_nulls_last("dept"), desc("age"))- Parameters:
columnName- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.1.0
-
desc
Returns a sort expression based on the descending order of the column.df.sort(asc("dept"), desc("age"))- Parameters:
columnName- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
desc_nulls_first
Returns a sort expression based on the descending order of the column, and null values appear before non-null values.df.sort(asc("dept"), desc_nulls_first("age"))- Parameters:
columnName- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.1.0
-
desc_nulls_last
Returns a sort expression based on the descending order of the column, and null values appear after non-null values.df.sort(asc("dept"), desc_nulls_last("age"))- Parameters:
columnName- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.1.0
-
approxCountDistinct
Deprecated.Use approx_count_distinct. Since 2.1.0.- Parameters:
e- (undocumented)- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
approxCountDistinct
Deprecated.Use approx_count_distinct. Since 2.1.0.- Parameters:
columnName- (undocumented)- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
approxCountDistinct
Deprecated.Use approx_count_distinct. Since 2.1.0.- Parameters:
e- (undocumented)rsd- (undocumented)- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
approxCountDistinct
Deprecated.Use approx_count_distinct. Since 2.1.0.- Parameters:
columnName- (undocumented)rsd- (undocumented)- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
approx_count_distinct
Aggregate function: returns the approximate number of distinct items in a group.- Parameters:
e- The column to count distinct values in. A column of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 2.1.0
-
approx_count_distinct
Aggregate function: returns the approximate number of distinct items in a group.- Parameters:
columnName- The name of the column to count distinct values in. A column of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 2.1.0
-
approx_count_distinct
Aggregate function: returns the approximate number of distinct items in a group.- Parameters:
rsd- maximum relative standard deviation allowed (default = 0.05). A column that evaluates to a double. Must be a constant.e- (undocumented)- Returns:
- Returns a column that evaluates to a long.
- Since:
- 2.1.0
-
approx_count_distinct
Aggregate function: returns the approximate number of distinct items in a group.- Parameters:
rsd- maximum relative standard deviation allowed (default = 0.05). A column that evaluates to a double. Must be a constant.columnName- (undocumented)- Returns:
- Returns a column that evaluates to a long.
- Since:
- 2.1.0
-
avg
Aggregate function: returns the average of the values in a group.- Parameters:
e- The column to average. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a numeric.
- Since:
- 1.3.0
-
avg
Aggregate function: returns the average of the values in a group.- Parameters:
columnName- The name of the column to average. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a numeric.
- Since:
- 1.3.0
-
collect_list
Aggregate function: returns a list of objects with duplicates.- Parameters:
e- The column to collect. A column of any type.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.6.0
- Note:
- The function is non-deterministic because the order of collected results depends on the order of the rows which may be non-deterministic after a shuffle.
-
collect_list
Aggregate function: returns a list of objects with duplicates.- Parameters:
columnName- The name of the column to collect. A column of any type.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.6.0
- Note:
- The function is non-deterministic because the order of collected results depends on the order of the rows which may be non-deterministic after a shuffle.
-
collect_set
Aggregate function: returns a set of objects with duplicate elements eliminated.- Parameters:
e- The column to collect. A column of any type.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.6.0
- Note:
- The function is non-deterministic because the order of collected results depends on the order of the rows which may be non-deterministic after a shuffle.
-
collect_set
Aggregate function: returns a set of objects with duplicate elements eliminated.- Parameters:
columnName- The name of the column to collect. A column of any type.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.6.0
- Note:
- The function is non-deterministic because the order of collected results depends on the order of the rows which may be non-deterministic after a shuffle.
-
collect_union
Aggregate function: returns the distinct union of the elements of an array-typed column across rows.The aggregation buffer holds only the distinct elements, so its size is bounded by the element universe rather than by the number of input rows. Null elements are dropped by default (IGNORE NULLS), matching
collect_set. WithRESPECT NULLS, a single null element is kept, in which case this is equivalent toarray_distinct(flatten(collect_list(e))). TheRESPECT NULLSclause is only available through SQL (e.g.expr("collect_union(col) RESPECT NULLS")).- Parameters:
e- The array column to collect the union of. A column of type array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.3.0
- Note:
- The function is non-deterministic because the order of collected results depends on the order of the rows which may be non-deterministic after a shuffle.
-
collect_union
Aggregate function: returns the distinct union of the elements of an array-typed column across rows.- Parameters:
columnName- The name of the array column to collect the union of. A column of type array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.3.0
- Note:
- The function is non-deterministic because the order of collected results depends on the order of the rows which may be non-deterministic after a shuffle.
-
count_min_sketch
Returns a count-min sketch of a column with the given esp, confidence and seed. The result is an array of bytes, which can be deserialized to aCountMinSketchbefore usage. Count-min sketch is a probabilistic data structure used for cardinality estimation using sub-linear space.- Parameters:
e- The column to compute the sketch on. A column that evaluates to an integral, string or binary.eps- The relative error, must be positive. A column that evaluates to a numeric. Must be a constant.confidence- The confidence, must be positive and less than 1.0. A column that evaluates to a numeric. Must be a constant.seed- The random seed. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
count_min_sketch
Returns a count-min sketch of a column with the given esp, confidence and seed. The result is an array of bytes, which can be deserialized to aCountMinSketchbefore usage. Count-min sketch is a probabilistic data structure used for cardinality estimation using sub-linear space.- Parameters:
e- The column to compute the sketch on. A column that evaluates to an integral, string or binary.eps- The relative error, must be positive. A column that evaluates to a numeric. Must be a constant.confidence- The confidence, must be positive and less than 1.0. A column that evaluates to a numeric. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.0.0
-
corr
Aggregate function: returns the Pearson Correlation Coefficient for two columns.- Parameters:
column1- The first column. A column that evaluates to a numeric.column2- The second column. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
corr
Aggregate function: returns the Pearson Correlation Coefficient for two columns.- Parameters:
columnName1- The name of the first column. A column that evaluates to a numeric.columnName2- The name of the second column. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
count
Aggregate function: returns the number of items in a group.- Parameters:
e- The column to count. A column of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
count
Aggregate function: returns the number of items in a group.- Parameters:
columnName- The name of the column to count. A column of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
countDistinct
Aggregate function: returns the number of distinct items in a group.An alias of
count_distinct, and it is encouraged to usecount_distinctdirectly.- Parameters:
expr- The first column. A column of any type.exprs- Additional columns. A column of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
countDistinct
public static Column countDistinct(String columnName, scala.collection.immutable.Seq<String> columnNames) Aggregate function: returns the number of distinct items in a group.An alias of
count_distinct, and it is encouraged to usecount_distinctdirectly.- Parameters:
columnName- first column to compute on. A column of any type.columnNames- additional columns to compute on. Columns of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.3.0
-
count_distinct
Aggregate function: returns the number of distinct items in a group.- Parameters:
expr- first column to compute on. A column of any type.exprs- additional columns to compute on. Columns of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.2.0
-
covar_pop
Aggregate function: returns the population covariance for two columns.- Parameters:
column1- first column to calculate covariance. A column that evaluates to a numeric.column2- second column to calculate covariance. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 2.0.0
-
covar_pop
Aggregate function: returns the population covariance for two columns.- Parameters:
columnName1- first column to calculate covariance. A column that evaluates to a numeric.columnName2- second column to calculate covariance. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 2.0.0
-
covar_samp
Aggregate function: returns the sample covariance for two columns.- Parameters:
column1- first column to calculate covariance. A column that evaluates to a numeric.column2- second column to calculate covariance. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 2.0.0
-
covar_samp
Aggregate function: returns the sample covariance for two columns.- Parameters:
columnName1- first column to calculate covariance. A column that evaluates to a numeric.columnName2- second column to calculate covariance. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 2.0.0
-
first
Aggregate function: returns the first value in a group.The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
e- column to fetch the first value for. A column of any type.ignoreNulls- if first value is null then look for first non-null value. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 2.0.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
first
Aggregate function: returns the first value of a column in a group.The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
columnName- column to fetch the first value for. A column of any type.ignoreNulls- if first value is null then look for first non-null value. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 2.0.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
first
Aggregate function: returns the first value in a group.The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
e- column to fetch the first value for. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
first
Aggregate function: returns the first value of a column in a group.The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
columnName- column to fetch the first value for. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
first_value
Aggregate function: returns the first value in a group.- Parameters:
e- column to fetch the first value for. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
first_value
Aggregate function: returns the first value in a group.The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
e- column to fetch the first value for. A column of any type.ignoreNulls- if first value is null then look for first non-null value. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
grouping
Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set.- Parameters:
e- column to check if it is aggregated. A column of any type.- Returns:
- Returns a column that evaluates to a byte.
- Since:
- 2.0.0
-
grouping
Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set.- Parameters:
columnName- column to check if it is aggregated. A column of any type.- Returns:
- Returns a column that evaluates to a byte.
- Since:
- 2.0.0
-
grouping_id
Aggregate function: returns the level of grouping, equals to(grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)- Parameters:
cols- columns to check for. Columns of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 2.0.0
- Note:
- The list of columns should match with grouping columns exactly, or empty (means all the grouping columns).
-
grouping_id
Aggregate function: returns the level of grouping, equals to(grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)- Parameters:
colName- the name of the first grouping column. A column of any type.colNames- the names of the remaining grouping columns. Columns of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 2.0.0
- Note:
- The list of columns should match with grouping columns exactly.
-
hll_sketch_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.- Parameters:
e- the column to compute the sketch on. A column that evaluates to an integral, a string or a binary.lgConfigK- the log-base-2 of K, where K is the number of buckets or slots for the HllSketch. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_sketch_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.- Parameters:
e- the column to compute the sketch on. A column that evaluates to an integral, a string or a binary.lgConfigK- the log-base-2 of K, where K is the number of buckets or slots for the HllSketch. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_sketch_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.- Parameters:
columnName- the name of the column to compute the sketch on. A column that evaluates to an integral, a string or a binary.lgConfigK- the log-base-2 of K, where K is the number of buckets or slots for the HllSketch. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_sketch_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.- Parameters:
e- the column to compute the sketch on. A column that evaluates to an integral, a string or a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_sketch_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.- Parameters:
columnName- the name of the column to compute the sketch on. A column that evaluates to an integral, a string or a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_union_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance. Throws an exception if sketches have different lgConfigK values and allowDifferentLgConfigK is set to false.- Parameters:
e- the column containing the HllSketch instances to merge. A column that evaluates to a binary.allowDifferentLgConfigK- allow sketches with different lgConfigK values to be merged. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_union_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance. Throws an exception if sketches have different lgConfigK values and allowDifferentLgConfigK is set to false.- Parameters:
e- the column containing the HllSketch instances to merge. A column that evaluates to a binary.allowDifferentLgConfigK- allow sketches with different lgConfigK values to be merged. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_union_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance. Throws an exception if sketches have different lgConfigK values and allowDifferentLgConfigK is set to false.- Parameters:
columnName- the name of the column containing the HllSketch instances to merge. A column that evaluates to a binary.allowDifferentLgConfigK- allow sketches with different lgConfigK values to be merged. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_union_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance. Throws an exception if sketches have different lgConfigK values.- Parameters:
e- the column containing the HllSketch instances to merge. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_union_agg
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance. Throws an exception if sketches have different lgConfigK values.- Parameters:
columnName- the name of the column containing the HllSketch instances to merge. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
kurtosis
Aggregate function: returns the kurtosis of the values in a group.- Parameters:
e- the column to compute the kurtosis on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
kurtosis
Aggregate function: returns the kurtosis of the values in a group.- Parameters:
columnName- the name of the column to compute the kurtosis on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
last
Aggregate function: returns the last value in a group.The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
e- the column to take the last value from. A column of any type.ignoreNulls- if true, returns the last non-null value; if all values are null, null is returned. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 2.0.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
last
Aggregate function: returns the last value of the column in a group.The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
columnName- the name of the column to take the last value from. A column of any type.ignoreNulls- if true, returns the last non-null value; if all values are null, null is returned. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 2.0.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
last
Aggregate function: returns the last value in a group.The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
e- column to fetch the last value for. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
last
Aggregate function: returns the last value of the column in a group.The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
columnName- (undocumented)- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
last_value
Aggregate function: returns the last value in a group.- Parameters:
e- column to fetch the last value for. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
last_value
Aggregate function: returns the last value in a group.The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
- Parameters:
e- column to fetch the last value for. A column of any type.ignoreNulls- whether to skip null values. A column that evaluates to a boolean.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
- Note:
- The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.
-
make_time
Create time from hour, minute and second fields. For invalid inputs it will throw an error.- Parameters:
hour- the hour to represent, from 0 to 23. A column that evaluates to an integer.minute- the minute to represent, from 0 to 59. A column that evaluates to an integer.second- the second to represent, from 0 to 59.999999. A column that evaluates to a decimal.- Returns:
- Returns a column that evaluates to a time.
- Since:
- 4.1.0
-
mode
Aggregate function: returns the most frequent value in a group.- Parameters:
e- target column to compute on. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.4.0
-
mode
Aggregate function: returns the most frequent value in a group.When multiple values have the same greatest frequency then either any of values is returned if deterministic is false or is not defined, or the lowest value is returned if deterministic is true.
- Parameters:
e- target column to compute on. A column of any type.deterministic- if there are multiple equally-frequent results then return the lowest. A boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
max
Aggregate function: returns the maximum value of the expression in a group.- Parameters:
e- the target column on which the maximum value is computed. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
-
max
Aggregate function: returns the maximum value of the column in a group.- Parameters:
columnName- (undocumented)- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
-
max_by
Aggregate function: returns the value associated with the maximum value of ord.- Parameters:
e- the column representing the values to be returned. A column of any type.ord- the column that needs to be maximized. A column of any orderable type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.3.0
- Note:
- The function is non-deterministic so the output order can be different for those associated
the same values of
e.
-
max_by
Aggregate function: returns an array of values associated with the topkvalues oford.The result array contains values in descending order by their associated ordering values. Returns null if there are no non-null ordering values.
- Parameters:
e- the column representing the values to be returned. A column of any type.ord- the column that needs to be maximized. A column of any orderable type.k- the number of top values to return. An integer. Must be a constant.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.2.0
- Note:
- The function is non-deterministic because the order of collected results depends on the
order of the rows which may be non-deterministic after a shuffle when there are ties in the
ordering expression., The maximum value of
kis 100000.
-
max_by
Aggregate function: returns an array of values associated with the topkvalues oford.The result array contains values in descending order by their associated ordering values. Returns null if there are no non-null ordering values.
- Parameters:
e- the column representing the values to be returned. A column of any type.ord- the column that needs to be maximized. A column of any orderable type.k- the number of top values to return. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.2.0
- Note:
- The function is non-deterministic because the order of collected results depends on the
order of the rows which may be non-deterministic after a shuffle when there are ties in the
ordering expression., The maximum value of
kis 100000.
-
mean
Aggregate function: returns the average of the values in a group. Alias for avg.- Parameters:
e- target column to compute on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
mean
Aggregate function: returns the average of the values in a group. Alias for avg.- Parameters:
columnName- (undocumented)- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
median
Aggregate function: returns the median of the values in a group.- Parameters:
e- target column to compute on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.4.0
-
min
Aggregate function: returns the minimum value of the expression in a group.- Parameters:
e- the target column on which the minimum value is computed. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
-
min
Aggregate function: returns the minimum value of the column in a group.- Parameters:
columnName- the name of the column on which the minimum value is computed. A column of an orderable type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
-
min_by
Aggregate function: returns the value associated with the minimum value of ord.- Parameters:
e- the column representing the values that will be returned. A column of any type.ord- the column that needs to be minimized. A column of an orderable type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.3.0
- Note:
- The function is non-deterministic so the output order can be different for those associated
the same values of
e.
-
min_by
Aggregate function: returns an array of values associated with the bottomkvalues oford.The result array contains values in ascending order by their associated ordering values. Returns null if there are no non-null ordering values.
- Parameters:
e- the column representing the values that will be returned. A column of any type.ord- the column that needs to be minimized. A column of an orderable type.k- the number of bottom values to return. An integer. Must be a constant.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.2.0
- Note:
- The function is non-deterministic because the order of collected results depends on the
order of the rows which may be non-deterministic after a shuffle when there are ties in the
ordering expression., The maximum value of
kis 100000.
-
min_by
Aggregate function: returns an array of values associated with the bottomkvalues oford.The result array contains values in ascending order by their associated ordering values. Returns null if there are no non-null ordering values.
- Parameters:
e- the column representing the values that will be returned. A column of any type.ord- the column that needs to be minimized. A column of an orderable type.k- the number of bottom values to return. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.2.0
- Note:
- The function is non-deterministic because the order of collected results depends on the
order of the rows which may be non-deterministic after a shuffle when there are ties in the
ordering expression., The maximum value of
kis 100000.
-
percentile
Aggregate function: returns the exact percentile(s) of numeric columnexprat the given percentage(s) with value range in [0.0, 1.0].- Parameters:
e- the column to compute the percentile on. A column that evaluates to a numeric or interval.percentage- the percentage in decimal, between 0.0 and 1.0. A column that evaluates to a numeric or an array. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
percentile
Aggregate function: returns the exact percentile(s) of numeric columnexprat the given percentage(s) with value range in [0.0, 1.0].- Parameters:
e- the column to compute the percentile on. A column that evaluates to a numeric or interval.percentage- the percentage in decimal, between 0.0 and 1.0. A column that evaluates to a numeric or an array. Must be a constant.frequency- the positive frequency with which to weight each value. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
percentile_approx
Aggregate function: returns the approximatepercentileof the numeric columncolwhich is the smallest value in the orderedcolvalues (sorted from least to greatest) such that no more thanpercentageofcolvalues is less than the value or equal to that value.If percentage is an array, each value must be between 0.0 and 1.0. If it is a single floating point value, it must be between 0.0 and 1.0.
The accuracy parameter is a positive numeric literal which controls approximation accuracy at the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the relative error of the approximation.
- Parameters:
e- the column to compute the approximate percentile on. A column that evaluates to a numeric or interval.percentage- the percentage in decimal, between 0.0 and 1.0. A column that evaluates to a numeric or an array. Must be a constant.accuracy- a positive numeric literal that controls approximation accuracy at the cost of memory. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.1.0
-
approx_percentile
Aggregate function: returns the approximatepercentileof the numeric columncolwhich is the smallest value in the orderedcolvalues (sorted from least to greatest) such that no more thanpercentageofcolvalues is less than the value or equal to that value.If percentage is an array, each value must be between 0.0 and 1.0. If it is a single floating point value, it must be between 0.0 and 1.0.
The accuracy parameter is a positive numeric literal which controls approximation accuracy at the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the relative error of the approximation.
- Parameters:
e- the column to compute the approximate percentile on. A column that evaluates to a numeric or interval.percentage- the percentage in decimal, between 0.0 and 1.0. A column that evaluates to a numeric or an array. Must be a constant.accuracy- a positive numeric literal that controls approximation accuracy at the cost of memory. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
product
Aggregate function: returns the product of all numerical elements in a group.- Parameters:
e- the column to compute the product on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.2.0
-
skewness
Aggregate function: returns the skewness of the values in a group.- Parameters:
e- the column to compute the skewness on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
skewness
Aggregate function: returns the skewness of the values in a group.- Parameters:
columnName- the name of the column to compute the skewness on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
std
Aggregate function: alias forstddev_samp.- Parameters:
e- the column to compute the standard deviation on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
stddev
Aggregate function: alias forstddev_samp.- Parameters:
e- the column to compute the standard deviation on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
stddev
Aggregate function: alias forstddev_samp.- Parameters:
columnName- the name of the column to compute the standard deviation on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
stddev_samp
Aggregate function: returns the sample standard deviation of the expression in a group.- Parameters:
e- the column to compute the sample standard deviation on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
stddev_samp
Aggregate function: returns the sample standard deviation of the expression in a group.- Parameters:
columnName- Name of the column to compute the sample standard deviation on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
stddev_pop
Aggregate function: returns the population standard deviation of the expression in a group.- Parameters:
e- The column to compute the population standard deviation on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
stddev_pop
Aggregate function: returns the population standard deviation of the expression in a group.- Parameters:
columnName- Name of the column to compute the population standard deviation on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
sum
Aggregate function: returns the sum of all values in the expression.- Parameters:
e- The column to sum. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a numeric or interval.
- Since:
- 1.3.0
-
sum
Aggregate function: returns the sum of all values in the given column.- Parameters:
columnName- Name of the column to sum. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a numeric or interval.
- Since:
- 1.3.0
-
sumDistinct
Deprecated.Use sum_distinct. Since 3.2.0.Aggregate function: returns the sum of distinct values in the expression.- Parameters:
e- (undocumented)- Returns:
- Returns a column that evaluates to a numeric or interval.
- Since:
- 1.3.0
-
sumDistinct
Deprecated.Use sum_distinct. Since 3.2.0.Aggregate function: returns the sum of distinct values in the expression.- Parameters:
columnName- (undocumented)- Returns:
- Returns a column that evaluates to a numeric or interval.
- Since:
- 1.3.0
-
sum_distinct
Aggregate function: returns the sum of distinct values in the expression.- Parameters:
e- The column to sum distinct values of. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a numeric or interval.
- Since:
- 3.2.0
-
theta_intersection_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by intersecting the Datasketches ThetaSketch instances in the input column via a Datasketches Intersection instance.- Parameters:
e- The column of Datasketches ThetaSketch instances to intersect. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_intersection_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by intersecting the Datasketches ThetaSketch instances in the input volumn via a Datasketches Intersection instance.- Parameters:
columnName- Name of the column of Datasketches ThetaSketch instances to intersect. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_sketch_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with thelgNomEntriesnominal entries.- Parameters:
e- The column to build the ThetaSketch from. A column that evaluates to a numeric, string, binary or array.lgNomEntries- The log-base-2 of nominal entries, which is the size of the sketch (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_sketch_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with thelgNomEntriesnominal entries.- Parameters:
e- The column to build the ThetaSketch from. A column that evaluates to a numeric, string, binary or array.lgNomEntries- The log-base-2 of nominal entries, which is the size of the sketch (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_sketch_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with thelgNomEntriesnominal entries.- Parameters:
columnName- Name of the column to build the ThetaSketch from. A column that evaluates to a numeric, string, binary or array.lgNomEntries- The log-base-2 of nominal entries, which is the size of the sketch (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_sketch_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the default value of 12 forlgNomEntries.- Parameters:
e- The column to build the ThetaSketch from. A column that evaluates to a numeric, string, binary or array.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_sketch_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the default value of 12 forlgNomEntries.- Parameters:
columnName- Name of the column to build the ThetaSketch from. A column that evaluates to a numeric, string, binary or array.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_union_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer.- Parameters:
e- The column containing binary ThetaSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union operation (must be between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_union_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer.- Parameters:
e- The column containing binary ThetaSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union operation (must be between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_union_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer.- Parameters:
columnName- The name of the column containing binary ThetaSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union operation (must be between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_union_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance. It is configured with the default value of 12 forlgNomEntries.- Parameters:
e- The column containing binary ThetaSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_union_agg
Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance. It is configured with the default value of 12 forlgNomEntries.- Parameters:
columnName- The name of the column containing binary ThetaSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
tuple_intersection_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
e- The column containing binary TupleSketch representations. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
e- The column containing binary TupleSketch representations. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
columnName- The name of the column containing binary TupleSketch representations. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. It is configured with the default mode of 'sum'.- Parameters:
e- The column containing binary TupleSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. It is configured with the default mode of 'sum'.- Parameters:
columnName- The name of the column containing binary TupleSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
e- The column containing binary TupleSketch representations. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
e- The column containing binary TupleSketch representations. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
columnName- The name of the column containing binary TupleSketch representations. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. It is configured with the default mode of 'sum'.- Parameters:
e- The column containing binary TupleSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance. It is configured with the default mode of 'sum'.- Parameters:
columnName- The name of the column containing binary TupleSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_double
public static Column tuple_sketch_agg_double(Column key, Column summary, Column lgNomEntries, Column mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode. The mode parameter specifies the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
key- the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summary- the summary values against which mode aggregations occur. A column that evaluates to a numeric.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.mode- the summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_double
public static Column tuple_sketch_agg_double(Column key, Column summary, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode. The mode parameter specifies the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
key- the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summary- the summary values against which mode aggregations occur. A column that evaluates to a numeric.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.mode- the summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_double
public static Column tuple_sketch_agg_double(String keyColumnName, String summaryColumnName, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode. The mode parameter specifies the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
keyColumnName- the name of the column containing the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summaryColumnName- the name of the column containing the summary values against which mode aggregations occur. A column that evaluates to a numeric.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.mode- the summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries. It uses the default mode of 'sum'.- Parameters:
key- the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summary- the summary values against which mode aggregations occur. A column that evaluates to a numeric.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_double
public static Column tuple_sketch_agg_double(String keyColumnName, String summaryColumnName, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries. It uses the default mode of 'sum'.- Parameters:
keyColumnName- the name of the column containing the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summaryColumnName- the name of the column containing the summary values against which mode aggregations occur. A column that evaluates to a numeric.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns. It uses the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
key- the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summary- the summary values against which mode aggregations occur. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns. It uses the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
keyColumnName- the name of the column containing the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summaryColumnName- the name of the column containing the summary values against which mode aggregations occur. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_integer
public static Column tuple_sketch_agg_integer(Column key, Column summary, Column lgNomEntries, Column mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode. The mode parameter specifies the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
key- the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summary- the summary values against which mode aggregations occur. A column that evaluates to an integral.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.mode- the summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_integer
public static Column tuple_sketch_agg_integer(Column key, Column summary, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode. The mode parameter specifies the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
key- the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summary- the summary values against which mode aggregations occur. A column that evaluates to an integral.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.mode- the summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_integer
public static Column tuple_sketch_agg_integer(String keyColumnName, String summaryColumnName, int lgNomEntries, String mode) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries and aggregation mode. The mode parameter specifies the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
keyColumnName- the name of the column containing the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summaryColumnName- the name of the column containing the summary values against which mode aggregations occur. A column that evaluates to an integral.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.mode- the summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries. It uses the default mode of 'sum'.- Parameters:
key- the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summary- the summary values against which mode aggregations occur. A column that evaluates to an integral.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_integer
public static Column tuple_sketch_agg_integer(String keyColumnName, String summaryColumnName, int lgNomEntries) Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with thelgNomEntriesnominal entries. It uses the default mode of 'sum'.- Parameters:
keyColumnName- the name of the column containing the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summaryColumnName- the name of the column containing the summary values against which mode aggregations occur. A column that evaluates to an integral.lgNomEntries- the log-base-2 of nominal entries (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns. It uses the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
key- the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summary- the summary values against which mode aggregations occur. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns. It uses the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
keyColumnName- the name of the column containing the key values against which unique counting occurs. A column that evaluates to an array, a binary, a numeric, or a string.summaryColumnName- the name of the column containing the summary values against which mode aggregations occur. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
e- the column containing binary TupleSketch representations to union. A column that evaluates to a binary.lgNomEntries- the log-base-2 of nominal entries for the union buffer (must be between 4 and 26). A column that evaluates to an integral. Must be a constant.mode- the summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
e- The input column containing binary TupleSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.mode- The summary mode: one of "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
columnName- The name of the input column containing binary TupleSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.mode- The summary mode: one of "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
e- The input column containing binary TupleSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
columnName- The name of the input column containing binary TupleSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
e- The input column containing binary TupleSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_double
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
columnName- The name of the input column containing binary TupleSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
e- The input column containing binary TupleSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.mode- The summary mode: one of "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
e- The input column containing binary TupleSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.mode- The summary mode: one of "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
columnName- The name of the input column containing binary TupleSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.mode- The summary mode: one of "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
e- The input column containing binary TupleSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
columnName- The name of the input column containing binary TupleSketch representations. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
e- The input column containing binary TupleSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_agg_integer
Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
columnName- The name of the input column containing binary TupleSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
kll_sketch_agg_bigint
Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column. The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).- Parameters:
e- The input column containing the values to aggregate. A column that evaluates to an integral.k- The parameter that controls the size and accuracy of the sketch. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_bigint
Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column. The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).- Parameters:
e- The input column containing the values to aggregate. A column that evaluates to an integral.k- The parameter that controls the size and accuracy of the sketch. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_bigint
Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column. The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).- Parameters:
columnName- The column containing bigint values to aggregate. A column that evaluates to an integral.k- The k parameter that controls size and accuracy (default 200, range 8-65535). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_bigint
Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column with default k value of 200.- Parameters:
e- The column containing bigint values to aggregate. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_bigint
Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column with default k value of 200.- Parameters:
columnName- The column containing bigint values to aggregate. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_float
Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column. The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).- Parameters:
e- The column containing float values to aggregate. A column that evaluates to a float.k- The k parameter that controls size and accuracy (default 200, range 8-65535). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_float
Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column. The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).- Parameters:
e- The column containing float values to aggregate. A column that evaluates to a numeric.k- The k parameter that controls size and accuracy (default 200, range 8-65535). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_float
Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column. The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).- Parameters:
columnName- The column containing float values to aggregate. A column that evaluates to a numeric.k- The k parameter that controls size and accuracy (default 200, range 8-65535). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_float
Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column with default k value of 200.- Parameters:
e- The column containing float values to aggregate. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_float
Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column with default k value of 200.- Parameters:
columnName- The column containing float values to aggregate. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_double
Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column. The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).- Parameters:
e- The column containing double values to aggregate. A column that evaluates to a float or double.k- The k parameter that controls size and accuracy (default 200, range 8-65535). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_double
Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column. The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).- Parameters:
e- The column containing double values to aggregate. A column that evaluates to a numeric.k- The k parameter that controls size and accuracy (default 200, range 8-65535). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_double
Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column. The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).- Parameters:
columnName- The column containing double values to aggregate. A column that evaluates to a numeric.k- The k parameter that controls size and accuracy (default 200, range 8-65535). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_double
Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column with default k value of 200.- Parameters:
e- The column containing double values to aggregate. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_agg_double
Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column with default k value of 200.- Parameters:
columnName- The column containing double values to aggregate. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_merge_agg_bigint
Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch. The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
e- The column containing binary KllLongsSketch representations to merge. A column that evaluates to a binary.k- The k parameter that controls size and accuracy of the merged sketch (range 8-65535). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_bigint
Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch. The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
e- The column containing binary KllLongsSketch representations to merge. A column that evaluates to a binary.k- The k parameter that controls size and accuracy of the merged sketch (range 8-65535). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_bigint
Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch. The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
columnName- The column containing binary KllLongsSketch representations. A column that evaluates to a binary.k- The k parameter that controls size and accuracy (range 8-65535). A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_bigint
Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch. If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
e- The column containing binary KllLongsSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_bigint
Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch. If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
columnName- The column containing binary KllLongsSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_float
Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch. The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
e- The column containing binary KllFloatsSketch representations. A column that evaluates to a binary.k- The k parameter that controls size and accuracy (range 8-65535). A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_float
Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch. The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
e- The column containing binary KllFloatsSketch representations. A column that evaluates to a binary.k- The k parameter that controls size and accuracy (range 8-65535). A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_float
Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch. The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
columnName- The column containing binary KllFloatsSketch representations. A column that evaluates to a binary.k- The k parameter that controls size and accuracy (range 8-65535). A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_float
Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch. If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
e- The column containing binary KllFloatsSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_float
Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch. If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
columnName- The column containing binary KllFloatsSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_double
Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch. The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
e- The column containing binary KllDoublesSketch representations. A column that evaluates to a binary.k- The k parameter that controls size and accuracy (range 8-65535). A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_double
Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch. The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
e- The column containing binary KllDoublesSketch representations. A column that evaluates to a binary.k- The k parameter that controls size and accuracy (range 8-65535). A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_double
Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch. The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
columnName- The column containing binary KllDoublesSketch representations. A column that evaluates to a binary.k- The k parameter that controls size and accuracy (range 8-65535). A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_double
Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch. If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
e- The column containing binary KllDoublesSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
kll_merge_agg_double
Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch. If k is not specified, the merged sketch adopts the k value from the first input sketch.- Parameters:
columnName- The column containing binary KllDoublesSketch representations. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.2
-
listagg
Aggregate function: returns the concatenation of non-null input values.- Parameters:
e- The target column to compute on. A column that evaluates to a string or binary.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
listagg
Aggregate function: returns the concatenation of non-null input values, separated by the delimiter.- Parameters:
e- The target column to compute on. A column that evaluates to a string or binary.delimiter- The delimiter used to separate the values. A column that evaluates to a string or binary. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
listagg_distinct
Aggregate function: returns the concatenation of distinct non-null input values.- Parameters:
e- the column to compute on. A column that evaluates to a string or binary.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
listagg_distinct
Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter.- Parameters:
e- the column to compute on. A column that evaluates to a string or binary.delimiter- the delimiter to separate the values. A column that evaluates to a string or binary. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
string_agg
Aggregate function: returns the concatenation of non-null input values. Alias forlistagg.- Parameters:
e- the column to compute on. A column that evaluates to a string or binary.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
string_agg
Aggregate function: returns the concatenation of non-null input values, separated by the delimiter. Alias forlistagg.- Parameters:
e- the column to compute on. A column that evaluates to a string or binary.delimiter- the delimiter to separate the values. A column that evaluates to a string or binary. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
string_agg_distinct
Aggregate function: returns the concatenation of distinct non-null input values. Alias forlistagg.- Parameters:
e- the column to compute on. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
string_agg_distinct
Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter. Alias forlistagg.- Parameters:
e- the column to compute on. A column that evaluates to a string or binary.delimiter- the delimiter to separate the values. A column that evaluates to a string or binary. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
variance
Aggregate function: alias forvar_samp.- Parameters:
e- the column to compute on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
variance
Aggregate function: alias forvar_samp.- Parameters:
columnName- (undocumented)- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
var_samp
Aggregate function: returns the unbiased variance of the values in a group.- Parameters:
e- the column to compute on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
var_samp
Aggregate function: returns the unbiased variance of the values in a group.- Parameters:
columnName- (undocumented)- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
var_pop
Aggregate function: returns the population variance of the values in a group.- Parameters:
e- the column to compute on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
var_pop
Aggregate function: returns the population variance of the values in a group.- Parameters:
columnName- (undocumented)- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
regr_avgx
Aggregate function: returns the average of the independent variable for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.- Parameters:
y- the dependent variable. A column that evaluates to a numeric.x- the independent variable. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
regr_avgy
Aggregate function: returns the average of the dependent variable for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.- Parameters:
y- the dependent variable. A column that evaluates to a numeric.x- the independent variable. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
regr_count
Aggregate function: returns the number of non-null number pairs in a group, whereyis the dependent variable andxis the independent variable.- Parameters:
y- the dependent variable. A column that evaluates to a numeric.x- the independent variable. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
regr_intercept
Aggregate function: returns the intercept of the univariate linear regression line for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.- Parameters:
y- The dependent variable. A column that evaluates to a numeric.x- The independent variable. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
regr_r2
Aggregate function: returns the coefficient of determination for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.- Parameters:
y- The dependent variable. A column that evaluates to a numeric.x- The independent variable. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
regr_slope
Aggregate function: returns the slope of the linear regression line for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.- Parameters:
y- The dependent variable. A column that evaluates to a numeric.x- The independent variable. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
regr_sxx
Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(x) for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.- Parameters:
y- The dependent variable. A column that evaluates to a numeric.x- The independent variable. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
regr_sxy
Aggregate function: returns REGR_COUNT(y, x) * COVAR_POP(y, x) for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.- Parameters:
y- The dependent variable. A column that evaluates to a numeric.x- The independent variable. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
regr_syy
Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(y) for non-null pairs in a group, whereyis the dependent variable andxis the independent variable.- Parameters:
y- The dependent variable. A column that evaluates to a numeric.x- The independent variable. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
any_value
Aggregate function: returns some value ofefor a group of rows.- Parameters:
e- The column to return some value from. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
any_value
Aggregate function: returns some value ofefor a group of rows. IfignoreNullsis true, returns only non-null values.- Parameters:
e- The column to return some value from. A column of any type.ignoreNulls- If true, returns only non-null values. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
count_if
Aggregate function: returns the number ofTRUEvalues for the expression.- Parameters:
e- The expression to count TRUE values of. A column that evaluates to a boolean.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
current_time
Returns the current time at the start of query evaluation. Note that the result will contain 6 fractional digits of seconds.- Returns:
- A time. Returns a column that evaluates to a time.
- Since:
- 4.1.0
-
current_time
Returns the current time at the start of query evaluation.- Parameters:
precision- An integer literal in the range [0..6], indicating how many fractional digits of seconds to include in the result. A column that evaluates to an integer. Must be a constant.- Returns:
- A time. Returns a column that evaluates to a time.
- Since:
- 4.1.0
-
histogram_numeric
Aggregate function: computes a histogram on numeric 'expr' using nb bins. The return value is an array of (x,y) pairs representing the centers of the histogram's bins. As the value of 'nb' is increased, the histogram approximation gets finer-grained, but may yield artifacts around outliers. In practice, 20-40 histogram bins appear to work well, with more bins being required for skewed or smaller datasets. Note that this function creates a histogram with non-uniform bin widths. It offers no guarantees in terms of the mean-squared-error of the histogram, but in practice is comparable to the histograms produced by the R/S-Plus statistical computing packages. Note: the output type of the 'x' field in the return value is propagated from the input value consumed in the aggregate function.- Parameters:
e- The column to compute the histogram on. A column that evaluates to a numeric.nBins- The number of histogram bins. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.5.0
-
every
Aggregate function: returns true if all values ofeare true.- Parameters:
e- The expression to evaluate. A column that evaluates to a boolean.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
bool_and
Aggregate function: returns true if all values ofeare true.- Parameters:
e- The expression to evaluate. A column that evaluates to a boolean.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
some
Aggregate function: returns true if at least one value ofeis true.- Parameters:
e- The expression to evaluate. A column that evaluates to a boolean.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
any
Aggregate function: returns true if at least one value ofeis true.- Parameters:
e- The expression to evaluate. A column that evaluates to a boolean.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
bool_or
Aggregate function: returns true if at least one value ofeis true.- Parameters:
e- column to check if at least one value is true. A column that evaluates to a boolean.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
bit_and
Aggregate function: returns the bitwise AND of all non-null input values, or null if none.- Parameters:
e- target column to compute on. A column that evaluates to an integral.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
bit_or
Aggregate function: returns the bitwise OR of all non-null input values, or null if none.- Parameters:
e- target column to compute on. A column that evaluates to an integral.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
bit_xor
Aggregate function: returns the bitwise XOR of all non-null input values, or null if none.- Parameters:
e- target column to compute on. A column that evaluates to an integral.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
counter_diff
Window function: computes the differences between consecutive cumulative counter values in a time series, thereby converting the counter from the cumulative to the delta format.Gracefully handles counter resets by returning NULL. Counter resets are detected when the counter value decreases.
Use the PARTITION BY clause of the window to separate independent counters. This is done by specifying all columns which uniquely identify a time series. These are typically the counter name and any attributes tied to the counter.
Use the ORDER BY clause of the window to order the observations by the associated timestamp in ascending order.
- Parameters:
value- A cumulative counter. Must be a numeric data type. Must be non-negative.- Returns:
- The difference between the current and previous counter value within the window partition, according to the order defined by the window's ORDER BY clause. Returns a column of the same type as the input.
- Since:
- 4.3.0
-
counter_diff
Window function: computes the differences between consecutive cumulative counter values in a time series, thereby converting the counter from the cumulative to the delta format.Gracefully handles counter resets by returning NULL. Counter resets are detected when the counter value decreases, or when the start time advances between rows.
Use the PARTITION BY clause of the window to separate independent counters. This is done by specifying all columns which uniquely identify a time series. These are typically the counter name and any attributes tied to the counter.
Use the ORDER BY clause of the window to order the observations by the associated timestamp in ascending order.
- Parameters:
value- A cumulative counter. Must be a numeric data type. Must be non-negative.startTime- A timestamp indicating when the counter was last set to zero. Used to signal counter resets.- Returns:
- The difference between the current and previous counter value within the window partition, according to the order defined by the window's ORDER BY clause. Returns a column of the same type as the input.
- Since:
- 4.3.0
-
cume_dist
Window function: returns the cumulative distribution of values within a window partition, i.e. the fraction of rows that are below the current row.N = total number of rows in the partition cumeDist(x) = number of values before (and including) x / N- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
dense_rank
Window function: returns the rank of rows within a window partition, without any gaps.The difference between rank and dense_rank is that denseRank leaves no gaps in ranking sequence when there are ties. That is, if you were ranking a competition using dense_rank and had three people tie for second place, you would say that all three were in second place and that the next person came in third. Rank would give me sequential numbers, making the person that came in third place (after the ties) would register as coming in fifth.
This is equivalent to the DENSE_RANK function in SQL.
- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.6.0
-
lag
Window function: returns the value that isoffsetrows before the current row, andnullif there is less thanoffsetrows before the current row. For example, anoffsetof one will return the previous row at any given point in the window partition.This is equivalent to the LAG function in SQL.
- Parameters:
e- the column to compute on. A column of any type.offset- number of rows to extend. A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
lag
Window function: returns the value that isoffsetrows before the current row, andnullif there is less thanoffsetrows before the current row. For example, anoffsetof one will return the previous row at any given point in the window partition.This is equivalent to the LAG function in SQL.
- Parameters:
columnName- name of column or expression.offset- number of rows to extend. A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
lag
Window function: returns the value that isoffsetrows before the current row, anddefaultValueif there is less thanoffsetrows before the current row. For example, anoffsetof one will return the previous row at any given point in the window partition.This is equivalent to the LAG function in SQL.
- Parameters:
columnName- name of column or expression.offset- number of rows to extend. A column that evaluates to an integer. Must be a constant.defaultValue- default value. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
lag
Window function: returns the value that isoffsetrows before the current row, anddefaultValueif there is less thanoffsetrows before the current row. For example, anoffsetof one will return the previous row at any given point in the window partition.This is equivalent to the LAG function in SQL.
- Parameters:
e- the column to compute on. A column of any type.offset- number of rows to extend. A column that evaluates to an integer. Must be a constant.defaultValue- default value. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
lag
Window function: returns the value that isoffsetrows before the current row, anddefaultValueif there is less thanoffsetrows before the current row.ignoreNullsdetermines whether null values of row are included in or eliminated from the calculation. For example, anoffsetof one will return the previous row at any given point in the window partition.This is equivalent to the LAG function in SQL.
- Parameters:
e- the column to compute on. A column of any type.offset- number of rows to extend. A column that evaluates to an integer. Must be a constant.defaultValue- default value. A column of any type.ignoreNulls- whether to ignore null values. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.2.0
-
lead
Window function: returns the value that isoffsetrows after the current row, andnullif there is less thanoffsetrows after the current row. For example, anoffsetof one will return the next row at any given point in the window partition.This is equivalent to the LEAD function in SQL.
- Parameters:
columnName- name of column or expression.offset- number of rows to extend. A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
lead
Window function: returns the value that isoffsetrows after the current row, andnullif there is less thanoffsetrows after the current row. For example, anoffsetof one will return the next row at any given point in the window partition.This is equivalent to the LEAD function in SQL.
- Parameters:
e- the column to compute on. A column of any type.offset- number of rows to extend. A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
lead
Window function: returns the value that isoffsetrows after the current row, anddefaultValueif there is less thanoffsetrows after the current row. For example, anoffsetof one will return the next row at any given point in the window partition.This is equivalent to the LEAD function in SQL.
- Parameters:
columnName- name of column or expression.offset- number of rows to extend. A column that evaluates to an integer. Must be a constant.defaultValue- default value. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
lead
Window function: returns the value that isoffsetrows after the current row, anddefaultValueif there is less thanoffsetrows after the current row. For example, anoffsetof one will return the next row at any given point in the window partition.This is equivalent to the LEAD function in SQL.
- Parameters:
e- the column to compute on. A column of any type.offset- number of rows to extend. A column that evaluates to an integer. Must be a constant.defaultValue- default value. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
lead
Window function: returns the value that isoffsetrows after the current row, anddefaultValueif there is less thanoffsetrows after the current row.ignoreNullsdetermines whether null values of row are included in or eliminated from the calculation. The default value ofignoreNullsis false. For example, anoffsetof one will return the next row at any given point in the window partition.This is equivalent to the LEAD function in SQL.
- Parameters:
e- The column to compute the lead value for. A column of any type.offset- Number of rows after the current row to look ahead. A column that evaluates to an integral. Must be a constant.defaultValue- Value to return when there are fewer thanoffsetrows after the current row. A column of any type. Must be a constant.ignoreNulls- Whether to skip null values when computing the result. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.2.0
-
nth_value
Window function: returns the value that is theoffsetth row of the window frame (counting from 1), andnullif the size of window frame is less thanoffsetrows.It will return the
offsetth non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.This is equivalent to the nth_value function in SQL.
- Parameters:
e- The column to extract the value from. A column of any type.offset- The 1-based row number within the window frame to use as the value. A column that evaluates to an integral. Must be a constant.ignoreNulls- Whether the nth value should skip nulls when determining which row to use. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.1.0
-
nth_value
Window function: returns the value that is theoffsetth row of the window frame (counting from 1), andnullif the size of window frame is less thanoffsetrows.This is equivalent to the nth_value function in SQL.
- Parameters:
e- The column to extract the value from. A column of any type.offset- The 1-based row number within the window frame to use as the value. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.1.0
-
ntile
Window function: returns the ntile group id (from 1 toninclusive) in an ordered window partition. For example, ifnis 4, the first quarter of the rows will get value 1, the second quarter will get 2, the third quarter will get 3, and the last quarter will get 4.This is equivalent to the NTILE function in SQL.
- Parameters:
n- The number of groups to divide the window partition into. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.4.0
-
percent_rank
Window function: returns the relative rank (i.e. percentile) of rows within a window partition.This is computed by:
(rank of row in its partition - 1) / (number of rows in the partition - 1)This is equivalent to the PERCENT_RANK function in SQL.
- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.6.0
-
rank
Window function: returns the rank of rows within a window partition.The difference between rank and dense_rank is that dense_rank leaves no gaps in ranking sequence when there are ties. That is, if you were ranking a competition using dense_rank and had three people tie for second place, you would say that all three were in second place and that the next person came in third. Rank would give me sequential numbers, making the person that came in third place (after the ties) would register as coming in fifth.
This is equivalent to the RANK function in SQL.
- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.4.0
-
row_number
Window function: returns a sequential number starting at 1 within a window partition.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.6.0
-
array
Creates a new array column. The input columns must all have the same data type.- Parameters:
cols- The columns to combine into an array. Each is a column of any type, and all must share the same data type.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.4.0
-
array
Creates a new array column. The input columns must all have the same data type.- Parameters:
colName- (undocumented)colNames- (undocumented)- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.4.0
-
map
Creates a new map column. The input columns must be grouped as key-value pairs, e.g. (key1, value1, key2, value2, ...). The key columns must all have the same data type, and can't be null. The value columns must all have the same data type.- Parameters:
cols- The columns grouped as key-value pairs (key1, value1, key2, value2, ...). Each is a column of any type; key columns must share a type and value columns must share a type.- Returns:
- Returns a column that evaluates to a map.
- Since:
- 2.0
-
named_struct
Creates a struct with the given field names and values.- Parameters:
cols- The field names and values grouped as pairs (name1, value1, name2, value2, ...). Names are columns that evaluate to a string; values are columns of any type.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 3.5.0
-
map_from_arrays
Creates a new map column. The array in the first column is used for keys. The array in the second column is used for values. All elements in the array for key should not be null.- Parameters:
keys- The array of keys for the map; elements must not be null. A column that evaluates to an array.values- The array of values for the map. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to a map.
- Since:
- 2.4
-
str_to_map
Creates a map after splitting the text into key/value pairs using delimiters. BothpairDelimandkeyValueDelimare treated as regular expressions.- Parameters:
text- The text to split into key/value pairs. A column that evaluates to a string.pairDelim- Delimiter used to split pairs, treated as a regular expression. A column that evaluates to a string.keyValueDelim- Delimiter used to split key and value, treated as a regular expression. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a map.
- Since:
- 3.5.0
-
str_to_map
Creates a map after splitting the text into key/value pairs using delimiters. ThepairDelimis treated as regular expressions.- Parameters:
text- The text to split into key/value pairs. A column that evaluates to a string.pairDelim- Delimiter used to split pairs, treated as a regular expression. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a map.
- Since:
- 3.5.0
-
str_to_map
Creates a map after splitting the text into key/value pairs using delimiters.- Parameters:
text- The text to split into key/value pairs. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a map.
- Since:
- 3.5.0
-
broadcast
Marks a DataFrame as small enough for use in broadcast joins.The following example marks the right DataFrame for broadcast hash join using
joinKey.// left and right are DataFrames left.join(broadcast(right), "joinKey")- Parameters:
df- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.5.0
-
coalesce
Returns the first column that is not null, or null if all inputs are null.For example,
coalesce(a, b, c)will return a if a is not null, or b if a is null and b is not null, or c if both a and b are null but c is not null.- Parameters:
e- the columns to work on. A column that evaluates to any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
-
input_file_name
Creates a string column for the file name of the current Spark task.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.6.0
-
isnan
Return true iff the column is NaN.- Parameters:
e- the column to check. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 1.6.0
-
isnull
Return true iff the column is null.- Parameters:
e- the column to check. A column that evaluates to any type.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 1.6.0
-
monotonicallyIncreasingId
Deprecated.Use monotonically_increasing_id(). Since 2.0.0.A column expression that generates monotonically increasing 64-bit integers.The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record number within each partition in the lower 33 bits. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.
As an example, consider a
DataFramewith two partitions, each with 3 records. This expression would return the following IDs:0, 1, 2, 8589934592 (1L << 33), 8589934593, 8589934594.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.4.0
-
monotonically_increasing_id
A column expression that generates monotonically increasing 64-bit integers.The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record number within each partition in the lower 33 bits. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.
As an example, consider a
DataFramewith two partitions, each with 3 records. This expression would return the following IDs:0, 1, 2, 8589934592 (1L << 33), 8589934593, 8589934594.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.6.0
-
nanvl
Returns col1 if it is not NaN, or col2 if col1 is NaN.Both inputs should be floating point columns (DoubleType or FloatType).
- Parameters:
col1- the first column to check. A column that evaluates to a numeric.col2- the column to return if the first is NaN. A column that evaluates to a numeric.- Returns:
- Returns a column of the same type as the first input.
- Since:
- 1.5.0
-
negate
Unary minus, i.e. negate the expression.// Select the amount column and negates all values. // Scala: df.select( -df("amount") ) // Java: df.select( negate(df.col("amount")) );- Parameters:
e- the column to negate. A column that evaluates to a numeric or interval.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
-
not
Inversion of boolean expression, i.e. NOT.// Scala: select rows that are not active (isActive === false) df.filter( !df("isActive") ) // Java: df.filter( not(df.col("isActive")) );- Parameters:
e- the column to invert. A column that evaluates to a boolean.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 1.3.0
-
rand
Generate a random column with independent and identically distributed (i.i.d.) samples uniformly distributed in [0.0, 1.0).- Parameters:
seed- the seed for the random generator.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
- Note:
- The function is non-deterministic in general case.
-
rand
Generate a random column with independent and identically distributed (i.i.d.) samples uniformly distributed in [0.0, 1.0).- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
- Note:
- The function is non-deterministic in general case.
-
randn
Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.- Parameters:
seed- the seed for the random generator.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
- Note:
- The function is non-deterministic in general case.
-
randn
Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
- Note:
- The function is non-deterministic in general case.
-
randstr
Returns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z. The string length must be a constant two-byte or four-byte integer (SMALLINT or INT, respectively).- Parameters:
length- the number of characters in the string to generate. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
randstr
Returns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z, with the chosen random seed. The string length must be a constant two-byte or four-byte integer (SMALLINT or INT, respectively).- Parameters:
length- the number of characters in the string to generate. A column that evaluates to an integral. Must be a constant.seed- the random seed to use. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
spark_partition_id
Partition ID.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.6.0
- Note:
- This is non-deterministic because it depends on data partitioning and task scheduling.
-
sqrt
Computes the square root of the specified float value.- Parameters:
e- the value to compute the square root of. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.3.0
-
sqrt
Computes the square root of the specified float value.- Parameters:
colName- the name of a numeric column to compute the square root of.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.5.0
-
try_add
Returns the sum ofleftandrightand the result is null on overflow. The acceptable input types are the same with the+operator.- Parameters:
left- the left operand. A column that evaluates to a numeric or interval.right- the right operand. A column that evaluates to a numeric or interval.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
try_avg
Returns the mean calculated from values of a group and the result is null on overflow.- Parameters:
e- the value to compute the mean of. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
try_divide
Returnsdividend/divisor. It always performs floating point division. Its result is always null ifdivisoris 0.- Parameters:
left- the dividend. A column that evaluates to a numeric or interval.right- the divisor. A column that evaluates to a numeric.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
try_mod
Returns the remainder ofdividend/divisor. Its result is always null ifdivisoris 0.- Parameters:
left- the dividend. A column that evaluates to a numeric.right- the divisor. A column that evaluates to a numeric.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
try_multiply
Returnsleft*rightand the result is null on overflow. The acceptable input types are the same with the*operator.- Parameters:
left- the multiplicand. A column that evaluates to a numeric or interval.right- the multiplier. A column that evaluates to a numeric or interval.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
try_subtract
Returnsleft-rightand the result is null on overflow. The acceptable input types are the same with the-operator.- Parameters:
left- the left operand. A column that evaluates to a numeric or interval.right- the right operand. A column that evaluates to a numeric or interval.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
try_sum
Returns the sum calculated from values of a group and the result is null on overflow.- Parameters:
e- the value to compute the sum of. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a numeric.
- Since:
- 3.5.0
-
struct
Creates a new struct column. If the input column is a column in aDataFrame, or a derived column expression that is named (i.e. aliased), its name would be retained as the StructField's name, otherwise, the newly generated StructField's name would be auto generated ascolwith a suffixindex + 1, i.e. col1, col2, col3, ...- Parameters:
cols- the columns to contain in the output struct. A column of any type.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 1.4.0
-
struct
Creates a new struct column that composes multiple input columns.- Parameters:
colName- the name of the first column to contain in the output struct.colNames- the names of the remaining columns to contain in the output struct.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 1.4.0
-
when
Evaluates a list of conditions and returns one of multiple possible result expressions. If otherwise is not defined at the end, null is returned for unmatched conditions.// Example: encoding gender string column into integer. // Scala: people.select(when(people("gender") === "male", 0) .when(people("gender") === "female", 1) .otherwise(2)) // Java: people.select(when(col("gender").equalTo("male"), 0) .when(col("gender").equalTo("female"), 1) .otherwise(2))- Parameters:
condition- the condition to evaluate. A column that evaluates to a boolean.value- the value to return when the condition is true. A literal value, or a column expression.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
bitwiseNOT
Deprecated.Use bitwise_not. Since 3.2.0.Computes bitwise NOT (~) of a number.- Parameters:
e- (undocumented)- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.4.0
-
bitwise_not
Computes bitwise NOT (~) of a number.- Parameters:
e- the target column to compute on. A column that evaluates to an integral.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.2.0
-
bit_count
Returns the number of bits that are set in the argument expr as an unsigned 64-bit integer, or NULL if the argument is NULL.- Parameters:
e- the target column to compute on. A column that evaluates to an integral or boolean.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
bit_get
Returns the value of the bit (0 or 1) at the specified position. The positions are numbered from right to left, starting at zero. The position argument cannot be negative.- Parameters:
e- the target column to compute on. A column that evaluates to an integral.pos- the bit position, numbered from right to left starting at zero. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to a byte.
- Since:
- 3.5.0
-
getbit
Returns the value of the bit (0 or 1) at the specified position. The positions are numbered from right to left, starting at zero. The position argument cannot be negative.- Parameters:
e- the target column to compute on. A column that evaluates to an integral.pos- the bit position, numbered from right to left starting at zero. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to a byte.
- Since:
- 3.5.0
-
expr
Parses the expression string into the column that it represents, similar toDataset.selectExpr(java.lang.String...).// get the number of words of each length df.groupBy(expr("length(word)")).count()- Parameters:
expr- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.5.0
-
abs
Computes the absolute value of a numeric value.- Parameters:
e- the value to compute the absolute value of. A column that evaluates to a numeric or interval.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.3.0
-
acos
- Parameters:
e- the value to compute the inverse cosine of. A column that evaluates to a double.- Returns:
- inverse cosine of
ein radians, as if computed byjava.lang.Math.acos. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
acos
- Parameters:
columnName- the value to compute the inverse cosine of.- Returns:
- inverse cosine of
columnName, as if computed byjava.lang.Math.acos. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
acosh
- Parameters:
e- the value to compute the inverse hyperbolic cosine of. A column that evaluates to a double.- Returns:
- inverse hyperbolic cosine of
e. Returns a column that evaluates to a double. - Since:
- 3.1.0
-
acosh
- Parameters:
columnName- the value to compute the inverse hyperbolic cosine of.- Returns:
- inverse hyperbolic cosine of
columnName. Returns a column that evaluates to a double. - Since:
- 3.1.0
-
asin
- Parameters:
e- the value to compute the inverse sine of. A column that evaluates to a double.- Returns:
- inverse sine of
ein radians, as if computed byjava.lang.Math.asin. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
asin
- Parameters:
columnName- the value to compute the inverse sine of.- Returns:
- inverse sine of
columnName, as if computed byjava.lang.Math.asin. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
asinh
- Parameters:
e- the value to compute the inverse hyperbolic sine of. A column that evaluates to a double.- Returns:
- inverse hyperbolic sine of
e. Returns a column that evaluates to a double. - Since:
- 3.1.0
-
asinh
- Parameters:
columnName- the value to compute the inverse hyperbolic sine of.- Returns:
- inverse hyperbolic sine of
columnName. Returns a column that evaluates to a double. - Since:
- 3.1.0
-
atan
- Parameters:
e- the value to compute the inverse tangent of. A column that evaluates to a double.- Returns:
- inverse tangent of
eas if computed byjava.lang.Math.atan. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atan
- Parameters:
columnName- the value to compute the inverse tangent of.- Returns:
- inverse tangent of
columnName, as if computed byjava.lang.Math.atan. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atan2
- Parameters:
y- coordinate on y-axis. A column that evaluates to a double.x- coordinate on x-axis. A column that evaluates to a double.- Returns:
- the theta component of the point (r, theta) in polar coordinates that
corresponds to the point (x, y) in Cartesian coordinates, as if computed by
java.lang.Math.atan2. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atan2
- Parameters:
y- coordinate on y-axisxName- coordinate on x-axis- Returns:
- the theta component of the point (r, theta) in polar coordinates that
corresponds to the point (x, y) in Cartesian coordinates, as if computed by
java.lang.Math.atan2. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atan2
- Parameters:
yName- coordinate on y-axisx- coordinate on x-axis- Returns:
- the theta component of the point (r, theta) in polar coordinates that
corresponds to the point (x, y) in Cartesian coordinates, as if computed by
java.lang.Math.atan2. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atan2
- Parameters:
yName- coordinate on y-axisxName- coordinate on x-axis- Returns:
- the theta component of the point (r, theta) in polar coordinates that
corresponds to the point (x, y) in Cartesian coordinates, as if computed by
java.lang.Math.atan2. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atan2
- Parameters:
y- coordinate on y-axisxValue- coordinate on x-axis- Returns:
- the theta component of the point (r, theta) in polar coordinates that
corresponds to the point (x, y) in Cartesian coordinates, as if computed by
java.lang.Math.atan2. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atan2
- Parameters:
yName- coordinate on y-axisxValue- coordinate on x-axis- Returns:
- the theta component of the point (r, theta) in polar coordinates that
corresponds to the point (x, y) in Cartesian coordinates, as if computed by
java.lang.Math.atan2. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atan2
- Parameters:
yValue- coordinate on y-axisx- coordinate on x-axis- Returns:
- the theta component of the point (r, theta) in polar coordinates that
corresponds to the point (x, y) in Cartesian coordinates, as if computed by
java.lang.Math.atan2. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atan2
- Parameters:
yValue- coordinate on y-axisxName- coordinate on x-axis- Returns:
- the theta component of the point (r, theta) in polar coordinates that
corresponds to the point (x, y) in Cartesian coordinates, as if computed by
java.lang.Math.atan2. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
atanh
- Parameters:
e- target column to compute on. A column that evaluates to a numeric.- Returns:
- inverse hyperbolic tangent of
e. Returns a column that evaluates to a double. - Since:
- 3.1.0
-
atanh
- Parameters:
columnName- target column to compute on.- Returns:
- inverse hyperbolic tangent of
columnName. Returns a column that evaluates to a double. - Since:
- 3.1.0
-
bin
An expression that returns the string representation of the binary value of the given long column. For example, bin("12") returns "1100".- Parameters:
e- target column to work on. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
bin
An expression that returns the string representation of the binary value of the given long column. For example, bin("12") returns "1100".- Parameters:
columnName- target column to work on.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
cbrt
Computes the cube-root of the given value.- Parameters:
e- target column to compute on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
cbrt
Computes the cube-root of the given column.- Parameters:
columnName- target column to compute on.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
ceil
Computes the ceiling of the given value ofetoscaledecimal places.- Parameters:
e- the value to compute the ceiling on. A column that evaluates to a numeric.scale- parameter to control the rounding behavior. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a long or decimal.
- Since:
- 3.3.0
-
ceil
Computes the ceiling of the given value ofeto 0 decimal places.- Parameters:
e- the value to compute the ceiling on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a long or decimal.
- Since:
- 1.4.0
-
ceil
Computes the ceiling of the given value ofcolumnNameto 0 decimal places.- Parameters:
columnName- the value to compute the ceiling on.- Returns:
- Returns a column that evaluates to a long or decimal.
- Since:
- 1.4.0
-
ceiling
Computes the ceiling of the given value ofetoscaledecimal places.- Parameters:
e- the value to compute the ceiling on. A column that evaluates to a numeric.scale- parameter to control the rounding behavior. A column that evaluates to an integer. Must be a constant.- Returns:
- Returns a column that evaluates to a long or decimal.
- Since:
- 3.5.0
-
ceiling
Computes the ceiling of the given value ofeto 0 decimal places.- Parameters:
e- the value to compute the ceiling on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a long or decimal.
- Since:
- 3.5.0
-
conv
Convert a number in a string column from one base to another.- Parameters:
num- a column to convert base for. A column that evaluates to a string.fromBase- from base number. A column that evaluates to an integer.toBase- to base number. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
cos
- Parameters:
e- angle in radians. A column that evaluates to a double.- Returns:
- cosine of the angle, as if computed by
java.lang.Math.cos. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
cos
- Parameters:
columnName- angle in radians. A column that evaluates to a double.- Returns:
- cosine of the angle, as if computed by
java.lang.Math.cos. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
cosh
- Parameters:
e- hyperbolic angle. A column that evaluates to a double.- Returns:
- hyperbolic cosine of the angle, as if computed by
java.lang.Math.cosh. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
cosh
- Parameters:
columnName- hyperbolic angle- Returns:
- hyperbolic cosine of the angle, as if computed by
java.lang.Math.cosh. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
cot
- Parameters:
e- angle in radians. A column that evaluates to a double.- Returns:
- cotangent of the angle. Returns a column that evaluates to a double.
- Since:
- 3.3.0
-
csc
- Parameters:
e- angle in radians. A column that evaluates to a double.- Returns:
- cosecant of the angle. Returns a column that evaluates to a double.
- Since:
- 3.3.0
-
e
Returns Euler's number.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
exp
Computes the exponential of the given value.- Parameters:
e- target column to compute on. A column that evaluates to a double.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
exp
Computes the exponential of the given column.- Parameters:
columnName- target column to compute on.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
expm1
Computes the exponential of the given value minus one.- Parameters:
e- column to calculate exponential for. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
expm1
Computes the exponential of the given column minus one.- Parameters:
columnName- column name to calculate exponential for. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
factorial
Computes the factorial of the given value.- Parameters:
e- a column to calculate factorial for. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.5.0
-
floor
Computes the floor of the given value ofetoscaledecimal places.- Parameters:
e- the target column to compute the floor on. A column that evaluates to a numeric.scale- the number of decimal places to control the rounding behavior. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a long or decimal.
- Since:
- 3.3.0
-
floor
Computes the floor of the given value ofeto 0 decimal places.- Parameters:
e- the target column to compute the floor on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a long or decimal.
- Since:
- 1.4.0
-
floor
Computes the floor of the given column value to 0 decimal places.- Parameters:
columnName- the target column name to compute the floor on. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a long or decimal.
- Since:
- 1.4.0
-
greatest
Returns the greatest value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.- Parameters:
exprs- columns to check for greatest value. A column that evaluates to any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
greatest
public static Column greatest(String columnName, scala.collection.immutable.Seq<String> columnNames) Returns the greatest value of the list of column names, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.- Parameters:
columnName- the first column name to check for greatest value. A column of a comparable type.columnNames- the remaining column names to check for greatest value. Columns of a comparable type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
hex
Computes hex value of the given column.- Parameters:
column- target column to work on. A column that evaluates to an integral, string or binary.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
unhex
Inverse of hex. Interprets each pair of characters as a hexadecimal number and converts to the byte representation of number.- Parameters:
column- target column to work on. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 1.5.0
-
hypot
Computessqrt(a^2^ + b^2^)without intermediate overflow or underflow.- Parameters:
l- a leg. A column that evaluates to a numeric.r- b leg. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
hypot
Computessqrt(a^2^ + b^2^)without intermediate overflow or underflow.- Parameters:
l- a leg. A column that evaluates to a numeric.rightName- b leg. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
hypot
Computessqrt(a^2^ + b^2^)without intermediate overflow or underflow.- Parameters:
leftName- a leg. A column that evaluates to a numeric.r- b leg. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
hypot
Computessqrt(a^2^ + b^2^)without intermediate overflow or underflow.- Parameters:
leftName- a leg. A column that evaluates to a numeric.rightName- b leg. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
hypot
Computessqrt(a^2^ + b^2^)without intermediate overflow or underflow.- Parameters:
l- a leg. A column that evaluates to a numeric.r- b leg. A column that evaluates to a numeric. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
hypot
Computessqrt(a^2^ + b^2^)without intermediate overflow or underflow.- Parameters:
leftName- The a leg of the triangle. A column that evaluates to a numeric.r- The b leg of the triangle. A column that evaluates to a numeric. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
hypot
Computessqrt(a^2^ + b^2^)without intermediate overflow or underflow.- Parameters:
l- The a leg of the triangle. A column that evaluates to a numeric. Must be a constant.r- The b leg of the triangle. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
hypot
Computessqrt(a^2^ + b^2^)without intermediate overflow or underflow.- Parameters:
l- The a leg of the triangle. A column that evaluates to a numeric. Must be a constant.rightName- The b leg of the triangle. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
least
Returns the least value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.- Parameters:
exprs- The values to be compared. Columns that evaluate to a comparable type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
least
Returns the least value of the list of column names, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.- Parameters:
columnName- The name of the first column to be compared. A column of a comparable type.columnNames- The names of the remaining columns to be compared. Columns of a comparable type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
ln
Computes the natural logarithm of the given value.- Parameters:
e- The value to compute the natural logarithm of. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
log
Computes the natural logarithm of the given value.- Parameters:
e- The value to compute the natural logarithm of. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
log
Computes the natural logarithm of the given column.- Parameters:
columnName- The name of the column to compute the natural logarithm of. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
log
Returns the first argument-base logarithm of the second argument.- Parameters:
base- The base of the logarithm. A column that evaluates to a numeric. Must be a constant.a- The value to compute the logarithm of. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
log
Returns the first argument-base logarithm of the second argument.- Parameters:
base- The base of the logarithm. A column that evaluates to a numeric. Must be a constant.columnName- The name of the column to compute the logarithm of. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
log10
Computes the logarithm of the given value in base 10.- Parameters:
e- The value to compute the base-10 logarithm of. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
log10
Computes the logarithm of the given value in base 10.- Parameters:
columnName- The name of the column to compute the base-10 logarithm of. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
log1p
Computes the natural logarithm of the given value plus one.- Parameters:
e- The value to compute the natural logarithm of the value plus one. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
log1p
Computes the natural logarithm of the given column plus one.- Parameters:
columnName- The name of the column to compute the natural logarithm of the value plus one. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
log2
Computes the logarithm of the given column in base 2.- Parameters:
expr- The value to compute the base-2 logarithm of. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.5.0
-
log2
Computes the logarithm of the given value in base 2.- Parameters:
columnName- a column to calculate logarithm for. A column that evaluates to a double.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.5.0
-
negative
Returns the negated value.- Parameters:
e- column to calculate negative value for. A column that evaluates to a numeric or interval.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
pi
Returns Pi.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
positive
Returns the value.- Parameters:
e- input value column. A column that evaluates to a numeric or interval.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
pow
Returns the value of the first argument raised to the power of the second argument.- Parameters:
l- the base number. A column that evaluates to a double.r- the exponent number. A column that evaluates to a double.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
pow
Returns the value of the first argument raised to the power of the second argument.- Parameters:
l- the base number. A column that evaluates to a double.rightName- the exponent number. A column that evaluates to a double.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
pow
Returns the value of the first argument raised to the power of the second argument.- Parameters:
leftName- the base number. A column that evaluates to a double.r- the exponent number. A column that evaluates to a double.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
pow
Returns the value of the first argument raised to the power of the second argument.- Parameters:
leftName- the base number.rightName- the exponent number.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
pow
Returns the value of the first argument raised to the power of the second argument.- Parameters:
l- the base number. A column that evaluates to a double.r- the exponent number. A column that evaluates to a double.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
pow
Returns the value of the first argument raised to the power of the second argument.- Parameters:
leftName- the base number.r- the exponent number.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
pow
Returns the value of the first argument raised to the power of the second argument.- Parameters:
l- the base number. A column that evaluates to a double.r- the exponent number. A column that evaluates to a double.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
pow
Returns the value of the first argument raised to the power of the second argument.- Parameters:
l- the base number.rightName- the exponent number.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
power
Returns the value of the first argument raised to the power of the second argument.- Parameters:
l- the base number. A column that evaluates to a double.r- the exponent number. A column that evaluates to a double.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
pmod
Returns the positive value of dividend mod divisor.- Parameters:
dividend- the column that contains dividend, or the specified dividend value. A column that evaluates to a numeric.divisor- the column that contains divisor, or the specified divisor value. A column that evaluates to a numeric.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
rint
Returns the double value that is closest in value to the argument and is equal to a mathematical integer.- Parameters:
e- target column to compute on. A column that evaluates to a double.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
rint
Returns the double value that is closest in value to the argument and is equal to a mathematical integer.- Parameters:
columnName- the numeric column name to round to the closest integer.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
round
Returns the value of the columnerounded to 0 decimal places with HALF_UP round mode.- Parameters:
e- the value to round. A column that evaluates to a numeric.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
round
Round the value ofetoscaledecimal places with HALF_UP round mode ifscaleis greater than or equal to 0 or at integral part whenscaleis less than 0.- Parameters:
e- the value to round. A column that evaluates to a numeric.scale- the number of decimal places to round to. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
round
Round the value ofetoscaledecimal places with HALF_UP round mode ifscaleis greater than or equal to 0 or at integral part whenscaleis less than 0.- Parameters:
e- the value to round. A column that evaluates to a numeric.scale- the number of decimal places to round to. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
bround
Returns the value of the columnerounded to 0 decimal places with HALF_EVEN round mode.- Parameters:
e- the value to round. A column that evaluates to a numeric.- Returns:
- Returns a column of the same type as the input.
- Since:
- 2.0.0
-
bround
Round the value ofetoscaledecimal places with HALF_EVEN round mode ifscaleis greater than or equal to 0 or at integral part whenscaleis less than 0.- Parameters:
e- the value to round. A column that evaluates to a numeric.scale- the number of decimal places to round to. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 2.0.0
-
bround
Round the value ofetoscaledecimal places with HALF_EVEN round mode ifscaleis greater than or equal to 0 or at integral part whenscaleis less than 0.- Parameters:
e- the value to round. A column that evaluates to a numeric.scale- the number of decimal places to round to. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
sec
- Parameters:
e- angle in radians. A column that evaluates to a double.- Returns:
- secant of the angle. Returns a column that evaluates to a double.
- Since:
- 3.3.0
-
shiftLeft
Deprecated.Use shiftleft. Since 3.2.0.Shift the given value numBits left. If the given value is a long value, this function will return a long value else it will return an integer value.- Parameters:
e- (undocumented)numBits- (undocumented)- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
shiftleft
Shift the given value numBits left. If the given value is a long value, this function will return a long value else it will return an integer value.- Parameters:
e- the value to shift. A column that evaluates to an integral.numBits- the number of bits to shift left. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.2.0
-
shiftRight
Deprecated.Use shiftright. Since 3.2.0.(Signed) shift the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value.- Parameters:
e- (undocumented)numBits- (undocumented)- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
shiftright
(Signed) shift the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value.- Parameters:
e- the value to shift. A column that evaluates to an integral.numBits- the number of bits to shift right. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.2.0
-
shiftRightUnsigned
Deprecated.Use shiftrightunsigned. Since 3.2.0.Unsigned shift the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value.- Parameters:
e- (undocumented)numBits- (undocumented)- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
shiftrightunsigned
Unsigned shift the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value.- Parameters:
e- the value to shift. A column that evaluates to an integral.numBits- the number of bits to shift right. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.2.0
-
sign
Computes the signum of the given value.- Parameters:
e- the value to compute the signum of. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
signum
Computes the signum of the given value.- Parameters:
e- the value to compute the signum of. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
signum
Computes the signum of the given column.- Parameters:
columnName- column to compute the signum on. A column that evaluates to a numeric or interval.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
sin
- Parameters:
e- angle in radians. A column that evaluates to a double.- Returns:
- sine of the angle, as if computed by
java.lang.Math.sin. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
sin
- Parameters:
columnName- angle in radians. A column that evaluates to a double.- Returns:
- sine of the angle, as if computed by
java.lang.Math.sin. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
sinh
- Parameters:
e- hyperbolic angle. A column that evaluates to a double.- Returns:
- hyperbolic sine of the given value, as if computed by
java.lang.Math.sinh. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
sinh
- Parameters:
columnName- hyperbolic angle. A column that evaluates to a double.- Returns:
- hyperbolic sine of the given value, as if computed by
java.lang.Math.sinh. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
tan
- Parameters:
e- angle in radians. A column that evaluates to a double.- Returns:
- tangent of the given value, as if computed by
java.lang.Math.tan. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
tan
- Parameters:
columnName- angle in radians. A column that evaluates to a double.- Returns:
- tangent of the given value, as if computed by
java.lang.Math.tan. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
tanh
- Parameters:
e- hyperbolic angle. A column that evaluates to a double.- Returns:
- hyperbolic tangent of the given value, as if computed by
java.lang.Math.tanh. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
tanh
- Parameters:
columnName- hyperbolic angle. A column that evaluates to a double.- Returns:
- hyperbolic tangent of the given value, as if computed by
java.lang.Math.tanh. Returns a column that evaluates to a double. - Since:
- 1.4.0
-
toDegrees
Deprecated.Use degrees. Since 2.1.0.- Parameters:
e- (undocumented)- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
toDegrees
Deprecated.Use degrees. Since 2.1.0.- Parameters:
columnName- (undocumented)- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
degrees
Converts an angle measured in radians to an approximately equivalent angle measured in degrees.- Parameters:
e- angle in radians. A column that evaluates to a double.- Returns:
- angle in degrees, as if computed by
java.lang.Math.toDegrees. Returns a column that evaluates to a double. - Since:
- 2.1.0
-
degrees
Converts an angle measured in radians to an approximately equivalent angle measured in degrees.- Parameters:
columnName- angle in radians. A column that evaluates to a double.- Returns:
- angle in degrees, as if computed by
java.lang.Math.toDegrees. Returns a column that evaluates to a double. - Since:
- 2.1.0
-
toRadians
Deprecated.Use radians. Since 2.1.0.- Parameters:
e- (undocumented)- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
toRadians
Deprecated.Use radians. Since 2.1.0.- Parameters:
columnName- (undocumented)- Returns:
- Returns a column that evaluates to a double.
- Since:
- 1.4.0
-
radians
Converts an angle measured in degrees to an approximately equivalent angle measured in radians.- Parameters:
e- angle in degrees. A column that evaluates to a double.- Returns:
- angle in radians, as if computed by
java.lang.Math.toRadians. Returns a column that evaluates to a double. - Since:
- 2.1.0
-
radians
Converts an angle measured in degrees to an approximately equivalent angle measured in radians.- Parameters:
columnName- angle in degrees. A column that evaluates to a double.- Returns:
- angle in radians, as if computed by
java.lang.Math.toRadians. Returns a column that evaluates to a double. - Since:
- 2.1.0
-
width_bucket
Returns the bucket number into which the value of this expression would fall after being evaluated. Note that input arguments must follow conditions listed below; otherwise, the method will return null.- Parameters:
v- value to compute a bucket number in the histogram. A column that evaluates to a double or interval.min- minimum value of the histogram. A column that evaluates to a double or interval.max- maximum value of the histogram. A column that evaluates to a double or interval.numBucket- the number of buckets. A column that evaluates to a long.- Returns:
- the bucket number into which the value would fall after being evaluated. Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
current_catalog
Returns the current catalog.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
current_database
Returns the current database.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
current_schema
Returns the current schema.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
current_path
Returns the current SQL path as a comma-separated list of qualified schema names.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.2.0
-
current_user
Returns the user name of current execution context.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
md5
Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string.- Parameters:
e- target column to compute on. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
sha1
Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.- Parameters:
e- target column to compute on. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
sha2
Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.- Parameters:
e- column to compute SHA-2 on. A column that evaluates to a binary.numBits- one of 224, 256, 384, or 512. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
crc32
Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint.- Parameters:
e- target column to compute on. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.5.0
-
hash
Calculates the hash code of given columns, and returns the result as an int column.- Parameters:
cols- one or more columns to compute on. A column of any type.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 2.0.0
-
xxhash64
Calculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column. The hash computation uses an initial seed of 42.- Parameters:
cols- one or more columns to compute on. A column of any type.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.0.0
-
assert_true
Returns null if the condition is true, and throws an exception otherwise.- Parameters:
c- The condition to check. A column that evaluates to a boolean.- Returns:
- Returns a column that always evaluates to NULL.
- Since:
- 3.1.0
-
assert_true
Returns null if the condition is true; throws an exception with the error message otherwise.- Parameters:
c- The condition to check. A column that evaluates to a boolean.e- The error message to throw. A column that evaluates to a string.- Returns:
- Returns a column that always evaluates to NULL.
- Since:
- 3.1.0
-
raise_error
Throws an exception with the provided error message.- Parameters:
c- The error message to throw. A column that evaluates to a string.- Returns:
- Returns a column that always evaluates to NULL.
- Since:
- 3.1.0
-
user
Returns the user name of current execution context.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
session_user
Returns the user name of current execution context.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
uuid
Returns an universally unique identifier (UUID) string. The value is returned as a canonical UUID 36-character string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
uuid
Returns an universally unique identifier (UUID) string. The value is returned as a canonical UUID 36-character string.- Parameters:
seed- The random number seed to use. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.1.0
-
hmac
Returns the keyed-hash message authentication code (HMAC) ofmessageusingkeyand the given hashalgorithm. The result is returned as raw MAC bytes; wrap it withhexorbase64for a textual value.- Parameters:
key- The secret key, as a binary value.message- The message to authenticate, as a binary value.algorithm- The hash algorithm. Valid values: SHA-224, SHA-256, SHA-384, SHA-512, SHA-1, MD5.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
hmac
Returns the keyed-hash message authentication code (HMAC) ofmessageusingkeyand SHA-256. The result is returned as raw MAC bytes; wrap it withhexorbase64for a textual value. To use a different algorithm, call the three-argument overload.- Parameters:
key- The secret key, as a binary value.message- The message to authenticate, as a binary value.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
aes_encrypt
public static Column aes_encrypt(Column input, Column key, Column mode, Column padding, Column iv, Column aad) Returns an encrypted value ofinputusing AES in givenmodewith the specifiedpadding. Key lengths of 16, 24 and 32 bits are supported. Supported combinations of (mode,padding) are ('ECB', 'PKCS'), ('GCM', 'NONE') and ('CBC', 'PKCS'). Optional initialization vectors (IVs) are only supported for CBC and GCM modes. These must be 16 bytes for CBC and 12 bytes for GCM. If not provided, a random vector will be generated and prepended to the output. Optional additional authenticated data (AAD) is only supported for GCM. If provided for encryption, the identical AAD value must be provided for decryption. The default mode is GCM.- Parameters:
input- The binary value to encrypt. A column that evaluates to a binary.key- The passphrase to use to encrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to encrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.padding- Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC. A column that evaluates to a string.iv- Optional initialization vector. Only supported for CBC and GCM modes. Valid values: None or "". 16-byte array for CBC mode. 12-byte array for GCM mode. A column that evaluates to a binary.aad- Optional additional authenticated data. Only supported for GCM mode. This can be any free-form input and must be provided for both encryption and decryption. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
aes_encrypt
Returns an encrypted value ofinput.- Parameters:
input- The binary value to encrypt. A column that evaluates to a binary.key- The passphrase to use to encrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to encrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.padding- Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC. A column that evaluates to a string.iv- Optional initialization vector. Only supported for CBC and GCM modes. Valid values: None or "". 16-byte array for CBC mode. 12-byte array for GCM mode. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column, Column)
-
aes_encrypt
Returns an encrypted value ofinput.- Parameters:
input- The binary value to encrypt. A column that evaluates to a binary.key- The passphrase to use to encrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to encrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.padding- Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column, Column)
-
aes_encrypt
Returns an encrypted value ofinput.- Parameters:
input- The binary value to encrypt. A column that evaluates to a binary.key- The passphrase to use to encrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to encrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column, Column)
-
aes_encrypt
Returns an encrypted value ofinput.- Parameters:
input- The binary value to encrypt. A column that evaluates to a binary.key- The passphrase to use to encrypt the data. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column, Column)
-
aes_decrypt
Returns a decrypted value ofinputusing AES inmodewithpadding. Key lengths of 16, 24 and 32 bits are supported. Supported combinations of (mode,padding) are ('ECB', 'PKCS'), ('GCM', 'NONE') and ('CBC', 'PKCS'). Optional additional authenticated data (AAD) is only supported for GCM. If provided for encryption, the identical AAD value must be provided for decryption. The default mode is GCM.- Parameters:
input- The binary value to decrypt. A column that evaluates to a binary.key- The passphrase to use to decrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.padding- Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC. A column that evaluates to a string.aad- Optional additional authenticated data. Only supported for GCM mode. This can be any free-form input and must be provided for both encryption and decryption. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
aes_decrypt
Returns a decrypted value ofinput.- Parameters:
input- The binary value to decrypt. A column that evaluates to a binary.key- The passphrase to use to decrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.padding- Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.aes_decrypt(Column, Column, Column, Column, Column)
-
aes_decrypt
Returns a decrypted value ofinput.- Parameters:
input- The binary value to decrypt. A column that evaluates to a binary.key- The passphrase to use to decrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.aes_decrypt(Column, Column, Column, Column, Column)
-
aes_decrypt
Returns a decrypted value ofinput.- Parameters:
input- The binary value to decrypt. A column that evaluates to a binary.key- The passphrase to use to decrypt the data. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.aes_decrypt(Column, Column, Column, Column, Column)
-
try_aes_decrypt
public static Column try_aes_decrypt(Column input, Column key, Column mode, Column padding, Column aad) This is a special version ofaes_decryptthat performs the same operation, but returns a NULL value instead of raising an error if the decryption cannot be performed.- Parameters:
input- The binary value to decrypt. A column that evaluates to a binary.key- The passphrase to use to decrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.padding- Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC. A column that evaluates to a string.aad- Optional additional authenticated data. Only supported for GCM mode. This can be any free-form input and must be provided for both encryption and decryption. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
try_aes_decrypt
Returns a decrypted value ofinput.- Parameters:
input- The binary value to decrypt. A column that evaluates to a binary.key- The passphrase to use to decrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.padding- Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.try_aes_decrypt(Column, Column, Column, Column, Column)
-
try_aes_decrypt
Returns a decrypted value ofinput.- Parameters:
input- The binary value to decrypt. A column that evaluates to a binary.key- The passphrase to use to decrypt the data. A column that evaluates to a binary.mode- Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB, GCM, CBC. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.try_aes_decrypt(Column, Column, Column, Column, Column)
-
try_aes_decrypt
Returns a decrypted value ofinput.- Parameters:
input- The binary value to decrypt. A column that evaluates to a binary.key- The passphrase to use to decrypt the data. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
- See Also:
-
org.apache.spark.sql.functions.try_aes_decrypt(Column, Column, Column, Column, Column)
-
sha
Returns a sha1 hash value as a hex string of thecol.- Parameters:
col- The value to hash. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
input_file_block_length
Returns the length of the block being read, or -1 if not available.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
input_file_block_start
Returns the start offset of the block being read, or -1 if not available.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
reflect
Calls a method with reflection.- Parameters:
cols- (undocumented)- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
java_method
Calls a method with reflection.- Parameters:
cols- (undocumented)- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
try_reflect
This is a special version ofreflectthat performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.- Parameters:
cols- (undocumented)- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
version
Returns the Spark version. The string contains 2 fields, the first being a release version and the second being a git revision.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
typeof
Return DDL-formatted type string for the data type of the input.- Parameters:
col- The value whose data type is returned. A column of any type.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
stack
Separatescol1, ...,colkintonrows. Uses column names col0, col1, etc. by default unless specified otherwise.- Parameters:
cols- The first column must be a constant integer for the number of rows, and the remaining columns are the input elements to be separated into rows.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
uniform
Returns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers. The provided numbers specifying the minimum and maximum values of the range must be constant. If both of these numbers are integers, then the result will also be an integer. Otherwise if one or both of these are floating-point numbers, then the result will also be a floating-point number.- Parameters:
min- Minimum value in the range. A column that evaluates to a numeric. Must be a constant.max- Maximum value in the range. A column that evaluates to a numeric. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
uniform
Returns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers, with the chosen random seed. The provided numbers specifying the minimum and maximum values of the range must be constant. If both of these numbers are integers, then the result will also be an integer. Otherwise if one or both of these are floating-point numbers, then the result will also be a floating-point number.- Parameters:
min- Minimum value in the range. A column that evaluates to a numeric. Must be a constant.max- Maximum value in the range. A column that evaluates to a numeric. Must be a constant.seed- Random number seed to use. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
random
Returns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).- Parameters:
seed- Random number seed to use. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
random
Returns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
bitmap_bit_position
Returns the bit position for the given input column.- Parameters:
col- The input column. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
bitmap_bucket_number
Returns the bucket number for the given input column.- Parameters:
col- The input column. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
bitmap_construct_agg
Returns a bitmap with the positions of the bits set from all the values from the input column. The input column will most likely be bitmap_bit_position().- Parameters:
col- The input column will most likely be bitmap_bit_position(). A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
bitmap_count
Returns the number of set bits in the input bitmap.- Parameters:
col- The input bitmap. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
bitmap_or_agg
Returns a bitmap that is the bitwise OR of all of the bitmaps from the input column. The input column should be bitmaps created from bitmap_construct_agg().- Parameters:
col- The input column should be bitmaps created from bitmap_construct_agg(). A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
bitmap_and_agg
Returns a bitmap that is the bitwise AND of all of the bitmaps from the input column. The input column should be bitmaps created from bitmap_construct_agg().- Parameters:
col- The input column should be bitmaps created from bitmap_construct_agg(). A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
ascii
Computes the numeric value of the first character of the string column, and returns the result as an int column.- Parameters:
e- The target column to work on. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
base64
Computes the BASE64 encoding of a binary column and returns it as a string column. This is the reverse of unbase64.- Parameters:
e- The target column to work on. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
bit_length
Calculates the bit length for the specified string column.- Parameters:
e- The source column or strings. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.3.0
-
concat_ws
Concatenates multiple input string columns together into a single string column, using the given separator.- Parameters:
sep- The words separator. A column that evaluates to a string. Must be a constant.exprs- The list of columns to work on. Each a column that evaluates to a string or an array of strings.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
- Note:
- Input strings which are null are skipped.
-
decode
Computes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32'). If either argument is null, the result will also be null.- Parameters:
value- The target column to work on. A column that evaluates to a binary.charset- The charset to use to decode to. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
encode
Computes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32'). If either argument is null, the result will also be null.- Parameters:
value- The target column to work on. A column that evaluates to a string.charset- The charset to use to encode. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 1.5.0
-
is_valid_utf8
Returns true if the input is a valid UTF-8 string, otherwise returns false.- Parameters:
str- A column of strings, each representing a UTF-8 byte sequence. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 4.0.0
-
make_valid_utf8
Returns a new string in which all invalid UTF-8 byte sequences, if any, are replaced by the Unicode replacement character (U+FFFD).- Parameters:
str- A column of strings, each representing a UTF-8 byte sequence. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
validate_utf8
Returns the input value if it corresponds to a valid UTF-8 string, or emits a SparkIllegalArgumentException exception otherwise.- Parameters:
str- A column of strings, each representing a UTF-8 byte sequence. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
try_validate_utf8
Returns the input value if it corresponds to a valid UTF-8 string, or NULL otherwise.- Parameters:
str- the input value. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
format_number
Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with HALF_EVEN round mode, and returns the result as a string column.If d is 0, the result has no decimal point or fractional part. If d is less than 0, the result will be null.
- Parameters:
x- the numeric value to be formatted. A column that evaluates to a numeric.d- the number of decimal places. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
format_string
Formats the arguments in printf-style and returns the result as a string column.- Parameters:
format- the format string that can contain embedded format tags. A column that evaluates to a string. Must be a constant.arguments- the values to be used in formatting. Columns that evaluate to any type.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
initcap
Returns a new string column by converting the first letter of each word to uppercase. Words are delimited by whitespace.For example, "hello world" will become "Hello World".
- Parameters:
e- the target column to work on. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
instr
Locate the position of the first occurrence of substr column in the given string. Returns null if either of the arguments are null.- Parameters:
str- the string to search in. A column that evaluates to a string.substring- the substring to search for. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.5.0
- Note:
- The position is not zero based, but 1 based index. Returns 0 if substr could not be found in str.
-
instr
Locate the position of the first occurrence of substr column in the given string. Returns null if either of the arguments are null.- Parameters:
str- the string to search in. A column that evaluates to a string.substring- the substring to search for. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 4.0.0
- Note:
- The position is not zero based, but 1 based index. Returns 0 if substr could not be found in str.
-
instr
Locate the position of the first occurrence ofsubstringinstr, starting the search from positionstart. Returns null if either of the arguments are null.- Parameters:
str- the string to search in. A column that evaluates to a string.substring- the substring to search for. A column that evaluates to a string.start- the position to start the search from. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 4.3.0
- Note:
- The position is not zero based, but 1 based index. Returns 0 if substr could not be found
in str., If
startis positive, the search proceeds forward. Ifstartis negative, the search proceeds backward from the end of the string. Ifstartis 0, returns 0.
-
instr
Locate the position of the first occurrence ofsubstringinstr, starting the search from positionstart. Returns null if either of the arguments are null.- Parameters:
str- the string to search in. A column that evaluates to a string.substring- the substring to search for. A column that evaluates to a string.start- the position to start the search from. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 4.3.0
- Note:
- The position is not zero based, but 1 based index. Returns 0 if substr could not be found
in str., If
startis positive, the search proceeds forward. Ifstartis negative, the search proceeds backward from the end of the string. Ifstartis 0, returns 0.
-
instr
Locate the position of theoccurrence-th occurrence ofsubstringinstr, starting the search from positionstart. Returns null if either of the arguments are null.- Parameters:
str- the string to search in. A column that evaluates to a string.substring- the substring to search for. A column that evaluates to a string.start- the position to start the search from. A column that evaluates to an integral. Must be a constant.occurrence- which occurrence of the substring to locate. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 4.3.0
- Note:
- The position is not zero based, but 1 based index. Returns 0 if substr could not be found
in str., If
startis positive, the search proceeds forward. Ifstartis negative, the search proceeds backward from the end of the string. Ifstartis 0, returns 0., Theoccurrenceparameter must be a positive integer.
-
instr
Locate the position of theoccurrence-th occurrence ofsubstringinstr, starting the search from positionstart. Returns null if either of the arguments are null.- Parameters:
str- the string to search in. A column that evaluates to a string.substring- the substring to search for. A column that evaluates to a string.start- the position to start the search from. A column that evaluates to an integral.occurrence- which occurrence of the substring to locate. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 4.3.0
- Note:
- The position is not zero based, but 1 based index. Returns 0 if substr could not be found
in str., If
startis positive, the search proceeds forward. Ifstartis negative, the search proceeds backward from the end of the string. Ifstartis 0, returns 0., Theoccurrenceparameter must be a positive integer.
-
length
Computes the character length of a given string or number of bytes of a binary string. The length of character strings include the trailing spaces. The length of binary strings includes binary zeros.- Parameters:
e- the target column to work on. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
len
Computes the character length of a given string or number of bytes of a binary string. The length of character strings include the trailing spaces. The length of binary strings includes binary zeros.- Parameters:
e- the target column to work on. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
lower
Converts a string column to lower case.- Parameters:
e- the target column to work on. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.3.0
-
levenshtein
Computes the Levenshtein distance of the two given string columns if it's less than or equal to a given threshold.- Parameters:
l- the first input column. A column that evaluates to a string.r- the second input column. A column that evaluates to a string.threshold- the maximum distance to compute. A column that evaluates to an integral. Must be a constant.- Returns:
- result distance, or -1. Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
levenshtein
Computes the Levenshtein distance of the two given string columns.- Parameters:
l- the first input column. A column that evaluates to a string.r- the second input column. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
jaro_winkler_similarity
Computes the Jaro-Winkler similarity between the two given string columns. The result is a double between 0.0 (no similarity) and 1.0 (identical).- Parameters:
l- A column that evaluates to a string.r- A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.3.0
-
locate
Locate the position of the first occurrence of substr.- Parameters:
substr- The substring to find. A column that evaluates to a string.str- A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.5.0
- Note:
- The position is not zero based, but 1 based index. Returns 0 if substr could not be found in str.
-
locate
Locate the position of the first occurrence of substr in a string column, after position pos.- Parameters:
substr- The substring to find. A column that evaluates to a string.str- A column that evaluates to a string.pos- The starting position. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.5.0
- Note:
- The position is not zero based, but 1 based index. returns 0 if substr could not be found in str.
-
lpad
Left-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.- Parameters:
str- A column that evaluates to a string.len- The length of the padded result. A column that evaluates to an integer.pad- The padding string. A column that evaluates to a string.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
lpad
Left-pad the binary column with pad to a byte length of len. If the binary column is longer than len, the return value is shortened to len bytes.- Parameters:
str- A column that evaluates to a binary.len- The byte length of the padded result. A column that evaluates to an integer.pad- The padding bytes. A column that evaluates to a binary.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.3.0
-
lpad
Left-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.- Parameters:
str- A column that evaluates to a string.len- The length of the padded result. A column that evaluates to an integer.pad- The padding string. A column that evaluates to a string.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
ltrim
Trim the spaces from left end for the specified string value.- Parameters:
e- A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
ltrim
Trim the specified character string from left end for the specified string column.- Parameters:
e- A column that evaluates to a string.trimString- The trim string. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.3.0
-
ltrim
Trim the specified character string from left end for the specified string column.- Parameters:
e- A column that evaluates to a string.trim- The trim string. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
octet_length
Calculates the byte length for the specified string column.- Parameters:
e- A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.3.0
-
collate
Marks a given column with specified collation.- Parameters:
e- A column that evaluates to a string.collation- The collation name. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
collation
Returns the collation name of a given column.- Parameters:
e- A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
rlike
Returns true ifstrmatchesregexp, or false otherwise.- Parameters:
str- A column that evaluates to a string.regexp- The regular expression pattern. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
regexp
Returns true ifstrmatchesregexp, or false otherwise.- Parameters:
str- A column that evaluates to a string.regexp- The regular expression pattern. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
regexp_like
Returns true ifstrmatchesregexp, or false otherwise.- Parameters:
str- A column that evaluates to a string.regexp- The regular expression pattern. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
regexp_count
Returns a count of the number of times that the regular expression patternregexpis matched in the stringstr.- Parameters:
str- target column to work on. A column that evaluates to a string.regexp- regex pattern to apply. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
regexp_extract
Extract a specific group matched by a Java regex, from the specified string column. If the regex did not match, or the specified group did not match, an empty string is returned. if the specified group index exceeds the group count of regex, an IllegalArgumentException will be thrown.- Parameters:
e- target column to work on. A column that evaluates to a string.exp- regex pattern to apply. A string. Must be a constant.groupIdx- matched group id. An integer. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
regexp_extract_all
Extract all strings in thestrthat match theregexpexpression and corresponding to the first regex group index.- Parameters:
str- target column to work on. A column that evaluates to a string.regexp- regex pattern to apply. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.5.0
-
regexp_extract_all
Extract all strings in thestrthat match theregexpexpression and corresponding to the regex group index.- Parameters:
str- target column to work on. A column that evaluates to a string.regexp- regex pattern to apply. A column that evaluates to a string.idx- matched group id. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.5.0
-
regexp_replace
Replace all substrings of the specified string value that match regexp with rep.- Parameters:
e- target column to work on. A column that evaluates to a string.pattern- regex pattern to apply. A string. Must be a constant.replacement- replacement string. A string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
regexp_replace
Replace all substrings of the specified string value that match regexp with rep, starting at the specified positionpos.- Parameters:
e- target column to work on. A column that evaluates to a string.pattern- regex pattern to apply. A string. Must be a constant.replacement- replacement string. A string. Must be a constant.pos- position to start replacement. The first position is 1. An integer. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.3.0
-
regexp_replace
Replace all substrings of the specified string value that match regexp with rep.- Parameters:
e- target column to work on. A column that evaluates to a string.pattern- regex pattern to apply. A column that evaluates to a string.replacement- replacement string. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.1.0
-
regexp_replace
Replace all substrings of the specified string value that match regexp with rep, starting at the specified positionpos.- Parameters:
e- target column to work on. A column that evaluates to a string.pattern- regex pattern to apply. A column that evaluates to a string.replacement- replacement string. A column that evaluates to a string.pos- position to start replacement. The first position is 1. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.3.0
-
regexp_substr
Returns the substring that matches the regular expressionregexpwithin the stringstr. If the regular expression is not found, the result is null.- Parameters:
str- target column to work on. A column that evaluates to a string.regexp- regex pattern to apply. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
regexp_instr
Searches a string for a regular expression and returns an integer that indicates the beginning position of the matched substring. Positions are 1-based, not 0-based. If no match is found, returns 0.- Parameters:
str- target column to work on. A column that evaluates to a string.regexp- regex pattern to apply. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
regexp_instr
Searches a string for a regular expression and returns an integer that indicates the beginning position of the matched substring. Positions are 1-based, not 0-based. If no match is found, returns 0.- Parameters:
str- target column to work on. A column that evaluates to a string.regexp- regex pattern to apply. A column that evaluates to a string.idx- matched group id. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
unbase64
Decodes a BASE64 encoded string column and returns it as a binary column. This is the reverse of base64.- Parameters:
e- target column to work on. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 1.5.0
-
rpad
Right-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.- Parameters:
str- target column to work on. A column that evaluates to a string.len- length of the final string. An integer. Must be a constant.pad- chars to append. A string. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
rpad
Right-pad the binary column with pad to a byte length of len. If the binary column is longer than len, the return value is shortened to len bytes.- Parameters:
str- target column to work on. A column that evaluates to a binary.len- byte length of the final binary. An integer. Must be a constant.pad- bytes to append. A binary. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.3.0
-
rpad
Right-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.- Parameters:
str- target column to work on. A column that evaluates to a string or binary.len- length of the final result. A column that evaluates to an integer.pad- chars or bytes to append. A column that evaluates to a string or binary.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
repeat
Repeats a string column n times, and returns it as a new string column.- Parameters:
str- target column to work on. A column that evaluates to a string.n- number of times to repeat value. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
repeat
Repeats a string column n times, and returns it as a new string column.- Parameters:
str- target column to work on. A column that evaluates to a string.n- number of times to repeat value. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
rtrim
Trim the spaces from right end for the specified string value.- Parameters:
e- target column to work on. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
rtrim
Trim the specified character string from right end for the specified string column.- Parameters:
e- target column to work on. A column that evaluates to a string.trimString- the trim string characters to trim. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.3.0
-
rtrim
Trim the specified character string from right end for the specified string column.- Parameters:
e- target column to work on. A column that evaluates to a string.trim- the trim string characters to trim. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
soundex
Returns the soundex code for the specified expression.- Parameters:
e- target column to work on. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
split
Splits str around matches of the given pattern.- Parameters:
str- a string expression to split. A column that evaluates to a string.pattern- a string representing a regular expression. The regex string should be a Java regular expression. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.5.0
-
split
Splits str around matches of the given pattern.- Parameters:
str- a string expression to split. A column that evaluates to a string.pattern- a column of string representing a regular expression. The regex string should be a Java regular expression. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.0.0
-
split
Splits str around matches of the given pattern.- Parameters:
str- a string expression to split. A column that evaluates to a string.pattern- a string representing a regular expression. The regex string should be a Java regular expression. A column that evaluates to a string.limit- an integer expression which controls the number of times the regex is applied.- limit greater than 0: The resulting array's length will not be more than limit, and the resulting array's last entry will contain all input beyond the last matched regex.
- limit less than or equal to 0:
regexwill be applied as many times as possible, and the resulting array can be of any size.
- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.0.0
-
split
Splits str around matches of the given pattern.- Parameters:
str- a string expression to split. A column that evaluates to a string.pattern- a column of string representing a regular expression. The regex string should be a Java regular expression. A column that evaluates to a string.limit- a column of integer expression which controls the number of times the regex is applied.- limit greater than 0: The resulting array's length will not be more than limit, and the resulting array's last entry will contain all input beyond the last matched regex.
- limit less than or equal to 0:
regexwill be applied as many times as possible, and the resulting array can be of any size.
- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.0.0
-
substring
Substring starts atposand is of lengthlenwhen str is String type or returns the slice of byte array that starts atposin byte and is of lengthlenwhen str is Binary type- Parameters:
str- target column to work on. A column that evaluates to a string or binary.pos- starting position in str. A column that evaluates to an integral. Must be a constant.len- length of chars. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
- Note:
- The position is not zero based, but 1 based index.
-
substring
Substring starts atposand is of lengthlenwhen str is String type or returns the slice of byte array that starts atposin byte and is of lengthlenwhen str is Binary type- Parameters:
str- target column to work on. A column that evaluates to a string or binary.pos- starting position in str. A column that evaluates to an integral.len- length of chars. A column that evaluates to an integral.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
- Note:
- The position is not zero based, but 1 based index.
-
substring_index
Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. substring_index performs a case-sensitive match when searching for delim.- Parameters:
str- target column to work on. A column that evaluates to a string.delim- delimiter of values. A column that evaluates to a string. Must be a constant.count- number of occurrences. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
overlay
Overlay the specified portion ofsrcwithreplace, starting from byte positionposofsrcand proceeding forlenbytes.- Parameters:
src- the string that will be replaced. A column that evaluates to a string or binary.replace- the substitution string. A column that evaluates to a string or binary.pos- the starting position in src. A column that evaluates to an integral.len- the number of bytes to replace in src. A column that evaluates to an integral.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.0.0
-
overlay
Overlay the specified portion ofsrcwithreplace, starting from byte positionposofsrc.- Parameters:
src- the string that will be replaced. A column that evaluates to a string or binary.replace- the substitution string. A column that evaluates to a string or binary.pos- the starting position in src. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.0.0
-
sentences
Splits a string into arrays of sentences, where each sentence is an array of words.- Parameters:
string- a string to be split. A column that evaluates to a string.language- a language of the locale. A column that evaluates to a string.country- a country of the locale. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.2.0
-
sentences
Splits a string into arrays of sentences, where each sentence is an array of words. The defaultcountry('') is used.- Parameters:
string- a string to be split. A column that evaluates to a string.language- a language of the locale. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.0.0
-
sentences
Splits a string into arrays of sentences, where each sentence is an array of words. The default locale is used.- Parameters:
string- a string to be split. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.2.0
-
translate
Translate any character in the src by a character in replaceString. The characters in replaceString correspond to the characters in matchingString. The translate will happen when any character in the string matches the character in thematchingString.- Parameters:
src- source column to work on. A column that evaluates to a string.matchingString- matching characters. A column that evaluates to a string. Must be a constant.replaceString- characters for replacement. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
trim
Trim the spaces from both ends for the specified string column.- Parameters:
e- The string column to trim. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
trim
Trim the specified character from both ends for the specified string column.- Parameters:
e- The string column to trim. A column that evaluates to a string.trimString- The trim string characters to trim. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.3.0
-
trim
Trim the specified character from both ends for the specified string column.- Parameters:
e- The string column to trim. A column that evaluates to a string.trim- The trim string characters to trim. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
upper
Converts a string column to upper case.- Parameters:
e- The input column to convert to upper case. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.3.0
-
to_binary
Converts the inputeto a binary value based on the suppliedformat. Theformatcan be a case-insensitive string literal of "hex", "utf-8", "utf8", or "base64". By default, the binary format for conversion is "hex" ifformatis omitted. The function returns NULL if at least one of the input parameters is NULL.- Parameters:
e- The input value to convert. A column that evaluates to a string.f- The format to use to convert the value. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
to_binary
Converts the inputeto a binary value based on the default format "hex". The function returns NULL if at least one of the input parameters is NULL.- Parameters:
e- The input value to convert. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
to_char
Converteto a string based on theformat. Throws an exception if the conversion fails. The format can consist of the following characters, case insensitive: '0' or '9': Specifies an expected digit between 0 and 9. A sequence of 0 or 9 in the format string matches a sequence of digits in the input value, generating a result string of the same length as the corresponding sequence in the format string. The result string is left-padded with zeros if the 0/9 sequence comprises more digits than the matching part of the decimal value, starts with 0, and is before the decimal point. Otherwise, it is padded with spaces. '.' or 'D': Specifies the position of the decimal point (optional, only allowed once). ',' or 'G': Specifies the position of the grouping (thousands) separator (,). There must be a 0 or 9 to the left and right of each grouping separator. '$': Specifies the location of the $ currency sign. This character may only be specified once. 'S' or 'MI': Specifies the position of a '-' or '+' sign (optional, only allowed once at the beginning or end of the format string). Note that 'S' prints '+' for positive values but 'MI' prints a space. 'PR': Only allowed at the end of the format string; specifies that the result string will be wrapped by angle brackets if the input value is negative.If
eis a datetime,formatshall be a valid datetime pattern, see Datetime Patterns. Ifeis a binary, it is converted to a string in one of the formats: 'base64': a base 64 string. 'hex': a string in the hexadecimal format. 'utf-8': the input binary is decoded to UTF-8 string.- Parameters:
e- The input value to convert. A column that evaluates to a numeric, date, timestamp or binary.format- The format to use to convert the value. A column that evaluates to a string. Must be a constant wheneis a numeric or binary value.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
to_varchar
Converteto a string based on theformat. Throws an exception if the conversion fails. The format can consist of the following characters, case insensitive: '0' or '9': Specifies an expected digit between 0 and 9. A sequence of 0 or 9 in the format string matches a sequence of digits in the input value, generating a result string of the same length as the corresponding sequence in the format string. The result string is left-padded with zeros if the 0/9 sequence comprises more digits than the matching part of the decimal value, starts with 0, and is before the decimal point. Otherwise, it is padded with spaces. '.' or 'D': Specifies the position of the decimal point (optional, only allowed once). ',' or 'G': Specifies the position of the grouping (thousands) separator (,). There must be a 0 or 9 to the left and right of each grouping separator. '$': Specifies the location of the $ currency sign. This character may only be specified once. 'S' or 'MI': Specifies the position of a '-' or '+' sign (optional, only allowed once at the beginning or end of the format string). Note that 'S' prints '+' for positive values but 'MI' prints a space. 'PR': Only allowed at the end of the format string; specifies that the result string will be wrapped by angle brackets if the input value is negative.If
eis a datetime,formatshall be a valid datetime pattern, see Datetime Patterns. Ifeis a binary, it is converted to a string in one of the formats: 'base64': a base 64 string. 'hex': a string in the hexadecimal format. 'utf-8': the input binary is decoded to UTF-8 string.- Parameters:
e- The input value to convert. A column that evaluates to a numeric, date, timestamp or binary.format- The format to use to convert the value. A column that evaluates to a string. Must be a constant wheneis a numeric or binary value.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
to_number
Convert string 'e' to a number based on the string format 'format'. Throws an exception if the conversion fails. The format can consist of the following characters, case insensitive: '0' or '9': Specifies an expected digit between 0 and 9. A sequence of 0 or 9 in the format string matches a sequence of digits in the input string. If the 0/9 sequence starts with 0 and is before the decimal point, it can only match a digit sequence of the same size. Otherwise, if the sequence starts with 9 or is after the decimal point, it can match a digit sequence that has the same or smaller size. '.' or 'D': Specifies the position of the decimal point (optional, only allowed once). ',' or 'G': Specifies the position of the grouping (thousands) separator (,). There must be a 0 or 9 to the left and right of each grouping separator. 'expr' must match the grouping separator relevant for the size of the number. '$': Specifies the location of the $ currency sign. This character may only be specified once. 'S' or 'MI': Specifies the position of a '-' or '+' sign (optional, only allowed once at the beginning or end of the format string). Note that 'S' allows '-' but 'MI' does not. 'PR': Only allowed at the end of the format string; specifies that 'expr' indicates a negative number with wrapping angled brackets.- Parameters:
e- The input string to convert to a number. A column that evaluates to a string.format- The format to use to convert the value. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a decimal.
- Since:
- 3.5.0
-
replace
Replaces all occurrences ofsearchwithreplace.- Parameters:
src- A column of strings to be replaced. A column that evaluates to a string.search- A column of strings. Ifsearchis not found instr,stris returned unchanged. A column that evaluates to a string.replace- A column of strings. Ifreplaceis not specified or is an empty string, nothing replaces the string that is removed fromstr. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
replace
Replaces all occurrences ofsearchwithreplace.- Parameters:
src- A column of strings to be replaced. A column that evaluates to a string.search- A column of strings. Ifsearchis not found insrc,srcis returned unchanged. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
split_part
Splitsstrby delimiter and return requested part of the split (1-based). If any input is null, returns null. ifpartNumis out of range of split parts, returns empty string. IfpartNumis 0, throws an error. IfpartNumis negative, the parts are counted backward from the end of the string. If thedelimiteris an empty string, thestris not split.- Parameters:
str- A column of strings to be split. A column that evaluates to a string.delimiter- The delimiter used for split. A column that evaluates to a string.partNum- The requested part of the split (1-based). A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
substr
Returns the substring ofstrthat starts atposand is of lengthlen, or the slice of byte array that starts atposand is of lengthlen.- Parameters:
str- The input from which to take the substring. A column that evaluates to a string or binary.pos- The starting position of the substring. A column that evaluates to an integral.len- The length of the substring. A column that evaluates to an integral.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
substr
Returns the substring ofstrthat starts atpos, or the slice of byte array that starts atpos.- Parameters:
str- The input from which to take the substring. A column that evaluates to a string or binary.pos- The starting position of the substring. A column that evaluates to an integral.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
try_parse_url
Extracts a part from a URL.- Parameters:
url- A column of strings, each representing a URL. A column that evaluates to a string.partToExtract- The part to extract from the URL. A column that evaluates to a string.key- The key of a query parameter in the URL. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
try_parse_url
Extracts a part from a URL.- Parameters:
url- A column of strings, each representing a URL. A column that evaluates to a string.partToExtract- The part to extract from the URL. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
parse_url
Extracts a part from a URL.- Parameters:
url- A column of strings, each representing a URL. A column that evaluates to a string.partToExtract- The part to extract from the URL. A column that evaluates to a string.key- The key of a query parameter in the URL. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
parse_url
Extracts a part from a URL.- Parameters:
url- A column representing a URL. A column that evaluates to a string.partToExtract- The part to extract from the URL. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
printf
Formats the arguments in printf-style and returns the result as a string column.- Parameters:
format- A format string that can contain embedded format tags. A column that evaluates to a string.arguments- The values to be used in formatting. Columns that evaluate to any type.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
url_decode
Decodes astrin 'application/x-www-form-urlencoded' format using a specific encoding scheme.- Parameters:
str- A URL-encoded string. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
try_url_decode
This is a special version ofurl_decodethat performs the same operation, but returns a NULL value instead of raising an error if the decoding cannot be performed.- Parameters:
str- A URL-encoded string. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
url_encode
Translates a string into 'application/x-www-form-urlencoded' format using a specific encoding scheme.- Parameters:
str- A string to encode. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
position
Returns the position of the first occurrence ofsubstrinstrafter positionstart. The givenstartand return value are 1-based.- Parameters:
substr- The substring to search for. A column that evaluates to a string.str- The string to search in. A column that evaluates to a string.start- The 1-based position to start the search from. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
position
Returns the position of the first occurrence ofsubstrinstrafter position1. The return value are 1-based.- Parameters:
substr- The substring to search for. A column that evaluates to a string.str- The string to search in. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
endswith
Returns a boolean. The value is True if str ends with suffix. Returns NULL if either input expression is NULL. Otherwise, returns False. Both str or suffix must be of STRING or BINARY type.- Parameters:
str- The string to test. A column that evaluates to a string or binary.suffix- The suffix to test for. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
startswith
Returns a boolean. The value is True if str starts with prefix. Returns NULL if either input expression is NULL. Otherwise, returns False. Both str or prefix must be of STRING or BINARY type.- Parameters:
str- The string to test. A column that evaluates to a string or binary.prefix- The prefix to test for. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
btrim
Removes the leading and trailing space characters fromstr.- Parameters:
str- The string to trim. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
btrim
Remove the leading and trailingtrimcharacters fromstr.- Parameters:
str- The string to trim. A column that evaluates to a string.trim- The trim string characters to trim. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
try_to_binary
This is a special version ofto_binarythat performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.- Parameters:
e- The string to convert. A column that evaluates to a string.f- The format to use for the conversion. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
try_to_binary
This is a special version ofto_binarythat performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.- Parameters:
e- The string to convert. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
try_to_number
Convert stringeto a number based on the string formatformat. Returns NULL if the stringedoes not match the expected format. The format follows the same semantics as the to_number function.- Parameters:
e- The string to convert. A column that evaluates to a string.format- The format used to convert the string to a number. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a decimal.
- Since:
- 3.5.0
-
char_length
Returns the character length of string data or number of bytes of binary data. The length of string data includes the trailing spaces. The length of binary data includes binary zeros.- Parameters:
str- Input column or strings. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
character_length
Returns the character length of string data or number of bytes of binary data. The length of string data includes the trailing spaces. The length of binary data includes binary zeros.- Parameters:
str- Input column or strings. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
chr
Returns the ASCII character having the binary equivalent ton. If n is larger than 256 the result is equivalent to chr(n % 256)- Parameters:
n- The code point. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
contains
Returns a boolean. The value is True if right is found inside left. Returns NULL if either input expression is NULL. Otherwise, returns False. Both left or right must be of STRING or BINARY type.- Parameters:
left- The input to check, may be NULL. A column that evaluates to a string or binary.right- The input to find, may be NULL. A column that evaluates to a string or binary.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
elt
Returns then-th input, e.g., returnsinput2whennis 2. The function returns NULL if the index exceeds the length of the array andspark.sql.ansi.enabledis set to false. Ifspark.sql.ansi.enabledis set to true, it throws ArrayIndexOutOfBoundsException for invalid indices.- Parameters:
inputs- The index followed by the inputs to select from. Columns where the first evaluates to an integral and the rest evaluate to strings or binaries.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
find_in_set
Returns the index (1-based) of the given string (str) in the comma-delimited list (strArray). Returns 0, if the string was not found or if the given string (str) contains a comma.- Parameters:
str- The given string to be found. A column that evaluates to a string.strArray- The comma-delimited list. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
like
Returns true if str matchespatternwithescapeChar, null if any arguments are null, false otherwise.- Parameters:
str- A column that evaluates to a string.pattern- The pattern to match. A column that evaluates to a string.escapeChar- The escape character. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
like
Returns true if str matchespatternwithescapeChar('\'), null if any arguments are null, false otherwise.- Parameters:
str- A column that evaluates to a string.pattern- The pattern to match. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
ilike
Returns true if str matchespatternwithescapeCharcase-insensitively, null if any arguments are null, false otherwise.- Parameters:
str- A column that evaluates to a string.pattern- The pattern to match. A column that evaluates to a string.escapeChar- The escape character. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
ilike
Returns true if str matchespatternwithescapeChar('\') case-insensitively, null if any arguments are null, false otherwise.- Parameters:
str- A column that evaluates to a string.pattern- The pattern to match. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
lcase
Returnsstrwith all characters changed to lowercase.- Parameters:
str- A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
ucase
Returnsstrwith all characters changed to uppercase.- Parameters:
str- A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
left
Returns the leftmostlen(lencan be string type) characters from the stringstr, iflenis less or equal than 0 the result is an empty string.- Parameters:
str- Input column or strings. A column that evaluates to a string or binary.len- The number of leftmost characters. A column that evaluates to an integral.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
right
Returns the rightmostlen(lencan be string type) characters from the stringstr, iflenis less or equal than 0 the result is an empty string.- Parameters:
str- Input column or strings. A column that evaluates to a string.len- The number of rightmost characters. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
quote
Returnsstrenclosed by single quotes and each instance of single quote in it is preceded by a backslash.- Parameters:
str- A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.1.0
-
hll_sketch_estimate
Returns the estimated number of unique values given the binary representation of a Datasketches HllSketch.- Parameters:
c- The binary representation of a Datasketches HllSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
hll_sketch_estimate
Returns the estimated number of unique values given the binary representation of a Datasketches HllSketch.- Parameters:
columnName- Name of the column containing the binary representation of a Datasketches HllSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
hll_union
Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object. Throws an exception if sketches have different lgConfigK values.- Parameters:
c1- The first binary representation of a Datasketches HllSketch. A column that evaluates to a binary.c2- The second binary representation of a Datasketches HllSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_union
Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object. Throws an exception if sketches have different lgConfigK values.- Parameters:
columnName1- Name of the column containing the first binary representation of a Datasketches HllSketch. A column that evaluates to a binary.columnName2- Name of the column containing the second binary representation of a Datasketches HllSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_union
Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object. Throws an exception if sketches have different lgConfigK values and allowDifferentLgConfigK is set to false.- Parameters:
c1- The first binary representation of a Datasketches HllSketch. A column that evaluates to a binary.c2- The second binary representation of a Datasketches HllSketch. A column that evaluates to a binary.allowDifferentLgConfigK- Allow sketches with different lgConfigK values to be merged (defaults to false). A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
hll_union
public static Column hll_union(String columnName1, String columnName2, boolean allowDifferentLgConfigK) Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object. Throws an exception if sketches have different lgConfigK values and allowDifferentLgConfigK is set to false.- Parameters:
columnName1- Name of the column containing the first binary representation of a Datasketches HllSketch. A column that evaluates to a binary.columnName2- Name of the column containing the second binary representation of a Datasketches HllSketch. A column that evaluates to a binary.allowDifferentLgConfigK- Allow sketches with different lgConfigK values to be merged (defaults to false). A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 3.5.0
-
theta_difference
Subtracts two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches AnotB object- Parameters:
c1- The first binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.c2- The second binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_difference
Subtracts two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches AnotB object- Parameters:
columnName1- Name of the column containing the first binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.columnName2- Name of the column containing the second binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_intersection
Intersects two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Intersection object- Parameters:
c1- The first binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.c2- The second binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_intersection
Intersects two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Intersection object- Parameters:
columnName1- Name of the column containing the first binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.columnName2- Name of the column containing the second binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_sketch_estimate
Returns the estimated number of unique values given the binary representation of a Datasketches ThetaSketch.- Parameters:
c- The binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.1.0
-
theta_sketch_estimate
Returns the estimated number of unique values given the binary representation of a Datasketches ThetaSketch.- Parameters:
columnName- Name of the column containing the binary representation of a Datasketches ThetaSketch.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.1.0
-
theta_union
Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object. It is configured with the default value of 12 forlgNomEntries.- Parameters:
c1- The first binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.c2- The second binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_union
Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object. It is configured with the default value of 12 forlgNomEntries.- Parameters:
columnName1- Name of the column containing the first binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.columnName2- Name of the column containing the second binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_union
Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer.- Parameters:
c1- The first binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.c2- The second binary representation of a Datasketches ThetaSketch. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union operation (must be between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_union
Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer.- Parameters:
columnName1- The first ThetaSketch column to union. A column that evaluates to a binary.columnName2- The second ThetaSketch column to union. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
theta_union
Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer.- Parameters:
c1- The first ThetaSketch column to union. A column that evaluates to a binary.c2- The second ThetaSketch column to union. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
tuple_difference_double
Subtracts two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches AnotB object. Returns elements in the first sketch that are not in the second sketch.- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column to subtract. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_difference_double
Subtracts two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches AnotB object. Returns elements in the first sketch that are not in the second sketch.- Parameters:
columnName1- The first TupleSketch column. A column that evaluates to a binary.columnName2- The second TupleSketch column to subtract. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_difference_integer
Subtracts two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches AnotB object. Returns elements in the first sketch that are not in the second sketch.- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column to subtract. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_difference_integer
Subtracts two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches AnotB object. Returns elements in the first sketch that are not in the second sketch.- Parameters:
columnName1- The first TupleSketch column. A column that evaluates to a binary.columnName2- The second TupleSketch column to subtract. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_double
Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
c1- The first TupleSketch column to intersect. A column that evaluates to a binary.c2- The second TupleSketch column to intersect. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_double
Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
columnName1- The first TupleSketch column to intersect. A column that evaluates to a binary.columnName2- The second TupleSketch column to intersect. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_double
Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
c1- The first TupleSketch column to intersect. A column that evaluates to a binary.c2- The second TupleSketch column to intersect. A column that evaluates to a binary.mode- The aggregation mode for numeric summaries (sum, min, max, alwaysone). A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_double
Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
columnName1- The first TupleSketch column to intersect. A column that evaluates to a binary.columnName2- The second TupleSketch column to intersect. A column that evaluates to a binary.mode- The aggregation mode for numeric summaries (sum, min, max, alwaysone). A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_double
Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
c1- The first TupleSketch column to intersect. A column that evaluates to a binary.c2- The second TupleSketch column to intersect. A column that evaluates to a binary.mode- The aggregation mode for numeric summaries (sum, min, max, alwaysone). A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_integer
Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
c1- The first TupleSketch column to intersect. A column that evaluates to a binary.c2- The second TupleSketch column to intersect. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_integer
Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
columnName1- The first TupleSketch column to intersect. A column that evaluates to a binary.columnName2- The second TupleSketch column to intersect. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_integer
Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
c1- The first TupleSketch column to intersect. A column that evaluates to a binary.c2- The second TupleSketch column to intersect. A column that evaluates to a binary.mode- The aggregation mode for numeric summaries (sum, min, max, alwaysone). A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_integer
public static Column tuple_intersection_integer(String columnName1, String columnName2, String mode) Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
columnName1- The first TupleSketch column to intersect. A column that evaluates to a binary.columnName2- The second TupleSketch column to intersect. A column that evaluates to a binary.mode- The aggregation mode for numeric summaries (sum, min, max, alwaysone). A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_integer
Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_sketch_estimate_double
Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with double summary data type.- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_estimate_double
Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with double summary data type.- Parameters:
columnName- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_estimate_integer
Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with integer summary data type.- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_estimate_integer
Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with integer summary data type.- Parameters:
columnName- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_summary_double
Aggregates the summary values from a Datasketches TupleSketch with double summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_summary_double
Aggregates the summary values from a Datasketches TupleSketch with double summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
columnName- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_summary_double
Aggregates the summary values from a Datasketches TupleSketch with double summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone).- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_summary_double
Aggregates the summary values from a Datasketches TupleSketch with double summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone).- Parameters:
columnName- The column containing a binary TupleSketch representation. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_summary_double
Aggregates the summary values from a Datasketches TupleSketch with double summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone).- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_summary_integer
Aggregates the summary values from a Datasketches TupleSketch with integer summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.2.0
-
tuple_sketch_summary_integer
Aggregates the summary values from a Datasketches TupleSketch with integer summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
columnName- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.2.0
-
tuple_sketch_summary_integer
Aggregates the summary values from a Datasketches TupleSketch with integer summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone).- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.2.0
-
tuple_sketch_summary_integer
Aggregates the summary values from a Datasketches TupleSketch with integer summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone).- Parameters:
columnName- The column containing a binary TupleSketch representation. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.2.0
-
tuple_sketch_summary_integer
Aggregates the summary values from a Datasketches TupleSketch with integer summary data type. The mode parameter specifies the aggregation mode (sum, min, max, alwaysone).- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.2.0
-
tuple_sketch_theta_double
Returns the theta value (sampling rate) from a Datasketches TupleSketch with double summary data type. The theta value represents the effective sampling rate of the sketch, between 0.0 and 1.0.- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_theta_double
Returns the theta value (sampling rate) from a Datasketches TupleSketch with double summary data type. The theta value represents the effective sampling rate of the sketch, between 0.0 and 1.0.- Parameters:
columnName- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_theta_integer
Returns the theta value (sampling rate) from a Datasketches TupleSketch with integer summary data type. The theta value represents the effective sampling rate of the sketch, between 0.0 and 1.0.- Parameters:
c- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_sketch_theta_integer
Returns the theta value (sampling rate) from a Datasketches TupleSketch with integer summary data type. The theta value represents the effective sampling rate of the sketch, between 0.0 and 1.0.- Parameters:
columnName- The column containing a binary TupleSketch representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 4.2.0
-
tuple_union_double
Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_double
Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
columnName1- The first TupleSketch column. A column that evaluates to a binary.columnName2- The second TupleSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_double
Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_double
Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
columnName1- The first TupleSketch column. A column that evaluates to a binary.columnName2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_double
Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.mode- The aggregation mode for numeric summaries (sum, min, max, alwaysone). A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_double
public static Column tuple_union_double(String columnName1, String columnName2, int lgNomEntries, String mode) Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
columnName1- The first TupleSketch column. A column that evaluates to a binary.columnName2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.mode- The aggregation mode for numeric summaries (sum, min, max, alwaysone). A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_double
Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral.mode- The aggregation mode for numeric summaries (sum, min, max, alwaysone). A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_integer
Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_integer
Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
columnName1- The first TupleSketch column. A column that evaluates to a binary.columnName2- The second TupleSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_integer
Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_integer
Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
columnName1- The first TupleSketch column. A column that evaluates to a binary.columnName2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries for the union buffer. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_integer
Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries. A column that evaluates to an integral. Must be a constant.mode- The summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_integer
public static Column tuple_union_integer(String columnName1, String columnName2, int lgNomEntries, String mode) Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
columnName1- The first TupleSketch column. A column that evaluates to a binary.columnName2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries. A column that evaluates to an integral. Must be a constant.mode- The summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_integer
Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
c1- The first TupleSketch column. A column that evaluates to a binary.c2- The second TupleSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries. A column that evaluates to an integral.mode- The summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_difference_theta_double
Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with double summary data type in the input columns using a Datasketches AnotB object. Returns elements in the TupleSketch that are not in the ThetaSketch.- Parameters:
c1- The TupleSketch column. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_difference_theta_double
Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with double summary data type in the input columns using a Datasketches AnotB object. Returns elements in the TupleSketch that are not in the ThetaSketch.- Parameters:
columnName1- The TupleSketch column. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_difference_theta_integer
Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with integer summary data type in the input columns using a Datasketches AnotB object. Returns elements in the TupleSketch that are not in the ThetaSketch.- Parameters:
c1- The TupleSketch column. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_difference_theta_integer
Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with integer summary data type in the input columns using a Datasketches AnotB object. Returns elements in the TupleSketch that are not in the ThetaSketch.- Parameters:
columnName1- The TupleSketch column. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_double
Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
c1- The TupleSketch column. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_double
Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
columnName1- The TupleSketch column. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_double
Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
c1- The TupleSketch column. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.mode- The summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_double
public static Column tuple_intersection_theta_double(String columnName1, String columnName2, String mode) Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
columnName1- The TupleSketch column. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.mode- The summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_double
Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
c1- The TupleSketch column. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.mode- The summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_integer
Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
c1- The TupleSketch column. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_integer
Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone). It is configured with the default mode of 'sum'.- Parameters:
columnName1- The TupleSketch column. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_integer
Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
c1- The TupleSketch column. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.mode- The summary mode: "sum", "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_integer
public static Column tuple_intersection_theta_integer(String columnName1, String columnName2, String mode) Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
columnName1- The TupleSketch column with integer summaries. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_intersection_theta_integer
Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object. The mode parameter specifies the aggregation mode for numeric summaries during intersection (sum, min, max, alwaysone).- Parameters:
c1- The TupleSketch column with integer summaries. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_double
Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
c1- The TupleSketch column with double summaries. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_double
Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
columnName1- The TupleSketch column with double summaries. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_double
Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
c1- The TupleSketch column with double summaries. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_double
public static Column tuple_union_theta_double(String columnName1, String columnName2, int lgNomEntries) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
columnName1- The TupleSketch column with double summaries. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_double
Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
c1- The TupleSketch column with double summaries. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.mode- The summary mode: sum (default), min, max, or alwaysone. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_double
public static Column tuple_union_theta_double(String columnName1, String columnName2, int lgNomEntries, String mode) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
columnName1- The TupleSketch column with double summaries. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.mode- The summary mode: sum (default), min, max, or alwaysone. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_double
public static Column tuple_union_theta_double(Column c1, Column c2, Column lgNomEntries, Column mode) Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
c1- The TupleSketch column with double summaries. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (between 4 and 26, defaults to 12). A column that evaluates to an integral.mode- The summary mode: sum (default), min, max, or alwaysone. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_integer
Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
c1- The TupleSketch column with integer summaries. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_integer
Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It is configured with the default values of 12 forlgNomEntriesand 'sum' for mode.- Parameters:
columnName1- The TupleSketch column with integer summaries. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_integer
Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
c1- The TupleSketch column with integer summaries. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_integer
public static Column tuple_union_theta_integer(String columnName1, String columnName2, int lgNomEntries) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer. It uses the default mode of 'sum'.- Parameters:
columnName1- The TupleSketch column with integer summaries. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_integer
Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
c1- The TupleSketch column with integer summaries. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.mode- The summary mode: sum (default), min, max, or alwaysone. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_integer
public static Column tuple_union_theta_integer(String columnName1, String columnName2, int lgNomEntries, String mode) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
columnName1- The TupleSketch column with integer summaries. A column that evaluates to a binary.columnName2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.mode- The summary mode: sum (default), min, max, or alwaysone. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
tuple_union_theta_integer
public static Column tuple_union_theta_integer(Column c1, Column c2, Column lgNomEntries, Column mode) Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object. It allows the configuration oflgNomEntrieslog nominal entries for the union buffer and the aggregation mode for numeric summaries (sum, min, max, alwaysone).- Parameters:
c1- The TupleSketch column with integer summaries. A column that evaluates to a binary.c2- The ThetaSketch column. A column that evaluates to a binary.lgNomEntries- The log-base-2 of nominal entries (must be between 4 and 26, defaults to 12). A column that evaluates to an integral. Must be a constant.mode- The summary mode: "sum" (default), "min", "max", or "alwaysone". A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
kll_sketch_to_string_bigint
Returns a string with human readable summary information about the KLL bigint sketch.- Parameters:
e- The KLL bigint sketch binary representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.1.0
-
kll_sketch_to_string_float
Returns a string with human readable summary information about the KLL float sketch.- Parameters:
e- The KLL float sketch binary representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.1.0
-
kll_sketch_to_string_double
Returns a string with human readable summary information about the KLL double sketch.- Parameters:
e- The KLL double sketch binary representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.1.0
-
kll_sketch_get_n_bigint
Returns the number of items collected in the KLL bigint sketch.- Parameters:
e- The KLL bigint sketch binary representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.1.0
-
kll_sketch_get_n_float
Returns the number of items collected in the KLL float sketch.- Parameters:
e- The KLL float sketch binary representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.1.0
-
kll_sketch_get_n_double
Returns the number of items collected in the KLL double sketch.- Parameters:
e- The KLL double sketch binary representation. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.1.0
-
kll_sketch_merge_bigint
Merges two KLL bigint sketch buffers together into one.- Parameters:
left- The first KLL bigint sketch. A column that evaluates to a binary.right- The second KLL bigint sketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_merge_float
Merges two KLL float sketch buffers together into one.- Parameters:
left- The first KLL float sketch. A column that evaluates to a binary.right- The second KLL float sketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_merge_double
Merges two KLL double sketch buffers together into one.- Parameters:
left- The first KLL double sketch. A column that evaluates to a binary.right- The second KLL double sketch. A column that evaluates to a binary.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
kll_sketch_get_quantile_bigint
Extracts a quantile value from a KLL bigint sketch given an input rank value. The rank can be a single value or an array.- Parameters:
sketch- The KLL bigint sketch binary representation. A column that evaluates to a binary.rank- The rank value(s) to extract (between 0.0 and 1.0). A column that evaluates to a numeric or an array. Must be a constant.- Returns:
- Returns a column that evaluates to a long, or an array of longs when
rankis an array. - Since:
- 4.1.0
-
kll_sketch_get_quantile_float
Extracts a quantile value from a KLL float sketch given an input rank value. The rank can be a single value or an array.- Parameters:
sketch- The KLL float sketch binary representation. A column that evaluates to a binary.rank- The rank value(s) to extract (between 0.0 and 1.0). A column that evaluates to a numeric or an array.- Returns:
- Returns a column that evaluates to a float, or an array of floats when
rankis an array. - Since:
- 4.1.0
-
kll_sketch_get_quantile_double
Extracts a quantile value from a KLL double sketch given an input rank value. The rank can be a single value or an array.- Parameters:
sketch- The KLL double sketch binary representation. A column that evaluates to a binary.rank- The rank value(s) to extract (between 0.0 and 1.0). A column that evaluates to a numeric or an array.- Returns:
- Returns a column that evaluates to a double, or an array of doubles when
rankis an array. - Since:
- 4.1.0
-
kll_sketch_get_rank_bigint
Extracts a rank value from a KLL bigint sketch given an input quantile value. The quantile can be a single value or an array.- Parameters:
sketch- The KLL bigint sketch binary representation. A column that evaluates to a binary.quantile- The quantile value(s) to lookup. A column that evaluates to an integral or an array.- Returns:
- Returns a column that evaluates to a double, or an array of doubles when
quantileis an array. - Since:
- 4.1.0
-
kll_sketch_get_rank_float
Extracts a rank value from a KLL float sketch given an input quantile value. The quantile can be a single value or an array.- Parameters:
sketch- The KLL float sketch binary representation. A column that evaluates to a binary.quantile- The quantile value(s) to lookup. A column that evaluates to a numeric or an array.- Returns:
- Returns a column that evaluates to a double, or an array of doubles when
quantileis an array. - Since:
- 4.1.0
-
kll_sketch_get_rank_double
Extracts a rank value from a KLL double sketch given an input quantile value. The quantile can be a single value or an array.- Parameters:
sketch- The KLL double sketch binary representation. A column that evaluates to a binary.quantile- The quantile value(s) to look up. A column that evaluates to a numeric or an array. Must be a constant.- Returns:
- Returns a column that evaluates to a double, or an array of doubles when
quantileis an array. - Since:
- 4.1.0
-
add_months
Returns the date that isnumMonthsafterstartDate.- Parameters:
startDate- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.numMonths- The number of months to add tostartDate, can be negative to subtract months. A column that evaluates to an integer.- Returns:
- A date, or null if
startDatewas a string that could not be cast to a date. Returns a column that evaluates to a date. - Since:
- 1.5.0
-
add_months
Returns the date that isnumMonthsafterstartDate.- Parameters:
startDate- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.numMonths- A column of the number of months to add tostartDate, can be negative to subtract months. A column that evaluates to an integer.- Returns:
- A date, or null if
startDatewas a string that could not be cast to a date. Returns a column that evaluates to a date. - Since:
- 3.0.0
-
curdate
Returns the current date at the start of query evaluation as a date column. All calls of current_date within the same query return the same value.- Returns:
- Returns a column that evaluates to a date.
- Since:
- 3.5.0
-
current_date
Returns the current date at the start of query evaluation as a date column. All calls of current_date within the same query return the same value.- Returns:
- Returns a column that evaluates to a date.
- Since:
- 1.5.0
-
current_timezone
Returns the current session local timezone.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
current_timestamp
Returns the current timestamp at the start of query evaluation as a timestamp column. All calls of current_timestamp within the same query return the same value.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 1.5.0
-
now
Returns the current timestamp at the start of query evaluation.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
localtimestamp
Returns the current timestamp without time zone at the start of query evaluation as a timestamp without time zone column. All calls of localtimestamp within the same query return the same value.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.3.0
-
date_format
Converts a date/timestamp/string to a value of string in the format specified by the date format given by the second argument.See Datetime Patterns for valid date and time format patterns
- Parameters:
dateExpr- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp or time.format- A patterndd.MM.yyyywould return a string like18.03.1993. A column that evaluates to a string.- Returns:
- A string, or null if
dateExprwas a string that could not be cast to a timestamp. Returns a column that evaluates to a string. - Throws:
IllegalArgumentException- if theformatpattern is invalid- Since:
- 1.5.0
- Note:
- Use specialized functions like
year(org.apache.spark.sql.Column)whenever possible as they benefit from a specialized implementation.
-
date_add
Returns the date that isdaysdays afterstart- Parameters:
start- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.days- The number of days to add tostart, can be negative to subtract days. A column that evaluates to an integer, short, or byte.- Returns:
- A date, or null if
startwas a string that could not be cast to a date. Returns a column that evaluates to a date. - Since:
- 1.5.0
-
date_add
Returns the date that isdaysdays afterstart- Parameters:
start- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.days- A column of the number of days to add tostart, can be negative to subtract days. A column that evaluates to an integer, short, or byte.- Returns:
- A date, or null if
startwas a string that could not be cast to a date. Returns a column that evaluates to a date. - Since:
- 3.0.0
-
dateadd
Returns the date that isdaysdays afterstart- Parameters:
start- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.days- A column of the number of days to add tostart, can be negative to subtract days. A column that evaluates to an integer, short, or byte.- Returns:
- A date, or null if
startwas a string that could not be cast to a date. Returns a column that evaluates to a date. - Since:
- 3.5.0
-
date_sub
Returns the date that isdaysdays beforestart- Parameters:
start- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.days- The number of days to subtract fromstart, can be negative to add days. A column that evaluates to an integer, short, or byte.- Returns:
- A date, or null if
startwas a string that could not be cast to a date. Returns a column that evaluates to a date. - Since:
- 1.5.0
-
date_sub
Returns the date that isdaysdays beforestart- Parameters:
start- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.days- A column of the number of days to subtract fromstart, can be negative to add days. A column that evaluates to an integer, short, or byte.- Returns:
- A date, or null if
startwas a string that could not be cast to a date. Returns a column that evaluates to a date. - Since:
- 3.0.0
-
datediff
Returns the number of days fromstarttoend.Only considers the date part of the input. For example:
datediff("2018-01-10 00:00:00", "2018-01-09 23:59:59") // returns 1- Parameters:
end- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.start- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.- Returns:
- An integer, or null if either
endorstartwere strings that could not be cast to a date. Negative ifendis beforestart. Returns a column that evaluates to an integer. - Since:
- 1.5.0
-
date_diff
Returns the number of days fromstarttoend.Only considers the date part of the input. For example:
date_diff("2018-01-10 00:00:00", "2018-01-09 23:59:59") // returns 1- Parameters:
end- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.start- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.- Returns:
- An integer, or null if either
endorstartwere strings that could not be cast to a date. Negative ifendis beforestart. Returns a column that evaluates to an integer. - Since:
- 3.5.0
-
date_from_unix_date
Create date from the number ofdayssince 1970-01-01.- Parameters:
days- The number of days since 1970-01-01. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a date.
- Since:
- 3.5.0
-
year
Extracts the year as an integer from a given date/timestamp/string.- Parameters:
e- The date, timestamp or string to extract the year from. A column that evaluates to a date, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
quarter
Extracts the quarter as an integer from a given date/timestamp/string.- Parameters:
e- The date, timestamp or string to extract the quarter from. A column that evaluates to a date, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
month
Extracts the month as an integer from a given date/timestamp/string.- Parameters:
e- The date, timestamp or string to extract the month from. A column that evaluates to a date, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
dayofweek
Extracts the day of the week as an integer from a given date/timestamp/string. Ranges from 1 for a Sunday through to 7 for a Saturday- Parameters:
e- The date, timestamp or string to extract the day of the week from. A column that evaluates to a date, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 2.3.0
-
dayofmonth
Extracts the day of the month as an integer from a given date/timestamp/string.- Parameters:
e- The date, timestamp or string to extract the day of the month from. A column that evaluates to a date, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
day
Extracts the day of the month as an integer from a given date/timestamp/string.- Parameters:
e- The date, timestamp or string to extract the day of the month from. A column that evaluates to a date, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
dayofyear
Extracts the day of the year as an integer from a given date/timestamp/string.- Parameters:
e- The date, timestamp or string to extract the day of the year from. A column that evaluates to a date, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
hour
Extracts the hours as an integer from a given date/time/timestamp/string. The input may also be a nanosecond-precision timestampTIMESTAMP_NTZ(p)orTIMESTAMP_LTZ(p)(pin[7, 9], since 4.3.0), in which case the sub-microsecond digits are ignored.- Parameters:
e- The column to extract the hours from. A column that evaluates to a date, time, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
extract
Extracts a part of the date/timestamp or interval source.- Parameters:
field- selects which part of the source should be extracted.source- a date, time, timestamp or interval column from wherefieldshould be extracted.- Returns:
- a part of the date/timestamp or interval source. Returns a column whose type depends on the
field to extract, e.g. an integer for
YEARand a decimal forSECOND. - Since:
- 3.5.0
-
date_part
Extracts a part of the date/timestamp or interval source.- Parameters:
field- selects which part of the source should be extracted, and supported string values are as same as the fields of the equivalent functionextract.source- a date/timestamp or time or interval column from wherefieldshould be extracted.- Returns:
- a part of the date/timestamp or interval source. Returns a column whose type depends on the
field to extract, e.g. an integer for
YEARand a decimal forSECOND. - Since:
- 3.5.0
-
datepart
Extracts a part of the date/timestamp or interval source.- Parameters:
field- selects which part of the source should be extracted, and supported string values are as same as the fields of the equivalent functionEXTRACT.source- a date/timestamp or interval column from wherefieldshould be extracted.- Returns:
- a part of the date/timestamp or interval source. Returns a column whose type depends on the
field to extract, e.g. an integer for
YEARand a decimal forSECOND. - Since:
- 3.5.0
-
last_day
Returns the last day of the month which the given date belongs to. For example, input "2015-07-27" returns "2015-07-31" since July 31 is the last day of the month in July 2015.- Parameters:
e- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.- Returns:
- A date, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to a date.
- Since:
- 1.5.0
-
minute
Extracts the minutes as an integer from a given date/time/timestamp/string. The input may also be a nanosecond-precision timestampTIMESTAMP_NTZ(p)orTIMESTAMP_LTZ(p)(pin[7, 9], since 4.3.0), in which case the sub-microsecond digits are ignored.- Parameters:
e- The column to extract the minutes from. A column that evaluates to a date, time, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
weekday
Returns the day of the week for date/timestamp (0 = Monday, 1 = Tuesday, ..., 6 = Sunday).- Parameters:
e- The column to extract the day of the week from. A column that evaluates to a date, timestamp or string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
make_date
- Parameters:
year- The year to build the date. A column that evaluates to an integral.month- The month to build the date. A column that evaluates to an integral.day- The day to build the date. A column that evaluates to an integral.- Returns:
- A date created from year, month and day fields. Returns a column that evaluates to a date.
- Since:
- 3.3.0
-
months_between
Returns number of months between datesstartandend.A whole number is returned if both inputs have the same day of month or both are the last day of their respective months. Otherwise, the difference is calculated assuming 31 days per month.
For example:
months_between("2017-11-14", "2017-07-14") // returns 4.0 months_between("2017-01-01", "2017-01-10") // returns 0.29032258 months_between("2017-06-01", "2017-06-16 12:00:00") // returns -0.5- Parameters:
end- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp.start- A date, timestamp or string. If a string, the data must be in a format that can cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp.- Returns:
- A double, or null if either
endorstartwere strings that could not be cast to a timestamp. Negative ifendis beforestart. Returns a column that evaluates to a double. - Since:
- 1.5.0
-
months_between
Returns number of months between datesendandstart. IfroundOffis set to true, the result is rounded off to 8 digits; it is not rounded otherwise.- Parameters:
end- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp.start- A date, timestamp or string. If a string, the data must be in a format that can cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp.roundOff- Whether to round off the result to 8 digits. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 2.4.0
-
next_day
Returns the first date which is later than the value of thedatecolumn that is on the specified day of the week.For example,
next_day('2015-07-27', "Sunday")returns 2015-08-02 because that is the first Sunday after 2015-07-27.- Parameters:
date- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.dayOfWeek- Case insensitive, and accepts: "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun". A column that evaluates to a string.- Returns:
- A date, or null if
datewas a string that could not be cast to a date or ifdayOfWeekwas an invalid value. Returns a column that evaluates to a date. - Since:
- 1.5.0
-
next_day
Returns the first date which is later than the value of thedatecolumn that is on the specified day of the week.For example,
next_day('2015-07-27', "Sunday")returns 2015-08-02 because that is the first Sunday after 2015-07-27.- Parameters:
date- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.dayOfWeek- A column of the day of week. Case insensitive, and accepts: "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun". A column that evaluates to a string.- Returns:
- A date, or null if
datewas a string that could not be cast to a date or ifdayOfWeekwas an invalid value. Returns a column that evaluates to a date. - Since:
- 3.2.0
-
second
Extracts the seconds as an integer from a given date/time/timestamp/string. The input may also be a nanosecond-precision timestampTIMESTAMP_NTZ(p)orTIMESTAMP_LTZ(p)(pin[7, 9], since 4.3.0), in which case the sub-microsecond digits are ignored.- Parameters:
e- The column to extract the seconds from. A column that evaluates to a date, time, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a timestamp. Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
weekofyear
Extracts the week number as an integer from a given date/timestamp/string.A week is considered to start on a Monday and week 1 is the first week with more than 3 days, as defined by ISO 8601
- Parameters:
e- The column to extract the week number from. A column that evaluates to a date, timestamp or string.- Returns:
- An integer, or null if the input was a string that could not be cast to a date. Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
from_unixtime
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the yyyy-MM-dd HH:mm:ss format.- Parameters:
ut- A number of a type that is castable to a long, such as string or integer. Can be negative for timestamps before the unix epoch- Returns:
- A string, or null if the input was a string that could not be cast to a long. Returns a column that evaluates to a string.
- Since:
- 1.5.0
-
from_unixtime
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.See Datetime Patterns for valid date and time format patterns
- Parameters:
ut- A number of a type that is castable to a long, such as string or integer. Can be negative for timestamps before the unix epochf- A date time pattern that the input will be formatted to- Returns:
- A string, or null if
utwas a string that could not be cast to a long orfwas an invalid date time pattern. Returns a column that evaluates to a string. - Since:
- 1.5.0
-
unix_timestamp
Returns the current Unix timestamp (in seconds) as a long.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 1.5.0
- Note:
- All calls of
unix_timestampwithin the same query return the same value (i.e. the current timestamp is calculated at the start of query evaluation).
-
unix_timestamp
Converts time string in format yyyy-MM-dd HH:mm:ss to Unix timestamp (in seconds), using the default timezone and the default locale.- Parameters:
s- A date, timestamp or string. If a string, the data must be in theyyyy-MM-dd HH:mm:ssformat- Returns:
- A long, or null if the input was a string not of the correct format. Returns a column that evaluates to a long.
- Since:
- 1.5.0
-
unix_timestamp
Converts time string with given pattern to Unix timestamp (in seconds).See Datetime Patterns for valid date and time format patterns
- Parameters:
s- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a string, date, or timestamp.p- A date time pattern detailing the format ofswhensis a string. A column that evaluates to a string.- Returns:
- A long, or null if
swas a string that could not be cast to a date orpwas an invalid format. Returns a column that evaluates to a long. - Since:
- 1.5.0
-
to_time
Parses a string value to a time value.- Parameters:
str- A string to be parsed to time. A column that evaluates to a string.- Returns:
- A time, or raises an error if the input is malformed. Returns a column that evaluates to a time.
- Since:
- 4.1.0
-
to_time
Parses a string value to a time value.See Datetime Patterns for valid time format patterns.
- Parameters:
str- A string to be parsed to time.format- A time format pattern to follow. A column that evaluates to a string.- Returns:
- A time, or raises an error if the input is malformed. Returns a column that evaluates to a time.
- Since:
- 4.1.0
-
to_timestamp
Converts to a timestamp by casting rules toTimestampType.- Parameters:
s- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a string, date, timestamp, or numeric.- Returns:
- A timestamp, or null if the input was a string that could not be cast to a timestamp. Returns a column that evaluates to a timestamp.
- Since:
- 2.2.0
-
to_timestamp
Converts time string with the given pattern to timestamp.See Datetime Patterns for valid date and time format patterns
- Parameters:
s- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a string, date, timestamp, or numeric.fmt- A date time pattern detailing the format ofswhensis a string. A column that evaluates to a string.- Returns:
- A timestamp, or null if
swas a string that could not be cast to a timestamp orfmtwas an invalid format. Returns a column that evaluates to a timestamp. - Since:
- 2.2.0
-
try_to_time
Parses a string value to a time value.- Parameters:
str- A string to be parsed to time. A column that evaluates to a string.- Returns:
- A time, or null if the input is malformed. Returns a column that evaluates to a time.
- Since:
- 4.1.0
-
try_to_time
Parses a string value to a time value.See Datetime Patterns for valid time format patterns.
- Parameters:
str- A string to be parsed to time.format- A time format pattern to follow. A column that evaluates to a string.- Returns:
- A time, or null if the input is malformed. Returns a column that evaluates to a time.
- Since:
- 4.1.0
-
try_to_timestamp
Parses theswith theformatto a timestamp. The function always returns null on an invalid input with/without ANSI SQL mode enabled. The result data type is consistent with the value of configurationspark.sql.timestampType.- Parameters:
s- Column values to convert. A column that evaluates to a string, date, timestamp, or numeric.format- Format to use to convert timestamp values. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
try_to_timestamp
Parses thesto a timestamp. The function always returns null on an invalid input with/without ANSI SQL mode enabled. It follows casting rules to a timestamp. The result data type is consistent with the value of configurationspark.sql.timestampType.- Parameters:
s- Column values to convert. A column that evaluates to a string, date, timestamp, or numeric.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
to_date
Converts the column intoDateTypeby casting rules toDateType.- Parameters:
e- Input column of values to convert. A column that evaluates to a string, date, or timestamp.- Returns:
- Returns a column that evaluates to a date.
- Since:
- 1.5.0
-
to_date
Converts the column into aDateTypewith a specified formatSee Datetime Patterns for valid date and time format patterns
- Parameters:
e- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a string, date, or timestamp.fmt- A date time pattern detailing the format ofewheneis a string. A column that evaluates to a string.- Returns:
- A date, or null if
ewas a string that could not be cast to a date orfmtwas an invalid format. Returns a column that evaluates to a date. - Since:
- 2.2.0
-
try_to_date
This is a special version ofto_datethat performs the same operation, but returns a NULL value instead of raising an error if date cannot be created.- Parameters:
e- Input column of values to convert. A column that evaluates to a string, date, or timestamp.- Returns:
- Returns a column that evaluates to a date.
- Since:
- 4.1.0
-
try_to_date
This is a special version ofto_datethat performs the same operation, but returns a NULL value instead of raising an error if date cannot be created.- Parameters:
e- Input column of values to convert. A column that evaluates to a string, date, or timestamp.fmt- Format to use to convert date values. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a date.
- Since:
- 4.1.0
-
unix_date
Returns the number of days since 1970-01-01.- Parameters:
e- Input column of values to convert. A column that evaluates to a date.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
unix_micros
Returns the number of microseconds since 1970-01-01 00:00:00 UTC.- Parameters:
e- Input column of values to convert. A column that evaluates to a timestamp.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
unix_nanos
Returns the number of nanoseconds since 1970-01-01 00:00:00 UTC for a nanosecond-precision timestamp (TIMESTAMP_LTZ(p)/TIMESTAMP_NTZ(p),pin[7, 9]). The result is a losslessDECIMAL(21, 0).- Parameters:
e- input column of nanosecond-precision timestamp values to convert. A column that evaluates to a timestamp.- Returns:
- Returns a column that evaluates to a decimal.
- Since:
- 4.3.0
-
unix_millis
Returns the number of milliseconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision.- Parameters:
e- input column of values to convert. A column that evaluates to a timestamp.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
unix_seconds
Returns the number of seconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision.- Parameters:
e- input column of values to convert. A column that evaluates to a timestamp.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
trunc
Returns date truncated to the unit specified by the format.For example,
trunc("2018-11-19 12:01:19", "year")returns 2018-01-01- Parameters:
date- A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a date.format- : 'year', 'yyyy', 'yy' to truncate by year, or 'month', 'mon', 'mm' to truncate by month Other options are: 'week', 'quarter'. A column that evaluates to a string.- Returns:
- A date, or null if
datewas a string that could not be cast to a date orformatwas an invalid value. Returns a column that evaluates to a date. - Since:
- 1.5.0
-
date_trunc
Returns timestamp truncated to the unit specified by the format.For example,
date_trunc("year", "2018-11-19 12:01:19")returns 2018-01-01 00:00:00- Parameters:
format- : 'year', 'yyyy', 'yy' to truncate by year, 'month', 'mon', 'mm' to truncate by month, 'day', 'dd' to truncate by day, Other options are: 'microsecond', 'millisecond', 'second', 'minute', 'hour', 'week', 'quarter'. A column that evaluates to a string.timestamp- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp.- Returns:
- A timestamp, or null if
timestampwas a string that could not be cast to a timestamp orformatwas an invalid value. Returns a column that evaluates to a timestamp. - Since:
- 2.3.0
-
from_utc_timestamp
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone. For example, 'GMT+1' would yield '2017-07-14 03:40:00.0'.- Parameters:
ts- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp.tz- A string detailing the time zone ID that the input should be adjusted to. It should be in the format of either region-based zone IDs or zone offsets. Region IDs must have the form 'area/city', such as 'America/Los_Angeles'. Zone offsets must be in the format '(+|-)HH:mm', for example '-08:00' or '+01:00'. Also 'UTC' and 'Z' are supported as aliases of '+00:00'. Other short names are not recommended to use because they can be ambiguous. A column that evaluates to a string.- Returns:
- A timestamp, or null if
tswas a string that could not be cast to a timestamp ortzwas an invalid value. Returns a column that evaluates to a timestamp. - Since:
- 1.5.0
-
from_utc_timestamp
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone. For example, 'GMT+1' would yield '2017-07-14 03:40:00.0'.- Parameters:
ts- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp.tz- A string detailing the time zone ID that the input should be adjusted to. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 2.4.0
-
to_utc_timestamp
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'.- Parameters:
ts- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp.tz- A string detailing the time zone ID that the input should be adjusted to. It should be in the format of either region-based zone IDs or zone offsets. Region IDs must have the form 'area/city', such as 'America/Los_Angeles'. Zone offsets must be in the format '(+|-)HH:mm', for example '-08:00' or '+01:00'. Also 'UTC' and 'Z' are supported as aliases of '+00:00'. Other short names are not recommended to use because they can be ambiguous. A column that evaluates to a string.- Returns:
- A timestamp, or null if
tswas a string that could not be cast to a timestamp ortzwas an invalid value. Returns a column that evaluates to a timestamp. - Since:
- 1.5.0
-
to_utc_timestamp
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'.- Parameters:
ts- A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such asyyyy-MM-ddoryyyy-MM-dd HH:mm:ss.SSSS. A column that evaluates to a timestamp.tz- A string detailing the time zone ID that the input should be adjusted to. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 2.4.0
-
window
public static Column window(Column timeColumn, String windowDuration, String slideDuration, String startTime) Bucketize rows into one or more time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The following example takes the average stock price for a one minute window every 10 seconds starting 5 seconds after the hour:val df = ... // schema => timestamp: TimestampType, stockId: StringType, price: DoubleType df.groupBy(window($"timestamp", "1 minute", "10 seconds", "5 seconds"), $"stockId") .agg(mean("price"))The windows will look like:
09:00:05-09:01:05 09:00:15-09:01:15 09:00:25-09:01:25 ...For a streaming query, you may use the function
current_timestampto generate windows on processing time.- Parameters:
timeColumn- The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType. A column that evaluates to a timestamp.windowDuration- A string specifying the width of the window, e.g.10 minutes,1 second. Checkorg.apache.spark.unsafe.types.CalendarIntervalfor valid duration identifiers. Note that the duration is a fixed length of time, and does not vary over time according to a calendar. For example,1 dayalways means 86,400,000 milliseconds, not a calendar day. A column that evaluates to a string.slideDuration- A string specifying the sliding interval of the window, e.g.1 minute. A new window will be generated everyslideDuration. Must be less than or equal to thewindowDuration. Checkorg.apache.spark.unsafe.types.CalendarIntervalfor valid duration identifiers. This duration is likewise absolute, and does not vary according to a calendar. A column that evaluates to a string.startTime- The offset with respect to 1970-01-01 00:00:00 UTC with which to start window intervals. For example, in order to have hourly tumbling windows that start 15 minutes past the hour, e.g. 12:15-13:15, 13:15-14:15... providestartTimeas15 minutes. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 2.0.0
-
window
Bucketize rows into one or more time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The windows start beginning at 1970-01-01 00:00:00 UTC. The following example takes the average stock price for a one minute window every 10 seconds:val df = ... // schema => timestamp: TimestampType, stockId: StringType, price: DoubleType df.groupBy(window($"timestamp", "1 minute", "10 seconds"), $"stockId") .agg(mean("price"))The windows will look like:
09:00:00-09:01:00 09:00:10-09:01:10 09:00:20-09:01:20 ...For a streaming query, you may use the function
current_timestampto generate windows on processing time.- Parameters:
timeColumn- The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType. A column that evaluates to a timestamp.windowDuration- A string specifying the width of the window, e.g.10 minutes,1 second. Checkorg.apache.spark.unsafe.types.CalendarIntervalfor valid duration identifiers. Note that the duration is a fixed length of time, and does not vary over time according to a calendar. For example,1 dayalways means 86,400,000 milliseconds, not a calendar day. A column that evaluates to a string.slideDuration- A string specifying the sliding interval of the window, e.g.1 minute. A new window will be generated everyslideDuration. Must be less than or equal to thewindowDuration. Checkorg.apache.spark.unsafe.types.CalendarIntervalfor valid duration identifiers. This duration is likewise absolute, and does not vary according to a calendar. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 2.0.0
-
window
Generates tumbling time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The windows start beginning at 1970-01-01 00:00:00 UTC. The following example takes the average stock price for a one minute tumbling window:val df = ... // schema => timestamp: TimestampType, stockId: StringType, price: DoubleType df.groupBy(window($"timestamp", "1 minute"), $"stockId") .agg(mean("price"))The windows will look like:
09:00:00-09:01:00 09:01:00-09:02:00 09:02:00-09:03:00 ...For a streaming query, you may use the function
current_timestampto generate windows on processing time.- Parameters:
timeColumn- The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType. A column that evaluates to a timestamp.windowDuration- A string specifying the width of the window, e.g.10 minutes,1 second. Checkorg.apache.spark.unsafe.types.CalendarIntervalfor valid duration identifiers. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 2.0.0
-
window_time
Extracts the event time from the window column.The window column is of StructType { start: Timestamp, end: Timestamp } where start is inclusive and end is exclusive. Since event time can support microsecond precision, window_time(window) = window.end - 1 microsecond.
- Parameters:
windowColumn- The window column (typically produced by window aggregation) of type StructType { start: Timestamp, end: Timestamp }. A column that evaluates to a struct.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.4.0
-
session_window
Generates session window given a timestamp specifying column.Session window is one of dynamic windows, which means the length of window is varying according to the given inputs. The length of session window is defined as "the timestamp of latest input of the session + gap duration", so when the new inputs are bound to the current session window, the end time of session window can be expanded according to the new inputs.
Windows can support microsecond precision. gapDuration in the order of months are not supported.
For a streaming query, you may use the function
current_timestampto generate windows on processing time.- Parameters:
timeColumn- The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType. A column that evaluates to a timestamp.gapDuration- A string specifying the timeout of the session, e.g.10 minutes,1 second. Checkorg.apache.spark.unsafe.types.CalendarIntervalfor valid duration identifiers. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 3.2.0
-
session_window
Generates session window given a timestamp specifying column.Session window is one of dynamic windows, which means the length of window is varying according to the given inputs. For static gap duration, the length of session window is defined as "the timestamp of latest input of the session + gap duration", so when the new inputs are bound to the current session window, the end time of session window can be expanded according to the new inputs.
Besides a static gap duration value, users can also provide an expression to specify gap duration dynamically based on the input row. With dynamic gap duration, the closing of a session window does not depend on the latest input anymore. A session window's range is the union of all events' ranges which are determined by event start time and evaluated gap duration during the query execution. Note that the rows with negative or zero gap duration will be filtered out from the aggregation.
Windows can support microsecond precision. gapDuration in the order of months are not supported.
For a streaming query, you may use the function
current_timestampto generate windows on processing time.- Parameters:
timeColumn- The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType. A column that evaluates to a timestamp.gapDuration- A column specifying the timeout of the session. It could be static value, e.g.10 minutes,1 second, or an expression/UDF that specifies gap duration dynamically based on the input row. A column that evaluates to a string or interval.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 3.2.0
-
timestamp_seconds
Converts the number of seconds from the Unix epoch (1970-01-01T00:00:00Z) to a timestamp.- Parameters:
e- unix time values. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.1.0
-
timestamp_millis
Creates timestamp from the number of milliseconds since UTC epoch.- Parameters:
e- unix time values. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
timestamp_micros
Creates timestamp from the number of microseconds since UTC epoch.- Parameters:
e- unix time values. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
timestamp_nanos
Creates a timestamp with the local time zone and nanosecond precision (TIMESTAMP_LTZ(9)) from the number of nanoseconds since UTC epoch.- Parameters:
e- nanosecond values since the UTC epoch. A column that evaluates to an integral or decimal.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.3.0
-
timestamp_diff
Gets the difference between the timestamps in the specified units by truncating the fraction part.- Parameters:
unit- the units of the difference between the given timestamps, e.g. 'YEAR', 'MONTH', 'DAY', 'HOUR'. A column that evaluates to a string. Must be a constant.start- A timestamp which the expression subtracts fromend. A column that evaluates to a timestamp.end- A timestamp from which the expression subtractsstart. A column that evaluates to a timestamp.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.0.0
-
timestamp_add
Adds the specified number of units to the given timestamp.- Parameters:
unit- the units of datetime to add, e.g. 'YEAR', 'MONTH', 'DAY', 'HOUR'. A column that evaluates to a string. Must be a constant.quantity- the number of units of time to add. A column that evaluates to an integral.ts- A timestamp to which to add. A column that evaluates to a timestamp.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
time_bucket
Returns the start of the fixed-size bucket ofbucketSizethat containsts, with buckets aligned to the default origin (1970-01-01 00:00:00). ForTIMESTAMP_NTZ, bucketing is performed in UTC. ForTIMESTAMP, year-month interval buckets and calendar-day components of day-time interval buckets align to the session time zone.- Parameters:
bucketSize- A day-time or year-month interval defining the bucket size. Must be positive and foldable.ts- A TIMESTAMP or TIMESTAMP_NTZ value to bucket.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.2.0
-
time_bucket
Returns the start of the fixed-size bucket ofbucketSizethat containsts, with buckets aligned toorigin. ForTIMESTAMP_NTZ, bucketing is performed in UTC. ForTIMESTAMP, year-month interval buckets and calendar-day components of day-time interval buckets align to the session time zone.- Parameters:
bucketSize- A day-time or year-month interval defining the bucket size. Must be positive and foldable.ts- A TIMESTAMP or TIMESTAMP_NTZ value to bucket.origin- Alignment anchor. Must be the same type astsand must be foldable.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.2.0
-
time_diff
Returns the difference between two times, measured in specified units. Throws a SparkIllegalArgumentException, in case the specified unit is not supported.- Parameters:
unit- A STRING representing the unit of the time difference. Supported units are: "HOUR", "MINUTE", "SECOND", "MILLISECOND", and "MICROSECOND". The unit is case-insensitive. A column that evaluates to a string.start- A starting TIME. A column that evaluates to a time.end- An ending TIME. A column that evaluates to a time.- Returns:
- The difference between
endandstarttimes, measured in specified units. Returns a column that evaluates to a long. - Since:
- 4.1.0
- Note:
- If any of the inputs is
NULL, the result isNULL.
-
time_trunc
Returnstimetruncated to theunit.- Parameters:
unit- A STRING representing the unit to truncate the time to. Supported units are: "HOUR", "MINUTE", "SECOND", "MILLISECOND", and "MICROSECOND". The unit is case-insensitive. A column that evaluates to a string.time- A TIME to truncate. A column that evaluates to a time.- Returns:
- A TIME truncated to the specified unit. Returns a column that evaluates to a time.
- Throws:
IllegalArgumentException- If theunitis not supported.- Since:
- 4.1.0
- Note:
- If any of the inputs is
NULL, the result isNULL.
-
time_from_seconds
Creates a TIME from the number of seconds since midnight.- Parameters:
e- seconds since midnight (0 to 86399.999999). A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a time.
- Since:
- 4.2.0
-
time_from_millis
Creates a TIME from the number of milliseconds since midnight.- Parameters:
e- milliseconds since midnight (0 to 86399999). A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a time.
- Since:
- 4.2.0
-
time_from_micros
Creates a TIME from the number of microseconds since midnight.- Parameters:
e- microseconds since midnight (0 to 86399999999). A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to a time.
- Since:
- 4.2.0
-
time_to_seconds
Extracts the number of seconds (including fractional seconds) from a TIME value. Returns a DECIMAL(14,6) to preserve microsecond precision.- Parameters:
e- TIME value to convert. A column that evaluates to a time.- Returns:
- Returns a column that evaluates to a decimal.
- Since:
- 4.2.0
-
time_to_millis
Extracts the number of milliseconds since midnight from a TIME value.- Parameters:
e- the TIME value to convert. A column that evaluates to a time.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.2.0
-
time_to_micros
Extracts the number of microseconds since midnight from a TIME value.- Parameters:
e- the TIME value to convert. A column that evaluates to a time.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 4.2.0
-
to_timestamp_ltz
Parses thetimestampexpression with theformatexpression to a timestamp with local time zone. Returns null with invalid input.- Parameters:
timestamp- the input column or strings. A column that evaluates to a date, timestamp or string.format- the format used to parse the timestamp values. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
to_timestamp_ltz
Parses thetimestampexpression with the default format to a timestamp with local time zone. The default format follows casting rules to a timestamp. Returns null with invalid input.- Parameters:
timestamp- the input column or strings. A column that evaluates to a date, timestamp or string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
to_timestamp_ntz
Parses thetimestamp_strexpression with theformatexpression to a timestamp without time zone. Returns null with invalid input.- Parameters:
timestamp- the input column or strings. A column that evaluates to a date, timestamp or string.format- the format used to parse the timestamp values. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
to_timestamp_ntz
Parses thetimestampexpression with the default format to a timestamp without time zone. The default format follows casting rules to a timestamp. Returns null with invalid input.- Parameters:
timestamp- the input column or strings. A column that evaluates to a date, timestamp or string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
to_unix_timestamp
Returns the UNIX timestamp of the given time.- Parameters:
timeExp- the input column or strings. A column that evaluates to a date, timestamp or string.format- the format used to convert the time values. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
to_unix_timestamp
Returns the UNIX timestamp of the given time.- Parameters:
timeExp- the input column or strings. A column that evaluates to a date, timestamp or string.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
monthname
Extracts the three-letter abbreviated month name from a given date/timestamp/string.- Parameters:
timeExp- the target date/timestamp to work on. A column that evaluates to a date, timestamp or string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
dayname
Extracts the three-letter abbreviated day name from a given date/timestamp/string.- Parameters:
timeExp- the target date/timestamp to work on. A column that evaluates to a date, timestamp or string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
array_contains
Returns true if the array containsvalue, false if not. Returns null if the array orvalueis null, or ifvalueis not found and the array contains a null element.- Parameters:
column- the target column containing the arrays. A column that evaluates to an array.value- the value to check for in the array. A column that evaluates to a value matching the array's element type.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 1.5.0
-
array_append
Returns an ARRAY containing all elements from the source ARRAY as well as the new element. The new element/column is located at end of the ARRAY.- Parameters:
column- the source column containing the array. A column that evaluates to an array.element- the value to append to the array. A column that evaluates to a value matching the array's element type.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.4.0
-
arrays_overlap
Returnstrueifa1anda2have at least one non-null element in common. If not and both the arrays are non-empty and any of them contains anull, it returnsnull. It returnsfalseotherwise.- Parameters:
a1- the first input array. A column that evaluates to an array.a2- the second input array. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 2.4.0
-
slice
Returns an array containing all the elements inxfrom indexstart(or starting from the end ifstartis negative) with the specifiedlength.- Parameters:
x- the array column to be sliced. A column that evaluates to an array.start- the starting index. A column that evaluates to an integer.length- the length of the slice. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
slice
Returns an array containing all the elements inxfrom indexstart(or starting from the end ifstartis negative) with the specifiedlength.- Parameters:
x- the array column to be sliced. A column that evaluates to an array.start- the starting index. A column that evaluates to an integer.length- the length of the slice. A column that evaluates to an integer.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.1.0
-
array_join
Concatenates the elements ofcolumnusing thedelimiter. Null values are replaced withnullReplacement.- Parameters:
column- the input column containing the array. A column that evaluates to an array.delimiter- the string used to join the array elements. A column that evaluates to a string.nullReplacement- the string used to replace null values. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.4.0
-
array_join
Concatenates the elements ofcolumnusing thedelimiter.- Parameters:
column- the input column containing the array. A column that evaluates to an array.delimiter- the string used to join the array elements. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.4.0
-
concat
Concatenates multiple input columns together into a single column. The function works with strings, binary and compatible array columns.- Parameters:
exprs- Input columns to concatenate. A column that evaluates to a string, binary or an array.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
- Note:
- Returns null if any of the input columns are null.
-
array_position
Locates the position of the first occurrence of the value in the given array as long. Returns null if either of the arguments are null.- Parameters:
column- The array to search. A column that evaluates to an array.value- The value to locate. A column.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 2.4.0
- Note:
- The position is not zero based, but 1 based index. Returns 0 if value could not be found in array.
-
element_at
Returns element of array at given index in value if column is array. Returns value for the given key in value if column is map.- Parameters:
column- The array or map to extract from. A column that evaluates to an array or a map.value- The 1-based index for arrays, or the key for maps. A column.- Returns:
- Returns a column of the element type of the input array, or the value type of the input map.
- Since:
- 2.4.0
-
try_element_at
(array, index) - Returns element of array at given (1-based) index. If Index is 0, Spark will throw an error. If index < 0, accesses elements from the last to the first. The function always returns NULL if the index exceeds the length of the array.(map, key) - Returns value for given key. The function always returns NULL if the key is not contained in the map.
- Parameters:
column- The array or map to extract from. A column that evaluates to an array or a map.value- The 1-based index for arrays, or the key for maps. A column.- Returns:
- Returns a column of the element type of the input array, or the value type of the input map.
- Since:
- 3.5.0
-
get
Returns element of array at given (0-based) index. If the index points outside of the array boundaries, then this function returns NULL.- Parameters:
column- The array to extract from. A column that evaluates to an array.index- The 0-based index. A column that evaluates to an integral.- Returns:
- Returns a column of the element type of the input array.
- Since:
- 3.4.0
-
array_sort
Sorts the input array in ascending order. Null elements will be placed at the end of the returned array. NaN is greater than any non-NaN elements for double/float type.The elements of the input array must be orderable. For example, when the array elements are structs, the default comparator compares the struct fields in schema order. Therefore, all fields in the struct must be orderable. If the default comparator does not support the input type, you can specify a custom comparator.
- Parameters:
e- The array to sort. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
array_sort
Sorts the input array based on the given comparator function. The comparator will take two arguments representing two elements of the array. It returns a negative integer, 0, or a positive integer as the first element is less than, equal to, or greater than the second element. If the comparator function returns null, the function will fail and raise an error.- Parameters:
e- The array to sort. A column that evaluates to an array.comparator- A binary comparator function that returns a negative integer, 0, or a positive integer as the first element is less than, equal to, or greater than the second element.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.4.0
-
array_remove
Remove all elements that equal to element from the given array.- Parameters:
column- The array to remove from. A column that evaluates to an array.element- The element to remove. A column.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
array_compact
Remove all null elements from the given array.- Parameters:
column- The array to compact. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.4.0
-
array_prepend
Returns an array containing value as well as all elements from array. The new element is positioned at the beginning of the array.- Parameters:
column- The array to prepend to. A column that evaluates to an array.element- The element to prepend. A column.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.5.0
-
array_distinct
Removes duplicate values from the array.- Parameters:
e- The array to deduplicate. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
array_intersect
Returns an array of the elements in the intersection of the given two arrays, without duplicates.- Parameters:
col1- The first array. A column that evaluates to an array.col2- The second array. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
array_insert
Adds an item into a given array at a specified position- Parameters:
arr- The array to insert into. A column that evaluates to an array.pos- The 1-based position at which to insert (negative counts from the end). A column that evaluates to an integral.value- The value to insert. A column.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.4.0
-
array_union
Returns an array of the elements in the union of the given two arrays, without duplicates.- Parameters:
col1- The first array. A column that evaluates to an array.col2- The second array. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
array_except
Returns an array of the elements in the first array but not in the second array, without duplicates. The order of elements in the result is not determined- Parameters:
col1- The first array. A column that evaluates to an array.col2- The second array. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
transform
Returns an array of elements after applying a transformation to each element in the input array.df.select(transform(col("i"), x => x + 1))- Parameters:
column- the input array column. A column that evaluates to an array.f- col => transformed_col, the lambda function to transform the input column.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.0.0
-
transform
Returns an array of elements after applying a transformation to each element in the input array.df.select(transform(col("i"), (x, i) => x + i))- Parameters:
column- the input array column. A column that evaluates to an array.f- (col, index) => transformed_col, the lambda function to transform the input column given the index. Indices start at 0.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.0.0
-
exists
Returns whether a predicate holds for one or more elements in the array.df.select(exists(col("i"), _ % 2 === 0))- Parameters:
column- the input array column. A column that evaluates to an array.f- col => predicate, the Boolean predicate to check the input column.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.0.0
-
forall
Returns whether a predicate holds for every element in the array.df.select(forall(col("i"), x => x % 2 === 0))- Parameters:
column- the input array column. A column that evaluates to an array.f- col => predicate, the Boolean predicate to check the input column.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.0.0
-
filter
Returns an array of elements for which a predicate holds in a given array.df.select(filter(col("s"), x => x % 2 === 0))- Parameters:
column- the input array column. A column that evaluates to an array.f- col => predicate, the Boolean predicate to filter the input column.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.0.0
-
filter
Returns an array of elements for which a predicate holds in a given array.df.select(filter(col("s"), (x, i) => i % 2 === 0))- Parameters:
column- the input array column. A column that evaluates to an array.f- (col, index) => predicate, the Boolean predicate to filter the input column given the index. Indices start at 0.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.0.0
-
aggregate
public static Column aggregate(Column expr, Column initialValue, scala.Function2<Column, Column, Column> merge, scala.Function1<Column, Column> finish) Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state. The final state is converted into the final result by applying a finish function.df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))- Parameters:
expr- the input array column. A column that evaluates to an array.initialValue- the initial value. A column of any type.merge- (combined_value, input_value) => combined_value, the merge function to merge an input value to the combined_value.finish- combined_value => final_value, the lambda function to convert the combined value of all inputs to final result.- Returns:
- Returns a column of the same type as the initial value.
- Since:
- 3.0.0
-
aggregate
public static Column aggregate(Column expr, Column initialValue, scala.Function2<Column, Column, Column> merge) Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x))- Parameters:
expr- the input array column. A column that evaluates to an array.initialValue- the initial value. A column of any type.merge- (combined_value, input_value) => combined_value, the merge function to merge an input value to the combined_value- Returns:
- Returns a column of the same type as the initial value.
- Since:
- 3.0.0
-
reduce
public static Column reduce(Column expr, Column initialValue, scala.Function2<Column, Column, Column> merge, scala.Function1<Column, Column> finish) Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state. The final state is converted into the final result by applying a finish function.df.select(reduce(col("i"), lit(0), (acc, x) => acc + x, _ * 10))- Parameters:
expr- the input array column. A column that evaluates to an array.initialValue- the initial value. A column of any type.merge- (combined_value, input_value) => combined_value, the merge function to merge an input value to the combined_value.finish- combined_value => final_value, the lambda function to convert the combined value of all inputs to final result.- Returns:
- Returns a column of the same type as the initial value.
- Since:
- 3.5.0
-
reduce
public static Column reduce(Column expr, Column initialValue, scala.Function2<Column, Column, Column> merge) Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.df.select(reduce(col("i"), lit(0), (acc, x) => acc + x))- Parameters:
expr- the input array column. A column that evaluates to an array.initialValue- the initial value. A column of any type.merge- (combined_value, input_value) => combined_value, the merge function to merge an input value to the combined_value- Returns:
- Returns a column of the same type as the initial value.
- Since:
- 3.5.0
-
zip_with
Merge two given arrays, element-wise, into a single array using a function. If one array is shorter, nulls are appended at the end to match the length of the longer array, before applying the function.df.select(zip_with(df1("val1"), df1("val2"), (x, y) => x + y))- Parameters:
left- the left input array column. A column that evaluates to an array.right- the right input array column. A column that evaluates to an array.f- (lCol, rCol) => col, the lambda function to merge two input columns into one column.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.0.0
-
transform_keys
Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new keys for the pairs.df.select(transform_keys(col("i"), (k, v) => k + v))- Parameters:
expr- the input map column. A column that evaluates to a map.f- (key, value) => new_key, the lambda function to transform the key of input map column- Returns:
- Returns a column that evaluates to a map.
- Since:
- 3.0.0
-
transform_values
Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new values for the pairs.df.select(transform_values(col("i"), (k, v) => k + v))- Parameters:
expr- the input map column. A column that evaluates to a map.f- (key, value) => new_value, the lambda function to transform the value of input map column- Returns:
- Returns a column that evaluates to a map.
- Since:
- 3.0.0
-
map_filter
Returns a map whose key-value pairs satisfy a predicate.df.select(map_filter(col("m"), (k, v) => k * 10 === v))- Parameters:
expr- the input map column. A column that evaluates to a map.f- (key, value) => predicate, the Boolean predicate to filter the input map column- Returns:
- Returns a column that evaluates to a map.
- Since:
- 3.0.0
-
map_zip_with
public static Column map_zip_with(Column left, Column right, scala.Function3<Column, Column, Column, Column> f) Merge two given maps, key-wise into a single map using a function.df.select(map_zip_with(df("m1"), df("m2"), (k, v1, v2) => k === v1 + v2))- Parameters:
left- the left input map column. A column that evaluates to a map.right- the right input map column. A column that evaluates to a map.f- (key, value1, value2) => new_value, the lambda function to merge the map values- Returns:
- Returns a column that evaluates to a map.
- Since:
- 3.0.0
-
explode
Creates a new row for each element in the given array or map column. Uses the default column namecolfor elements in the array andkeyandvaluefor elements in the map unless specified otherwise.- Parameters:
e- the target column to explode. A column that evaluates to an array or a map.- Returns:
- Returns a column of the element type of the input array, or the key and value columns of the input map.
- Since:
- 1.3.0
-
explode_outer
Creates a new row for each element in the given array or map column. Uses the default column namecolfor elements in the array andkeyandvaluefor elements in the map unless specified otherwise. Unlike explode, if the array/map is null or empty then null is produced.- Parameters:
e- the target column to explode. A column that evaluates to an array or a map.- Returns:
- Returns a column of the element type of the input array, or the key and value columns of the input map.
- Since:
- 2.2.0
-
posexplode
Creates a new row for each element with position in the given array or map column. Uses the default column nameposfor position, andcolfor elements in the array andkeyandvaluefor elements in the map unless specified otherwise.- Parameters:
e- the target column to explode. A column that evaluates to an array or a map.- Returns:
- Returns the position column and a column of the element type of the input array, or the position column and the key and value columns of the input map.
- Since:
- 2.1.0
-
posexplode_outer
Creates a new row for each element with position in the given array or map column. Uses the default column nameposfor position, andcolfor elements in the array andkeyandvaluefor elements in the map unless specified otherwise. Unlike posexplode, if the array/map is null or empty then the row (null, null) is produced.- Parameters:
e- the target column to explode. A column that evaluates to an array or a map.- Returns:
- Returns the position column and a column of the element type of the input array, or the position column and the key and value columns of the input map.
- Since:
- 2.2.0
-
inline
Creates a new row for each element in the given array of structs.- Parameters:
e- the target column to explode. A column that evaluates to an array of structs.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 3.4.0
-
inline_outer
Creates a new row for each element in the given array of structs. Unlike inline, if the array is null or empty then null is produced for each nested column.- Parameters:
e- the target column to explode. A column that evaluates to an array of structs.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 3.4.0
-
get_json_object
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object. It will return null if the input json string is invalid.- Parameters:
e- the JSON string column. A column that evaluates to a string.path- the JSON path to extract. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.6.0
-
json_tuple
Creates a new row for a json column according to the given field names.- Parameters:
json- the JSON string column. A column that evaluates to a string.fields- the field names to extract. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 1.6.0
-
from_json
public static Column from_json(Column e, StructType schema, scala.collection.immutable.Map<String, String> options) (Scala-specific) Parses a column containing a JSON string into aStructTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema to use when parsing the json string. A string, StructType or DataType. Must be a constant.options- options to control how the json is parsed. Accepts the same options as the json data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 2.1.0
-
from_json
public static Column from_json(Column e, DataType schema, scala.collection.immutable.Map<String, String> options) (Scala-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema to use when parsing the json string. A string, StructType or DataType. Must be a constant.options- options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column of the type given by the schema (a struct, array, or map).
- Since:
- 2.2.0
-
from_json
(Java-specific) Parses a column containing a JSON string into aStructTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema to use when parsing the json string. A string, StructType or DataType. Must be a constant.options- options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 2.1.0
-
from_json
(Java-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema to use when parsing the json string. A string, StructType or DataType. Must be a constant.options- options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column of the type given by the schema (a struct, array, or map).
- Since:
- 2.2.0
-
from_json
Parses a column containing a JSON string into aStructTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema to use when parsing the json string. A string, StructType or DataType. Must be a constant.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 2.1.0
-
from_json
Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema to use when parsing the json string. A string, StructType or DataType. Must be a constant.- Returns:
- Returns a column of the type given by the schema (a struct, array, or map).
- Since:
- 2.2.0
-
from_json
(Java-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema as a DDL-formatted string. A string, StructType or DataType. Must be a constant.options- options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column of the type given by the schema (a struct, array, or map).
- Since:
- 2.1.0
-
from_json
public static Column from_json(Column e, String schema, scala.collection.immutable.Map<String, String> options) (Scala-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema as a DDL-formatted string. A string, StructType or DataType. Must be a constant.options- options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column of the type given by the schema (a struct, array, or map).
- Since:
- 2.3.0
-
from_json
(Scala-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypeofStructTypes with the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema to use when parsing the json string. A column that evaluates to a string.- Returns:
- Returns a column of the type given by the schema (a struct, array, or map).
- Since:
- 2.4.0
-
from_json
(Java-specific) Parses a column containing a JSON string into aMapTypewithStringTypeas keys type,StructTypeorArrayTypeofStructTypes with the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing JSON data. A column that evaluates to a string.schema- the schema to use when parsing the json string. A column that evaluates to a string.options- options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column of the type given by the schema (a struct, array, or map).
- Since:
- 2.4.0
-
try_parse_json
Parses a JSON string and constructs a Variant value. Returns null if the input string is not a valid JSON value.- Parameters:
json- a string column that contains JSON data. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a variant.
- Since:
- 4.0.0
-
parse_json
Parses a JSON string and constructs a Variant value.- Parameters:
json- a string column that contains JSON data. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a variant.
- Since:
- 4.0.0
-
to_variant_object
Converts a column containing nested inputs (array/map/struct) into a variants where maps and structs are converted to variant objects which are unordered unlike SQL structs. Input maps can only have string keys.- Parameters:
col- a column with a nested schema or column name. A column that evaluates to a struct, array, map, or variant.- Returns:
- Returns a column that evaluates to a variant.
- Since:
- 4.0.0
-
is_variant_null
Check if a variant value is a variant null. Returns true if and only if the input is a variant null and false otherwise (including in the case of SQL NULL).- Parameters:
v- a variant column. A column that evaluates to a variant.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 4.0.0
-
is_valid_variant
Check if a variant value is valid. Returns true if the variant is valid, false if it is malformed, and NULL if the input is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 4.2.0
-
variant_delete
public static Column variant_delete(Column v, Column path, scala.collection.immutable.Seq<Column> paths) Removes fields or array elements from a variant at the given JSONPath locations. Multiple paths are applied left to right. Returns NULL ifvis NULL; NULL paths are skipped.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the column containing the first JSONPath string. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A column that evaluates to a string.paths- additional JSONPath arguments, applied afterpathin order. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a variant.
- Since:
- 5.0.0
-
variant_delete
public static Column variant_delete(Column v, String path, scala.collection.immutable.Seq<String> paths) Removes fields or array elements from a variant at the given JSONPath locations. Multiple paths are applied left to right. Returns NULL ifvis NULL; NULL paths are skipped.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the first JSONPath identifying a deletion target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A string. Must be a constant.paths- additional JSONPath strings, applied afterpathin order. A string. Must be a constant.- Returns:
- Returns a column that evaluates to a variant.
- Since:
- 5.0.0
-
variant_insert
Inserts a value into a variant at the given JSONPath location. An object path adds a new field (error if it already exists); an array path inserts at the index, shifting later elements right. Missing intermediate keys are created. Throws an error if a path segment hits a value of an incompatible type. Returns NULL if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the column containing the JSONPath string identifying the insertion target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A column that evaluates to a string.value- the value to insert. Any expression castable to variant.- Returns:
- Returns a column that evaluates to a variant.
- Since:
- 4.3.0
-
variant_insert
Inserts a value into a variant at the given JSONPath location. An object path adds a new field (error if it already exists); an array path inserts at the index, shifting later elements right. Missing intermediate keys are created. Throws an error if a path segment hits a value of an incompatible type. Returns NULL if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the JSONPath identifying the insertion target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A string. Must be a constant.value- the value to insert. Any expression castable to variant.- Returns:
- Returns a column that evaluates to a variant.
- Since:
- 4.3.0
-
try_variant_insert
Inserts a value into a variant at the given JSONPath location. An object path adds a new field; an array path inserts at the index, shifting later elements right. Missing intermediate keys are created. Returns NULL if the field already exists or a path segment hits a value of an incompatible type, or if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the column containing the JSONPath string identifying the insertion target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A column that evaluates to a string.value- the value to insert. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
try_variant_insert
Inserts a value into a variant at the given JSONPath location. An object path adds a new field; an array path inserts at the index, shifting later elements right. Missing intermediate keys are created. Returns NULL if the field already exists or a path segment hits a value of an incompatible type, or if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the JSONPath identifying the insertion target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A string. Must be a constant.value- the value to insert. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
variant_set
Sets or upserts a value in a variant at the given JSONPath location. An existing object field or array element at the target is replaced. A missing field, array index, or intermediate path is created. Throws an error if a path segment hits a value of an incompatible type. Returns NULL if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the column containing the JSONPath string identifying the set target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A column that evaluates to a string.value- the value to set. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
variant_set
Sets or upserts a value in a variant at the given JSONPath location. An existing object field or array element at the target is replaced. A missing field, array index, or intermediate path is created. Throws an error if a path segment hits a value of an incompatible type. Returns NULL if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the JSONPath identifying the set target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A string. Must be a constant.value- the value to set. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
variant_set
Sets or upserts a value in a variant at the given JSONPath location. An existing object field or array element at the target is replaced. A missing field, array index, or intermediate path is created, unlesscreateIfMissingis false, in which case the variant is left unchanged. Throws an error if a path segment hits a value of an incompatible type. Returns NULL if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the column containing the JSONPath string identifying the set target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A column that evaluates to a string.value- the value to set. Any expression castable to variant.createIfMissing- whether to create missing keys or out-of-range array indices. A boolean. Must be a constant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
variant_set
Sets or upserts a value in a variant at the given JSONPath location. An existing object field or array element at the target is replaced. A missing field, array index, or intermediate path is created, unlesscreateIfMissingis false, in which case the variant is left unchanged. Throws an error if a path segment hits a value of an incompatible type. Returns NULL if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the JSONPath identifying the set target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed. A string. Must be a constant.value- the value to set. Any expression castable to variant.createIfMissing- whether to create missing keys or out-of-range array indices. A boolean. Must be a constant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
try_variant_set
Sets or upserts a value in a variant at the given JSONPath location. An existing object field or array element at the target is replaced. A missing field, array index, or intermediate path is created. Returns NULL if a path segment hits a value of an incompatible type, or if any argument is NULL.- Parameters:
v- a variant column.path- the column containing the JSONPath string identifying the set target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed.value- the value to set. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
try_variant_set
Sets or upserts a value in a variant at the given JSONPath location. An existing object field or array element at the target is replaced. A missing field, array index, or intermediate path is created. Returns NULL if a path segment hits a value of an incompatible type, or if any argument is NULL.- Parameters:
v- a variant column.path- the JSONPath identifying the set target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed.value- the value to set. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
try_variant_set
Sets or upserts a value in a variant at the given JSONPath location. An existing object field or array element at the target is replaced. A missing field, array index, or intermediate path is created, unlesscreateIfMissingis false, in which case the variant is left unchanged. Returns NULL if a path segment hits a value of an incompatible type, or if any argument is NULL.- Parameters:
v- a variant column.path- the column containing the JSONPath string identifying the set target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed.value- the value to set. Any expression castable to variant.createIfMissing- whether to create missing keys or out-of-range array indices.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
try_variant_set
Sets or upserts a value in a variant at the given JSONPath location. An existing object field or array element at the target is replaced. A missing field, array index, or intermediate path is created, unlesscreateIfMissingis false, in which case the variant is left unchanged. Returns NULL if a path segment hits a value of an incompatible type, or if any argument is NULL.- Parameters:
v- a variant column.path- the JSONPath identifying the set target. A valid path should start with$and is followed by one or more segments like[123],.name,['name'], or["name"]. The root path$is not allowed.value- the value to set. Any expression castable to variant.createIfMissing- whether to create missing keys or out-of-range array indices.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
variant_array_append
Appends a value to the array in a variant at the given JSONPath location. Returns the variant unchanged if a path key or index is absent. Throws an error if a path segment hits a value of an incompatible type or the target is not an array. Returns NULL if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the column containing the JSONPath string identifying the target array. A valid path should start with$and is followed by zero or more segments like[123],.name,['name'], or["name"]. A column that evaluates to a string.value- the value to append. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
variant_array_append
Appends a value to the array in a variant at the given JSONPath location. Returns the variant unchanged if a path key or index is absent. Throws an error if a path segment hits a value of an incompatible type or the target is not an array. Returns NULL if any argument is NULL.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the JSONPath identifying the target array. A valid path should start with$and is followed by zero or more segments like[123],.name,['name'], or["name"]. A string. Must be a constant.value- the value to append. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
try_variant_array_append
Appends a value to the array in a variant at the given JSONPath location. Returns the variant unchanged if a path key or index is absent. Returns NULL if a path segment hits a value of an incompatible type, the target is not an array, or if any argument is NULL.- Parameters:
v- a variant column.path- the column containing the JSONPath string identifying the target array. A valid path should start with$and is followed by zero or more segments like[123],.name,['name'], or["name"].value- the value to append. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
try_variant_array_append
Appends a value to the array in a variant at the given JSONPath location. Returns the variant unchanged if a path key or index is absent. Returns NULL if a path segment hits a value of an incompatible type, the target is not an array, or if any argument is NULL.- Parameters:
v- a variant column.path- the JSONPath identifying the target array. A valid path should start with$and is followed by zero or more segments like[123],.name,['name'], or["name"].value- the value to append. Any expression castable to variant.- Returns:
- (undocumented)
- Since:
- 4.3.0
-
variant_get
Extracts a sub-variant fromvaccording topathstring, and then cast the sub-variant totargetType. Returns null if the path does not exist. Throws an exception if the cast fails.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the extraction path. A valid path should start with$and is followed by zero or more segments like[123],.name,['name'], or["name"]. A string. Must be a constant.targetType- the target data type to cast into, in a DDL-formatted string. A string. Must be a constant.- Returns:
- Returns a column of the type specified by the
targetTypeargument. - Since:
- 4.0.0
-
variant_get
Extracts a sub-variant fromvaccording topathcolumn, and then cast the sub-variant totargetType. Returns null if the path does not exist. Throws an exception if the cast fails.- Parameters:
v- a variant column. A column that evaluates to a variant.path- the column containing the extraction path strings. A valid path string should start with$and is followed by zero or more segments like[123],.name,['name'], or["name"]. A column that evaluates to a string.targetType- the target data type to cast into, in a DDL-formatted string. A string. Must be a constant.- Returns:
- Returns a column of the type specified by the
targetTypeargument. - Since:
- 4.0.0
-
try_variant_get
Extracts a sub-variant fromvaccording topathstring, and then cast the sub-variant totargetType. Returns null if the path does not exist or the cast fails..- Parameters:
v- a variant column. A column that evaluates to a variant.path- the extraction path. A valid path should start with$and is followed by zero or more segments like[123],.name,['name'], or["name"]. A string. Must be a constant.targetType- the target data type to cast into, in a DDL-formatted string. A string. Must be a constant.- Returns:
- Returns a column of the type specified by the
targetTypeargument. - Since:
- 4.0.0
-
try_variant_get
Extracts a sub-variant fromvaccording topathcolumn, and then cast the sub-variant totargetType. Returns null if the path does not exist or the cast fails..- Parameters:
v- a variant column. A column that evaluates to a variant.path- the column containing the extraction path strings. A valid path string should start with$and is followed by zero or more segments like[123],.name,['name'], or["name"]. A column that evaluates to a string.targetType- the target data type to cast into, in a DDL-formatted string. A string. Must be a constant.- Returns:
- Returns a column of the type specified by the
targetTypeargument. - Since:
- 4.0.0
-
schema_of_variant
Returns schema in the SQL format of a variant.- Parameters:
v- a variant column. A column that evaluates to a variant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
schema_of_variant_agg
Returns the merged schema in the SQL format of a variant column.- Parameters:
v- a variant column. A column that evaluates to a variant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
schema_of_json
Parses a JSON string and infers its schema in DDL format.- Parameters:
json- a JSON string. A string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.4.0
-
schema_of_json
Parses a JSON string and infers its schema in DDL format.- Parameters:
json- a foldable string column containing a JSON string. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.4.0
-
schema_of_json
Parses a JSON string and infers its schema in DDL format using options.- Parameters:
json- a foldable string column containing JSON data. A column that evaluates to a string.options- options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- a column with string literal containing schema in DDL format. Returns a column that evaluates to a string.
- Since:
- 3.0.0
-
json_array_length
Returns the number of elements in the outermost JSON array.NULLis returned in case of any other valid JSON string,NULLor an invalid JSON.- Parameters:
e- the JSON array string column. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
json_object_keys
Returns all the keys of the outermost JSON object as an array. If a valid JSON object is given, all the keys of the outermost object will be returned as an array. If it is any other valid JSON string, an invalid JSON string or an empty string, the function returns null.- Parameters:
e- the JSON object string column. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.5.0
-
to_json
(Scala-specific) Converts a column containing aStructType,ArrayTypeor aMapTypeinto a JSON string with the specified schema. Throws an exception, in the case of an unsupported type.- Parameters:
e- a column containing a struct, an array, a map, or a variant. A column that evaluates to a struct, array, map, or variant.options- options to control how the struct column is converted into a json string. accepts the same options and the json data source. See Data Source Option in the version you use. Additionally the function supports theprettyoption which enables pretty JSON generation. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.1.0
-
to_json
(Java-specific) Converts a column containing aStructType,ArrayTypeor aMapTypeinto a JSON string with the specified schema. Throws an exception, in the case of an unsupported type.- Parameters:
e- a column containing a struct, an array, a map, or a variant. A column that evaluates to a struct, array, map, or variant.options- options to control how the struct column is converted into a json string. accepts the same options and the json data source. See Data Source Option in the version you use. Additionally the function supports theprettyoption which enables pretty JSON generation. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.1.0
-
to_json
Converts a column containing aStructType,ArrayTypeor aMapTypeinto a JSON string with the specified schema. Throws an exception, in the case of an unsupported type.- Parameters:
e- a column containing a struct, an array, a map, or a variant. A column that evaluates to a struct, array, map, or variant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 2.1.0
-
mask
Masks the given string value. The function replaces characters with 'X' or 'x', and numbers with 'n'. This can be useful for creating copies of tables with sensitive information removed.- Parameters:
input- string value to mask. Supported types: STRING, VARCHAR, CHAR. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
mask
Masks the given string value. The function replaces upper-case characters with specific character, lower-case characters with 'x', and numbers with 'n'. This can be useful for creating copies of tables with sensitive information removed.- Parameters:
input- string value to mask. Supported types: STRING, VARCHAR, CHAR. A column that evaluates to a string.upperChar- character to replace upper-case characters with. Specify NULL to retain original character. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
mask
Masks the given string value. The function replaces upper-case and lower-case characters with the characters specified respectively, and numbers with 'n'. This can be useful for creating copies of tables with sensitive information removed.- Parameters:
input- string value to mask. Supported types: STRING, VARCHAR, CHAR. A column that evaluates to a string.upperChar- character to replace upper-case characters with. Specify NULL to retain original character. A column that evaluates to a string.lowerChar- character to replace lower-case characters with. Specify NULL to retain original character. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
mask
Masks the given string value. The function replaces upper-case, lower-case characters and numbers with the characters specified respectively. This can be useful for creating copies of tables with sensitive information removed.- Parameters:
input- string value to mask. Supported types: STRING, VARCHAR, CHAR. A column that evaluates to a string.upperChar- character to replace upper-case characters with. Specify NULL to retain original character. A column that evaluates to a string.lowerChar- character to replace lower-case characters with. Specify NULL to retain original character. A column that evaluates to a string.digitChar- character to replace digit characters with. Specify NULL to retain original character. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
mask
public static Column mask(Column input, Column upperChar, Column lowerChar, Column digitChar, Column otherChar) Masks the given string value. This can be useful for creating copies of tables with sensitive information removed.- Parameters:
input- string value to mask. Supported types: STRING, VARCHAR, CHAR. A column that evaluates to a string.upperChar- character to replace upper-case characters with. Specify NULL to retain original character. A column that evaluates to a string.lowerChar- character to replace lower-case characters with. Specify NULL to retain original character. A column that evaluates to a string.digitChar- character to replace digit characters with. Specify NULL to retain original character. A column that evaluates to a string.otherChar- character to replace all other characters with. Specify NULL to retain original character. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
size
Returns length of array or map.This function returns -1 for null input only if spark.sql.ansi.enabled is false and spark.sql.legacy.sizeOfNull is true. Otherwise, it returns null for null input. With the default settings, the function returns null for null input.
- Parameters:
e- the target column. A column that evaluates to an array or a map.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 1.5.0
-
cardinality
Returns length of array or map. This is an alias ofsizefunction.This function returns -1 for null input only if spark.sql.ansi.enabled is false and spark.sql.legacy.sizeOfNull is true. Otherwise, it returns null for null input. With the default settings, the function returns null for null input.
- Parameters:
e- the target column. A column that evaluates to an array or a map.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
sort_array
Sorts the input array for the given column in ascending order, according to the natural ordering of the array elements. Null elements will be placed at the beginning of the returned array.- Parameters:
e- the array column to sort. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.5.0
-
sort_array
Sorts the input array for the given column in ascending or descending order, according to the natural ordering of the array elements. NaN is greater than any non-NaN elements for double/float type. Null elements will be placed at the beginning of the returned array in ascending order or at the end of the returned array in descending order.- Parameters:
e- the array column to sort. A column that evaluates to an array.asc- whether to sort in ascending order. A column that evaluates to a boolean. Must be a constant.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 1.5.0
-
array_min
Returns the minimum value in the array. NaN is greater than any non-NaN elements for double/float type. NULL elements are skipped.- Parameters:
e- the array column. A column that evaluates to an array.- Returns:
- Returns a column of the element type of the input array.
- Since:
- 2.4.0
-
array_max
Returns the maximum value in the array. NaN is greater than any non-NaN elements for double/float type. NULL elements are skipped.- Parameters:
e- the input column. A column that evaluates to an array.- Returns:
- Returns a column of the element type of the input array.
- Since:
- 2.4.0
-
array_size
Returns the total number of elements in the array. The function returns null for null input.- Parameters:
e- the input column. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
array_agg
Aggregate function: returns a list of objects with duplicates.- Parameters:
e- the input column. A column that evaluates to any type.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.5.0
- Note:
- The function is non-deterministic because the order of collected results depends on the order of the rows which may be non-deterministic after a shuffle.
-
shuffle
Returns a random permutation of the given array.- Parameters:
e- the input column. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
- Note:
- The function is non-deterministic.
-
shuffle
Returns a random permutation of the given array.- Parameters:
e- the input column. A column that evaluates to an array.seed- the seed for the random generator. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.0.0
- Note:
- The function is non-deterministic.
-
reverse
Returns a reversed string or an array with reverse order of elements.- Parameters:
e- the input column. A column that evaluates to a string, a binary, or an array.- Returns:
- Returns a column of the same type as the input.
- Since:
- 1.5.0
-
flatten
Creates a single array from an array of arrays. If a structure of nested arrays is deeper than two levels, only one level of nesting is removed.- Parameters:
e- the input column. A column that evaluates to an array of arrays.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
sequence
Generate a sequence of integers from start to stop, incrementing by step.- Parameters:
start- the starting value (inclusive) of the sequence. A column that evaluates to an integral, a date, or a timestamp.stop- the last value (inclusive) of the sequence. A column that evaluates to an integral, a date, or a timestamp.step- the value to add to the current element to get the next element. A column that evaluates to an integral or interval.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
sequence
Generate a sequence of integers from start to stop, incrementing by 1 if start is less than or equal to stop, otherwise -1.- Parameters:
start- the starting value (inclusive) of the sequence. A column that evaluates to an integral, a date, or a timestamp.stop- the last value (inclusive) of the sequence. A column that evaluates to an integral, a date, or a timestamp.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
array_repeat
Creates an array containing the left argument repeated the number of times given by the right argument.- Parameters:
left- the value to repeat. A column that evaluates to any type.right- the number of times to repeat the value. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
array_repeat
Creates an array containing the left argument repeated the number of times given by the right argument.- Parameters:
e- the value to repeat. A column that evaluates to any type.count- the number of times to repeat the value. A column that evaluates to an integral. Must be a constant.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
map_contains_key
Returns true if the map contains the key.- Parameters:
column- the input column. A column that evaluates to a map.key- the key to check for. A column that evaluates to the map's key type. Must be a constant.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.3.0
-
map_keys
Returns an unordered array containing the keys of the map.- Parameters:
e- the input column. A column that evaluates to a map.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.3.0
-
map_values
Returns an unordered array containing the values of the map.- Parameters:
e- the input column. A column that evaluates to a map.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.3.0
-
map_entries
Returns an unordered array of all entries in the given map.- Parameters:
e- the input column. A column that evaluates to a map.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.0.0
-
map_from_entries
Returns a map created from the given array of entries.- Parameters:
e- the array of entries to convert. A column that evaluates to an array of structs, each with a key and value field.- Returns:
- Returns a column that evaluates to a map.
- Since:
- 2.4.0
-
arrays_zip
Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.- Parameters:
e- the columns of arrays to be merged. Each is a column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 2.4.0
-
map_concat
Returns the union of all the given maps.- Parameters:
cols- the maps to merge. Each is a column that evaluates to a map.- Returns:
- Returns a column that evaluates to a map.
- Since:
- 2.4.0
-
from_csv
public static Column from_csv(Column e, StructType schema, scala.collection.immutable.Map<String, String> options) Parses a column containing a CSV string into aStructTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing CSV data. A column that evaluates to a string.schema- the schema to use when parsing the CSV string. A string, StructType or DataType. Must be a constant.options- options to control how the CSV is parsed. accepts the same options and the CSV data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 3.0.0
-
from_csv
(Java-specific) Parses a column containing a CSV string into aStructTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing CSV data. A column that evaluates to a string.schema- the schema to use when parsing the CSV string. A column that evaluates to a string.options- options to control how the CSV is parsed. accepts the same options and the CSV data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 3.0.0
-
schema_of_csv
Parses a CSV string and infers its schema in DDL format.- Parameters:
csv- a CSV string. A string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.0.0
-
schema_of_csv
Parses a CSV string and infers its schema in DDL format.- Parameters:
csv- a foldable string column containing a CSV string. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.0.0
-
schema_of_csv
Parses a CSV string and infers its schema in DDL format using options.- Parameters:
csv- a foldable string column containing a CSV string. A column that evaluates to a string.options- options to control how the CSV is parsed. accepts the same options and the CSV data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- a column with string literal containing schema in DDL format. Returns a column that evaluates to a string.
- Since:
- 3.0.0
-
to_csv
(Java-specific) Converts a column containing aStructTypeinto a CSV string with the specified schema. Throws an exception, in the case of an unsupported type.- Parameters:
e- a column containing a struct. A column that evaluates to a string.options- options to control how the struct column is converted into a CSV string. It accepts the same options and the CSV data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.0.0
-
to_csv
Converts a column containing aStructTypeinto a CSV string with the specified schema. Throws an exception, in the case of an unsupported type.- Parameters:
e- a column containing a struct. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.0.0
-
from_xml
Parses a column containing a XML string into the data type corresponding to the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing XML data. A column that evaluates to a string.schema- the schema to use when parsing the XML string. A string, StructType or DataType. Must be a constant.options- options to control how the XML is parsed. accepts the same options and the XML data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 4.0.0
-
from_xml
(Java-specific) Parses a column containing a XML string into aStructTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing XML data. A column that evaluates to a string.schema- the schema as a DDL-formatted string. A string, StructType or DataType. Must be a constant.options- options to control how the XML is parsed. accepts the same options and the xml data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 4.0.0
-
from_xml
(Java-specific) Parses a column containing a XML string into aStructTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing XML data. A column that evaluates to a string.schema- the schema to use when parsing the XML string. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 4.0.0
-
from_xml
(Java-specific) Parses a column containing a XML string into aStructTypewith the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing XML data. A column that evaluates to a string.schema- the schema to use when parsing the XML string. A column that evaluates to a string.options- options to control how the XML is parsed. accepts the same options and the XML data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 4.0.0
-
from_xml
Parses a column containing a XML string into the data type corresponding to the specified schema. Returnsnull, in the case of an unparseable string.- Parameters:
e- a string column containing XML data. A column that evaluates to a string.schema- the schema to use when parsing the XML string. A string, StructType or DataType. Must be a constant.- Returns:
- Returns a column that evaluates to a struct.
- Since:
- 4.0.0
-
schema_of_xml
Parses a XML string and infers its schema in DDL format.- Parameters:
xml- a XML string. A string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
schema_of_xml
Parses a XML string and infers its schema in DDL format.- Parameters:
xml- a foldable string column containing a XML string. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
schema_of_xml
Parses a XML string and infers its schema in DDL format using options.- Parameters:
xml- a foldable string column containing XML data. A column that evaluates to a string.options- options to control how the xml is parsed. accepts the same options and the XML data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- a column with string literal containing schema in DDL format. Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
to_xml
(Java-specific) Converts a column containing aStructTypeinto a XML string with the specified schema. Throws an exception, in the case of an unsupported type.- Parameters:
e- a column containing a struct. A column that evaluates to a string.options- options to control how the struct column is converted into a XML string. It accepts the same options as the XML data source. See Data Source Option in the version you use. A map of string options. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
to_xml
Converts a column containing aStructTypeinto a XML string with the specified schema. Throws an exception, in the case of an unsupported type.- Parameters:
e- a column containing a struct. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 4.0.0
-
years
(Java-specific) A transform for timestamps and dates to partition data into years.- Parameters:
e- the target column to transform. A column that evaluates to a date or a timestamp.- Returns:
- (undocumented)
- Since:
- 3.0.0
-
months
(Java-specific) A transform for timestamps and dates to partition data into months.- Parameters:
e- the target column to transform. A column that evaluates to a date or a timestamp.- Returns:
- (undocumented)
- Since:
- 3.0.0
-
days
(Java-specific) A transform for timestamps and dates to partition data into days.- Parameters:
e- the target column to transform. A column that evaluates to a date or a timestamp.- Returns:
- (undocumented)
- Since:
- 3.0.0
-
xpath
Returns a string array of values within the nodes of xml that match the XPath expression.- Parameters:
xml- the XML column to evaluate. A column that evaluates to a string.path- the XPath expression to match. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 3.5.0
-
xpath_boolean
Returns true if the XPath expression evaluates to true, or if a matching node is found.- Parameters:
xml- the XML column to evaluate. A column that evaluates to a string.path- the XPath expression to match. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
xpath_double
Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.- Parameters:
xml- the XML column to evaluate. A column that evaluates to a string.path- the XPath expression to match. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
xpath_number
Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.- Parameters:
xml- the XML column to evaluate. A column that evaluates to a string.path- the XPath expression to match. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a double.
- Since:
- 3.5.0
-
xpath_float
Returns a float value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.- Parameters:
xml- the XML column to evaluate. A column that evaluates to a string.path- the XPath expression to match. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a float.
- Since:
- 3.5.0
-
xpath_int
Returns an integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.- Parameters:
xml- the XML column to evaluate. A column that evaluates to a string.path- the XPath expression to match. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 3.5.0
-
xpath_long
Returns a long integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.- Parameters:
xml- the XML column to evaluate. A column that evaluates to a string.path- the XPath expression to match. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a long.
- Since:
- 3.5.0
-
xpath_short
Returns a short integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.- Parameters:
xml- the XML column to evaluate. A column that evaluates to a string.path- the XPath expression to match. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a short.
- Since:
- 3.5.0
-
xpath_string
Returns the text contents of the first xml node that matches the XPath expression.- Parameters:
xml- the XML column to evaluate. A column that evaluates to a string.path- the XPath expression to match. A column that evaluates to a string. Must be a constant.- Returns:
- Returns a column that evaluates to a string.
- Since:
- 3.5.0
-
hours
(Java-specific) A transform for timestamps to partition data into hours.- Parameters:
e- target date or timestamp column to work on. A column that evaluates to a date or timestamp.- Returns:
- (undocumented)
- Since:
- 3.0.0
-
convert_timezone
Converts the timestamp without time zonesourceTsfrom thesourceTztime zone totargetTz.- Parameters:
sourceTz- the time zone for the input timestamp. If it is missed, the current session time zone is used as the source time zone. A column that evaluates to a string.targetTz- the time zone to which the input timestamp should be converted. A column that evaluates to a string.sourceTs- a timestamp without time zone. A column that evaluates to a timestamp.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
convert_timezone
Converts the timestamp without time zonesourceTsfrom the current time zone totargetTz.- Parameters:
targetTz- the time zone to which the input timestamp should be converted. A column that evaluates to a string.sourceTs- a timestamp without time zone. A column that evaluates to a timestamp.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
make_dt_interval
Make DayTimeIntervalType duration from days, hours, mins and secs.- Parameters:
days- the number of days, positive or negative. A column that evaluates to an integral.hours- the number of hours, positive or negative. A column that evaluates to an integral.mins- the number of minutes, positive or negative. A column that evaluates to an integral.secs- the number of seconds with the fractional part in microsecond precision. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
make_dt_interval
Make DayTimeIntervalType duration from days, hours and mins.- Parameters:
days- the number of days, positive or negative. A column that evaluates to an integral.hours- the number of hours, positive or negative. A column that evaluates to an integral.mins- the number of minutes, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
make_dt_interval
Make DayTimeIntervalType duration from days and hours.- Parameters:
days- the number of days, positive or negative. A column that evaluates to an integral.hours- the number of hours, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
make_dt_interval
Make DayTimeIntervalType duration from days.- Parameters:
days- the number of days, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
make_dt_interval
Make DayTimeIntervalType duration.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
try_make_interval
public static Column try_make_interval(Column years, Column months, Column weeks, Column days, Column hours, Column mins, Column secs) This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.- Parameters:
years- the number of years, positive or negative. A column that evaluates to an integral.months- the number of months, positive or negative. A column that evaluates to an integral.weeks- the number of weeks, positive or negative. A column that evaluates to an integral.days- the number of days, positive or negative. A column that evaluates to an integral.hours- the number of hours, positive or negative. A column that evaluates to an integral.mins- the number of minutes, positive or negative. A column that evaluates to an integral.secs- the number of seconds with the fractional part in microsecond precision. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 4.0.0
-
make_interval
public static Column make_interval(Column years, Column months, Column weeks, Column days, Column hours, Column mins, Column secs) Make interval from years, months, weeks, days, hours, mins and secs.- Parameters:
years- the number of years, positive or negative. A column that evaluates to an integral.months- the number of months, positive or negative. A column that evaluates to an integral.weeks- the number of weeks, positive or negative. A column that evaluates to an integral.days- the number of days, positive or negative. A column that evaluates to an integral.hours- the number of hours, positive or negative. A column that evaluates to an integral.mins- the number of minutes, positive or negative. A column that evaluates to an integral.secs- the number of seconds with the fractional part in microsecond precision. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
try_make_interval
public static Column try_make_interval(Column years, Column months, Column weeks, Column days, Column hours, Column mins) This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.- Parameters:
years- the number of years, positive or negative. A column that evaluates to an integral.months- the number of months, positive or negative. A column that evaluates to an integral.weeks- the number of weeks, positive or negative. A column that evaluates to an integral.days- the number of days, positive or negative. A column that evaluates to an integral.hours- the number of hours, positive or negative. A column that evaluates to an integral.mins- the number of minutes, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 4.0.0
-
make_interval
public static Column make_interval(Column years, Column months, Column weeks, Column days, Column hours, Column mins) Make interval from years, months, weeks, days, hours and mins.- Parameters:
years- the number of years, positive or negative. A column that evaluates to an integral.months- the number of months, positive or negative. A column that evaluates to an integral.weeks- the number of weeks, positive or negative. A column that evaluates to an integral.days- the number of days, positive or negative. A column that evaluates to an integral.hours- the number of hours, positive or negative. A column that evaluates to an integral.mins- the number of minutes, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
try_make_interval
public static Column try_make_interval(Column years, Column months, Column weeks, Column days, Column hours) This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.- Parameters:
years- the number of years, positive or negative. A column that evaluates to an integral.months- the number of months, positive or negative. A column that evaluates to an integral.weeks- the number of weeks, positive or negative. A column that evaluates to an integral.days- the number of days, positive or negative. A column that evaluates to an integral.hours- the number of hours, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 4.0.0
-
make_interval
public static Column make_interval(Column years, Column months, Column weeks, Column days, Column hours) Make interval from years, months, weeks, days and hours.- Parameters:
years- the number of years, positive or negative. A column that evaluates to an integral.months- the number of months, positive or negative. A column that evaluates to an integral.weeks- the number of weeks, positive or negative. A column that evaluates to an integral.days- the number of days, positive or negative. A column that evaluates to an integral.hours- the number of hours, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
try_make_interval
This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.- Parameters:
years- the number of years, positive or negative. A column that evaluates to an integral.months- the number of months, positive or negative. A column that evaluates to an integral.weeks- the number of weeks, positive or negative. A column that evaluates to an integral.days- the number of days, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 4.0.0
-
make_interval
Make interval from years, months, weeks and days.- Parameters:
years- the number of years, positive or negative. A column that evaluates to an integral.months- the number of months, positive or negative. A column that evaluates to an integral.weeks- the number of weeks, positive or negative. A column that evaluates to an integral.days- the number of days, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
try_make_interval
This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.- Parameters:
years- the number of years, positive or negative. A column that evaluates to an integral.months- the number of months, positive or negative. A column that evaluates to an integral.weeks- the number of weeks, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 4.0.0
-
make_interval
Make interval from years, months and weeks.- Parameters:
years- The number of years, positive or negative. A column that evaluates to an integral.months- The number of months, positive or negative. A column that evaluates to an integral.weeks- The number of weeks, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
try_make_interval
This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.- Parameters:
years- The number of years, positive or negative. A column that evaluates to an integral.months- The number of months, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 4.0.0
-
make_interval
Make interval from years and months.- Parameters:
years- The number of years, positive or negative. A column that evaluates to an integral.months- The number of months, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
try_make_interval
This is a special version ofmake_intervalthat performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.- Parameters:
years- The number of years, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 4.0.0
-
make_interval
Make interval from years.- Parameters:
years- The number of years, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
make_interval
Make interval.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
make_timestamp
public static Column make_timestamp(Column years, Column months, Column days, Column hours, Column mins, Column secs, Column timezone) Create timestamp from years, months, days, hours, mins, secs and timezone fields. The result data type is consistent with the value of configurationspark.sql.timestampType. If the configurationspark.sql.ansi.enabledis false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 to 12. A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.timezone- The time zone identifier. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
make_timestamp
public static Column make_timestamp(Column years, Column months, Column days, Column hours, Column mins, Column secs) Create timestamp from years, months, days, hours, mins and secs fields. The result data type is consistent with the value of configurationspark.sql.timestampType. If the configurationspark.sql.ansi.enabledis false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 to 12. A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
make_timestamp
Create a local date-time from date, time, and timezone fields.- Parameters:
date- The date to represent, in valid DATE format. A column that evaluates to a date.time- The time to represent, in valid TIME format. A column that evaluates to a time.timezone- The time zone identifier. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.1.0
-
make_timestamp
Create a local date-time from date and time fields.- Parameters:
date- The date to represent, in valid DATE format. A column that evaluates to a date.time- The time to represent, in valid TIME format. A column that evaluates to a time.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.1.0
-
try_make_timestamp
public static Column try_make_timestamp(Column years, Column months, Column days, Column hours, Column mins, Column secs, Column timezone) Try to create a timestamp from years, months, days, hours, mins, secs and timezone fields. The result data type is consistent with the value of configurationspark.sql.timestampType. The function returns NULL on invalid inputs.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 to 12. A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.timezone- The time zone identifier. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.0.0
-
try_make_timestamp
public static Column try_make_timestamp(Column years, Column months, Column days, Column hours, Column mins, Column secs) Try to create a timestamp from years, months, days, hours, mins, and secs fields. The result data type is consistent with the value of configurationspark.sql.timestampType. The function returns NULL on invalid inputs.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 to 12. A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.0.0
-
try_make_timestamp
Try to create a local date-time from date, time, and timezone fields.- Parameters:
date- The date to represent, in valid DATE format. A column that evaluates to a date.time- The time to represent, in valid TIME format. A column that evaluates to a time.timezone- The time zone identifier. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.1.0
-
try_make_timestamp
Try to create a local date-time from date and time fields.- Parameters:
date- The date to represent, in valid DATE format. A column that evaluates to a date.time- The time to represent, in valid TIME format. A column that evaluates to a time.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.1.0
-
make_timestamp_ltz
public static Column make_timestamp_ltz(Column years, Column months, Column days, Column hours, Column mins, Column secs, Column timezone) Create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields. If the configurationspark.sql.ansi.enabledis false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 to 12. A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.timezone- The time zone identifier. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
make_timestamp_ltz
public static Column make_timestamp_ltz(Column years, Column months, Column days, Column hours, Column mins, Column secs) Create the current timestamp with local time zone from years, months, days, hours, mins and secs fields. If the configurationspark.sql.ansi.enabledis false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 (January) to 12 (December). A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
try_make_timestamp_ltz
public static Column try_make_timestamp_ltz(Column years, Column months, Column days, Column hours, Column mins, Column secs, Column timezone) Try to create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields. The function returns NULL on invalid inputs.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 (January) to 12 (December). A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.timezone- The time zone identifier. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.0.0
-
try_make_timestamp_ltz
public static Column try_make_timestamp_ltz(Column years, Column months, Column days, Column hours, Column mins, Column secs) Try to create the current timestamp with local time zone from years, months, days, hours, mins and secs fields. The function returns NULL on invalid inputs.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 (January) to 12 (December). A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.0.0
-
make_timestamp_ntz
public static Column make_timestamp_ntz(Column years, Column months, Column days, Column hours, Column mins, Column secs) Create local date-time from years, months, days, hours, mins, secs fields. If the configurationspark.sql.ansi.enabledis false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 (January) to 12 (December). A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 3.5.0
-
make_timestamp_ntz
Create a local date-time from date and time fields.- Parameters:
date- The date to represent, in valid DATE format. A column that evaluates to a date.time- The time to represent, in valid TIME format. A column that evaluates to a time.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.1.0
-
try_make_timestamp_ntz
public static Column try_make_timestamp_ntz(Column years, Column months, Column days, Column hours, Column mins, Column secs) Try to create a local date-time from years, months, days, hours, mins, secs fields. The function returns NULL on invalid inputs.- Parameters:
years- The year to represent, from 1 to 9999. A column that evaluates to an integral.months- The month-of-year to represent, from 1 (January) to 12 (December). A column that evaluates to an integral.days- The day-of-month to represent, from 1 to 31. A column that evaluates to an integral.hours- The hour-of-day to represent, from 0 to 23. A column that evaluates to an integral.mins- The minute-of-hour to represent, from 0 to 59. A column that evaluates to an integral.secs- The second-of-minute and its micro-fraction to represent, from 0 to 60. A column that evaluates to a numeric.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.0.0
-
try_make_timestamp_ntz
Try to create a local date-time from date and time fields.- Parameters:
date- The date to represent, in valid DATE format. A column that evaluates to a date.time- The time to represent, in valid TIME format. A column that evaluates to a time.- Returns:
- Returns a column that evaluates to a timestamp.
- Since:
- 4.1.0
-
make_ym_interval
Make year-month interval from years, months.- Parameters:
years- The number of years, positive or negative. A column that evaluates to an integral.months- The number of months, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
make_ym_interval
Make year-month interval from years.- Parameters:
years- The number of years, positive or negative. A column that evaluates to an integral.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
make_ym_interval
Make year-month interval.- Returns:
- Returns a column that evaluates to an interval.
- Since:
- 3.5.0
-
bucket
(Java-specific) A transform for any type that partitions by a hash of the input column.- Parameters:
numBuckets- The number of buckets. A column that evaluates to an integral. Must be a constant.e- The input column to partition. A column of any type.- Returns:
- (undocumented)
- Since:
- 3.0.0
-
bucket
(Java-specific) A transform for any type that partitions by a hash of the input column.- Parameters:
numBuckets- The number of buckets. Must be a constant.e- The input column to partition. A column of any type.- Returns:
- (undocumented)
- Since:
- 3.0.0
-
ifnull
Returnscol2ifcol1is null, orcol1otherwise.- Parameters:
col1- The column to test for null. A column of any type.col2- The column to return when col1 is null. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
isnotnull
Returns true ifcolis not null, or false otherwise.- Parameters:
col- The column to check. A column of any type.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
equal_null
Returns same result as the EQUAL(=) operator for non-null operands, but returns true if both are null, false if one of the them is null.- Parameters:
col1- The first column to compare. A column of any type.col2- The second column to compare. A column of any type.- Returns:
- Returns a column that evaluates to a boolean.
- Since:
- 3.5.0
-
nullif
Returns null ifcol1equals tocol2, orcol1otherwise.- Parameters:
col1- The value to return if it is not equal tocol2. A column of any type.col2- The value compared withcol1. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
nullifzero
Returns null ifcolis equal to zero, orcolotherwise.- Parameters:
col- The input value. A column that evaluates to a numeric.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
nvl
Returnscol2ifcol1is null, orcol1otherwise.- Parameters:
col1- The value to return if it is not null. A column of any type.col2- The value to return ifcol1is null. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
nvl2
Returnscol2ifcol1is not null, orcol3otherwise.- Parameters:
col1- The value that determines which branch to return. A column of any type.col2- The value to return ifcol1is not null. A column of any type.col3- The value to return ifcol1is null. A column of any type.- Returns:
- Returns a column of the same type as the input.
- Since:
- 3.5.0
-
zeroifnull
Returns zero ifcolis null, orcolotherwise.- Parameters:
col- The input value. A column that evaluates to a numeric.- Returns:
- Returns a column of the same type as the input.
- Since:
- 4.0.0
-
st_asbinary
Returns the input GEOGRAPHY or GEOMETRY value in WKB format.- Parameters:
geo- A geospatial value, either a GEOGRAPHY or a GEOMETRY. A column that evaluates to a geography or geometry.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.1.0
-
st_asbinary
Returns the input GEOGRAPHY or GEOMETRY value in WKB format using the specified endianness.- Parameters:
geo- A geospatial value, either a GEOGRAPHY or a GEOMETRY. A column that evaluates to a geography or geometry.endianness- The endianness of the output WKB, 'NDR' for little-endian or 'XDR' for big-endian. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
st_asbinary
Returns the input GEOGRAPHY or GEOMETRY value in WKB format using the specified endianness.- Parameters:
geo- A geospatial value, either a GEOGRAPHY or a GEOMETRY. A column that evaluates to a geography or geometry.endianness- The endianness of the output WKB, 'NDR' for little-endian or 'XDR' for big-endian. A column that evaluates to a string.- Returns:
- Returns a column that evaluates to a binary.
- Since:
- 4.2.0
-
st_geogfromwkb
Parses the WKB description of a geography and returns the corresponding GEOGRAPHY value.- Parameters:
wkb- A value in WKB format, representing a GEOGRAPHY value. A column that evaluates to a binary.- Returns:
- (undocumented)
- Since:
- 4.1.0
-
st_geomfromwkb
Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.- Parameters:
wkb- A value in WKB format, representing a GEOMETRY value. A column that evaluates to a binary.- Returns:
- (undocumented)
- Since:
- 4.1.0
-
st_geomfromwkb
Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.- Parameters:
wkb- A value in WKB format, representing a GEOMETRY value. A column that evaluates to a binary.srid- The SRID value of the geometry. A column that evaluates to an integer.- Returns:
- (undocumented)
- Since:
- 4.2.0
-
st_geomfromwkb
Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.- Parameters:
wkb- A value in WKB format, representing a GEOMETRY value. A column that evaluates to a binary.srid- The SRID value of the geometry. A column that evaluates to an integer.- Returns:
- (undocumented)
- Since:
- 4.2.0
-
st_setsrid
Returns a new GEOGRAPHY or GEOMETRY value whose SRID is the specified SRID value.- Parameters:
geo- A geospatial value, either a GEOGRAPHY or a GEOMETRY. A column that evaluates to a geography or geometry.srid- The new SRID of the geospatial value. A column that evaluates to an integer.- Returns:
- (undocumented)
- Since:
- 4.1.0
-
st_setsrid
Returns a new GEOGRAPHY or GEOMETRY value whose SRID is the specified SRID value.- Parameters:
geo- A geospatial value, either a GEOGRAPHY or a GEOMETRY. A column that evaluates to a geography or geometry.srid- The new SRID of the geospatial value. A column that evaluates to an integer.- Returns:
- (undocumented)
- Since:
- 4.1.0
-
st_srid
Returns the SRID of the input GEOGRAPHY or GEOMETRY value.- Parameters:
geo- A geospatial value, either a GEOGRAPHY or a GEOMETRY. A column that evaluates to a geography or geometry.- Returns:
- Returns a column that evaluates to an integer.
- Since:
- 4.1.0
-
udaf
public static <IN,BUF, UserDefinedFunction udafOUT> (Aggregator<IN, BUF, OUT> agg, scala.reflect.api.TypeTags.TypeTag<IN> evidence$3) Obtains aUserDefinedFunctionthat wraps the givenAggregatorso that it may be used with untyped Data Frames.val agg = // Aggregator[IN, BUF, OUT] // declare a UDF based on agg val aggUDF = udaf(agg) val aggData = df.agg(aggUDF($"colname")) // register agg as a named function spark.udf.register("myAggName", udaf(agg))- Parameters:
agg- the typed Aggregatorevidence$3- (undocumented)- Returns:
- a UserDefinedFunction that can be used as an aggregating expression.
- Since:
- 3.0.0
- Note:
- The input encoder is inferred from the input type IN.
-
udaf
public static <IN,BUF, UserDefinedFunction udafOUT> (Aggregator<IN, BUF, OUT> agg, Encoder<IN> inputEncoder) Obtains aUserDefinedFunctionthat wraps the givenAggregatorso that it may be used with untyped Data Frames.Aggregator<IN, BUF, OUT> agg = // custom Aggregator Encoder<IN> enc = // input encoder // declare a UDF based on agg UserDefinedFunction aggUDF = udaf(agg, enc) DataFrame aggData = df.agg(aggUDF($"colname")) // register agg as a named function spark.udf.register("myAggName", udaf(agg, enc))- Parameters:
agg- the typed AggregatorinputEncoder- a specific input encoder to use- Returns:
- a UserDefinedFunction that can be used as an aggregating expression
- Since:
- 3.0.0
- Note:
- This overloading takes an explicit input encoder, to support UDAF declarations in Java.
-
udf
public static <RT> UserDefinedFunction udf(scala.Function0<RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$4) Defines a Scala closure of 0 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$4- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1> UserDefinedFunction udf(scala.Function1<A1, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$5, scala.reflect.api.TypeTags.TypeTag<A1> evidence$6) Defines a Scala closure of 1 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$5- (undocumented)evidence$6- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1, UserDefinedFunction udfA2> (scala.Function2<A1, A2, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$7, scala.reflect.api.TypeTags.TypeTag<A1> evidence$8, scala.reflect.api.TypeTags.TypeTag<A2> evidence$9) Defines a Scala closure of 2 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$7- (undocumented)evidence$8- (undocumented)evidence$9- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1, UserDefinedFunction udfA2, A3> (scala.Function3<A1, A2, A3, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$10, scala.reflect.api.TypeTags.TypeTag<A1> evidence$11, scala.reflect.api.TypeTags.TypeTag<A2> evidence$12, scala.reflect.api.TypeTags.TypeTag<A3> evidence$13) Defines a Scala closure of 3 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$10- (undocumented)evidence$11- (undocumented)evidence$12- (undocumented)evidence$13- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1, UserDefinedFunction udfA2, A3, A4> (scala.Function4<A1, A2, A3, A4, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$14, scala.reflect.api.TypeTags.TypeTag<A1> evidence$15, scala.reflect.api.TypeTags.TypeTag<A2> evidence$16, scala.reflect.api.TypeTags.TypeTag<A3> evidence$17, scala.reflect.api.TypeTags.TypeTag<A4> evidence$18) Defines a Scala closure of 4 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$14- (undocumented)evidence$15- (undocumented)evidence$16- (undocumented)evidence$17- (undocumented)evidence$18- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1, UserDefinedFunction udfA2, A3, A4, A5> (scala.Function5<A1, A2, A3, A4, A5, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$19, scala.reflect.api.TypeTags.TypeTag<A1> evidence$20, scala.reflect.api.TypeTags.TypeTag<A2> evidence$21, scala.reflect.api.TypeTags.TypeTag<A3> evidence$22, scala.reflect.api.TypeTags.TypeTag<A4> evidence$23, scala.reflect.api.TypeTags.TypeTag<A5> evidence$24) Defines a Scala closure of 5 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$19- (undocumented)evidence$20- (undocumented)evidence$21- (undocumented)evidence$22- (undocumented)evidence$23- (undocumented)evidence$24- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1, UserDefinedFunction udfA2, A3, A4, A5, A6> (scala.Function6<A1, A2, A3, A4, A5, A6, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$25, scala.reflect.api.TypeTags.TypeTag<A1> evidence$26, scala.reflect.api.TypeTags.TypeTag<A2> evidence$27, scala.reflect.api.TypeTags.TypeTag<A3> evidence$28, scala.reflect.api.TypeTags.TypeTag<A4> evidence$29, scala.reflect.api.TypeTags.TypeTag<A5> evidence$30, scala.reflect.api.TypeTags.TypeTag<A6> evidence$31) Defines a Scala closure of 6 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$25- (undocumented)evidence$26- (undocumented)evidence$27- (undocumented)evidence$28- (undocumented)evidence$29- (undocumented)evidence$30- (undocumented)evidence$31- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1, UserDefinedFunction udfA2, A3, A4, A5, A6, A7> (scala.Function7<A1, A2, A3, A4, A5, A6, A7, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$32, scala.reflect.api.TypeTags.TypeTag<A1> evidence$33, scala.reflect.api.TypeTags.TypeTag<A2> evidence$34, scala.reflect.api.TypeTags.TypeTag<A3> evidence$35, scala.reflect.api.TypeTags.TypeTag<A4> evidence$36, scala.reflect.api.TypeTags.TypeTag<A5> evidence$37, scala.reflect.api.TypeTags.TypeTag<A6> evidence$38, scala.reflect.api.TypeTags.TypeTag<A7> evidence$39) Defines a Scala closure of 7 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$32- (undocumented)evidence$33- (undocumented)evidence$34- (undocumented)evidence$35- (undocumented)evidence$36- (undocumented)evidence$37- (undocumented)evidence$38- (undocumented)evidence$39- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1, UserDefinedFunction udfA2, A3, A4, A5, A6, A7, A8> (scala.Function8<A1, A2, A3, A4, A5, A6, A7, A8, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$40, scala.reflect.api.TypeTags.TypeTag<A1> evidence$41, scala.reflect.api.TypeTags.TypeTag<A2> evidence$42, scala.reflect.api.TypeTags.TypeTag<A3> evidence$43, scala.reflect.api.TypeTags.TypeTag<A4> evidence$44, scala.reflect.api.TypeTags.TypeTag<A5> evidence$45, scala.reflect.api.TypeTags.TypeTag<A6> evidence$46, scala.reflect.api.TypeTags.TypeTag<A7> evidence$47, scala.reflect.api.TypeTags.TypeTag<A8> evidence$48) Defines a Scala closure of 8 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$40- (undocumented)evidence$41- (undocumented)evidence$42- (undocumented)evidence$43- (undocumented)evidence$44- (undocumented)evidence$45- (undocumented)evidence$46- (undocumented)evidence$47- (undocumented)evidence$48- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1, UserDefinedFunction udfA2, A3, A4, A5, A6, A7, A8, A9> (scala.Function9<A1, A2, A3, A4, A5, A6, A7, A8, A9, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$49, scala.reflect.api.TypeTags.TypeTag<A1> evidence$50, scala.reflect.api.TypeTags.TypeTag<A2> evidence$51, scala.reflect.api.TypeTags.TypeTag<A3> evidence$52, scala.reflect.api.TypeTags.TypeTag<A4> evidence$53, scala.reflect.api.TypeTags.TypeTag<A5> evidence$54, scala.reflect.api.TypeTags.TypeTag<A6> evidence$55, scala.reflect.api.TypeTags.TypeTag<A7> evidence$56, scala.reflect.api.TypeTags.TypeTag<A8> evidence$57, scala.reflect.api.TypeTags.TypeTag<A9> evidence$58) Defines a Scala closure of 9 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$49- (undocumented)evidence$50- (undocumented)evidence$51- (undocumented)evidence$52- (undocumented)evidence$53- (undocumented)evidence$54- (undocumented)evidence$55- (undocumented)evidence$56- (undocumented)evidence$57- (undocumented)evidence$58- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
public static <RT,A1, UserDefinedFunction udfA2, A3, A4, A5, A6, A7, A8, A9, A10> (scala.Function10<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$59, scala.reflect.api.TypeTags.TypeTag<A1> evidence$60, scala.reflect.api.TypeTags.TypeTag<A2> evidence$61, scala.reflect.api.TypeTags.TypeTag<A3> evidence$62, scala.reflect.api.TypeTags.TypeTag<A4> evidence$63, scala.reflect.api.TypeTags.TypeTag<A5> evidence$64, scala.reflect.api.TypeTags.TypeTag<A6> evidence$65, scala.reflect.api.TypeTags.TypeTag<A7> evidence$66, scala.reflect.api.TypeTags.TypeTag<A8> evidence$67, scala.reflect.api.TypeTags.TypeTag<A9> evidence$68, scala.reflect.api.TypeTags.TypeTag<A10> evidence$69) Defines a Scala closure of 10 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)evidence$59- (undocumented)evidence$60- (undocumented)evidence$61- (undocumented)evidence$62- (undocumented)evidence$63- (undocumented)evidence$64- (undocumented)evidence$65- (undocumented)evidence$66- (undocumented)evidence$67- (undocumented)evidence$68- (undocumented)evidence$69- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.3.0
-
udf
Defines a Java UDF0 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF1 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF2 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF3 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF4 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF5 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF6 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF7 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF8 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF9 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Defines a Java UDF10 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().- Parameters:
f- (undocumented)returnType- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.3.0
-
udf
Deprecated.Scala `udf` method with return type parameter is deprecated. Please use Scala `udf` method without return type parameter. Since 3.0.0.Defines a deterministic user-defined function (UDF) using a Scala closure. For this variant, the caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the APIUserDefinedFunction.asNondeterministic().Note that, although the Scala closure can have primitive-type function argument, it doesn't work well with null values. Because the Scala closure is passed in as Any type, there is no type information for the function arguments. Without the type information, Spark may blindly pass null to the Scala closure with primitive-type argument, and the closure will see the default value of the Java type for the null argument, e.g.
udf((x: Int) => x, IntegerType), the result is 0 for null input.- Parameters:
f- A closure in ScaladataType- The output data type of the UDF- Returns:
- (undocumented)
- Since:
- 2.0.0
-
callUDF
Deprecated.Use call_udf.Call an user-defined function.- Parameters:
udfName- (undocumented)cols- (undocumented)- Returns:
- (undocumented)
- Since:
- 1.5.0
-
call_udf
Call an user-defined function. Example:import org.apache.spark.sql._ val df = Seq(("id1", 1), ("id2", 4), ("id3", 5)).toDF("id", "value") val spark = df.sparkSession spark.udf.register("simpleUDF", (v: Int) => v * v) df.select($"id", call_udf("simpleUDF", $"value"))- Parameters:
udfName- (undocumented)cols- (undocumented)- Returns:
- (undocumented)
- Since:
- 3.2.0
-
call_function
Call a SQL function.- Parameters:
funcName- function name that follows the SQL identifier syntax (can be quoted, can be qualified)cols- the expression parameters of function- Returns:
- (undocumented)
- Since:
- 3.5.0
-
unwrap_udt
Unwrap UDT data type column into its underlying type.- Parameters:
column- the UDT column to unwrap. A column that evaluates to a user-defined type.- Returns:
- (undocumented)
- Since:
- 3.4.0
-
vector_cosine_similarity
Returns the cosine similarity between two float vectors.- Parameters:
left- first vector column. A column that evaluates to an array.right- second vector column. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to a float.
- Since:
- 4.3.0
-
vector_inner_product
Returns the inner product (dot product) between two float vectors.- Parameters:
left- first vector column. A column that evaluates to an array.right- second vector column. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to a float.
- Since:
- 4.3.0
-
vector_l2_distance
Returns the Euclidean (L2) distance between two float vectors.- Parameters:
left- first vector column. A column that evaluates to an array.right- second vector column. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to a float.
- Since:
- 4.3.0
-
vector_norm
Returns the Lp norm of a float vector. Degree defaults to 2.0 if unspecified.- Parameters:
vector- input vector column. A column that evaluates to an array.degree- norm degree (1.0 for L1, 2.0 for L2, infinity norm). A column that evaluates to a float.- Returns:
- Returns a column that evaluates to a float.
- Since:
- 4.3.0
-
vector_norm
Returns the Lp norm of a float vector using degree 2.0 (Euclidean norm).- Parameters:
vector- input vector column. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to a float.
- Since:
- 4.3.0
-
vector_normalize
Normalizes a float vector to unit length. Degree defaults to 2.0 if unspecified.- Parameters:
vector- input vector column. A column that evaluates to an array.degree- norm degree (1.0 for L1, 2.0 for L2, infinity norm). A column that evaluates to a float.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.3.0
-
vector_normalize
Normalizes a float vector to unit length using degree 2.0 (Euclidean norm).- Parameters:
vector- input vector column. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.3.0
-
vector_avg
Aggregate function: returns the element-wise mean of float vectors in a group.- Parameters:
col- input vector column. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.3.0
-
vector_sum
Aggregate function: returns the element-wise sum of float vectors in a group.- Parameters:
col- input vector column. A column that evaluates to an array.- Returns:
- Returns a column that evaluates to an array.
- Since:
- 4.3.0
-