Avro Functions

This page lists all avro functions available in Spark SQL.

from_avro

from_avro(child, jsonFormatSchema, options) - Converts a binary Avro value into a Catalyst value.

Arguments:

  • child - A binary value containing Avro-encoded data.
  • jsonFormatSchema - A constant string with the Avro schema in JSON format used to interpret the data.
  • options - A constant map of string keys and values holding conversion options.

Examples:

> SELECT from_avro(s, '{"type": "record", "name": "struct", "fields": [{ "name": "u", "type": ["int","string"] }]}', map()) IS NULL AS result FROM (SELECT NAMED_STRUCT('u', NAMED_STRUCT('member0', member0, 'member1', member1)) AS s FROM VALUES (1, NULL), (NULL,  'a') tab(member0, member1));
 [false]

Note:

The specified schema must match actual schema of the read data, otherwise the behavior is undefined: it may fail or return arbitrary result. To deserialize the data with a compatible and evolved schema, the expected Avro schema can be set via the corresponding option. CHAR/VARCHAR come from the spark.sql.catalyst.type property on Avro string fields, and for map keys from spark.sql.catalyst.mapKey.type on the Avro map schema (both written by Spark's Avro writer). Assignment is accepted when CHAR/VARCHAR are first-class (spark.sql.charVarchar.standardSemantics.enabled or spark.sql.preserveCharVarcharTypeInfo). Then STRING values are assigned (CHAR padded). Overflow is a malformed record: FAILFAST fails with EXCEED_LIMIT_LENGTH as the cause; PERMISSIVE returns a struct with every field NULL (NULL for a non-struct schema). When CHAR/VARCHAR are not first-class, CHAR/VARCHAR in the Avro schema are read as STRING without padding or length checks. Avro ENUM symbols are not assigned. CHAR/VARCHAR map-key length checks require spark.sql.charVarchar.standardSemantics.enabled and do not pad or trim.

Since: 4.0.0


schema_of_avro

schema_of_avro(jsonFormatSchema, options) - Returns schema in the DDL format of the avro schema in JSON string format.

Arguments:

  • jsonFormatSchema - A constant string with the Avro schema in JSON format.
  • options - A constant map of string keys and values holding conversion options.

Examples:

> SELECT schema_of_avro('{"type": "record", "name": "struct", "fields": [{"name": "u", "type": ["int", "string"]}]}', map());
 STRUCT<u: STRUCT<member0: INT, member1: STRING> NOT NULL>

Since: 4.0.0


to_avro

to_avro(child[, jsonFormatSchema]) - Converts a Catalyst binary input value into its corresponding Avro format result.

Arguments:

  • child - A value to encode into Avro binary format.
  • jsonFormatSchema - An optional constant string with the Avro schema in JSON format used to encode the value. If omitted, the schema is inferred from the input type.

Examples:

> SELECT to_avro(s, '{"type": "record", "name": "struct", "fields": [{ "name": "u", "type": ["int","string"] }]}') IS NULL FROM (SELECT NULL AS s);
 [true]
> SELECT to_avro(s) IS NULL FROM (SELECT NULL AS s);
 [true]

Note:

to_avro serializes the current string content. It does not pad CHAR values or trim VARCHAR values.

Since: 4.0.0