Avro Functions¶

This page lists all avro functions available in Spark SQL.

---

from_avro¶

from_avro(child, jsonFormatSchema, options) - Converts a binary Avro value into a Catalyst value.

Examples:

> SELECT from_avro(s, '{"type": "record", "name": "struct", "fields": [{ "name": "u", "type": ["int","string"] }]}', map()) IS NULL AS result FROM (SELECT NAMED_STRUCT('u', NAMED_STRUCT('member0', member0, 'member1', member1)) AS s FROM VALUES (1, NULL), (NULL,  'a') tab(member0, member1));
 [false]

Note:

The specified schema must match actual schema of the read data, otherwise the behavior is undefined: it may fail or return arbitrary result. To deserialize the data with a compatible and evolved schema, the expected Avro schema can be set via the corresponding option.

Since: 4.0.0

schema_of_avro¶

schema_of_avro(jsonFormatSchema, options) - Returns schema in the DDL format of the avro schema in JSON string format.

Examples:

> SELECT schema_of_avro('{"type": "record", "name": "struct", "fields": [{"name": "u", "type": ["int", "string"]}]}', map());
 STRUCT<u: STRUCT<member0: INT, member1: STRING> NOT NULL>

Since: 4.0.0

to_avro¶

to_avro(child[, jsonFormatSchema]) - Converts a Catalyst binary input value into its corresponding Avro format result.

Examples:

> SELECT to_avro(s, '{"type": "record", "name": "struct", "fields": [{ "name": "u", "type": ["int","string"] }]}') IS NULL FROM (SELECT NULL AS s);
 [true]
> SELECT to_avro(s) IS NULL FROM (SELECT NULL AS s);
 [true]

Since: 4.0.0