Protobuf Functions

This page lists all protobuf functions available in Spark SQL.

from_protobuf

from_protobuf(data, messageName, descFilePath, options) - Converts a binary Protobuf value into a Catalyst value.

Arguments:

  • data - The binary Protobuf value to convert.
  • messageName - A constant string naming the Protobuf message to look for in the descriptor file.
  • descFilePath - Optional. A constant string or binary value with the Protobuf descriptor file, created by protoc with --descriptor_set_out and --include_imports. If omitted, the message must be resolvable otherwise.
  • options - Optional. A constant map of string key-value pairs controlling the conversion. By default no options are set.

Examples:

> SELECT from_protobuf(s, 'Person', '/path/to/descriptor.desc', map()) IS NULL AS result FROM (SELECT NAMED_STRUCT('name', name, 'id', id) AS s FROM VALUES ('John Doe', 1), (NULL,  2) tab(name, id));
 [false]

Note:

The specified Protobuf schema must match actual schema of the read data, otherwise the behavior is undefined: it may fail or return arbitrary result. To deserialize the data with a compatible and evolved schema, the expected Protobuf schema can be set via the corresponding option.

Since: 4.0.0


to_protobuf

to_protobuf(child, messageName, descFilePath, options) - Converts a Catalyst binary input value into its corresponding Protobuf format result.

Arguments:

  • child - The Catalyst input value to convert to Protobuf binary.
  • messageName - A constant string naming the Protobuf message to serialize to.
  • descFilePath - Optional. A constant string or binary value with the Protobuf descriptor file, created by protoc with --descriptor_set_out and --include_imports. If omitted, the message must be resolvable otherwise.
  • options - Optional. A constant map of string key-value pairs controlling the conversion. By default no options are set.

Examples:

> SELECT to_protobuf(s, 'Person', '/path/to/descriptor.desc', map('emitDefaultValues', 'true')) IS NULL FROM (SELECT NULL AS s);
 [true]

Since: 4.0.0