Apache Kudu Security

Kudu uses an internal PKI system to issue X.509 certificates to servers in the cluster. Connections between peers who have both obtained certificates will use TLS for authentication, which doesn’t require contacting the Kerberos KDC. These certificates are only used for internal communication among Kudu servers, and between Kudu clients and servers. The certificates are never presented in a public facing protocol.

By using internally-issued certificates, Kudu offers strong authentication which scales to huge clusters, and allows TLS encryption to be used without requiring you to manually deploy certificates on every node.

After authenticating to a secure cluster, the Kudu client will automatically request an authentication token from the Kudu master. An authentication token encapsulates the identity of the authenticated user and carries the master’s RSA signature so that its authenticity can be verified.

This token will be used to authenticate subsequent connections. By default, authentication tokens are only valid for seven days, so that even if a token were compromised, it could not be used indefinitely. For the most part, authentication tokens should be completely transparent to users. By using authentication tokens, Kudu takes advantage of strong authentication without paying the scalability cost of communicating with a central authority for every connection.

When used with distributed compute frameworks such as Spark, authentication tokens can simplify configuration and improve security. For example, the Kudu Spark connector will automatically retrieve an authentication token during the planning stage, and distribute the token to tasks. This allows Spark to work against a secured Kudu cluster where only the planner node has Kerberos credentials.

Users running client Kudu applications must first run the kinit command to obtain a Kerberos ticket-granting ticket. For example:

Once authenticated, you use the same client code to read from and write to Kudu servers with and without Kerberos configuration.

Kudu authentication is designed to scale to thousands of nodes, which requires avoiding unnecessary coordination with a central authentication authority (such as the Kerberos KDC). Instead, Kudu servers and clients will use Kerberos to establish initial trust with the Kudu master, and then use alternate credentials for subsequent connections. In particular, the master will issue internal X.509 certificates to servers, and temporary authentication tokens to clients.

Apache Ranger models tabular objects stored in a Kudu cluster in the following hierarchy:

In Ranger, privileges are also associated with specific actions. Access to Kudu tables may rely on privileges on the following actions:

There are two additional access types:

If a user has the ALL privilege on a resource, they implicitly have privileges to perform any action on that resource (except those that require users to be a delegated admin, see below). Also, if a user is granted any privilege, they are able to perform actions requiring METADATA (e.g. opening the table) without having to explicitly grant METADATA privilege to them.

Finally, Ranger supports a delegate admin flag which is independent of the action types (it’s not implied by ALL and doesn’t imply METADATA). This is similar to the GRANT OPTION part of ALL WITH GRANT OPTION in SQL as it is required to modify privileges in Ranger and change the owner of a Kudu table.

While the action types are hierarchical, in terms of privilege evaluation, Ranger doesn’t have the concept of hierarchy. For instance, if a user has SELECT privilege on a database, it does not imply that the user has SELECT privileges on every table belonging to that database. On the other hand, Ranger supports privilege wildcard matching. For example, db=a→table=* matches all the tables that belong to database a. Therefore, in Ranger users actually need the SELECT privilege granted on db=a→table=*→column=* to allow SELECT on every table and every column in database a.

Nevertheless, with Ranger integration, when a Kudu master receives a request, it consults Ranger to determine what privileges a user has. And the required policies documented in the policy section are enforced to determine whether the user is authorized to perform the requested action or not.

In addition to granting privileges to a user by username, privileges can also be granted to table owners using the special {OWNER} username. These policies are evaluated only when a user tries to perform an action on a table that they own. For example, a policy can be defined for the {OWNER} user and db=→table= resource, and it will automatically be applied when any table is accessed by its owner. This way administrators don’t need to choose between creating policies one by one for each table, and granting access to a wide range of users.

Rather than having every tablet server communicate directly with the underlying authorization service (Ranger), privileges are propagated and checked via authorization tokens. These tokens encapsulate what privileges a user has on a given table. Tokens are generated by the master and returned to Kudu clients upon opening a Kudu table. Kudu clients automatically attach authorization tokens when sending requests to tablet servers.

Similar to the validity interval for authentication tokens, to limit the window of potential unwanted access if a token becomes compromised, authorization tokens are valid for five minutes by default. The acquisition and renewal of a token is hidden from the user, as Kudu clients automatically retrieve new tokens when existing tokens expire.

When a tablet server that has been configured to enforce fine-grained access control receives a request, it checks the privileges in the attached token, rejecting it if the privileges are not sufficient to perform the requested operation, or if it is invalid (e.g. expired).

It may be desirable to allow certain users to view and modify any data stored in Kudu. Such users can be specified via the --trusted_user_acl master configuration. Trusted users can perform any operation that would otherwise require fine-grained privileges, without Kudu consulting the authorization service.

Additionally, some services that interact with Kudu may authorize requests on behalf of their end users. For example, Apache Impala authorizes queries on behalf of its users, and sends requests to Kudu as the Impala service user, commonly "impala". Since Impala authorizes requests on its own, to avoid extraneous communication between the authorization service and Kudu, the Impala service user should be listed as a trusted user.

On the other hand, privilege cache in Kudu master is disabled with Ranger integration, since Ranger provides client side cache the use privileges and can periodically poll the privilege store for any changes. When a change is detected, the cache will be automatically updated.

The following authorization policy is enforced by Kudu masters.

The following authorization policy is enforced by Kudu tablet servers.

Encryption in transit can be configured on Kudu servers using the --rpc_encryption flag, which can be set to required, optional, or disabled. By default, the flag is set to optional. When required, Kudu will reject unencrypted connections. When optional, Kudu will attempt to use encryption. Same as authentication, when disabled or encryption fails for optional, Kudu will only allow unencrypted connections from trusted subnets and reject any unencrypted connections from publicly routable IPs. To secure a cluster, use --rpc_encryption=required.

It’s also possible to encrypt data at rest. Kudu supports AES-128-CTR, AES-192-CTR, and AES-256-CTR ciphers to encrypt data. Each physical file is encrypted with a unique key (File Key), which in turn is encrypted with the server’s own key (Server Key), which is encrypted by the Cluster Key stored in a third-party Key Management Service (KMS). Kudu supports Apache Ranger KMS and Apache Hadoop KMS (they are API-compatible).

Encryption at rest can be enabled with the --encrypt_data_at_rest=true flag. As the default key provider is NOT secure (it stores the Server Keys in cleartext and a Cluster Key is not used), the key provider should be set to ranger-kms using the encryption_key_provider flag and its URL set with ranger_kms_url. Before starting the server, a key must exist in the key provider with the same name as passed to Kudu with the --encryption_cluster_key_name flag.

When data is encrypted, CLI tools accessing the file system directly need to be provided with the same flags and the instance file from a data, WAL, or metadata directory must also be set with the --instance_file flag, for example:

The Kudu web UI can be configured to use secure HTTPS encryption by providing each server with TLS certificates. See Configuring a Secure Kudu Cluster for more information on web UI HTTPS configuration.

To prevent sensitive data from being exposed in the web UI, all row data is redacted. Table metadata, such as table names, column names, and partitioning information is not redacted. The web UI can be completely disabled by setting the --webserver_enabled=false flag on Kudu servers.