Status

Current state: "Under Discussion"

Discussion thread: https://lists.apache.org/thread/x1qn1k5qvcpv3k2brk209zp4w6ndbgjt

JIRA: KAFKA-20144 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-1142 introduced the LIST_CONFIG_RESOURCES RPC as a way for clients to discover which Kafka resources have non-default configuration properties. It was built on top of KIP-1000, which originally added support for client-metrics resources.

The current implementation requires DESCRIBE_CONFIGS on the CLUSTER resource for all resource types (GROUP, TOPIC, BROKER, BROKER_LOGGER, CLIENT_METRICS). This creates an unintentional permission anomaly with other, well-established RPCs:

• LIST_GROUPS: a user needs only DESCRIBE on CLUSTER to discover which consumer groups exist.

• LIST_CONFIG_RESOURCES for groups: the same user needs the elevated DESCRIBE_CONFIGS on CLUSTER to discover which of those groups have custom configurations — a strictly weaker piece of information.

Similarly for topics:

• METADATA: a user needs only DESCRIBE on a topic (or DESCRIBE on CLUSTER) to learn that a topic exists.

• LIST_CONFIG_RESOURCES for topics: the same user needs DESCRIBE_CONFIGS on CLUSTER.

This anomaly has a direct user-visible impact: kafka-configs.sh --describe --entity-type groups silently returns incomplete results for users who hold DESCRIBE on CLUSTER but not DESCRIBE_CONFIGS on CLUSTER. Specifically, groups that were configured via kafka-configs.sh --alter but have no active consumers are completely invisible to such users, because LIST_CONFIG_RESOURCES is the only RPC that can discover them.

This KIP introduces a new API version that aligns the authorization model with the actual semantics of the API and with existing peer APIs, eliminating the anomaly described above.

Public Interfaces

Changed authorization semantics for LIST_CONFIG_RESOURCES

A new API version (version 2) is introduced for LIST_CONFIG_RESOURCES (API key 74). The wire format is unchanged. The only change is the required ACL operation.

Version ≤ 1 (unchanged):

Version 2+:

• CLIENT_METRICS, BROKER, and BROKER_LOGGER are cluster-scoped resources and require cluster-level Describe. They are silently omitted if the caller lacks it.
• In version 2, the response is always NONE (success). Resources the caller is not authorized to see are silently omitted rather than returning CLUSTER_AUTHORIZATION_FAILED.

API version bump

ListConfigResourcesRequest.json valid versions updated from 0-1 to 0-2. The version comment is updated accordingly. No new fields are added.

Proposed Changes

KafkaApis.handleListConfigResources

• Version ≤ 1: no change to existing behavior.

• Version 2+:

  1. Check DESCRIBE on CLUSTER.

  2. If the check passes, return all requested resource types as today.

  3. If the check fails:

     • For GROUP and TOPIC: perform per-resource DESCRIBE checks. Return only the resources the caller is authorized to see.

     • For CLIENT_METRICS, BROKER, and BROKER_LOGGER: silently omit these resource types from the response.

  4. Always return NONE as the top-level error code (no CLUSTER_AUTHORIZATION_FAILED).

  5. If the result set is empty and the caller holds DescribeConfigs on CLUSTER but not Describe on CLUSTER, emit a DEBUG-level log message explaining that DescribeConfigs on CLUSTER is not sufficient for version 2+ and that the caller must be granted Describe on CLUSTER (or per-resource Describe on GROUP/TOPIC) to receive results.

This mirrors the behavior of LIST_GROUPS (API key 16), which returns only the groups a caller is authorized to see rather than failing the entire request.

Diagnostics / logging

The broker and the kafka-configs.sh tool each emit a diagnostic message, but under different conditions because the tool operates as a plain client and cannot inspect which ACLs the principal holds.

Broker (KafkaApis) — DEBUG level

Emitted when the result set is empty and the broker has determined that the principal holds DescribeConfigs on CLUSTER but not Describe on CLUSTER. No message is emitted when the caller holds neither permission, because an empty result is the expected and correct outcome in that situation.

LIST_CONFIG_RESOURCES v2 returned an empty result. Grant Describe on CLUSTER (or per-resource Describe on GROUP/TOPIC) to receive results.

kafka-configs.sh (ConfigCommand) — WARN level

The tool receives only the response from the broker; it cannot determine why the result is empty. It therefore emits a warning unconditionally whenever the returned list is empty.

LIST_CONFIG_RESOURCES returned an empty result. Grant Describe on CLUSTER (or per-resource Describe on GROUP/TOPIC) to receive results.

Documentation

docs/security/authorization-and-acls.md

Replace the single-row entry for LIST_CONFIG_RESOURCES:

LIST_CONFIG_RESOURCES (74) | DescribeConfigs | Cluster

with:

LIST_CONFIG_RESOURCES (74) | Describe        | Cluster      
LIST_CONFIG_RESOURCES (74) | Describe        | Group
LIST_CONFIG_RESOURCES (74) | Describe        | Topic

docs/getting-started/upgrade.md

add under Notable changes in 4.4.0

LIST_CONFIG_RESOURCES (API key 74) version 2 changes the required ACL from DESCRIBE_CONFIGS on Cluster to Describe on Cluster. For GROUP and TOPIC resource types, per-resource Describe is also accepted and returns a filtered result set. Users who hold only DESCRIBE_CONFIGS on Cluster must be granted Describe to use the new API version. Clients using API version ≤ 1 are unaffected. For further details, please refer to KIP-1298.

Compatibility, Deprecation, and Migration Plan

Wire format: unchanged. No new fields are added in version 2.

Behavioral change: version 2 requires Describe instead of DescribeConfigs. Clients that negotiate version 2 but whose principal holds only DESCRIBE_CONFIGS (and not DESCRIBE) will receive an empty success response rather than an authorization error.

Migration path:

1. Upgrade brokers to the release that ships version 2 support.
2. Grant Describe on CLUSTER to principals that currently hold only DESCRIBE_CONFIGS on CLUSTER and need to use LIST_CONFIG_RESOURCES.
3. Migrate clients to version 2.

Clients that remain on version ≤ 1 are completely unaffected; the old behavior is preserved indefinitely for those versions.

Test Plan

Unit tests:

• v2, cluster DESCRIBE held: full result set returned for all resource types.

• v2, only DESCRIBE_CONFIGS held: empty success response (no CLUSTER_AUTHORIZATION_FAILED).

• v2, per-resource fallback: caller holds Describe on specific groups/topics but not on CLUSTER — only those resources are returned.

• v2, cluster-scoped resources without cluster DESCRIBE: CLIENT_METRICS, BROKER, BROKER_LOGGER silently omitted.

• v ≤ 1, DESCRIBE_CONFIGS held: full result set returned (existing behavior preserved).

• v ≤ 1, DESCRIBE_CONFIGS denied: CLUSTER_AUTHORIZATION_FAILED returned (existing behavior preserved).

Integration tests:

• End-to-end: principal with only DESCRIBE on CLUSTER uses kafka-configs.sh --list and receives the full resource list.
• End-to-end: principal with only per-resource DESCRIBE on specific groups/topics receives a filtered list.

Rejected Alternatives

Make DESCRIBE imply DESCRIBE_CONFIGS at the ACL engine level

One might consider solving this by making DESCRIBE imply DESCRIBE_CONFIGS in the ACL implication chain (analogous to how ALTER already implies DESCRIBE). This would automatically allow any holder of DESCRIBE to pass any DESCRIBE_CONFIGS check. However, this change would affect every single RPC and resource type that uses DESCRIBE_CONFIGS across the entire broker — DESCRIBE_CONFIGS (API 32), ALTER_CONFIGS (API 33), INCREMENTAL_ALTER_CONFIGS (API 44), and more — potentially granting unintended read access to configurations on topics, groups, and brokers cluster-wide. The blast radius is far too large for what is a narrow, targeted fix. This KIP instead changes only the authorization policy of LIST_CONFIG_RESOURCES itself.

Accept both DESCRIBE and DESCRIBE_CONFIGS without a version bump

An earlier proposal kept the same wire format (no new API version) and changed only the server-side authorization policy to accept either Describe or DescribeConfigs:

This was rejected for several reasons:

1. Inconsistency with other resource-listing APIs - LIST_GROUPS (API 16) and METADATA (API 3) check only Describe — they never accept DescribeConfigs as an alternative. Accepting both operations in LIST_CONFIG_RESOURCES would make it with a dual-AclOperation pattern, which is confusing and hard to reason about when setting up ACLs.
2. Increased complexity - Handler code would need to evaluate two alternative AclOperations per resource type, which is not done anywhere else in the codebase. A clean version bump with a single required operation is simpler to implement, test, and document.