Master KIP

KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum (Accepted)

Status

Current state: Approved.

Discussion thread: here

JIRA: KAFKA-9082

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

With work underway to migrate configuration metadata away from ZooKeeper (KIP-500), components and tools relying on direct ZK access must be updated to issue administrative calls through AK's broker via KafkaAdminClient. ConfigCommand, a tool used for describing and altering configs for various entities (topics, users, clients, brokers, and broker-loggers), is one such tool requiring update. Partial AK support has been implemented, however additional functionality is necessary to maintain ZK-parity. This KIP proposes to relax the --zookeeper restriction for all afflicted commands+flags.

In addition, the ConfigCommand tool would benefit from quality-of-life change to shorten entity specification on the command line.

Proposed Changes

There are 3 major functions of the config command:

1. Alter an entity's config by adding/updating/removing config entries. The given entity name may either be a specific entity (via --entity-name) or, if supported, the default config for all entities (via --entity-default).
2. Describing an entity's config. Like altering, the given entity name may either be a specific entity (via --entity-name) or, if supported, the default config for all entities (via --entity-default).
3. Describing all entity configs. This happens when no entity identifier is provided (i.e. --entity-name and --entity-default omitted).

Which functions are supported, and which are proposed by this KIP (green) is as follows:

*AK broker-side functionality requiring future KIP

For the quality-of-life change, proposed is the shortening of specifying entity type and name into a single flag:

• --entity-type <TYPES> --entity-name <NAME> can be shortened to --<TYPE> <NAME>
  • E.g. --entity-type topics --entity-name some-topic can be shortened to --topic some-topic
• --entity-type <TYPES> --entity-default can be shortened to --<TYPE>-defaults
  • E.g. --entity-type brokers --entity-default can be shortened to --broker-defaults

Note --entity-type, --entity-name, and --entity-default will NOT be deprecated/removed.

Public Interfaces

Summary of the public interfaces proposed. Note these are all additions, no existing commands/flags are removed or changed.

• Support all broker descriptions for AK:

  • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type brokers <note --entity-name omitted>

• Support all broker-logger descriptions for AK:

  • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type broker-loggers <note --entity-name omitted>

• Support topic config commands for AK:
  • Specific topic description:
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type topics --entity-name some-topic
  • All topic descriptions:
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type topics <note --entity-name omitted>
  • Altering topic config:
    • bin/kafka-configs.sh --bootstrap-server x --alter --entity-type topics --entity-name topic <--add-config/--delete-config ...>
• Support user/client config commands for AK:
  • Specific user/client descriptions:
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type users --entity-name some-user
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type users --entity-default
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type users --entity-name some-user --entity-type clients --entity-name some-client
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type users --entity-name some-user --entity-type clients --entity-default
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type users --entity-name default --entity-type clients --entity-name some-client
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type users --entity-name default --entity-type clients --entity-default
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type clients --entity-name some-client
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type clients --entity-default
  • All user/client descriptions:
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type users <note --entity-name omitted>
    • bin/kafka-configs.sh --bootstrap-server x --describe --entity-type clients <note --entity-name omitted>
  • Altering user/client config:
    • bin/kafka-configs.sh --bootstrap-server x --alter <same enumerations as user/client descriptions above>
• Add abbreviation helper flags:
  • --entity-type topics --entity-name some-topic => --topic some-topic
  • --entity-type users --entity-name some-user => --user some-user
  • --entity-type users --entity-default => --user-defaults
  • --entity-type clients --entity-name some-client => --client some-client
  • --entity-type clients --entity-default => --client-defaults
  • --entity-type brokers --entity-name 10 => --broker 10
  • --entity-type brokers --entity-default => --broker-defaults
  • --entity-type broker-loggers --entity-name 10 => --broker-logger 10

Compatibility, Deprecation, and Migration Plan

There is no impact to current usage, the semantics of all existing combinations of flags remain unchanged.

Rejected Alternatives

None.