Status

Current state: Accepted

Discussion thread: Discussion, Voting

JIRA:  KAFKA-9612 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

We'd like to support more complex configurations for some dynamically configurable options. The best way to describe these configurations is with a structured value like JSON or a nested list. It's possible to specify these configurations in server.properties, and to configure them in code with an AdminClient, but not using the CLI tools.

kafka-configs.sh currently makes it easy to set simple dynamic configs, but doesn't make it possible to set structured values more complex than a list. The current supported input for the --add-config option is:

Key Value pairs of configs to add. Square brackets can be used to group values which contain commas: 'k1=v1,k2=[v1,v2,v2],k3=v3'.

The implementation makes it impossible to set a value that contains commas (,)  and square brackets ([]) together and rules out structured formats like JSON.

Proposed Changes

We will add an option to kafka-configs.sh (ConfigCommand.scala) to accept a properties file and add the properties from the file.

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-default \
  --alter --add-config-file new.properties

Public Interfaces

We will add the --add-config-file option to kafka-configs.sh. It will be mutually exclusive with --add-config.

Compatibility, Deprecation, and Migration Plan

Since we're adding a new option to the command, existing use cases will be unaffected.

Rejected Alternatives

Allow specification of complex configs using --add-config

It would be possible to make the existing --add-config option do more sophisticated parsing of its input, for example to do brace matching. That would allow us to support things like JSON and nested lists:

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-default \
  --alter --add-config 'k1={"key1":"val1","key2":"val2"},k2=[[1,2],[3,4]]'

However, there are several downsides to this. It quickly becomes unwieldy as the values being set become more complex. Shell escaping of various special characters becomes an issue. This also presupposes that the formats we want to support will have balanced braces.

Use multiple --add-config options, each with a single argument

We could remove the ambiguity of commas and square brackets by removing the current logic about splitting the configurations into key/value pairs. Setting multiple configs could be achieved by multiple calls to kafka-configs.sh or by allowing --add-config to be specified multiple times.

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-default \
  --alter \
  --add-config k1=v1 \
  --add-config k2=v1,v2,v3 \
  --add-config k3=v3 \
  --add-config 'k4={"key1": "val1", "key2": "val2"}' \
  --add-config 'k5=[[1,2],[3,4]]'

However this would break existing multi-key uses of kafka-configs.sh.

Allow configuration from STDIN as well as from a file

It might be nice to enable input from standard input, to enable pipe/grep workflows. However, it's not clear that such workflows would be common. A script that would use such a workflow can send the output of the pipe to a temporary file and load that instead. Since the rest of the commands in the Kafka CLI don't support specifying STDIN, we won't support that here either, for consistency. STDIN support across the entire toolset can be added as a separate KIP if the need arises.

grep 'complex.property' server.properties | bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-default \
  --alter --add-config-file -

Allow --add-config and --add-config-file at the same time

Instead of specifying that --add-config and --add-config-file are mutually exclusive, we could accept both. This creates ambiguity about what happens if a configuration is specified in both places. We could specify how this is resolved (eg. "values supplied in --add-config take precedence over values supplied in the file"), but this seems more complex than is warranted. Users can call kafka-configs.sh multiple times if they want to add configs from multiple sources.

There is no similar ambiguity with --delete-config and --delete-config-file, since there's no value to conflict.

Include a --delete-config-file option

Some systems like Kubernetes allow the cleanup of resources by deleting with the same config that was used to create the resources. This is not an exactly parallel situation, because deleting a configuration property doesn't necessarily put the system in the same state that it was in before the addition.

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-default \
  --alter --delete-config-file old.properties

grep 'complex.property' server.properties | bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-default \
  --alter --delete-config-file -

Use a format other than a properties file for the configs

Since dynamic configurations are stored in ZooKeeper in JSON format, it might make sense to accept a JSON file in that same format. This seems like it is tying the implementation details too closely to the interface. Users already use properties files to specify configurations. Introducing a different format would add complexity.