This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Current state: Under Discussion [One of "Under Discussion", "Accepted", "Rejected"]
Discussion thread: here
Planned release: 2.0
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Describe the problems you are trying to solve.
- KIP-4 defines the high level motivation for using the admin client and KAFKA-3268 with its subtasks acts as the umbrella JIRA for this KIP.
- The current implementation of the
kafka.admin.ConfigCommand) which is used by
kafka-configs.shconnects to Zookeeper directly. This prevents the tool from being used in deployments where only the brokers are exposed to clients (i.e. where the zookeeper servers are intentionally not exposed).
- There is a general push to refactor/rewrite/replace tools which need Zookeeper access with equivalents which use the
AdminClientAPI. Thus it is necessary to change the
ConfigCommandso that it no longer talks to Zookeeper directly, but via an intermediating broker.
- Makes the ConfigCommand transparent to the authorization mechanism therefore enables higher level of security. The broker will be able to authorize config admin requests.
AdminClientAPI currently lacks any functionality for changing broker, user and client configurations which is possible with the current Zookeeper based
- Changing the ConfigCommand will increase the user experience for KIP-226 for the above mentioned reasons.
Command Line Tools And Arguments
The options accepted by
kafka-configs.sh command will change:
--zookeeperwill be removed, it's functionality won't be available. The design decision behind is that the ConfigCommand tool will be rewritten in the tools module which doesn't depend on the core module. This makes it hard to provide backward compatibility with the current ConfigCommand.
--bootstrap-serverwas added in - KAFKA-6494Getting issue details... STATUS and will be used here.
--adminclient.configoption will be added and should be used similarly to other tools, such as --producer.config in the console-producer. This parses a config file and initializes the admin client used internally.
--adminclient-propertyoption will be added. It is a key=value list that will be parsed by the command. It initializes the internal admin client.
A new tool, called scram-credentials.sh will be added. The need for this broker is when people use zookeeper as a credentials store for SCRAM (and currently users have no other option), then direct interaction with zookeeper is required to set up the initial credentials with inter-broker communication. The functionality of the tool will cover the following:
- Add, remove and list SCRAM credentials directly with zookeeper
- Will continue to use the
--zookeeperoption to specify the zookeeper host
- Works similarly to the old config command. Examples:
- Add new credentials:
bin/scram-credentials.sh --zookeeper localhost:2181 --add 'SCRAM-SHA-256=[iterations=8192,password=alice-secret],SCRAM-SHA-512=[password=alice-secret]' --username alice
- Describe credentials:
bin/scram-credentials.sh --zookeeper localhost:2181 --describe --username alice
- Remove credentials:
bin/scram-credentials.sh --zookeeper localhost:2181 --delete 'SCRAM-SHA-512' --username alice
- Add new credentials:
Wire Format Types
5: User (new)
6: Client (new)
A new type needs to be added to transfer quota values. Since the protocol classes in Kafka already uses ByteBuffers it is logical to use their functionality for serializing doubles. The serialization is basically a representation of the specified floating-point value according to the IEEE 754 floating-point "double format" bit layout. The ByteBuffer serializer writes eight bytes containing the given double value, in Big Endian byte order, into this buffer at the current position, and then increments the position by eight.
The implementation will be defined in
org.apache.kafka.common.protocol.types with the other protocol types and it will have no default value much like the other types available in the protocol.
The justification for a new protocol is that a quota is quite different from a broker or topic config because a quota can sometimes be identified a simple user, client or even a (user,client) tuple while a topic or a broker config can be identified only by the topic's name or the broker's ID. Moreover quotas have their own well defined types.
- Can be sent to any broker
- If the
nameis empty it means that listing the default quota is asked. Responses will be returned the same way for defaults.
- If the
quota_typearray is empty, all quotas are returned. Otherwise, quotas with the provided types are returned.
- Authorization: "DescribeQuotas" can only be interpreted on the "Cluster" resource and represented by the DescribeConfigs ACL due to the similarity in use cases. Unauthorized requests will receive an appropriate AuthorizationFailed error code.
- Can be sent to any broker
nameis empty it means that altering a default quota is asked.
- Authorization: "AlterQuotas" can only be interpreted on the "Cluster" resource and represented by the AlterConfigs ACL due to the similarity in use cases. Unauthorized requests will receive an appropriate AuthorizationFailed error code.
- For tools that allow users to alter quota configs, a validation/dry-run mode where validation errors are reported but no creation is attempted is available via the
- The AlterQuotas protocol has an incremental semantics. By this we mean that the request will update only those quotas which are sent in the request.
- Removing quotas will be done by sending a NaN as the value.
This request needs some change as currently the --add-config operation of ConfigCommand would do incremental operations in Zookeeper but the AlterConfigs protocol sets the whole properties object. The purpose of this change to add an boolean parameter to the request so that it can specify the behavior (incremental or set) which needs to be executed.
- The default value of
incremental_updateis false. That means that the request will wipe the node's data and sets what is sent in the request.
- Setting the
incremental_updateflag to true makes sure that existing configs are not deleted.
- Deleting a config in incremental mode is done by sending an empty string as value.
- Other existing semantics aren't changed.
New Command Line Interface
kafka-config.sh command line interface will change a little bit in terms of help message and response format as we will use argparse4j for parsing arguments.
As seen above, the describe format becomes more organized and it will also return default properties (as the protocol currently supports that). In case of alter we will also do an extra describe after executing the alter and print the most fresh state.
Compatibility, Deprecation, And Migration Plan
Firstly, the --zookeeper option will be removed from kafka-configs.sh and the backing code will be replaced. Therefore every user will need to change
--adminclient.config. SCRAM update will be done by the newly introduced
scram-credentials.sh tool. Other existing behavior will be kept.
Secondly, users as of this KIP would be able to describe all topics or brokers in one step but can't do it for clients and users. For those who have this use case would still need to use the old command for a while (see below). The reason for this change is currently MetadataRequest provides enough information about topics and brokers so it's possible to describe all of them in one step but there's no such information about clients and users.
Finally, backward compatibilty (for instance a 2.0 client wants to admin a 1.0 server) will be impacted as some of the protocols are newly created and doesn't exist in old servers. In these cases users should continue using the scala version of the ConfigCommand by putting the core jar on their classpath.
The old command could be launched through kafka-run-class.sh like this:
Communicating through the broker instead of Zookeeper allows us to give more protection to Zookeeper as it could be hidden behind a firewall and you can only allow the broker through it. Also this would allow much finer grain authorization and audit of admin operations in the brokers.
From the CLI point of view the impact should be minimal as only the
--zookeeper option will change but we can assume the Zookeeper is a more protected resource than the
CLIENT ports of the brokers, therefore we can assume that they have knowledge about it and change with minimal effort.
From the compatibility point of view there might be a bigger impact as mentioned above. Since the command now uses the wire protocols (including some newly introduced ones) the backward compatibility will be impacted. That means that a user can't use a 2.0 client to administrate a 1.0 broker as in the older broker some of the wire protocols don't exist. This again should be acceptable most of the users as most of the admin commands require the core jar on the classpath which means that most of the time the commands are executed from an environment with the same version of the brokers. Therefore the old tool should still be available.
kafka.admin.ConfigCommand will print a warning message saying it is deprecated and will be removed in a future version.
Special Migration Tools
There are no tools required.
Removal Of The Existing Behavior
--zookeeper option will be removed with this change as it has minimal impact on the current users.
Listing multiple users' and clients' quotas at once won't be possible. If this is required, users would need to use the old tool.
Most of the functionality can be covered with end-to-end tests. There will be unit tests also to verify the protocol and the broker side logic.
At this moment this ConfigCommand can describe all the topics and all the brokers with one command but can't describe all clients or users. The reason for this is that we can gain the required information for topics and brokers by a MetadataRequest but we have no such protocols for getting a list of users or clients.