Current state: Accepted (vote thread)
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
kafka-topics.sh uses only direct Zookeeper connections which is not really desired compared to the AdminClient. This change would aim to add capability to the TopicCommand to be able to connect to a broker using the AdminClient.
This is part of KIP-4 which outlines the importance of exposing admin operations via the Kafka protocol:
- Allows clients in any language to administrate Kafka (Wire protocol is supported by any language)
- Provides public client for performing admin operations
- Ensures integration test code in other projects and clients maintains compatibility
- Prevents users from needing to use the Command classes and work around standard output and system exits
- Removing the need for admin scripts (
kafka-topics.sh, etc) to talk directly to Zookeeper.
- Allows ZNodes to be completely locked down via ACLs
- Further hides the Zookeeper details of Kafka
A few extra options will be added to kafka-configs.sh:
--bootstrap-server option will be added to accept config changes. This will would accept a list of brokers that the internal AdminClient would use.
--command-config option will be also added. This would accept a file argument that points to the AdminClient connection properties file (such as SSL, buffers, etc.).
The change proposed in this KIP is to add an extra option as stated above and to migrate create, delete, list and describe operations to use a broker connection. This would be a backward compatible change, leaving the zookeeper option available and fully working until a further point in time but would deprecate it as part of this KIP.
Specific behavior changes
--zookeepertogether would result in an exception as they should be mutually exclusive.
- Deleting an internal topic is allowed by the protocol and thus by this command as well. If this feature is not desirable, then ALCs must be set accordingly on the server side or the
delete.topic.enable=falsetopic level config must be set.
Compatibility, Deprecation, And Migration Plan
This KIP won't implement topic config alternation as that is deprecated in the TopicCommand and should be done by
kafka-configs.sh. The only alternation we allow is changing the partition number for topics.
No other existing behavior would be removed and the implementation would be done in a backward compatible way.
Also retrieving the list of topics that are marked for deletion won't be implemented now as currently it's not possible to retrieve via any protocols. This conversation is part of KIP-142. The implementation will add a note regarding this in its output.
As part of the
--zookeeper option deprecation we will mark it as deprecated in the command help but also print out a warning message about using this deprecated option.
The existing tests will be run with the
--bootstrap-server mode too. Additionally we can refactor some of the
kafka-topics.sh usages in the smokes to use the AdminClient mode.
Topics Marked For Deletion
KafkaAdminClient.listTopics() uses the Metadata protocol to acquire topic information. The returned response however won't contain the topics that are under deletion but couldn't complete yet (for instance because of some replicas offline), therefore it is not possible to implement the current command's "marked for deletion" feature. To get around this there were several alternatives that can be seen below but during the discussion we decided to keep this work in KIP-142. The idea in this KIP was that we could introduce some changes in the Metadata protocol, such as:
- Cache topics that are under deletion but some of their replicas are offline.
- Create a new error, called
- Bump the Metadata request version. The format of the protocol won't change, only the fact that there is a new Error type that we're introducing, but that requires bumping the protocol as old clients won't be able to handle it and most probably end up in an
- Smarten up the
KafkaApis.handleTopicMetadataRequestto also return the list of topics under deletion with the above error
At an early stage of the KIP discussion it occured that there is a need for a protocol that would handle topic partition changes, such as increasing the partition number. It got rejected as a similar api, called CreatePartitions already exists and we don't need a new protocol.
For archiving purposes here is the protocol:
- From the authorization perspective using this protocol would require an ALTER operation on a Topic resource. Both currently available and therefore can be used.