Current state: "Under Discussion"
Discussion thread: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Topic deletion in Kafka removes data and the operation is not reversible (i.e. there is no "undelete" operation). Hence, there is a valid use-case to keep the flag "delete.topic.enable" as "false" in
server.properties configs, so that someone doesn't accidentally delete a topic, which can then lead to outages.
But sometimes, there are legit client use-cases, where the users want to deprecate a topic and those topics need to be deleted from the cluster. Currently, the process to do this operation is very cumbersome (with "delete.topic.enable" flag set as "false"):
1. Change the `server.properties` config on each broker with: `delete.topic.enable=true` 2. Roll the cluster 3. Run Kafka Admin commands to delete the topics 4. Revert the `server.properties` config change for `delete.topic.enable` 5. Roll the cluster again
There are other hacky workarounds as well, like:
1. Set Topic retention of the topic-to-be-deleted to a small value to flush the data 2. Invoke `zkcli.sh rmr /brokers/topic/topic-to-be-deleted` 3. When Topic metadata is no longer available, rm the topic dirs on the brokers
The above process is pretty risky and can lead to unavailability of one or more topics if any mistake happens in executing the above commands.
KIP-226 added support for dynamic update of broker configuration. In this KIP, we propose to extend the support to dynamic update of "delete.topic.enable" config flag.
Use case: To allow deletion of topics from the cluster dynamically without restarting the brokers (and otherwise keeping the config "delete.topic.enable" as "false" in the static server.properties config).
Default for whole cluster (/configs/brokers/<default>)
Broker-level config (for testing) (/configs/brokers/<broker-id>)
Dynamic update changes:
TopicDeletionManager.isDeleteTopicEnabled will be updated on dynamic flag reconfiguration. This variable flag will be used at all places in lieu of
Sample Workflow to delete a topic after the proposed changes:
bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config delete.topic.enable=true
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic TopicToBeDeleted
bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --delete-config delete.topic.enable
Compatibility, Deprecation, and Migration Plan
- There won't be any impact on existing users.
- There won't be any change of current behavior.
- No migration tool required