Current state: Accepted
Vote thread: here
Discussion thread: here
JIRA: KAFKA-6435 – Application Reset Tool might delete incorrect internal topics
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Users may want to specify which internal topics should be deleted. At present, the streams reset tool deletes all topics that start with "<application.id>-" and there are no options to control it.
The `--internal-topics` option is especially useful when there are prefix conflicts between applications, e.g. "app" and "app-v2". In this case, if we want to reset "app", the reset tool's default behaviour will delete both the internal topics of "app" and "app-v2" (since both are prefixed by "app-"). With the `--internal-topics` option, we can provide internal topic names for "app" and delete the internal topics for "app" without deleting the internal topics for "app-v2".
Add an option to the streams reset tool.
--internal-topics <String: list> Comma-separated list of internal topics
Add an internal-topics option to streams reset tool.
The `--internal-topics` option will take a comma-separated list of internal topics to be deleted. These topics must be a subset of those flagged as internal during a `--dry-run` without the `internal-topics` option. If omitted, the tool will default to the original behaviour, i.e. delete all topics that start with "<application.id>-".
Usage of the "internal-topic" options will involve:
- Run `kafka-streams-application-reset.sh` with `--dry-run` to check which internal topics will be deleted.
- If everything looks okay, run the script again without `--dry-run`. Otherwise, run it with `--internal-topics` to specify the internal topics to be deleted.
Compatibility, Deprecation, and Migration Plan
If the new option is not specified, the stream resetter's behaviour will remain unchanged.
Disallow "dash" in the application.id
This fixes the example given above in the motivation, but does not allow users to specify the internal topics. Also, it will raise a lot of backward compatibility issues.
Have a "negative prefix" option
This may be more convenient than having to list out relevant internal topics which may be tedious. However, the internal-topics option is a lot more explicit and it's usage will be more familiar because of it's similarity to other options (e.g. input-topics and intermediate-topics). Having displayed the internal-topics marked for deletion in the dry-run, it is more straightforward to simply specify the subset to be deleted instead of finding a prefix (or prefixes) to omit everything by.