...
Binary log format
No changes expected
The network protocol and api behavior
No changes expected
Any class in the public packages under clientsConfiguration, especially client configuration
org/apache/kafka/common/serialization
No changes expected
org/apache/kafka/common
No changes expected
org/apache/kafka/common/errors
InvalidNamespaceException will be added.
org/apache/kafka/clients/producer
No changes expected
org/apache/kafka/clients/consumer (eventually, once stable)
No changes expected
Monitoring
Command line tools and arguments
Add create, list and delete for namespaces to kafka-topics and AdminUtils.
An optional "namespace" argument will be added to kafka-topics, and kafka-configs. If namespace argument is not provided, a default namespace of "" will be used. This will help in keeping the current behavior of Kafka and cli tools intact.
- Anything else that will likely break existing users in some way when they upgrade
- To avoid changing request/ response formats and public APIs, this KIP proposes to append namespaces to topic names. However, some separator is required to be able to parse namespace and topic name from a topic name. We need an out of band character (or string) to make sure that we are not breaking any existing topic. Kafka does not disallow any characters as of now for topic names. That makes it hard to come up with some character (or string) that is guaranteed to be not present in any of existing topics. We suggest to make this demarcation string configurable at cluster level, with a default of "::". However, if the users end up setting this to a string that appears in some existing topics, then this will lead to unexpected behavior.None
Proposed Changes
After considering a few approaches, listed in Rejected Alternatives section, below is what we think is the least obtrusive approach to support namespaces in Kafka. We suggest to represent namespaces at storage layer, i.e., storage layout of Zookeeper entities and logs on disk. Internal and public APIs can pass around namespaces, as part of, prepended to, topic names. However, we need someway to separate namespace and topic. This can be done by using a delimiter character that is not allowed in Kafka topics. Kafka allows a topic name to contain characters only from [a-zA-Z0-9\\._\\-]. We suggest to have ":" as the delimiting char, but can be any of the following.
...
- <namespace>:<topic>
- <namespace>%<topic>
- <namespace>@<topic>
- <namespace>|<topic>
- <namespace>#<topic>
- <namespace>*<topic>
- <namespace>~<topic>
- <namespace>$<topic>
- <namespace>^<topic>
- <namespace>&<topic>
- <namespace>><topic>
Namespace Name Restrictions
...
Multi-tiered Namespaces
Namespace can have any char in [a-zA-Z0-9\\.]. The "." in namespaces will be used to separate tiers. For instance, a namespace "org.apache.kafka" will translate to "org/apache/kafka".
Create
List
Delete
Changes in Storage Layouts
Following are the changes in layout structures on ZK and disks.
...