Table of Contents |
---|
Status
Current state: Under DiscussionAdopted
Discussion thread: here
JIRA:
(0.10.2.0) Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-4565
(0.11.0.0) Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-4636
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Code Block | ||
---|---|---|
| ||
{ "version": 4, "jmx_port": 9999, "timestamp": 2233345666, "host": "localhost", "port": 9092, "rack": "rack1", "listener._security._protocol._map": { "PLAINTEXT": "PLAINTEXT", "SSL": "SSL", "SASL_PLAINTEXT": "SASL_PLAINTEXT", "SASL_SSL": "SASL_SSL" }, "endpoints": [ "CLIENT://cluster1.foo.com:9092", "REPLICATION: //broker1.replication.local:9093", "INTERNAL_PLAINTEXT: //broker1.local:9094", "INTERNAL_SASL://broker1.local:9095" ] } |
Protocol
Version 2 3 of UpdateMetadataRequest
will be introduced and the elements of the end_points
array would also have a listener_name field.
Code Block |
---|
UpdateMetadata Request (Version: 23) => controller_id controller_epoch [partition_states] [live_brokers] controller_id => INT32 controller_epoch => INT32 partition_states => topic partition controller_epoch leader leader_epoch [isr] zk_version [replicas] topic => STRING partition => INT32 controller_epoch => INT32 leader => INT32 leader_epoch => INT32 isr => INT32 zk_version => INT32 replicas => INT32 live_brokers => id [end_points] id => INT32 end_points => port host listener_name (new) security_protocol_type port => INT32 host => STRING listener_name => String (new) security_protocol_type => INT16 |
...
The exception is ZooKeeper-based consumers. These consumers retrieve the broker registration information directly from ZooKeeper directly and would have to be updated to map from the listener name to the security protocoland will choose the first listener with PLAINTEXT as the security protocol (the only security protocol they support).
Compatibility, Deprecation, and Migration Plan
...
For users upgrading, they should only use listener names once all the brokers and ZooKeeper-based consumers have been upgraded to a version that supports listener names. ZooKeeper-based consumers will use the first listener with PLAINTEXT as the security protocol, so listener ordering is important in such cases.
Rejected Alternatives
- Instead of adding the
listener.security.protocol.map
config, we could extend the protocol part of the listener definition to include both the listener name and security protocol. For example,CLIENT+SASL_PLAINTEXT://192.1.1.8:9092.
This is appealing from a clarity perspective (the listeners are fully defined in a single config value), but it may lead to duplication betweenlisteners
andadvertised.listeners
. A way to avoid that issue (at the cost of loss of symmetry) would be foradvertised.listeners
to only include the listener name (we can infer the security protocol by looking at thelisteners
entry with the same name). - Assume that
listener.security.protocol.map
is the same in every broker. The slight benefit in terms of smaller broker registration JSON is not worth the additional operational complexity when it comes to changing the config values in a running cluster (two rolling upgrades would be needed in some simple cases). - Using hard-coded listener domains for internal and replication traffic. The config format is simpler and there's less scope for hard to understand configs. The main disadvantage is that it's a bit too specific and may need to be extended again as more sophisticated use cases appear. The current proposal is more general and it seems like a natural evolution of the existing system.
...