Status
Current state: Under Discussion
Discussion thread: here
JIRA: KAFKA-4565
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
During the 0.9.0.0 release cycle, support for multiple listeners per broker was introduced. Each listener is associated with a security protocol, ip/host and port. When combined with the advertised listeners mechanism, there is a fair amount of flexibility with one limitation: at most one listener per security protocol in each of the two configs (listeners
and advertised.listeners
).
In some environments, one may want to differentiate between external clients, internal clients and replication traffic independently of the security protocol for cost, performance and security reasons. A couple of examples that illustrate this:
- Replication traffic is assigned to a separate network interface so that it does not interfere with client traffic.
- External traffic goes through a proxy/load-balancer (security, flexibility) while internal traffic hits the brokers directly (performance, cost).
As such, we propose that Kafka brokers should be able to define multiple listeners for the same security protocol for binding (i.e. listeners) and
sharing (i.e. advertised.listeners)
so that internal, external and replication traffic can be separated if required.
Public Interfaces
A new broker config listener.security.protocol.map
will be introduced so that we can map a protocol label to a security protocol. The config value should be in the CSV Map format that is currently used by max.connections.per.ip.overrides
. The config value should follow map semantics: each key should only appear once, but values may appear multiple times. For example, the config could be defined in the following way to match the existing behaviour:
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
To ensure compatibility with existing configs, we propose the above as the default value for the new config. It's worth mentioning that the config value should be the same in every broker in the Kafka cluster for it to work as expected. This is also the case for a number of existing Kafka broker configs and since Kafka doesn't support cluster configs at this point, it seems acceptable.
The next step is to change the validation of advertised.listeners
and listeners
so that the protocol label has to be one of the keys in listener.security.protocol.map
(only security protocols are allowed currently). For example, the following would configure a broker with two different host:port pairs mapped to the same security protocol in two cases:
listener.security.protocol.map=CLIENT:SASL_PLAINTEXT,REPLICATION:PLAINTEXT,INTERNAL_PLAINTEXT:PLAINTEXT,INTERNAL_SASL:SASL_PLAINTEXT advertised.listeners=CLIENT://cluster1.foo.com:9092,REPLICATION://broker1.replication.local:9093,INTERNAL_PLAINTEXT://broker1.local:9094,INTERNAL_SASL://broker1.local:9095 listeners=CLIENT://192.1.1.8:9092,REPLICATION://10.1.1.5:9093,INTERNAL_PLAINTEXT://10.1.1.5:9094,INTERNAL_SASL://10.1.1.5:9095
We then introduce a second broker config as an alternative to security.inter.broker.protocol:
inter.broker.protocol.label=REPLICATION
It is an error to set both security.inter.broker.protocol
and inter.broker.protocol.label
at the same time. inter.broker.protocol.label
will be null
by default, which means that PLAINTEXT
will be used by default (as is currently the case).
There are a couple more interfaces that need to be updated slightly to support protocol labels. The first is the broker registration data stored in ZooKeeper. Protocol labels would be used instead of security protocols in version 4 of the format:
{ "version":4, "jmx_port":9999, "timestamp":2233345666, "host":"localhost", “port”:9092, "rack":"rack1", "endpoints": [ "CLIENT://cluster1.foo.com:9092", “REPLICATION://broker1.replication.local:9093”, “INTERNAL_PLAINTEXT://broker1.local:9094”, "INTERNAL_SASL://broker1.local:9095" ] }
The second and final interface change is to the UpdateMetadataRequest
protocol type. Version 2 would have a protocol_label
field instead of security_protocol_type
:
UpdateMetadata Request (Version: 2) => controller_id controller_epoch [partition_states] [live_brokers] controller_id => INT32 controller_epoch => INT32 partition_states => topic partition controller_epoch leader leader_epoch [isr] zk_version [replicas] topic => STRING partition => INT32 controller_epoch => INT32 leader => INT32 leader_epoch => INT32 isr => INT32 zk_version => INT32 replicas => INT32 live_brokers => id [end_points] id => INT32 end_points => port host protocol_label (instead of security_protocol_type) port => INT32 host => STRING protocol_label => String (instead of security_protocol_type => INT16)
Note that protocol labels only exist in the brokers, clients never see them.
Proposed Changes
We would have to change a number of places in the code that currently use SecurityProtocol
as a key to use SecurityLabel
instead. A few examples:
- Acceptor thread
- Metadata request handler
- ReplicaManager
- Broker
The changes are mostly mechanical and don't affect public API.
As stated previously, clients never see protocol labels and will make metadata requests exactly as before. The difference is that the list of endpoints they get back is restricted to the protocol label of the endpoint where they made the request. In the example above, if a client makes a metadata request to cluster1.foo.com:9092
, it will be mapped to the 192.1.1.8:9092
interface. The broker will return SASL_PLAINTEXT://cluster1.foo.com:9092
in the metadata response. It returned the entry in advertised.listeners
with the same protocol label, but it translated the protocol label to the associated security protocol before returning it to the client.
Compatibility, Deprecation, and Migration Plan
As mentioned previously, the default value of listener.security.protocol.map
maps the existing security protocols to a label with the same name:
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
As such, the change is backwards compatible.
For users upgrading, they should only use protocol labels once all the brokers have been upgraded.
Future work
- Different security protocol settings per listener. For example, one may want to configure SSL differently for internal versus external traffic.
Rejected Alternatives
- Using hard-coded listener domains for internal and replication traffic. The config format is simpler and there's less scope for hard to understand configs. The main disadvantage is that it's a bit too specific and may need to be extended again as more sophisticated use cases appear. The current proposal is more general and it seems like a natural evolution of the existing system.