Current state: Under Discussion
JIRA: Will create a JIRA after discussion
Released: <Kafka Version>
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
At the moment Kafka supports 3 main types of security protocol: Plain, SSL, SASL and its variant SASL_SSL and SASL_PLAINTEXT. Behind-the-scenes Kerberos is the only supported authentication mechanism and SSL is the only supported transport protocol. Anyone who is willing to enable security in Kafka they need to properly config Kerberos and SSL. This is sometimes not desirable for users who already have their own security mechanism installed e.g. token based authentication or other SASL mechs. Due to current design of Kafka users with their own choice of authentication mechanism have no way to reuse what they have but have to employ Kerberos. It will be good for Kafka to provide a plug-able way for users to implement their own security mechanism.
A new security protocol is introduced in org.apache.kafka.common.protocol.SecurityProtocol
A new config is added to client properties named customized.network.channelbuilder which can be set to an implementation of org.apache.kafka.common.network.ChannelBuilder. This config takes effect only when security.protocol is set to CUSTOMIZED. Once CUSTOMIZED is used on client side a new listener needs to be added in broker: CUSTOMIZED://host:port and security.inter.broker.protocol is set to CUSTOMIZED if CUSTOMIZED security protocol is the only supported protocol.
With the new property customized.network.channelbuilder introduced users are able to provide their own implementation of ChannelBuilder which contains their own logic of authentication.
Since org.apache.kafka.common.network.ChannelBuilder is exposed to public constructor is not a good way to pass parameters. Method configure is a better place to pass in necessary parameters
When SecurityProtocol is set to CUSTOMIZED ChannelBuilders will read the class name from customized.network.channelbuilder and then instantiate it.
With above changes users who want to include their own authentication logic can set security.protocol to CUSTOMIZED, customized.network.channelbuilder to XYZChannelBuilder where the authentication and secure transport logic resides and add a new listener CUSTOMIZED://host:port in broker.
Compatibility, Deprecation, and Migration Plan
It will not impact any of existing clients. When clients upgrade to new version, they can set customized.network.channelbuilder
New SASL mechanism
As discussed in KIP-43 an alternative way is to provide a plug-able SASL. But this implementation is bound to SASL implementation only. Authentication users supply must comply with SASL. This KIP provides higher level of abstraction as it is security protocol agnostic and up to users how they want to implement it. This KIP can certainly coexist with KIP-43 which is more focus on SASL mechanism extension.