This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: Rejected as already being implemented as part of KIP-853: KRaft Controller Membership Changes
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Kafka Admin clients which support KIP-919 are able to target KRaft controllers directly. This was introduced to allow performing administrative operations without involving brokers, such as DESCRIBE_QUORUM and INCREMENTAL_ALTER_CONFIGS (for log4j level changes). However, we don’t allow users to configure advertised.listeners for clients to connect to controllers, as it is a forbidden configuration for controllers. Without this configuration, the clients may not be able to connect to controllers if they are sitting behind the NAT network while the client is in the external network.
Currently Kafka controllers’ endpoint information are based on the listeners configurations. When bootstrap.controller is set, the AdminClient uses DescribeClusterRequest to obtain the cluster topology. The response for this request contains the endpoint information for the active controller and the subsequent requests from the client are sent to this endpoint. This means if the listeners configuration for controllers is set to localhost which is in most cases, the AdminClient would end up sending the subsequent requests to localhost.
The listeners configuration can be set with the actual hostname address of the controllers so that the clients on the external network can use it to talk to them. However this would mean that when a controller starts up, it would immediately try to bind to the configured address, which may not be possible due to various reasons. For example, when running on Kubernetes, the DNS may not be updated yet with the dynamic IP of the new pod, if the controller just had been restarted. Another example would be Kafka cluster running in a dedicated production environment and admin clients running in an operational network that is linked to the production network but it is not the same network. In this case, the user might use some bastion hosts to forward the connection through. The controllers talk with each using their IPs or internal hostnames but the admin clients might need to talk to them through some bastion DNS names. In both examples, the controllers may not be able to bind to the host address and result in crashing with a fatal error. Yet another example where the current implementation might not work is when the broker is running with multiple different networks attached. In that case, it needs to listen on all interfaces (i.e. on 0.0.0.0) rather than on a single hostname or IP address that will be later advertised.
Public Interfaces
advertised.listeners configuration
We will remove the current restriction on configuring advertised.listeners for controllers. If advertised.listeners is not defined, it will fallback to listeners configuration like brokers. Currently, the endpoint for a controller is created using the listeners defined in the controller.listener.names. This will be changed to use the advertised.listeners instead. So when advertised.listeners is not defined, the configuration will fall back to the current behavior of using the listeners specified in the controller.listeners.names.
Proposed Changes
When defining listeners for controllers, various validation checks take place, one of which is to restrict defining advertised.listeners. As already mentioned, this validation check will be removed but also a new validation will be put for controller’s advertised.listeners. The listeners defined in advertised.listeners must appear in the configured controller.listener.names. There are also validations for requiring configured values for listeners to appear in controller.listener.names and vice versa. These validations will stay the same.
Controllers will continue using the addresses specified in controller.quorum.voters to communicate with each other and brokers will continue using these addresses to communicate with the controllers. This allows you to configure the discovery mechanism for inter-controller and broker to controller communication separately from how admin clients connect to the controllers.
Currently, both brokers and combined nodes are restricted from including any controller listeners in their advertised.listeners configurations. This restriction will be lifted for combined nodes, allowing users to advertise both broker and controller listeners. For example, users may have a controller quorum that includes both dedicated controller nodes and combined nodes. They would initialise AdminClient with bootstrap.controller set to their endpoints that must be of the controller type. However, this restriction will remain for brokers because we do not want to advertise any controller listeners from the brokers. Requests intended for controllers that are sent to brokers are forwarded to the controllers, and the responses are handled by the brokers.
Compatibility, Deprecation, and Migration Plan
This change will not impact existing users in terms of controller configuration and operation. There is no old behavior to phase out; we are simply introducing a new configuration option for controllers. If this option is not set, it will fall back to the existing behavior, using the listeners for controller endpoints in the responses to AdminClient requests.
Test Plan
We will update the existing unit tests for Kafka configurations and add new test cases to the existing Admin integration tests to cover scenarios where a client communicates with controllers configured with advertised.listeners. Additionally, we will update the documentation and the controller.properties template file with a commented-out advertised.listeners field, along with a description.
Rejected Alternatives
We could potentially change the controller to build DescribeClusterResponse using the endpoints from controller.quorum.voters. However, this configuration is intended for internal communication within the quorum, particularly for quorum elections, and it doesn't seem appropriate for use by admin clients. Additionally, this approach would not allow users the flexibility to separately configure how clients communicate with controllers versus how controllers and brokers communicate with each other.