Current state: Approved
Discussion thread: here
Pull Request: https://github.com/apache/kafka/pull/4429
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Kafka Connect has a REST interface for managing and monitoring connectors. Currently this REST interface is using only plain HTTP without any encryption and authentication. This is not ideal because:
- The interface might be used to transfer sensitive information (e.g. passwords in Kafka Connect connector configurations)
- Anyone with access to the interface can add connectors (which can forward sensitive messages from Kafka brokers somewhere else)
Extending the REST interface with support for SSL / TLS encryption and SSL / TLS client authentication could address these issues.
This KIP proposes enabling SSL support in the Jetty HTTP server. Jetty already supports SSL / TLS. So the main work in this KIP will be around enabling and configuring SSL / TLS.
Users will be able to configure REST listeners using a new configuration option
listeners. It will allow to configure the protocol (which will be either HTTP or HTTPS), host and port. Users will be allowed to configure either HTTP listener or HTTPS listener or both. For example:
listeners parameter is configured, the existing options
rest.port will be ignored. The fields
rest.port will be marked as deprecated.
The HTTPS listener (when configured in
listeners) will by default use the SSL configuration from the
ssl.* options. In case the user needs to use different SSL configuration for connecting to Kafka brokers and for the REST interface, the default settings can be overridden by using the prefix
listeners.https. - for example:
rest.advertised.port options will continue to be used as today to specify the connection address which should be used by other workers. In addition a new option
rest.advertised.listener will define whether other workers should connect using HTTP or HTTPS protocols. In case HTTPS is selected, the connecting worker will use the SSL configuration from the existing
ssl.* options. Even in case when
rest.advertised.port options are not specified this field will be used to define which protocol should be advertised to other workers in combination with the appropriate hostname and port from the listener field.
This proposal doesn't include any authorization / ACL features. Only encryption and authentication. Authorization / ACLs should be subject of separate KIP in order to keep the scope of this KIP under control. It also doesn't add any other authentication options than SSL/TLS client authentication.
Configuration of SSL / TLS for the Kafka Connect REST interface tries to follow the configuration for other SSL / TLS enabled server interfaces. It will be done through the properties configuration file for the distributed Kafka Connect workers.
Following new options will be added:
|null||List of REST listeners in the format |
Configures the listener used for communication between workers. Valid values are either
When the listeners configuration is not defined or when it contains
|none||Valid values are |
This is the only authentication option suggested as part of this KIP.
Following existing options will be affected by this KIP:
rest.port will be marked as deprecated. The
listeners field would be the one preferred for the long term future.
Following existing options will be reused by this KIP without any changes:
|null||This field will be reused without any changes.|
Migration Plan and Compatibility
This KIP is a new implementation and doesn't have any backwards compatibility issues or special requirements on migration from older versions. Existing Kafka Connect installation would work in the same way as before this change. Without the SSL configuration, the REST interface will continue to be configured as today - i.e. without SSL / TLS.
- The first version of this KIP suggested only single REST listener - either HTTP or HTTPS. This was changed based on the feedback from the discussion