Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-10731 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

SSL certificates are typically short-lived, and while Kafka brokers support dynamic SSL certificate reloads via dynamic configuration (KIP-226), consumers and producers still require disruptive restarts to apply updated certificates. The ability to automatically auto-reload SSL certificate when they are updated eliminates such disruptions and allows for smooth application operations particularly in environments where certificates are rotated automatically by agents.

The goal is to improve system uptime and reduce operational overhead caused by restarts. 

The brokers would benefit from the proposed improvement by eliminating the need to run kafka-configs to dynamically update the listeners configuration for certificate rotation.

Public Interfaces

Add an configuration property that can be used by brokers, consumer, producers:

ssl.auto.reload: When set to 'true', the keystore and truststore directories are monitored for changes and automatically reload the SSL Engine Factory with the updated keystore/truststore files. The files must keep their original names and paths. Defaults to 'false'.

Configuration example

The following block can be present in reconfig-server.properties, consumer.properties or producer.properties.

security.protocol=SSL
ssl.key.password=<password>
ssl.keystore.location=/path/to/keystore.p12
ssl.keystore.password=<password>
ssl.truststore.location=/path/to/ca.p12
ssl.truststore.password=<password>
# new option
ssl.auto.reload=true

Proposed Changes

The current SSL implementation already supports reconfiguration and reloading, which is used when a broker’s configuration is dynamically updated. However, the missing functionality for SSL auto-reload lies in the ability to automatically trigger a reconfiguration when changes occur in the keystore or truststore files. 

The changes will be limited to the SslFactory. Brokers, controllers, consumers, producers, and admin clients which rely on it, will benefit from the update.

Inspired by Spring Boot's SSL hot reload feature, the proposed changes would trigger an SslFactory reconfiguration whenever the configured keystore or truststore files are modified. Java's WatchService API can be used for file changes monitoring. Each SslFactory when configured with ssl.auto.reload=true will register its keystore and truststore files, along with its reconfiguration method, with the Watcher service. The service will then trigger reconfiguration whenever changes are detected in the registered files. A single java application can have multiple SslFactory instances but only one Watcher service thread.

Successive modifications to the keystore and truststore files (such as deletion and recreation) can lead to unnecessary consecutive SslFactory  reconfigurations. To prevent this, a 30-second quiet period is enforced for the monitored files. This ensures that no action is triggered unless at least 30 seconds have passed since the last change.

Compatibility, Deprecation, and Migration Plan

No impact on existing users.

Test Plan

The change can be tested by running a broker, consumer, and producer with the new configuration enabled. Then, update we the keystore file for each component and verify that the new files trigger a new SslFactory reconfiguration without any issues.

Rejected Alternatives

Instead of using a shared Watcher service, each SslFactory could have its own, allowing for behavior to be customized on a per-factory basis, notably a configurable quiet period. However, this approach would result in higher resource consumption.