Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-10731 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

SSL certificates are typically short-lived, and while Kafka brokers support dynamic SSL certificate reloads via dynamic configuration (KIP-226), consumers and producers still require disruptive restarts to apply updated certificates. The ability to automatically hot-reload SSL certificate when they are updated eliminates such disruptions and allows for smooth application operations particularly in environments where certificates are rotated automatically by agents.

The goal is to improve system uptime and reduce operational overhead caused by restarts. 

The brokers would benefit from the proposed improvement by eliminating the need to run kafka-configs to dynamically update the listeners configuration for certificate rotation.

Public Interfaces

Add an configuration property that can be used by brokers, consumer, producers:

  • ssl.hot.reload="true|false"

Configuration example

The following block can be present in reconfig-server.properties, consumer.properties or producer.properties.

security.protocol=SSL
ssl.key.password=<password>
ssl.keystore.location=/path/to/keystore.p12
ssl.keystore.password=<password>
ssl.truststore.location=/path/to/ca.p12
ssl.truststore.password=<password>
# new option
ssl.hot.reload=true


Proposed Changes

The current SSL implementation already supports reconfiguration and reloading, which is used when a broker’s configuration is dynamically updated. However, the missing functionality for SSL hot reloading lies in the ability to automatically trigger a reconfiguration when changes occur in the keystore or truststore files. 

The changes will be limited to the SslFactory. Brokers, consumers, and producers, which rely on it, will benefit from the update.

Inspired by Spring Boot's SSL hot reload feature, the proposed changes would trigger an SslFactory reconfiguration whenever the configured keystore or truststore files are modified. Java's WatchService API can be used for file changes monitoring. Each SslFactory when configured with ssl.hot.reload=true will register its keystore and truststore files, along with its reconfiguration method, with the Watcher service. The service will then trigger reconfiguration whenever changes are detected in the registered files.

To prevent redundant reconfigurations, a quiet period is enforced across watched files, ensuring that related changes do not trigger unnecessary reconfigurations.


Compatibility, Deprecation, and Migration Plan

No impact on existing users.

Test Plan

The change can be tested by running a broker, consumer, and producer with the new configuration enabled. Then, update the keystore for each component and verify that the new certificates are seamlessly rotated without any issues.

Rejected Alternatives

Instead of using a shared Watcher service, each SslFactory could have its own, allowing for behavior to be customized on a per-factory basis, notably a configurable quiet period. However, this approach would result in higher resource consumption.

  • No labels