Status

Current state: Under Discussion

Discussion thread: here

JIRA:  KAFKA-10731 - Add Support for SSL hot reload

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka clients authenticate with brokers using TLS. The SSL credentials (keystore and truststore) are loaded once at startup and never refreshed. When certificates expire or are rotated, the only way to apply new credentials today is to restart every client. In environments that use short-lived certificates this forces frequent restarts, adds operational complexity, and increases the risk of downtime.

This KIP adds opt-in SSL hot reload to Kafka clients. When enabled, the client monitors its keystore and truststore files and automatically reconfigures its SslFactory when a change is detected, with no restart required.

Prior work: KIP-1119

KIP-1119 ("Add support for SSL auto reload") addressed the same problem. That work was not merged, for two reasons that shaped the design choices made in this KIP:

1. Reliability of Java's WatchService in Kubernetes

KIP-1119 used WatchService to detect file changes. WatchService relies on the Linux inotify kernel subsystem. However, inotify does not emit events for bind mounts, which is how Kubernetes mounts ConfigMaps and Secrets into pods. This means WatchService would silently miss certificate rotations in the most common Kubernetes deployment pattern, the exact environment where automatic reload is most needed.

2. Transient inconsistency during rotation

A certificate rotation is not atomic. An operator (or automation tool like cert-manager) must write the new keystore and the new truststore in two separate file operations. There is always a window (sometimes several seconds) between the two writes. If the reload fires after the first write but before the second, the client loads a mismatched pair: a new keystore with an old truststore (or vice versa). Each file is individually valid, so no exception is thrown, but the pair is semantically inconsistent. New TLS handshakes can fail until the second file is also written and the client reloads again.

Public Interfaces

New Configuration Properties

All three keys are optional and off by default. Existing configurations require no changes.

New Metrics

Group: ssl-metrics

New Internal Classes

These classes are in org.apache.kafka.common.security.ssl and are not exposed in any public API surface.

Proposed Changes

Why polling instead of WatchService

The WatchService approach was rejected for this KIP for the same reason it blocked KIP-1119: inotify does not fire on bind-mounted files in Kubernetes, which is the primary deployment environment for this feature. Polling with Files.getLastModifiedTime() is less elegant but works uniformly across bare-metal, VMs, Docker, and Kubernetes, regardless of how the volume is mounted.

The polling interval defaults to 60 seconds. For production workloads with automated rotation tools this is more than sufficient; for testing or fast-rotating certificates it can be reduced via ssl.hotreload.poll.interval.seconds.

Why content hashing instead of modification timestamps

The initial polling implementation was comparing file modification timestamps (mtime) between cycles. This approach has two reliability concerns that make it unsuitable as the sole detection mechanism.

First, Kubernetes mounts Secrets and ConfigMaps using a chain of symbolic links through versioned directories. While the JVM correctly follows this chain when reading mtime, some container runtimes and volume configurations have been observed to produce unreliable or stale mtime values on the terminal file, depending on the CSI driver in use.

Second, rotation tooling that copies files while preserving the original mtime (cp -p, rsync --times, certain Ansible modules) can replace a file's contents without advancing its modification timestamp. A timestamp-based poller would silently miss such a rotation.

This KIP uses the SHA-256 digest of each file's byte contents as the change signal. A change is detected when the digest differs from the previous cycle. This is unaffected by symlinks, copy semantics, or filesystem metadata reliability — only the actual file contents matter. The computed digest is also published as a metric (see below).

Why debounce

A certificate rotation involves writing two files (keystore and truststore) in two separate operations. Without a grace period, the poller may detect the first write and reload immediately, loading an inconsistent pair. The debounce mechanism delays the reload until the filesystem has been quiet for a configurable window.

The logic is:

1. The poller detects a file change.
2. Instead of notifying listeners immediately, it schedules a one-shot "notify" task to run after ssl.hotreload.debounce.seconds.
3. If another file change is detected before the task fires, the task is cancelled and a new one is scheduled.
4. When the task finally fires, it notifies all listeners. By this time both files have settled.

Setting ssl.hotreload.debounce.seconds=0 disables this behaviour entirely and restores immediate-fire semantics. The default of 5 seconds covers typical rotation tooling (cert-manager, Vault Secrets Operator, ..) without introducing a noticeable delay.

Note: the debounce addresses the case where two files are updated seconds apart during an automated rotation. It is not a substitute for a deliberate, multi-step CA rotation procedure. For the latter, operators should use the SHA-256 hash metrics described below to verify each step has propagated before proceeding.

Why SHA-256 metrics and operational metrics

Publishing metrics serves two independent purposes.

The first is operational alerting. The ssl-hot-reload-poll-last-run-ms and ssl-hot-reload-poll-failure-total metrics allow operators and platform teams to alert on the poller's health. A stale last-run timestamp means the polling thread has died and hot reload is silently no longer functioning. A rising failure count means each poll attempt is throwing an error (which could indicate for example a file permission change or disk problem introduced after the client started). Without these metrics, neither failure mode is visible to the operator.

The second is CA rotation observability, specifically for mTLS deployments.

In mTLS setups, a safe CA rotation requires multiple deliberate steps: adding the new CA alongside the existing one in every node's truststore, waiting until every node trusts both CAs, rotating the server certificates, and only then removing the old CA. Each step must be fully propagated across all nodes before the next step begins. Proceeding too early can break intra-cluster communication.

Today there is no way for an operator to verify, without filesystem access to each individual node, that a hot-reload cycle has completed and that a node has loaded the expected certificate material. The truststore and keystore SHA-256 hash metrics fill this gap. By querying the hash metrics across all nodes via JMX or a Prometheus scraper, an operator can confirm that every node reports the same expected fingerprint before proceeding to the next rotation step.

The SslMaterialPollerRegistry

A single Kafka application can create many SslFactory instances (one per consumer, one per producer, etc.). The naive approach would create one polling thread per SslFactory. This is wasteful in the common case where all factories share the same SSL configuration.

Two extremes were considered:

• One global singleton poller: efficient, but invalid if the application connects to multiple clusters using different certificates.
• One poller per factory: safe but redundant.

This KIP takes a middle path. SslMaterialPollerRegistry is a process-wide registry keyed by the tuple (keystorePath, truststorePath, pollIntervalSeconds, debounceSeconds). Two factories with identical keys share one poller and one polling thread. Each factory registers a listener (its reconfigure callback) with that shared poller. When a factory is closed it deregisters its listener; the poller is stopped and removed from the registry when its last listener is removed.

This means:

• Applications with a single SSL configuration (the common case) use exactly one polling thread, regardless of how many consumers and producers they create.
• Applications that connect to multiple clusters with different certificates get independent pollers, one per distinct configuration. Isolation is preserved: a file change for cluster A does not trigger a reload for cluster B.

Changes to SslFactory

• Add field Runnable onSslMaterialChange to hold the listener reference (required for deregistration, which uses reference equality).
• In configure(): if hot reload is enabled, deregister any previous listener and call SslMaterialPollerRegistry.getInstance().register(configs, this.onSslMaterialChange).
• In close(): call SslMaterialPollerRegistry.getInstance().deregister(configs, this.onSslMaterialChange) to clean up. This ensures no thread leak when a producer or consumer is closed.

Compatibility, Deprecation, and Migration Plan

• The feature is opt-in and disabled by default. No existing behaviour changes unless ssl.hotreload.enable=true is set.
• The feature applies to clients only: producers, consumers, and admin clients. Broker-side hot reload is already possible via dynamic configuration and is out of scope for this KIP.

Test Plan

• Verify that changing a truststore file (simulated by updating its modification time) triggers SslFactory reconfiguration.
• Verify that when two factories watch different files, changing one file only reloads the corresponding factory.
• Verify that hot reload disabled (ssl.hotreload.enable=false) produces no reconfiguration even when files change.
• Verify that the debounce timer resets when a second file change arrives within the window.
• Verify that two changes in rapid succession produce exactly one listener invocation, not two.
• Verify that setting debounce=0 calls the listener immediately.
• Verify that two factories with the same SSL config share one poller (one polling thread).
• Verify that two factories with different SSL configs get independent pollers.
• Verify that closing the last factory for a given config removes the poller from the registry.

This was materialized in 8 new tests. in SslFactoryTest.

Rejected Alternatives

Using WatchService

Rejected because inotify does not emit events for bind-mounted files. Kubernetes mounts ConfigMaps and Secrets as bind mounts. This is the most common deployment environment for this feature, so a mechanism that silently fails there is not acceptable.

Very good article I found when I was surprised the no event was emitted in Kubernetes setup: https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/

One SslMaterialPoller per SslFactory (no registry)

Simple to implement and fully isolated, but creates one polling thread per producer and consumer even when they all watch the same two files. In applications with many clients this is wasteful. Rejected in favour of the registry approach.

One global singleton poller

Efficient for the common case (that was the choice for KIP-1119), but incorrect when a single application connects to multiple clusters with different SSL credentials. A global poller can only watch one set of files; it cannot serve multiple distinct configurations. Rejected in favor of the registry approach.