This is a mini-KIP proposal to allow a legacy Kafka in the 3.9.2 maintenance release to invoke an AlterConfigPolicy
in the same way when running in ZooKeeper mode and KRaft mode, thereby facilitating migration to KRaft.

Status

Current state: "Adopted"

Discussion thread: here-2 and here-1 

JIRA: KAFKA-19026 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

When processing an Incremental Alter Config on a Config entry of type List with OpType.SUBTRACT
the metadata passed to  AlterConfigPolicy.validate contains

• in KRaft mode : the config that would result AFTER the subtraction
• in ZK mode : the config as if the opType was OpType.SET, with no indication that actually the value would be removed

Also OpType.APPEND behaves inconsistently in this regard.

The behaviour shown in the ZK case makes it impossible to write a policy that fully protects topics from being left with an empty cleanup policy
(when that happens, no retention is applied and partitions can grow indefinitely).

As discussed on the dev mailing list, to avoid introducing a potentially breaking change in 3.9.2, this KIP proposes to fix the the adoption of a flag to control the policy behavior.

Public Interfaces

Add a new boolean configuration for kafka servers

alter.config.policy.kraft.compatibility.enable

The default value would be false, meaning nothing changes in 3.9.2. Conversely, setting it to true would align the behavior with KRaft policy validation

Proposed Changes

If the server config entry

alter.config.policy.kraft.compatibility.enable

is set to true, then the invocation of an AlterConfigPolicy from ZkAdminManager will be fixed to match the invocation from KRaft's ConfigurationControlManager,
as proposed and tested in the PR associated with the JIRA  https://github.com/apache/kafka/pull/19263

If the property is not explicitly set to true, then the existing behaviour is preserved.

Compatibility, Deprecation, and Migration Plan

• The new configuration entry will not be ported to 4.x versions where there is no ZkAdminManager component
• What impact (if any) will there be on existing users? 
  • none as the configuration would be false by default
• If we are changing behavior how will we phase out the older behavior?
  • not planned as Zookeper mode is phased out
• After a KRaft migration, the configuration can be removed from server.properties 
  • KRaft servers will ignore it however

Test Plan

Junit testing leveraging the @ClusterTest extension

Rejected Alternatives

Implementing the ZkAdminManager fix as a default behavior was rejected through PR and mailing list discussion, where backward compatibility was preferred to correctness.