Status

Current state: "Adopted"

Discussion thread: here

JIRA: None (don't think this needs a Jira ticket)

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka 3.7.x was meant to be the last minor release within the 3.x series followed then by a 4.0 release where Apache Zookeeper will be removed from the code base.

During the "Road to Kafka 4.0" discussion in the mailing list (https://lists.apache.org/thread/mxltb2k5y17q89vg0k9492dfc8z66jw1), it became apparent that there are some features that are not yet present in KRaft that the community is relying on to successfully operate and run Kafka. Thus, making it either hard or not feasible for a subset of the community to move to KRaft (or at least fully test KRaft) before Zookeeper's complete removal in 4.0.0. Some community members might only be able to try KRaft when Zookeeper is gone, leaving only a downgrade as a way out path if something doesn't work as expected (instead of config switching)

This KIP proposes to create at least one more minor version within the 3.x series ,3.8.0 release, and the scope of this KIP is to define which are the respective KIPs (mostly around KRaft) that should be part of it so the community can safely migrate to a potential 4.0 version.

This way community members can safely test KRaft without the need to perform downgrades or manual workarounds, as ZooKeeper is meant to be removed in 4.0.

Public Interfaces

No direct interfaces will be changed by this KIP

Proposed Changes

We propose the following:

• Have a release 3.8.x after 3.7.0
  • The aim of this release would be KRaft strategic feature parity with Zookeeper
• Once 3.8.0 is branched, immediately start with the 4.0.x release
• 3.8.x would be the last minor release within the 3.x series.

This assumes all must have features / KIPs are implemented in 3.8. Otherwise we would need to inject a 3.9 version in between 3.8 and 4.0

Scope

The following KIPs were identified by the community as blockers items that would need to be present in a 3.x release before moving to a potential 4.0 release.

We would aim to have all these in a 3.8 release. However, in the case that this is not possible and some of these blockers are still not resolved, we should then create a new minor version on the 3.x series (namely 3.9).

The following list is not an exhaustive list of new features 3.8 would contain (this release will most probably contain other features and KIPs).

Timeline

Release 3.7.0: TBD, probably early January 2024

Release 3.8.0: 3 to 4 months after 3.7 (or earlier if all required KIPs are already merged)

Release 4.0.0: 3 to 4 months after 3.8 branch is created

This assumes all must have features / KIPs are implemented in 3.8. Otherwise we would need to inject a 3.9 version in between 3.8 and 4.0

Compatibility, Deprecation, and Migration Plan

Not relevant for this KIP

Test Plan

This release will follow the same testing plan as usual, but we encourage community members who haven't migrated yet to KRaft, to test the Release Candidates (or even specific trunk builds) in order to detect as many potential problems with KRaft when used under these, potentially, new scenarios.

Rejected Alternatives

Some rejected alternatives are:

Parallel Development

• Have a multiple release heads: keep working on 3.x branches while also working on 4.x branches.
  • This will create a huge overhead in terms of management and conflict resolution

Mark 4.0 as Experimental Release

• Mark release 4.0 as experimental and work towards a 4.x which would be stable
  • We never did anything like this
  • This would create confusion
  • Risk of losing credibility among the community