This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: None yet

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Before Kafka 2.8.0 all features seemed to be released without any type of qualification into the project. During 2.8.0 it was the first time that a feature was released with "early access". This was done in the context of KRaft as a replacement for ZooKeeper (https://lists.apache.org/thread/m16d7om3lry70zw9b1t6fyw99tlppkd3). However, we never really defined what "early access" meant, for the developers nor for the end users.

An older instance of an intention of releasing a feature in a qualified mode was Tiered Storage (https://lists.apache.org/thread/w5w55jyko9zvh3fxxh4lrs6z7trx3nzk). Here, "Preview" was used as a qualifier to signify that the feature wouldn't be complete and the target was a release in the future.

Since then, we have released several features into Kafka with either "Early Access" or "Preview" qualifiers, sometimes a feature even went through "Early Access" first and "Preview" later (https://github.com/apache/kafka-site/pull/619).

We need to agree on the number of steps and what each of these steps mean for developers as well as for end users of said features.  We had some discussion in the mailing list and gathered some feedback here: https://lists.apache.org/thread/5z6rxvs9m0bro5ssmtg8qcgdk40882bz

The proposal is to create 4 levels, 1 being the lowest and 4 being the most mature level. By having these levels, the release manager can easily know if KIPs are present in a release (and under which category) of if they need to be postponed to the next one. Only KIPs that are declared level 2 and onwards can be released in a given release, KIPs that remain in Level 1 are to be pushed to further releases.

Public Interfaces

No changes in public interfaces

Proposed Changes

Graduation Steps

In order to reduce bike-shedding on the names of the steps, I propose to first use placeholder ones to define the hierarchy of steps and the implications for each of them. This step system is not meant to ensure a bug-free feature (more than we already try to ensure), but rather express the level of community input needed for a given feature.

Level 1

This is the "farther" from production a feature is. The feature is not yet completed by any means and it's not yet usable. For completion sake, in order to define graduation steps of a feature, we need to start by step 0 which is when the feature is being made.

The feature is not yet stable and it is not meant to be used by end users.

Level 2

The feature is now usable, but it might not be completed yet. This is intended for bigger features that are delivered in multiple milestones. As long as the feature reaches a milestone that enables end users to use the feature, its developers can decide to graduate from level 1 to level 2.

A feature in level 2 should be usable for end users, it should contain complete flows and it shouldn't, knowingly, leave Kafka clusters in any type of corrupt state. It might be that only a subset of the functionality is available or implemented. End users are encouraged to start testing this feature and to provide feedback and bug reports for it.

The API for this feature is not to be considered stable, developers reserve the right to change the API to fix any kind of bugs or inconsistencies found during testing. It is not encouraged to build tools or applications that would rely on these APIs. The fact that the API can change doesn't exempt from updating the KIP and submitting it for a new VOTE thread if needed.

This level is optional, and a feature does not need to go through it to reach the next level.

Level 3

Mildly complex features might reach this level directly from level 1, while more complex ones would reach it from level 2.

A feature in level 3 should be feature-complete (as per the KIP that defined it), it can still be evolved in terms of, for example, adding more and better metrics, documentation, or improving certain usability aspects of the feature. The feature is intended to work in different types of production environments, but there might be some aspects that might need some more testing and validation. The purpose of this level is to request help from the community to test this feature under said scenarios to confirm the feature works as expected. For example, help might be needed to test the feature in extreme high volume environments.

The API for this feature is to be considered stable and it can't change without undergoing a full deprecation cycle (at least 12 months in deprecated state after being removed in a "major" release).

This level is optional, and a feature must not go through it in order to reach the latest level.

Level 4

This is the final stage of the feature. Any feature might reach to this level directly, but it's only recommended for simpler and smaller features.

A feature in level 4 should be feature-complete (as per the KIP that defined it) and it must have full documentation, test suites and if it applies, system tests. If the feature went through "Level 3", all potential problems found are addressed. This level doesn't offer any more guarantees that any other feature that is already released in Kafka, bugs might still occur.

The API for this feature is to be considered stable and it can't change without undergoing a full deprecation cycle (at least 12 months in deprecated state after being removed in a "major" release).

Table

[1]: Implicitly when a KIP is approved and development starts, the feature is effectively in "Level 1"

[2]: As a final state of graduation all features aspire to get to this level.

Examples

After KIP-853: KRaft Controller Membership Changes got approved, it would have been positioned at Level 1. It stayed in that level for a while. In between, Kafka 3.8 release happened and it remained at Level 2. After that, it got promoted to Level 2 just in time to be taken in Kafka 3.9 release.

KIP-848: The Next Generation of the Consumer Rebalance Protocol on the other hand reached Level 2 at the time of Kafka 3.7. Improvements were done after that and when it was time to release Kafka 3.8 the KIP got promoted to Level 3.

An example of a feature in Level 4 could be KIP-390: Support Compression Level that reached this level during Kafka 3.8, and jumped directly from Level 1 to Level 4.

Names

1. "In Development"
2. "Early Access"
3. "Preview"
4. "Production Ready"

Alternatively, if we want to be a bit more playful we could go with a theme borrowed from the book industry (as an homage to Franz Kafka):

1. "In Development"
2. "Manuscript"
3. "Pre-print"
4. "Published"

When can a feature change levels?

Features can only move in one direction on the step system, level 1 to level 4, and can't go back to the previous level. A feature can be promoted to a "higher" level at any release (including patch releases).

Documentation

A new page in the documentation page will be created containing the agreed graduation steps for features and the implications for each of them. This new page should be under "Key Concepts" (also known as "Getting Started") in its own section. Right now it would be "1.7 Graduation Steps for Features".

How to track level changes

KIPs that require to go through the different graduation levels, must include a new section (Graduation Steps Log) containing the change log for the graduation steps and their date/release.

When preparing a release plan, the release manager should look at this section to capture the KIPs state.

If the KIP changes graduation step while a release is being prepared, the driver of the KIP should communicate this in the release thread.

Each graduation step can be stated by the developer "owning" the KIP. The developer might decide to seek consensus via a VOTE thread to promote to level 3 and 4.

Compatibility, Deprecation, and Migration Plan

Does not apply

Test Plan

Does not apply

Rejected Alternatives

The use of "early availability" and "general availability" have been rejected because, this being an open source project, all features are by default available, Kafka is not ruled by a centralized entity that decides which users get which features. "Early availability" and "general availability" give away the impression that features are not able to be used by end users until we decide to toggle some switch.

Using "alpha", "beta" and "gamma" has been rejected because of the widespread use and understanding of what "alpha" and "beta" mean in software development.