Status

Current state: "Accepted"

Discussion thread: here

Vote thread: here

JIRA: KAFKA-18422 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka has evolved significantly since its initial release 14 years ago, introducing numerous key features over time. As users run different Kafka versions, many seek to upgrade to access new features and bug fixes. However, our documentation currently lacks a recommended upgrade path, leaving users uncertain about the best approach. This KIP focuses on the the upgrade path for Kafka clients and Kafka Streams, and briefly covers Kafka Connect; the KIP does not cover broker upgrades though.

This KIP is twofold:

• Define client/Streams cut-off versions, for which direct upgrades to 4.x and newer client/Streams versions are not officially supported any longer.
• Define bridge release versions, that we recommend for upgrading very old client/Streams applications to 4.x in multiple steps.

There is a few reasons why we need to stop support very old client/Streams versions:

• Maintaining and testing backward compatibility is becoming more and more difficult, and effectively impractical.
• Dropping direct upgrade support for older versions, incentivized users to upgrade more frequently.
• There should be very few users with very old versions, and it's very unlikely that these users would jump forward to 4.x.

By defining a recommended upgrade path, we can help users navigate upgrades more confidently while also reducing the complexity of our testing matrix. This, in turn, allows us to limit direct upgrades from significantly older versions, making it easier to maintain compatibility across supported versions.

Additionally, it's important to clarify that this effort is not about defining what upgrades are technically possible—such determinations involve many nuances. Rather, our goal is to establish what we officially recommend, providing guidance that balances user needs with maintainability.

Background

Kafka does already provide certain backward/forward compatibility guarantees, that impact this KIP, and we want to summarize them here. We also need to distinguish different types of compatibility guarantees, namely client-broker protocol compatibility, (Java) API compatibility, and runtime compatibility (which is especially important for Kafka Streams, as it maintains state).

• Protocol Compatibility:
  In general, Kafka brokers and clients are backward and forward compatible. With the 4.0 release, however, older RPC versions are dropped via KIP-896, and thus, 2.1 becomes the oldest supported version compatible with 4.0, ie, 2.0 clients (or older) do not work with 4.0 brokers any longer, and 4.0 clients, require broker version 2.1 or newer.
  
  
• API Compatibility:
  Major releases allow us to remove deprecated Java APIs, with an additional guarantee that we keep deprecate APIs for at least 3 prior versions (i.e., we provide a 1 year backward compatibility). In practice this implies, that we did remove everything that was deprecated up-to-including version 3.6, with the 4.0 release, and we only provide API backward compatibility for version 3.7, 3.8, and 3.9, when upgrading to 4.x. Similarly, API backward compatibility is only guaranteed for versions 2.6, 2.7, and 2.8, when upgrading to 3.x. – We also want to call out, that the 1.x release series, had only two releases namely 1.0 and 1.1, and thus, API backward compatibility for 2.x does cover 1.1, 1.0, and 0.11.x releases (to meet our 1-year backward compatibility guarantee).
  
  
• Runtime Compatibility:
  In general we maintain runtime compatibility, and for certain upgrades, a two-rolling bounce upgrade path is necessary to bridge between older and newer version for online upgrades. With 4.0, we are dropping support for some features like "eager rebalancing" and EOSv1 and thus online upgrade from certain older version to 4.0 won't be supported any longer. Note, that it might still be possible to upgrade older versions in an offline fashion, by stopping an application completely and restarting it afterwards. This KIP does focuses on online rolling-bounce upgrades only.

Next, we want to call out major changes in Kafka Streams (and when they happend), that impact the recommended upgrade path:

• "Eager" vs "cooperative" rebalancing:
  
  • "Cooperative" rebalancing was added with Kafka Streams 2.4 release.
  • "Eager" rebalancing is removed with the 4.0 release.
  • Impact:
    • Kafka Streams application using "eager" rebalancing, cannot upgrade to 4.0 directly, but need to switch to "cooperative" rebalancing before upgrading to 4.0.
    • Older Kafka Streams applications that do only support "eager" rebalancing (ie, 2.3 and older), first need to upgrade to a bridge release and switch to "cooperative" rebalancing, before they can be further upgraded to 4.0.
• EOSv1 vs EOSv2:
  • EOSv2 was added with Kafka Streams 2.6 release (and requires broker version 2.5).
  • EOSv1 is removed with the 4.0 release.
  • Impact:
    • Kafka Streams applications using EOSv1, cannot upgrade to 4.0 directly, but EOSv2 must be enabled before upgrading to 4.0.
    • Older Kafka Streams application that do only support EOSv1 (ie, 2.5 and older), first need to upgrade to a bridge release and enable EOSv2, before they can be further upgraded to 4.0.

Public Interfaces

Only documentation changes.

Proposed Changes

Add the following content to https://kafka.apache.org/documentation/#upgrade

Client Upgrade Paths

Below is the recommended upgrade path for client/streams from an API and runtime compatibility point of view to upgrade from old version to 4.x. We omit the bug-fix version number, and recommend to use the latest bug-fix version available for all cases.

Note:

• When planning a client upgrade, broker compatibility must be considered. In Kafka broker version 4.x, the client baseline is 2.1, as proposed in KIP-896. Therefore, users have to at least upgrade Kafka client to 2.1 before upgrade broker to 4.x. For Kafka Streams, please refer to Streams API broker compatibility.
• In Kafka Streams, upgrading from specific versions may need two-rolling bounce. For more information, please check Kafka Streams Upgrade Guide.
• The lowest supported version for upgrade is 0.11.x, as earlier versions (such as 0.10.x) introduced breaking changes in Kafka Streams, and Kafka Streams itself was not available before version 0.10.0.

Kafka Clients:

API compatibility is not fully taken into account in the table below. For a smooth API compatible upgrade path, 0.11.x - 2.5.x clients should first upgraded to 2.8.x, and usage of all deprecated APIs should be removed, before upgrading to 3.9.x.

For clients 2.6.x to 3.3.x, upgrading to 3.9.x and removing the usage of deprecated APIs to ensure an API compatible upgrade to 4.x, is recommended.

Kafka Connect:

API compatibility is not fully taken into account in the table below. For a smooth API compatible upgrade path, 0.11.x - 2.5.x clients should first upgraded to 2.8.x, and usage of all deprecated APIs should be removed, before upgrading to 3.9.x.

For connect 2.6.x to 3.3.x, upgrading to 3.9.x and removing the usage of deprecated APIs to ensure an API compatible upgrade to 4.x, is recommended.

Kafka Streams:

For Kafka Streams, API compatibility has higher impact compared to clients, and thus, we recommend an API compatible upgrade path. This implies that older Kafka Streams applications need to use two bridge releases, before an upgrade to 4.x is recommended.

Compatibility, Deprecation, and Migration Plan

This KIP focuses on documenting the client upgrade path. It does not require compatibility, deprecation, or migration plans.

Test Plan

To align with this client upgrade path, adjustments are needed for the end-to-end tests:

• In the streams_upgrade_test.py, there are existing tests for older versions like LATEST_0_11, LATEST_1_0, LATEST_1_1 and LATEST_2_0, which are no longer part of our recommended upgrade path. These tests can be removed, which will help reduce maintenance overhead.

Rejected Alternative

The client upgrade path discussed in https://github.com/apache/kafka/pull/18193 outlines several upgrade scenarios, some of which are not ideal.

• [0.11.x - 3.9.x] → 4.x: This path implies that users upgrading from version 0.11.x or higher can directly upgrade to the latest 4.x release. However, it is highly unlikely that users are still on version 0.11 and planning to upgrade directly to 4.0. Supporting such an upgrade path presents significant challenges due to the age of version 0.11.x and the complexities involved in maintaining compatibility with versions that are no longer widely in use. Additionally, for users starting from such an old version, the upgrade process would typically span multiple intermediate steps over an extended period to ensure stability.

• [0.7.x - 2.0.x] → [2.1.x - 3.9.x] → 4.0.x: This path is also not recommended as it is unclear why version 2.0 is included in the first bucket, as users generally associate major version numbers with significant feature changes. Most users would likely upgrade to 2.0.x version as their initial step. Additionally, the inclusion of 0.7.x in this path is not ideal, as Kafka Streams have breaking changes in the 0.10.x releases, and Kafka Streams functionality did not exist prior to version 0.10.0.