Status

Discussion threadhttps://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9
Vote threadhttps://lists.apache.org/thread/towpvvvrf691wynlrtfkkt0r5csqh4lk
JIRA
Release1.18


Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As one of the release 2.0 efforts, the release managers were discussing about our API life cycle policies. There have been FLIP-196 and FLIP-197 that are relevant to this topic. These two FLIPs defined the stability guarantees of the programming APIs with various different stability annotations, and the promotion process. A recap of the conclusion is following:

Stability:
@Internal API: can change between major/minor/patch releases.
@Experimental API: can change between major/minor/patch releases.
@PublicEvolving API: can change between major/minor releases.
@Public API: can only change between major releases.

Promotion:
An @Experimental API should be promoted to @PublicEvolving after two releases, and a @PublicEvolving API should be promoted to @Public API after two releases, unless there is a documented reason not to do so.

One thing did not mention in these two FLIPs is the API deprecation process, which is in fact critical and fundamental to how the stability guarantees to the end user because the sensible API stability is all about removing existing APIs. For example, if we want to change a method ResultFoo foo(ArgumentFoo arg) to ResultBar bar(ArgumentBar arg), there are two ways:

  1. Mark method foo as deprecated and add the new method bar. At some point later, remove the method foo.
  2. Simply change the API in place, that basically means removing method foo and add method bar at the same time, i.e. replace method foo with method bar.

In the first option, users are given a period with stability guarantee to migrate from foo to bar. For the second option, this migration period is effectively zero. A zero migration period is problematic because end users may need a feature/bug fix from a new Flink version, but they cannot upgrade right away due to some backwards compatible changes, even though these changes perfectly comply with the API stability guarantees defined above. With a migration period, users will be able to better arrange their migration timing, which makes the upgrade more smooth. Therefore, ideally the migration period should never be 0, even for the @Experimental APIs.

As you may see, the migration period is fundamental to the API stability guarantees for the end users. And it is essentially how long a deprecated API can be removed from the source code. This FLIP proposes an API deprecation process for Flink.

Public Interfaces

  • Always add a "Since X.X.X" comment to indicate when was a class / interface / method marked as deprecated. So we can easily track when should the deprecated code be cleaned up in the deprecated list generated by Java doc. We should use ArchUnit to enforce this rule is possible.
  • In the release notes, add a section about the deprecation to notify the users about the source code removal in advance.

In addition, a @Public interface can only be marked as deprecated if

  • There are one or more @Public interfaces that serve as replacement of its functionality, or
  • The functionality of the @Public interface is going to be removed completely without any full or partial replacement.

Proposed Changes

The migration periods

API AnnotationMigration PeriodDESCRIPTION with Examples
ExperimentalAt least 1 patch release for the affected minor release.

Deprecation process:

  • Assuming an experimental API foo is introduced in 1.18.0, and is carried over to 1.19.0.
  • API foo is marked as @Deprecated in 1.19.1, with a new method bar introduced as a replacement
  • The source code of API foo SHOULD be removed in 1.19.2 or after.

User migration path:

  • If user is on 1.19.0, when they upgrade to 1.19.1, their application won't break immediately, they do a planned migration from foo to bar. So when they upgrade to 1.19.2, things won't break.
  • If user is on 1.18.x, they should be able to upgrade to 1.19.1 without being broken. Then take an active migration from foo to bar. After that, upgrading to 1.19.2 won't break the application.

NOTES:

  • Exception for zero migration period. If an experimental API is found defective and not working at all, we can replace it with a new API immediately with a zero migration period, because keeping a broken API for backwards compatibility does not make sense. However, this case should be extremely rare. In most cases, API in such status will be defined as WIP and therefore not announced to the end users.
  • In general, there is no need to backport the deprecation of foo in 1.19.1 to 1.18.x, unless the deprecated API is in "NOT WORKING" status mentioned above.
PublicEvolving

At least 1 minor release.

Deprecation Process:

  • Assuming a PublicEvolving API foo is introduced in 1.18.0, and is carried over to 1.19.0.
  • API foo is marked as @Deprecated in 1.20.0, with a new method bar as its replacement.
  • The source code of API foo needs to be kept in 1.20.0, and should be removed in 1.21.0 or after. 

User migration path:

  • If a user is on 1.18.0 or 1.19.0, they can upgrade to 1.20.x without being broken. When the users is on 1.20.x, they should do a planned migration from method foo to method bar. After that, migration to 1.21+ should not break things.

NOTES:

  • If there is a major version bump before 1 minor releases in the current major version are reached, the major version should keep the source code in its own minor version until two minor versions are reached. For example, in the above case, if Flink 2.0 is released after 1.19 instead of 1.20, then method foo should be marked as Deprecated in 2.0 and removed in 2.1 or after.
PublicAt least 2 minor releases.

Deprecation Process:

  • Assuming a Public API foo is introduced in 1.18.0, and is carried over to 1.19.0.
  • API foo is marked as @Deprecated in 1.20.0, with a new method bar as its replacement.
  • The source code of API foo needs to be kept in 1.20.0 and 1.21.0, and all the 1.x releases, and should be removed in 2.0 or after. 

User migration path:

  • If a user is on 1.18.0 or 1.19.0, they can upgrade to 1.20.x and 1.21.x without being broken. When the users is on 1.20.x / 1.21.x, they should do a planned migration from method foo to method bar. After that, migration to 1.22+ (including 2.0) should be smooth.

NOTES:

  • If there is a major version bump before 2 minor releases in the current major version are reached, the major version should keep the source code in its own minor version until two minor versions are reached. For example, in the above case, if Flink 2.0 is released after 1.20, then the deprecated source code of foo will be kept in 2.0 and all the 2.x versions. It can only be removed in 3.0.


Note that the same requirement for API change is still needed for all the Experimental/PublicEvolving/Public API changes.

Compatibility, Deprecation, and Migration Plan

  • The changed proposed in the FLIP are all about processes and conventions of the source code deprecation and removal, which are not clearly defined at the moment. So there should be no compatibility, deprecation or migration issue.

Test Plan

We can potentially add ArchUnit tests to ensure the deprecation information is provided correctly.

Rejected Alternatives

N/A