Release Managers

Jiangjie Qin Xintong Song Martijn Visser Jark Wu 

Release Sync

  • The release sync is scheduled 8am Wed (GMT+2)  / 2pm Wed (GMT+8) / 11pm Tues (GMT-7). Not every week will we have release sync. Please subscribe to the Google Calendar for updates.
  • Everyone is welcome to join the release sync via Google Meet.

Timeline Plan

  • Preview Release: End of Sep, 2024
  • Feature Freeze: End of Nov, 2024
  • Release: End of Dec, 2024 or Early Jan 2025

Work Items

Status (Updated: 10/08)


Done

(tick)

Will make it

(green star)

In danger

(star)

Very unlikely

(red star)

Won't make it

(minus)

State unclear

(question)

Total
Breaking changes170003020
Features112200015

Breaking Changes (Targeting to finish by the Preview Release)

Name / JIRA / FLIPContributorReviewerStateLast UpdateDescription / Note

Programming

API








Remove DataSet APIXu Huang

Weijie Guo 

(tick)

 

PR in review

Remove Deprecated methods/fields/classes in DataStream API

Xu Huang

Weijie Guo 

(tick)

 

Waiting for CI

FLINK-29741 - Getting issue details... STATUS

Xu Huang

Weijie Guo 

(tick)

 

  • Table dependencies has been moved to the Table module as internal interfaces. 
  • All other dependencies have been migrated.

FLINK-33747 - Getting issue details... STATUS

Yanquan LyuQingsheng Ren (tick)

 

  • Sink V1 has been removed

FLINK-36369 - Getting issue details... STATUS


Jark Wu 

(tick)

  

All deprecated APIs classes had been moved to another package, as internal interfaces.
Remove deprecated methods in Table API

Jark Wu 

(minus)

  

Can't finish the work within the preview release(this part of the work will be processed as much as possible in 2.0)

FLINK-36245 - Getting issue details... STATUS

Yanquan Lyu

Qingsheng Ren 

(tick)

 

  • SourceFunction has been relocated to package "org.apache.flink.streaming.api.functions.source.legacy"

  • SinkFunction has been relocated to package "org.apache.flink.streaming.api.functions.sink.legacy"

FLINK-14068 - Getting issue details... STATUS

Matthias Pohl 

Chesnay Schepler 

(tick)

 

Replace o.a.f.api.common.time.Time  by java.time.Duration  (deprecate public API in 1.19, removal in 2.0).

FLINK-14068 - Getting issue details... STATUS

Matthias Pohl 

Chesnay Schepler 

(tick)

Replace o.a.f.streaming.api.windowing.time.Time  by java.time.Duration  (deprecate public API in 1.19, removal in 2.0)

Configuration

FLINK-33750 - Getting issue details... STATUS

Xuannan Su  

Chesnay Schepler 

Zhu Zhu 

(tick)

 

All of the following configurations have been completely removed:

  • @Public configurations that have been deprecated for at least two minor release
  • @PublicEvolving configurations that have been deprecated for at least one minor release
Introduce a clean configuration layer

FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Zhu Zhu 

Junrui Lee 

Zhu Zhu 
(tick)

 

Goals:
1. All configuration can be set using key-value pairs

2. Avoid exposing unnecessary configuration (i.e. the entire ExecutionConfig) to user functions

FLINK-23620 - Getting issue details... STATUS

Junrui Lee 

Chesnay Schepler 

Zhu Zhu 

(tick)

 

Use a YAML compliant configuration parser.

Note: While this can be introduced as an optional feature in 1.x (i.e., first parse with standard YAML parser, fall back to custom parsing as a backup), the goal for 2.0 would be to remove the custom parsing logic.

Revisit all config option types and default values

Xuannan Su 

Junrui Lee 

Zakelly Lan

Jane Chan 

Dian Fu 


(tick)

 

The removal of deprecated ConfigOptions is covered by the work item above.

FLINK-33750 - Getting issue details... STATUS

State



Upgrade FRocksDB

FLINK-35573 - Getting issue details... STATUS

yue ma 

Zakelly Lan 

(tick)

 


FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package

FLINK-4602 - Getting issue details... STATUS


Zakelly Lan 

(minus)

 

Update RocksDB statebackend package

Since the public apis in old package are not deprecated in 1.20, we can only do deprecation in 2.0. So there will be no breaking changes in 2.0.


FLIP-398: Improve Serialization Configuration And Usage In Flink

Zhanghao Chen

Fang Yong 
Weijie Guo 

(tick)

  

Removed deprecated APIs in 2.0-preview

REST API

FLINK-32377 - Getting issue details... STATUS

FLINK-32688 - Getting issue details... STATUS

Matthias Pohl 

Chesnay Schepler 

(tick)

 

Remove deprecated fields from ExceptionHistory API (all-exceptions, timestamp, root-exception, truncated)

FLINK-36147 - Getting issue details... STATUS

Matthias Pohl 

Chesnay Schepler 

(tick)

Removes location field from REST API (was replaced by endpoint field)

Java Version



Make Java 17 default

Martijn Visser 


(minus)

 


Drop support for Java 8

Martijn Visser 


(tick)

 

Support for Java 8 has been dropped

Features (Targeting to finish by the Feature Freeze)

Name / JIRA / FLIPContributorReviewerStateLast UpdateDescription / Note
Introduce DataStream API V2

Weijie Guo 

Xintong Song 

(green star)

 

This is an either in-place or smooth replacement of DataStream API. It differs from the current DataStream API mainly in the following aspects.

  • Users define the data processing only through implementing a series of generic process functions
  • No exposes of the operator internals (mailbox thread model, barrier alignment, etc.)
  • No dependencies on internal implementations and/or 3rd party dependencies.

Progress: 

Basic stateless & stateful ds v2 framework: Done

Generalized Watermark: FLIP in preparation

Delta Join



Jark Wu 

(red star)

 

Delta join is a new kind of stream-stream join implementation without maintaining any states. 

Progress: FLIP in preparation.

State compatibility for SQL jobs



Jark Wu 

(minus)

 

State compatibility of upgrading Flink version for SQL jobs. This can be a good API story and milestone for Flink SQL for 2.x series. 

FLIP-473 Introduce New SQL Operators Based on Asynchronous State APIs


lincoln lee 

(green star)

 

SQL operator implementations based on the new Disaggregated State Management.

Progress: Join operator has been supported. Will support other 2 commly used operator before release.

Disaggregated State Management

Yuan Mei Zakelly Lan Yanfei Lei Hangxiang Yu 

Yuan Mei 

(green star)

 

Disaggregated State Backend/Management in the cloud-native era.

  • able to support very large states with a reasonable TPS
  • able to support state Sharing and Querying without interfering with normal data processing
  • Support fast recovering/rescaling in the cloud-native era

Note: Only the public API related part is must-have for release 2.0.

Progress: The new db and async state API & implementation are done. Local disk cache is done. Now working on checkpoint & recovery part.

FLIP-478 Introduce Config Option Scope

Zhu Zhu 

Junrui Lee 

Zhu Zhu 

(green star)

 

Clarify the scopes of configuration options, i.e. job config, cluster config or both, to avoid misuses of configuration. The scopes should be displayed in Flink configuration documentation.

FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

Matthias Pohl 

Zdenek Tison(tick)

 


FLIP-370: Support Balanced Tasks Scheduling

RocMarshal 

Rui Fan (star)

  • The slot level balancing strategy is completed
  • Progress in pending: 
    • The failover strategy for balancing tasks scheduling is in discussing.
FLIP-398: Improve Serialization Configuration And Usage In Flink

Zhanghao Chen 

Weijie Guo 

Fang Yong 

(green star)

 

In development:

  • Parametriezed type extractor to allow switching between Kryo-backed serializors (for backwards-compatibility) and dedicated ones for collection types
  • Dedicated serializer support for common Java collection types
FLIP-472: Aligning timeout logic in the AdaptiveScheduler's WaitingForResources and Executing states

Zdenek Tison

Matthias Pohl 

(tick)

 


Upgrade checkstyle

Martijn Visser 


(green star)

 

The major work about Checkstyle upgrade was done in 1.20 (Checkstyle was bumped to 9.3), in 2.0 the only thing to be done is just version bump




Adaptive optimization for batch jobs

FLIP-468: Introducing StreamGraph-Based Job Submission

FLIP-469: Supports Adaptive Optimization of StreamGraph

Junrui Lee 

Zhu Zhu 

(green star)

 

FLIP-468 is done

FLIP-469 is in development, target to finish in 2024-12

FLIP-470: Support Adaptive Broadcast Join

Xia Sun 

Zhu Zhu 

(green star)

 

in development, target to finish in 2024-12
FLIP-475: Support Adaptive Skewed Join Optimization

Junrui Lee 

Lei Yang 

Zhu Zhu 

(green star)

 

in development, target to finish in 2024-12

FLINK-33163 - Getting issue details... STATUS

Martijn Visser 


(green star)

 


Deprecate support for Java 11 (to-be discussed via FLIP)

Martijn Visser 


(green star)

 


Upgrade Calcite to 1.33.0 FLINK-31362 - Getting issue details... STATUS

Sergey Nuyanzin 

Timo Walther 

(tick)



Upgrade Calcite to 1.34.0 FLINK-31836 - Getting issue details... STATUS

Sergey Nuyanzin 

Timo Walther 

(tick)



Status / Follow-ups

11/05/24

  • Status
    • 2.0-preview released
    • Features
    • Testability Blockers
      • 6 Testability Blockers(2 resolved compared to last time), 5 of which are not being looked into
  • Previous Actions
    • Start a FLIP discussion for deprecating Java 11 Martijn Visser 
    • Update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
  • Discussion
    • Feature Freeze Date

10/09/24

  • Status
    • Breaking Changes
      • 17 completed
      • 2 will be completed in release 2.0
      • 1 will not happen in release 2.0
    • No open blocker issues with FixVersion 2.0-preview
    • Testability Blockers
      • 7 Testability Blockers(1 resolved compared to last time), 4 of which are not being looked into
  • Previous Actions
    • Ping contributors of work items listed as Breaking Changes to create Jira tickets under "2.0-preview" Xintong Song 
    • Start a FLIP discussion for deprecating Java 11 Martijn Visser 
    • Update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
  • Discussion
    • Create RC1 for 2.0 Preview
      • All blockers resolved
      • Release process: PR merged
      • Release announcement: PR in review, so far all comments addressed
    • Hive 3.1 does not support Java 11 FLINK-36437 - Getting issue details... STATUS
  • Action

09/25/24

  • Status
    • 8 Testability Blockers, 4 of which are not being looked into
    • Breaking Changes
      • Most are in code review
  • Previous Actions
    • Create a Jira ticket for checking deprecated APIs before the Preview Release Jiangjie Qin 
    • Find a solution for PyFlink tests depending on external connectors that uses deprecated APIs. Xintong Song Dian Fu
    • Review PR for FLINK-36207 - Getting issue details... STATUS Xintong Song 
    • Share the list of tasks for deprecated API removal  FLINK-36306 - Getting issue details... STATUS Weijie Guo
    • Looking into the preview release process FLINK-36360 - Getting issue details... STATUS Xintong Song 
    • Start working on the release announcement (Sep. 25) Xintong Song 
    • Ping contributors of work items listed as Breaking Changes to create Jira tickets under "2.0-preview" Xintong Song 
    • Start a FLIP discussion for deprecating Java 11 Martijn Visser 
    • Update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
  • Discussion
    • Progress of dropping Java 8 supports and making Java 17 the default?
    • Preview release process FLINK-36360 - Getting issue details... STATUS
      • Tried basically everything except for publishing the artifacts and building the images
    • Connector migration plan
      • 2.0-preview: Kafka connector
      • 2.0.0: a few most commonly used connectors
      • 2.3.0 (1 year since ): 80% of first-party connectors
    • Release Announcement
      • I've started collecting contents and working on the first draft
      • Will soon open a PR, based on which everyone can proposed contents to add
    • Time Plan
      • Sep. 30 - Breaking changes due
      • Oct. 1~7 - Public holidays in China
      • Oct. 8 - Branch cutting, RC Vote, etc.

09/11/24

  • Status
  • Previous Actions
    • Revisit and remove REST API and Metrics related legacy work items Xintong Song 
    • Create a 2.0-Preview Jira version Xintong Song 
    • Start a FLIP discussion for deprecating Java 11 Martijn Visser 
    • Update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
    • Create a Jira ticket for checking deprecated APIs before the Preview Release Jiangjie Qin 
  • Discussion
    • Risks for the preview release
      • PyFlink tests depends on external connectors, which depend on deprecated APIs. Removing deprecated APIs causes CI failures. This blocks many tasks including removal of Scala APIs and deprecated config options, and replacing Time with Duration.
        • Offline reached out to Dian Fu after the release sync. According to him, the related test cases are for testing the connectors. As the connectors are moved from Flink's main repo to external repos, the tests should be moved too. He will work on it. 
        • To quickly unblock other tasks, developers can disable the problematic pyflink test cases for now. They will be re-enabled after moved to external connector repos.
    • FLINK-36207 - Getting issue details... STATUS
      • Blocked by Scala removal. Weijie Guo is currently working on it, and should finish it by this weekend.
      • We'll disable Japicmp for @Deprecated APIs until the Flink 2.0 release.
      • We'll recheck if every breaking change is expected, by turning the check back on locally before the release and check all the reported issues. FLINK-36261 - Getting issue details... STATUS
    • "2.0-preview" Jira version
      • Tickets targeting the preview release should set the fixVersion to 2.0-preview
      • Tickets resolved before the preview release should set the fixVersion to 2.0-preview
    • For breaking changes that will be postponed to Release 3.0
      • A "3.0.0" Jira version has been created. Please set the target fixVersion accordingly.
      • There's also an umbrella for release 3.0 breaking changes. It would be helpful for preparing the next major version bump if we gather all breaking changes in one place. FLINK-36200 - Getting issue details... STATUS
    • Release process for Flink 2.0 (& Preview)
      • We need to start looking into the release process right away
        • We need to enrich the release guide for major releases.
        • We need to do a dry-run for branch cutting, building, etc., except for the final deploying / publishing stage.
      • We will start working on the release announcement 1-2 weeks later, so we know better which features going to make it
  • Actions
    • Find a solution for PyFlink tests depending on external connectors that uses deprecated APIs. Xintong Song Dian Fu
    • Review PR for FLINK-36207 - Getting issue details... STATUS Xintong Song 
    • Share the list of tasks for deprecated API removal Weijie Guo 
    • Ping contributors of work items listed as Breaking Changes to create Jira tickets under "2.0-preview" Xintong Song 
    • Looking into the preview release process Xintong Song 
    • Start working on the release announcement (Sep. 25)

08/28/24

  • Status
  • Discussion
    • Work items
      • Java version supports
        • Support Java 21
        • Make Java 17 (Preview Release)
        • Drop Java 8 (Preview Release)
        • Start a FLIP discussion for deprecating Java 11. Martijn Visser 
          • We will not remove support for Java 11 directly, because it's still the default java version in 1.20 and has not been deprecated.
      • REST API and Metrics related legacy work items can be removed.
        • Will remove after revisiting and checking for APIs already deprecated and can be simply removed. Xintong Song 
      • We'll keep collecting new features.
    • Preview Release
      • Complete breaking changes in the Preview Release, by the end of September
        • For Source/SinkFunction and legacy TableSource/SinkFunctions, there are a lot internal usages (planner, 1st-party connectors, tests, etc.) that may not be migrated soonish. We will first move the public APIs to internal interfaces, and gradually remove them later.
        • We should update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
      • Create a 2.0-Preview Jira version Xintong Song 
      • Create a Jira ticket for checking deprecated APIs before the Preview Release. Jiangjie Qin 
        • Add scripts for gathering all deprecated APIs
        • Make sure all deprecated APIs are either removed, or we understand why they are not removed.
        • We can leverage the scripts to generated API differences between versions (e.g., 2.0-Preview vs. 2.0.0)

08/14/24

  • Status
  • Discussion
    • Preview Release
      • Try to include all API breaking changes
      • Won't do
        • Feature freeze
        • Feature testing
        • Resolve all blockers
      • Will do
        • PMC vote (e.g., legal checks)
        • Announce and promote
      • Maybe define a new version "2.0-Preview" on Jira and mark all API-breaking changes are blocker for it?
    • Time Plan
      • Preview release: End of Sep.
      • Feature freeze: End of Nov.
      • Formal release: End of Dec. or early Jan.
    • Work Items
      • API-breaking: targeting to finish by the preview release
      • Non-breaking: targeting to finish by the feature freeze date, following time-based rules
  • Follow-ups 
    • Update work item list: separate into two lists, and remove outdated items
    • Create the "2.0-Preview" release and move relevant tickets to it?
    • Start collecting feature plans
  • No labels