Release Managers

Jiangjie Qin Xintong Song Martijn Visser Jark Wu 

Release Sync

  • The release sync is scheduled 8am Wed (GMT+2)  / 2pm Wed (GMT+8) / 11pm Tues (GMT-7). Not every week will we have release sync. Please subscribe to the Google Calendar for updates.
  • Everyone is welcome to join the release sync via Google Meet.

Timeline Plan

Work Items

Status (Updated: 01/21)


Done

(tick)

Will make it

(green star)

In danger

(star)

Very unlikely

(red star)

Won't make it

(minus)

State unclear

(question)

Total
Breaking changes180002020
Features130005018

Breaking Changes (Targeting to finish by the Preview Release)

Name / JIRA / FLIPContributorReviewerStateDocumentationX-team VerificationLast UpdateDescription / Note

Programming

API








Remove DataSet APIXu Huang(tick)

Not Needed

FLINK-37047 - Getting issue details... STATUS

 

Done

Remove Deprecated methods/fields/classes in DataStream API

Xu Huang(tick)

Not Needed

FLINK-37048 - Getting issue details... STATUS

 

Done

FLINK-29741 - Getting issue details... STATUS

Xu Huang(tick)

Not Needed

FLINK-37049 - Getting issue details... STATUS

 

  • Table dependencies has been moved to the Table module as internal interfaces. 
  • All other dependencies have been migrated.

FLINK-33747 - Getting issue details... STATUS

Yanquan LyuQingsheng Ren (tick)

Not Needed

FLINK-37050 - Getting issue details... STATUS

 

  • Sink V1 has been removed

FLINK-36369 - Getting issue details... STATUS


(tick)


FLINK-37051 - Getting issue details... STATUS

  

All deprecated APIs classes had been moved to another package, as internal interfaces.

FLINK-36476 - Getting issue details... STATUS

xuyang(tick)

FLINK-37116 - Getting issue details... STATUS


   

This part of the work has been processed about 70% in 2.0(Can't finish all within this release)

FLINK-36245 - Getting issue details... STATUS

Yanquan Lyu(tick)

FLINK-37162 - Getting issue details... STATUS

FLINK-37052 - Getting issue details... STATUS

 

  • SourceFunction has been relocated to package "org.apache.flink.streaming.api.functions.source.legacy"
  • SinkFunction has been relocated to package "org.apache.flink.streaming.api.functions.sink.legacy"

FLINK-14068 - Getting issue details... STATUS

Matthias Pohl 

(tick)

Not Needed (documented in FLINK-14068 release notes)

FLINK-37053 - Getting issue details... STATUS

 

Replace o.a.f.api.common.time.Time  by java.time.Duration  (deprecate public API in 1.19, removal in 2.0).

FLINK-14068 - Getting issue details... STATUS

(tick)

Not Needed (documented in FLINK-14068 release notes)

FLINK-37053 - Getting issue details... STATUS

Replace o.a.f.streaming.api.windowing.time.Time  by java.time.Duration  (deprecate public API in 1.19, removal in 2.0)

Configuration

FLINK-33750 - Getting issue details... STATUS

(tick)

Not Needed


 

All of the following configurations have been completely removed:

  • @Public configurations that have been deprecated for at least two minor release
  • @PublicEvolving configurations that have been deprecated for at least one minor release
Introduce a clean configuration layer

FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
FLIP-391: Deprecate RuntimeContext#getExecutionConfig
(tick)

Not Needed

FLINK-37055 - Getting issue details... STATUS

 

Goals:
1. All configuration can be set using key-value pairs

2. Avoid exposing unnecessary configuration (i.e. the entire ExecutionConfig) to user functions

FLINK-23620 - Getting issue details... STATUS

(tick)

FLINK-37150 - Getting issue details... STATUS

FLINK-37056 - Getting issue details... STATUS

 

Use a YAML compliant configuration parser.

Note: While this can be introduced as an optional feature in 1.x (i.e., first parse with standard YAML parser, fall back to custom parsing as a backup), the goal for 2.0 would be to remove the custom parsing logic.

Revisit all config option types and default values


(tick)

Not Needed


 

The removal of deprecated ConfigOptions is covered by the work item above.

FLINK-33750 - Getting issue details... STATUS

State



Upgrade FRocksDB

FLINK-35573 - Getting issue details... STATUS

(tick)

Not Needed

FLINK-37058 - Getting issue details... STATUS

 



Zakelly Lan 

(minus)



 

Update RocksDB statebackend package

Since the public apis in old package are not deprecated in 1.20, we can only do deprecation in 2.0. So there will be no breaking changes in 2.0.


Zhanghao Chen

(tick)

FLINK-37015 - Getting issue details... STATUS

FLINK-37060 - Getting issue details... STATUS

  

Removed deprecated APIs in 2.0-preview

REST API

FLINK-32377 - Getting issue details... STATUS

FLINK-32688 - Getting issue details... STATUS

(tick)

Not Needed (new schema documented in REST API docs)

FLINK-37061 - Getting issue details... STATUS

 

Remove deprecated fields from ExceptionHistory API (all-exceptions, timestamp, root-exception, truncated)

FLINK-36147 - Getting issue details... STATUS

(tick)

Not Needed (new schema documented in REST API docs)

FLINK-37061 - Getting issue details... STATUS

Removes location field from REST API (was replaced by endpoint field)

Java Version



Make Java 17 default


(minus)



 


Drop support for Java 8


(tick)

Not Needed

FLINK-37063 - Getting issue details... STATUS

 

Support for Java 8 has been dropped

Features (Targeting to finish by the Feature Freeze)

Name / JIRA / FLIPContributorReviewerStateDocumentationX-team VerificationLast UpdateDescription / Note
Introduce DataStream API V2

(tick)

FLINK-37113 - Getting issue details... STATUS

FLINK-37067 - Getting issue details... STATUS

 

This is an either in-place or smooth replacement of DataStream API. It differs from the current DataStream API mainly in the following aspects.

  • Users define the data processing only through implementing a series of generic process functions
  • No exposes of the operator internals (mailbox thread model, barrier alignment, etc.)
  • No dependencies on internal implementations and/or 3rd party dependencies.

Progress

Generalized Watermark: Done

Event Time Support: Done

Window & Join: Done

Delta Join



(minus)



 

Delta join is a new kind of stream-stream join implementation without maintaining any states. 

Progress: FLIP in preparation.

State compatibility for SQL jobs



(minus)



 

State compatibility of upgrading Flink version for SQL jobs. This can be a good API story and milestone for Flink SQL for 2.x series. 

xuyang

lincoln lee 

(tick)

Not Needed

FLINK-37068 - Getting issue details... STATUS

 

SQL operator implementations based on the new Disaggregated State Management.

Progress: Has supported several commly used operators in release 2.0.0, covers 10+ stateful query of nexmark queries.

Disaggregated State Management

(tick)

FLINK-36439 - Getting issue details... STATUS

FLINK-37069 - Getting issue details... STATUS

 

Disaggregated State Backend/Management in the cloud-native era.

  • able to support very large states with a reasonable TPS
  • able to support state Sharing and Querying without interfering with normal data processing
  • Support fast recovering/rescaling in the cloud-native era

Note: Only the public API related part is must-have for release 2.0.

Progress: First version complete. Working on documentations and examples.

FLIP-478 Introduce Config Option Scope

(minus)



 

Clarify the scopes of configuration options, i.e. job config, cluster config or both, to avoid misuses of configuration. The scopes should be displayed in Flink configuration documentation.

FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveSchedulerZdenek Tison(tick)

Not Needed

FLINK-37070 - Getting issue details... STATUS

 


FLIP-370: Support Balanced Tasks Scheduling


(minus)



Won't make it in Flink 2.0

FLIP-398: Improve Serialization Configuration And Usage In Flink(tick)

FLINK-37015 - Getting issue details... STATUS

FLINK-37060 - Getting issue details... STATUS

 

Dedicated serializer support for common Java collection types: PR in review.

Progress: Done

FLIP-472: Aligning timeout logic in the AdaptiveScheduler's WaitingForResources and Executing states

Zdenek Tison

(tick)

Not Needed

FLINK-37072 - Getting issue details... STATUS

 


FLINK-36470 - Getting issue details... STATUS


(tick)

Not Needed


The major work about Checkstyle upgrade was done in 1.20 (Checkstyle was bumped to 9.3), in 2.0 the only thing to be done is just version bump




Adaptive optimization for batch jobs

FLIP-468: Introducing StreamGraph-Based Job Submission

FLIP-469: Supports Adaptive Optimization of StreamGraph

(tick)

FLINK-37149 - Getting issue details... STATUS

FLINK-37073 - Getting issue details... STATUS

 


FLIP-470: Support Adaptive Broadcast Join(tick)

FLINK-37160 - Getting issue details... STATUS

 


FLIP-475: Support Adaptive Skewed Join Optimization(tick)

FLINK-37161 - Getting issue details... STATUS

 


FLINK-33163 - Getting issue details... STATUS


(tick)



 


Deprecate support for Java 11 (to-be discussed via FLIP)


(minus)



 


Upgrade Calcite to 1.33.0 FLINK-31362 - Getting issue details... STATUS

(tick)


FLINK-37074 - Getting issue details... STATUS



Upgrade Calcite to 1.34.0 FLINK-31836 - Getting issue details... STATUS

(tick)


FLINK-37075 - Getting issue details... STATUS




(tick)

FLINK-36761 - Getting issue details... STATUS

FLINK-37186 - Getting issue details... STATUS



(tick)

FLINK-37087 - Getting issue details... STATUS

FLINK-37134 - Getting issue details... STATUS


FLINK-37133 - Getting issue details... STATUS

(tick)

FLINK-37187 - Getting issue details... STATUS

FLINK-37188 - Getting issue details... STATUS



Status / Follow-ups

01/22/25

  • Status
    • Feature Developing Progress
    • Testability Blockers
      • 3 Testability Blockers(1 resolved, 1 created, 1 changes the priority to non-blocker compared to last time), 2 of which are not being looked into
  • Discussion
    • Documentation build on official website is down, testers need to build docs locally
    • Complete documentation and x-team testing instructions by Jan 31
    • Chinese New Year Holiday season: Jan 28 - Feb 4
      • Next release sync: Feb 12 ?

01/15/25

  • Status
    • Feature Developing Progress
    • Testability Blockers
      • 4 Testability Blockers, 2 of which are not being looked into(no change compared to last time)
  • Discussion
    • Feature Freeze
      • Jan 20
      • Remind the date on dev@ after release sync, and collect for potential exceptions
    • Branch Cutting
      • As soon as feature freeze
    • Documentation
      • Update wiki and create Jira (if needed) by Jan 17
      • Complete by Jan 31
    • Release Testing
      • Add instructions (or close tickets if not needed) by Jan 31
    • Release announcement: Start drafting in February
    • Chinese New Year holiday season: Jan 28 - Feb 4

01/08/25

  • Status
    • Feature Developing Progress
    • Testability Blockers
      • 4 Testability Blockers, 2 of which are not being looked into
  • Discussion
    • Feature Freeze Date
      • Extend to Jan 20?
      • Developing of the following features can benefit from 1 extra week
        • Disaggregated State Management
        • DataStream V2
        • Adaptive optimizations for batch jobs
        • Serialization Improvements
    • We'll cut the release branch immediately after feature freeze
    • Release testing
      • Process
        1. RM creates Jira tickets for all listed features and assign feature developers
        2. Feature developers add testing instructions and un-assign themselves, or close the ticket if testing is not needed
        3. RM / feature developer find other contributors for performing the tests
      • Complete instructions before end of January
    • Add documentation and x-team verification information on wiki, for both breaking changes and features
      • Documentation - developer should fill in:
        • "Not Needed", if the feature does not involve any user documentation changes
        • Jira ticket, if the documentation changes are needed but not yet completed
        • Jira ticket and documentation link, if the documentation changes are completed
      • X-team verification - release manager should create and fill in the Jira tickets
    • Release Announcement: Start drafting in Februray
    • Chinese New Year holiday season: Jan 28 - Feb 4

12/18/24

  • Status
    • Testability Blockers
      • 4 Testability Blockers(1 resolved compared to last time, 1 changes the priority to non-blocker), 3 of which are not being looked into
  • Previous Actions
  • Discussion

11/05/24

  • Status
    • 2.0-preview released
    • Features
    • Testability Blockers
      • 6 Testability Blockers(2 resolved compared to last time), 5 of which are not being looked into
  • Previous Actions
    • Start a FLIP discussion for deprecating Java 11 Martijn Visser 
    • Update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
  • Discussion
    • Feature Freeze Date

10/09/24

  • Status
    • Breaking Changes
      • 17 completed
      • 2 will be completed in release 2.0
      • 1 will not happen in release 2.0
    • No open blocker issues with FixVersion 2.0-preview
    • Testability Blockers
      • 7 Testability Blockers(1 resolved compared to last time), 4 of which are not being looked into
  • Previous Actions
    • Ping contributors of work items listed as Breaking Changes to create Jira tickets under "2.0-preview" Xintong Song 
    • Start a FLIP discussion for deprecating Java 11 Martijn Visser 
    • Update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
  • Discussion
    • Create RC1 for 2.0 Preview
      • All blockers resolved
      • Release process: PR merged
      • Release announcement: PR in review, so far all comments addressed
    • Hive 3.1 does not support Java 11 FLINK-36437 - Getting issue details... STATUS
  • Action

09/25/24

  • Status
    • 8 Testability Blockers, 4 of which are not being looked into
    • Breaking Changes
      • Most are in code review
  • Previous Actions
    • Create a Jira ticket for checking deprecated APIs before the Preview Release Jiangjie Qin 
    • Find a solution for PyFlink tests depending on external connectors that uses deprecated APIs. Xintong Song Dian Fu
    • Review PR for FLINK-36207 - Getting issue details... STATUS Xintong Song 
    • Share the list of tasks for deprecated API removal  FLINK-36306 - Getting issue details... STATUS Weijie Guo
    • Looking into the preview release process FLINK-36360 - Getting issue details... STATUS Xintong Song 
    • Start working on the release announcement (Sep. 25) Xintong Song 
    • Ping contributors of work items listed as Breaking Changes to create Jira tickets under "2.0-preview" Xintong Song 
    • Start a FLIP discussion for deprecating Java 11 Martijn Visser 
    • Update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
  • Discussion
    • Progress of dropping Java 8 supports and making Java 17 the default?
    • Preview release process FLINK-36360 - Getting issue details... STATUS
      • Tried basically everything except for publishing the artifacts and building the images
    • Connector migration plan
      • 2.0-preview: Kafka connector
      • 2.0.0: a few most commonly used connectors
      • 2.3.0 (1 year since ): 80% of first-party connectors
    • Release Announcement
      • I've started collecting contents and working on the first draft
      • Will soon open a PR, based on which everyone can proposed contents to add
    • Time Plan
      • Sep. 30 - Breaking changes due
      • Oct. 1~7 - Public holidays in China
      • Oct. 8 - Branch cutting, RC Vote, etc.

09/11/24

  • Status
  • Previous Actions
    • Revisit and remove REST API and Metrics related legacy work items Xintong Song 
    • Create a 2.0-Preview Jira version Xintong Song 
    • Start a FLIP discussion for deprecating Java 11 Martijn Visser 
    • Update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
    • Create a Jira ticket for checking deprecated APIs before the Preview Release Jiangjie Qin 
  • Discussion
    • Risks for the preview release
      • PyFlink tests depends on external connectors, which depend on deprecated APIs. Removing deprecated APIs causes CI failures. This blocks many tasks including removal of Scala APIs and deprecated config options, and replacing Time with Duration.
        • Offline reached out to Dian Fu after the release sync. According to him, the related test cases are for testing the connectors. As the connectors are moved from Flink's main repo to external repos, the tests should be moved too. He will work on it. 
        • To quickly unblock other tasks, developers can disable the problematic pyflink test cases for now. They will be re-enabled after moved to external connector repos.
    • FLINK-36207 - Getting issue details... STATUS
      • Blocked by Scala removal. Weijie Guo is currently working on it, and should finish it by this weekend.
      • We'll disable Japicmp for @Deprecated APIs until the Flink 2.0 release.
      • We'll recheck if every breaking change is expected, by turning the check back on locally before the release and check all the reported issues. FLINK-36261 - Getting issue details... STATUS
    • "2.0-preview" Jira version
      • Tickets targeting the preview release should set the fixVersion to 2.0-preview
      • Tickets resolved before the preview release should set the fixVersion to 2.0-preview
    • For breaking changes that will be postponed to Release 3.0
      • A "3.0.0" Jira version has been created. Please set the target fixVersion accordingly.
      • There's also an umbrella for release 3.0 breaking changes. It would be helpful for preparing the next major version bump if we gather all breaking changes in one place. FLINK-36200 - Getting issue details... STATUS
    • Release process for Flink 2.0 (& Preview)
      • We need to start looking into the release process right away
        • We need to enrich the release guide for major releases.
        • We need to do a dry-run for branch cutting, building, etc., except for the final deploying / publishing stage.
      • We will start working on the release announcement 1-2 weeks later, so we know better which features going to make it
  • Actions
    • Find a solution for PyFlink tests depending on external connectors that uses deprecated APIs. Xintong Song Dian Fu
    • Review PR for FLINK-36207 - Getting issue details... STATUS Xintong Song 
    • Share the list of tasks for deprecated API removal Weijie Guo 
    • Ping contributors of work items listed as Breaking Changes to create Jira tickets under "2.0-preview" Xintong Song 
    • Looking into the preview release process Xintong Song 
    • Start working on the release announcement (Sep. 25)

08/28/24

  • Status
  • Discussion
    • Work items
      • Java version supports
        • Support Java 21
        • Make Java 17 (Preview Release)
        • Drop Java 8 (Preview Release)
        • Start a FLIP discussion for deprecating Java 11. Martijn Visser 
          • We will not remove support for Java 11 directly, because it's still the default java version in 1.20 and has not been deprecated.
      • REST API and Metrics related legacy work items can be removed.
        • Will remove after revisiting and checking for APIs already deprecated and can be simply removed. Xintong Song 
      • We'll keep collecting new features.
    • Preview Release
      • Complete breaking changes in the Preview Release, by the end of September
        • For Source/SinkFunction and legacy TableSource/SinkFunctions, there are a lot internal usages (planner, 1st-party connectors, tests, etc.) that may not be migrated soonish. We will first move the public APIs to internal interfaces, and gradually remove them later.
        • We should update the deprecation process, to require migrating of internal usages before the deprecation. Xintong Song 
      • Create a 2.0-Preview Jira version Xintong Song 
      • Create a Jira ticket for checking deprecated APIs before the Preview Release. Jiangjie Qin 
        • Add scripts for gathering all deprecated APIs
        • Make sure all deprecated APIs are either removed, or we understand why they are not removed.
        • We can leverage the scripts to generated API differences between versions (e.g., 2.0-Preview vs. 2.0.0)

08/14/24

  • Status
  • Discussion
    • Preview Release
      • Try to include all API breaking changes
      • Won't do
        • Feature freeze
        • Feature testing
        • Resolve all blockers
      • Will do
        • PMC vote (e.g., legal checks)
        • Announce and promote
      • Maybe define a new version "2.0-Preview" on Jira and mark all API-breaking changes are blocker for it?
    • Time Plan
      • Preview release: End of Sep.
      • Feature freeze: End of Nov.
      • Formal release: End of Dec. or early Jan.
    • Work Items
      • API-breaking: targeting to finish by the preview release
      • Non-breaking: targeting to finish by the feature freeze date, following time-based rules
  • Follow-ups 
    • Update work item list: separate into two lists, and remove outdated items
    • Create the "2.0-Preview" release and move relevant tickets to it?
    • Start collecting feature plans
  • No labels