Table of contents

Jira Boards

Flink 1.18 Burndown: 1.18 Burndown - Agile Board - ASF JIRA (apache.org)

Sync meeting

  • The release sync will start from April 4th, 2023, at 10am (UTC+2) and 4pm (UTC+8). The release sync happens bi-weekly at first, and will be adjusted to weekly as we approaching the feature freeze date. 
  • Feel free to join on Zoom meeting (with passcode eSj55R if required). iCalendar file can be downloaded via this link. 
  • Due to the limit of Zoom, a meeting session is limited to 40 mins. We set up another Zoom meeting (with passcode jhS5w7 if required) as a backup if we cannot finish discussions within 40 mins in the first meeting session. 

Timeline

  • Feature Freeze
  • Release
    • End of September 2023


Highlight features

please feel free to add/suggest.

Features

List of features announced by contributors and committers that are likely to be ready for the feature freeze:n

NOTICE: It's preferred if only new features end up there and not all bugs/tasks separately, so that the page is not over bloated. Of course, unless fixing a bug is a really big or important one equivalent to implementing a completely new feature. A good rule of thumb would be that each entry in the page could (but does not have to) be later on included in a release blog post.

Legend

State

symbolmeaningcomment
(big grin)validatedthrough cross team testing
(tick)donewell documented with a complete test coverage
(green star)will make itthere is no reason this effort should not go into 1.18
(star)in dangerthere are some concerns the effort could be ready for the feature freeze of 1.18
(red star)very unlikelythere are severe concerns the effort could make it to 1.18
(minus)won't make itit was decided against adding this for the 1.18 release. working on the effort has been stopped.
(question)state unclear
(blue star)independentas the artifact could be released independent of Apache Flink

X-Team verification

symbolmeaning
(tick)done
(blue star)not required

Feature Stage

Please align with the list on the Apache Flink Roadmap (https://flink.apache.org/roadmap.html).

  • MVP: Have a look, consider whether this can help you in the future.
  • Beta: You can benefit from this, but you should carefully evaluate the feature.
  • Ready and Evolving: Ready to use in production, but be aware you may need to make some adjustments to your application and setup in the future, when you upgrade Flink.
  • Stable: Unrestricted use in production
  • Reaching End-of-Life: Stable, still feel free to use, but think about alternatives. Not a good match for new long-lived projects.
  • Deprecated: Start looking for alternatives now 

Summary

Numbers are based on the items in the list below, not on the tickets 


(big grin)(tick) (green star)(star)(red star)(minus)(question)(blue star)∑Remaining weeks
05/02/2023

11




3211
05/16/2023

11




328
06/13/2023
3237
221354
06/27/2023
3225
4
1352
07/25/2023
18141
111
450

List

Feel free to add categories.

Runtime
Name/JIRA IssueResponsible ContributorReviewer / committer availableFeature StageNoteState%UpdatedImplementedDocumentedX-team verified
Shuffle









FLINK-31634 - Getting issue details... STATUS

Yuxin Tan 

Wencong Liu 

Xintong Song 


done(tick) 100%24-07-2023

(tick) FLINK-32675 - Getting issue details... STATUS

see https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/#hybrid-shuffle


Quantify metrics of shuffle(i.e. network) memory

Weijie Guo 

Xintong Song 



(minus)
13-06-2023


Coordination









FLINK-32486 - Getting issue details... STATUS

Lijie Wang 

Zhu Zhu 



(green star)100%17-07-2023

(tick) No documentation needed


Supports deciding the parallelism of sources according to runtime information (e.g. DPP result)

Lijie Wang 

Zhu Zhu 



(minus)
19-06-2023


Supports JobMaster status recovery after JobMaster failover

Lijie Wang 

Zhu Zhu 



(minus)
19-06-2023


FLINK-32468 - Getting issue details... STATUS

Chesnay Schepler 

Matthias Pohl 


  • Pekko 1.0.0 RC2 passed our tests successfully
  • Ok to merge after the feature freeze in the next days
(tick) 100%26-07-2023
(tick) No documentation needed

FLINK-32678 - Getting issue details... STATUS

State & Checkpoint









FLINK-31238 - Getting issue details... STATUS

yue ma 

Yun Tang / ？



(minus)
17-07-2023


FLIP-306: Unified File Merging Mechanism for Checkpoints

FLINK-32070 - Getting issue details... STATUS

Zakelly Lan 

@yinhan

Yanfei Lei 

Yuan Mei 



(minus)
25-07-2023


FLIP-263: schema compatibility

FLINK-29844 - Getting issue details... STATUS

Hangxiang Yu 

Yuan Mei 



(minus)80%25-07-2023


FLINK-30235 - Getting issue details... STATUS

@Rui xia

Yuan Mei 



(minus)
13-06-2023


FLINK-26372 - Getting issue details... STATUS

Yanfei Lei 

Yuan Mei 



(minus) 
30-05-2023


FLINK-24402 - Getting issue details... STATUS

Yanfei Lei 

Yuan Mei 



(tick) 
25-07-2023
(tick) Documented as Metric

FLINK-32326 - Getting issue details... STATUS

Stefan Richter 

 Stefan Richter 



(tick) 
25-07-2023
(tick) No documentation needed

FLINK-32345 - Getting issue details... STATUS

Stefan Richter 

 Stefan Richter 



(tick) 
25-07-2023
(tick) No documentation needed

REST











Support CPU profiler on taskmanager

Yu Chen 

Yun Tang 



(minus)
13-06-2023


SQL
Name/JIRA IssueResponsible ContributorReviewer / committer availableFeature StageNoteState%UpdatedImplementedDocumentedX-team verified
Calcite Update

FLINK-27998 - Getting issue details... STATUS

Sergey Nuyanzin 

Timo Walther 
done(tick) 100%25-07-2023
(tick) No documentation needed

FLINK-28744 - Getting issue details... STATUS

Sergey Nuyanzin 

Timo Walther 


done(tick) 100%25-07-2023
(tick) No documentation needed

FLINK-29319 - Getting issue details... STATUS

Sergey Nuyanzin 

Timo Walther 


done(tick) 100%25-07-2023
(tick) No documentation needed

FLINK-31833 - Getting issue details... STATUS

ron liu 

Godfrey He 

(question)50%25-07-2023
(question) Unclear if it needed. I asked on the ticket.

FLINK-24909 - Getting issue details... STATUS

Sergey Nuyanzin 

Shengkai Fang 


done(tick) 100%25-07-2023
(tick) No documentation needed

FLIP-300 FLINK-31487 - Getting issue details... STATUS

lincoln lee 



done(tick) 100%25-07-2023

(tick) FLINK-32674 - Getting issue details... STATUS


FLIP-292: Enhance COMPILED PLAN to support operator-level state TTL configuration

FLINK-31791 - Getting issue details... STATUS

Jane Chan 



done(tick) 100%25-07-2023

(tick) FLINK-31957 - Getting issue details... STATUS


FLIP-302: Support TRUNCATE TABLE statement in batch mode

luoyuxia 



done, docs missing

(green star)

90%01-08-2023

(green star) FLINK-32287 - Getting issue details... STATUS


FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

Mang Zhang 

luoyuxia 
done, docs missing

(green star)

90%01-08-2023

(green star)  FLINK-32581 - Getting issue details... STATUS


FLIP-303: Support REPLACE TABLE AS SELECT statement

luoyuxia 



done(tick) 100%01-08-2023

(tick) See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-or-replace-table


FLIP-311: Support Call Stored Procedure

luoyuxia 



done, docs missing

(green star)

90%01-08-2023

(green star) FLINK-32356 - Getting issue details... STATUS

(tick) See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/show/#show-procedures


FLIP-308: Support Time Travel

Feng Jin 

luoyuxia 
done, docs missing

(green star)

90%01-08-2023

(green star) FLINK-32475 - Getting issue details... STATUS


FLINK-27237 - Getting issue details... STATUS

ron liuluoyuxia 
done(tick)100%25-07-2023


(tick) See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#add & https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#drop

&https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/show/#show-partitions


FLIP-296 FLINK-31535 - Getting issue details... STATUS

Kui Yuan 

Yun Tang 
done(tick)100%25-07-2023
(tick) See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/time_attributes/#advanced-watermark-features

FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

Fang Yong 

Benchao Li 
done(tick)100%25-07-2023
(tick) https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/

FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

FLINK-30025 - Getting issue details... STATUS

Jing Ge 

Jark Wu 
done(tick)100%25-07-2023
(tick) No documentation needed
Connectors









FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

Feng Jin Hang Ruan

 Leonard Xu 


done, some follow ups next release, docs missing

(green star)

20%25-07-2023

(star) FLINK-32653 - Getting issue details... STATUS


FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source

Hongshun Wang 

Leonard Xu 


done(tick)100%25-07-2023
(tick) https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/ & https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/

FLIP-294: Support Customized Catalog Modification Listener

Fang Yong 

Leonard Xu 
done

(green star)

100%25-07-2023

https://issues.apache.org/jira/browse/FLINK-32676
(star) FLINK-32676 - Getting issue details... STATUS


FLIP-314: Support Customized Job Lineage Listener

Fang Yong 

Leonard Xu 


under discussion

(minus)

10%25-07-2023


FLINK-26603 - Getting issue details... STATUS

luoyuxia 

lincoln lee 
done(tick)100%25-07-2023
(tick) No documentation needed

FLINK-32548 - Getting issue details... STATUS

Cai Liuyang 

Rui Fan 

Piotr Nowojski 

Rui Fan 


done(green star)100%25-07-2023

(star) FLINK-32705 - Getting issue details... STATUS  


FLINK-32524 - Getting issue details... STATUS

Rui Fan 

Piotr Nowojski Stefan Richter 


done

(tick)100%25-07-2023
see row above
Resource Management









FLINK-31316 - Getting issue details... STATUS

David Morávek 

David Morávek 


done, documentation missing(green star)80%25-07-2023

(star) FLINK-32671 - Getting issue details... STATUS


FLINK-32165 - Getting issue details... STATUS

Chesnay Schepler 

Chesnay Schepler 



(green star)90%25-07-2023
(tick) No documentation needed

FLINK-29344 - Getting issue details... STATUS

Chesnay Schepler 

Chesnay Schepler 



(minus)60%25-07-2023


Runtime

FLINK-31508 - Getting issue details... STATUS

Panagiotis Garefalakis 

Piotr Nowojski 


done, some follow ups next release, docs missing(green star)90%25-07-2023

(star) FLINK-31889 - Getting issue details... STATUS


FLINK-26522 - Getting issue details... STATUS /FLIP-285: Leader Election refactoring

Matthias Pohl 

Chesnay Schepler 



(tick)100%25-07-2023
(tick) No documentation needed

FLINK-32678 - Getting issue details... STATUS

Misc











FLINK-15736 - Getting issue details... STATUS

Chesnay Schepler 

Chesnay Schepler 



(green star)100%25-07-2023
(question) Checking with the contributor

FLINK-32032 - Getting issue details... STATUS

Martijn Visser 

Matthias Pohl 



(tick)100%25-07-2023
(tick) No documentation needed

FLINK-32678 - Getting issue details... STATUS

Status / Follow-ups

04/04/2023

  • Is Zoom OK for everyone? 
    • Looks good for now, except the 40 min limit. One possible solution is to create two meetings as a backup. Usually release syncs can finish within 40 mins.
  • Retrospective of 1.17
    • Has been summarized in 1.17 Release, at the end of the page.
  • Daily work divisions
    • JIRA ticket (Mainly focus on new blockers, progress of existing blockers each week) (Qingsheng Ren )
    • CI (Failures reported in #build channel and not reported) (Sergey Nuyanzin )
    • Benchmark (Check if there's regression in the #flink-dev-benchmarks channel) Konstantin Knauf 
    • Mailing list monitoring (Jing Ge )
    • This division doesn't mean that other RMs are now allowed to do the work. All RMs should take responsibilities on all of these daily issues if they are available. 
  • Feature freezing date is set to July 11, 2023. 

04/18/2023

  • CI instabilities
  • Version management
    • Qingsheng Ren will remove 1.15 related resources within this week. 

05/30/2023

  • Remind in the mailing list about updating the release wiki page.

06/13/2023

06/27/2023

07/11/2023

  • Feature freeze date is extended until July 24, 2023, which is a Monday
  • Blocker / Critical issues

07/25/2023

  • Deprecate APIs in 1.18
    • Purely adding annotations / JavaDocs: go directly because they are not new features
    • Works requiring changes to code base
      • Create a thread in ML for receiving applications including: 
        • FLIP ID and current status (in discussion / vote passed / developing)
        • Why this deprecation must be done in 1.18
      • All 2.0 unrelated requests will be denied
      • To be discussed: deadline of these deprecation works. 1.17 release testing lasted 3 weeks, so what about 2 weeks for feature freeze of these deprecations?
      • Get 2.0 RMs involved in evaluating cases
    • Decided not to give more extension, as we couldn't see a lot of benefits to 2.0 giving another 2 weeks
  • Cross-team testing
    • Goes in parallel with additional deprecation works above
    • will go normally, for about 3 weeks
    • Add a reminder about documentation before x-team test
  • Cutting branch will be delayed until at least 2 weeks later we will target at the next Tuesday. 
  • Blocker / Critical issues

08/01/2023

  • ical event doesn't work anymore
  • Requests to merge into master :
    • FLINK-32098 - Getting issue details... STATUS
    • FLINK-28045 - Getting issue details... STATUS  Deprecating SourceFunction API
  • Blockers/Critical (last 7 days)
    • FLINK-32686 - Getting issue details... STATUS Performance regression, PR has been approved and waiting for CI
    • FLINK-32685 - Getting issue details... STATUS Performance regression, Weijie Guo is looking into it
    • FLINK-32557 - Getting issue details... STATUS 1.18 API deprecations, maybe downgrade?
    • FLINK-32678 - Getting issue details... STATUS Release testing issue
    • FLINK-32223 - Getting issue details... STATUS Commit merged to master after feature freeze
    • FLINK-32662 - Getting issue details... STATUS  
    • FLINK-31168 - Getting issue details... STATUS : PR provided (test-code issue)
    • FLINK-32715 - Getting issue details... STATUS : PR provided (test-code issue)
    • FLINK-32663 - Getting issue details... STATUS  
    • FLINK-32144 - Getting issue details... STATUS : Test-code issue. Lijie will try to fix it
    • FLINK-32681 - Getting issue details... STATUS : Looks like a test instability caused by the underlying OS (Stefan Richter is looking into it)

08/08/2023

  • Blockers / Critical issues (last 7 days)
    • FLINK-32768 - Getting issue details... STATUS blocker, Junrui is working on it
    • FLINK-32730 - Getting issue details... STATUS blocker, Dalong is working on it
    • FLINK-31268 - Getting issue details... STATUS blocker, PR is ready and Chesnay is reviewing it.
    • FLINK-32751 - Getting issue details... STATUS critical, looks like an old one
  • Benchmark: TPC-DS result? Qingsheng Ren 
  • Documentation: Remind contributors to finish docs
  • Release announcement
  • Cutting the branch: we'll wait for the status of blockers


08/15/2023

  • Blockers / Critical issues (last 7 days)
    • FLINK-32730 - Getting issue details... STATUS blocker, Dalong is working on it, waiting for Lincoln Lee to review
    • FLINK-31268 - Getting issue details... STATUS blocker, PR is ready and Chesnay is reviewing it.
  • Benchmark: TPC-DS result? Qingsheng Ren 
  • Cutting the branch: we'll wait for the status of blockers

08/22/2023

  • Blocker / Critical issues (last 7 days)
  • Cutting 1.18 branch
    • The branch will be cut today in order to unblock the master branch Jing Ge 
  • Release testing
    • Issues are mistakenly assigned to the original author of new features. Qingsheng Ren will correct them. The original author should provide instruction / docs to help volunteers to do the verification
    • We target to finish all tests in 2 weeks. We'll make sure that all tickets are assigned on next Tuesday, and close them all before the next Tuesday after. 
  • Release Announcement
    • All contributors with the link can edit / comment now
  • Release Managing
    • Sergey Nuyanzin will clone the umbrella issue we used in 1.17 for tracking task assignees and the entire progress of promoting the release.

08/29/2023

  • Release Testing
    • 2 unassigned issues: 
    • Progress 50%, with 15 open / 30 total
    • Try to close all before next Tuesday
  • Blocker / Critical issues (last 7 days)
  • CI status
    • release-1.18 cron build?

09/05/2023

  • Release Testing: FLINK-32726 - Getting issue details... STATUS
  • Blocker / Critical issues (last 7 days)
  • CI status

last ci with nightly (20230903.4) is green https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=52961&view=results

09/12/2023

  • Release Testing: FLINK-32726 - Getting issue details... STATUS
  • Blocker / Critical issues (last 7 days)
  • CI status
    • last green was on 3rd of September, after that blocked because of blocker above


09/19/2023

  • No labels