Table of contents

Jira Boards

Flink 1.20 Burndown: 1.20 Burndown - Agile Board - ASF JIRA (apache.org)

Sync meeting

• The release sync will start from April 9, 2024, at 9.30am (UTC+2) and 3.30pm (UTC+8). The release sync happens bi-weekly at first, and adjusted to weekly from June 4, 2024, as we approaching the feature freeze date. 
• Feel free to join on Google Meet, and attendees are encouraged to fill out the topics to be discussed at the bottom of this page a day in advance to make it easier for everyone to understand the background of the topic. 

Timeline

• Feature Freeze
  
  • June 15, 2024, 00:00 CEST(UTC+2) 
• Release
  • End of July 2024

Highlight features

please feel free to add/suggest.

Features

List of features announced by contributors and committers that are likely to be ready for the feature freeze:n

NOTICE: It's preferred if only new features end up there and not all bugs/tasks separately, so that the page is not over bloated. Of course, unless fixing a bug is a really big or important one equivalent to implementing a completely new feature. A good rule of thumb would be that each entry in the page could (but does not have to) be later on included in a release blog post.

Legend

State

symbol	meaning	comment
	validated	through cross team testing
	done	well documented with a complete test coverage
	will make it	there is no reason this effort should not go into 1.20
	in danger	there are some concerns the effort could be ready for the feature freeze of 1.20
	very unlikely	there are severe concerns the effort could make it to 1.20
	won't make it	it was decided against adding this for the 1.20 release. working on the effort has been stopped.
	state unclear
	independent	as the artifact could be released independent of Apache Flink

X-Team verification

symbol	meaning
	done
	not required

Feature Stage

Please align with the list on the Apache Flink Roadmap (https://flink.apache.org/roadmap.html).

• MVP: Have a look, consider whether this can help you in the future.
• Beta: You can benefit from this, but you should carefully evaluate the feature.
• Ready and Evolving: Ready to use in production, but be aware you may need to make some adjustments to your application and setup in the future, when you upgrade Flink.
• Stable: Unrestricted use in production
• Reaching End-of-Life: Stable, still feel free to use, but think about alternatives. Not a good match for new long-lived projects.
• Deprecated: Start looking for alternatives now 

Summary

Numbers are based on the items in the list below, not on the tickets 

									∑	Remaining weeks
2024-04-23			5	1					6	8
2024-05-07			6	2					8	6
2024-05-21		1	7	2					10	4
2024-06-04		3	8		2				13	2
2024-06-11		6	8			2			16	1
2024-06-18		13				3			16	0

List

Feel free to add categories.

Runtime
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
Shuffle
FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn FLINK-35533 - Getting issue details... STATUS	Yuxin Tan	Xintong Song  Weijie Guo					2024-06-12
Coordination
FLIP-383: Support Job Recovery from JobMaster Failures for Batch Jobs FLINK-33892 - Getting issue details... STATUS	Junrui Lee	Zhu Zhu					2024-05-31
State & Checkpoint
FLIP-306: Unified File Merging Mechanism for Checkpoints FLINK-32070 - Getting issue details... STATUS	Jinzhong Li  Yanfei Lei  Hangxiang Yu	Yanfei Lei Hangxiang Yu  Zakelly Lan					2024-06-18
FLINK-31238 - Getting issue details... STATUS	yue ma  Stefan Richter  Roman Khachatryan	Piotr Nowojski  Zakelly Lan  Yun Tang					2024-06-04
FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0	yue ma  Yanfei Lei	Yun Tang  Zakelly Lan  Roman Khachatryan					2024-06-04
FLINK-26050 - Getting issue details... STATUS	Roman Khachatryan	Stefan Richter				99%	2024-06-18		To be updated
REST
Web UI
FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI FLINK-29481 - Getting issue details... STATUS	Rui Fan	RocMarshal					2024-05-16
SQL
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
FLIP-436: Introduce Catalog-related Syntax FLINK-34914 - Getting issue details... STATUS	Yubin Li	Jane Chan					2024-06-18
FLIP-376: Add DISTRIBUTED BY clause FLINK-33494 - Getting issue details... STATUS	Jim Hughes	Timo Walther					2024-06-18
FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines FLINK-35187 - Getting issue details... STATUS	ron liu@Feng Jin	ron liu		MVP for 1.20 completed, docs are preparing		95%	2024-06-11
FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table FLINK-35345 - Getting issue details... STATUS	ron liu @Feng Jin	ron liu				100%	2024-06-11
Connectors
FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction FLINK-35378 - Getting issue details... STATUS	Martijn Visser	Sergey Nuyanzin				100%	2024-06-13
FLIP-445: Support dynamic parallelism inference for HiveSource FLINK-35293 - Getting issue details... STATUS	Xia Sun	Zhu Zhu					2024-05-22
FLIP-451: Introduce timeout configuration to AsyncSink API FLINK-35435 - Getting issue details... STATUS	Ahmed Hamdy	Danny Cranmer					2024-05-26
Resource Management
Runtime
FLIP-398: Improve Serialization Configuration And Usage In Flink Add built-in Map/List/Set serializers FLINK-34037 - Getting issue details... STATUS	Zhanghao Chen	Weijie Guo Fang Yong					2024-06-12
FLIP-443: Interruptible timers firing FLINK-20217 - Getting issue details... STATUS	Piotr Nowojski	Roman Khachatryan					2024-06-14

Status / Follow-ups

04/09/24

• Feature freeze date – June 15 all set?
  • Discussed on ML and we are all aligned on the date 👍🏼
  • Robert Metzger feature list looks pretty light currently
  • TODOUfuk Celebi to ping Confluent teams to add features to Wiki
• Flink 1.19 retro
  • TODO Ufuk Celebi to reach out to release managers of Flink 1.19 and ask for feedback
  • TODO Robert Metzger to ask about mailing list monitoring
• Daily work divisions
  • JIRA ticket (Mainly focus on new blockers, progress of existing blockers each week)
    • Ufuk Celebi 
  • CI (Failures reported in #build channel and not reported)
    • Weijie Guo 
  • Benchmark (Check if there's regression in the #flink-dev-benchmarks channel)
    • Rui Fan
  • Mailing list monitoring
    • Robert Metzger 
• Blockers of 1.20
  • FLINK-35040 - Getting issue details... STATUS
    • It seems Java 11 regresses, and Java 8 and Java 17 are fine.
    • Robert Metzger Any relevant change recently? 
    • Rui Fan applied for a Jenkins account to check for a JVM version upgrade.
  • FLINK-35009 - Getting issue details... STATUS
    • Since Transformation  is an Internal API indeed, and this only breaks the kafka-connector test code. I tend to this should not be a blocker at all.
  • FLINK-34997 - Getting issue details... STATUS
    • This one is an unstable test case, seems to be that docker-compose binaries are sometimes missing in the Azure VMs. 

04/23/24

• Blockers of 1.20
  •   FLINK-35040 - Getting issue details... STATUS
    • It seems the performance regresses after FLINK-34955 - Getting issue details... STATUS is merged.
    • The performance will be recovered after reverting common-ios version.
    • 2 concerns:
      • I didn't find any common-ios code cost the time during run the benchmark on my Local. (async-profiler wall mode)
      • FLINK-34955 - Getting issue details... STATUS seems necessary and fixes some bugs.
    • TODO: Try generate the flame graph on benchmark server Rui Fan 
  • FLINK-35041 - Getting issue details... STATUS
    • I can't reproduce this by running IncrementalRemoteKeyedStateHandleTest itself, but I can reliably reproduce it (within about a minute) by running the entire org.apache.flink.runtime.state package in the mode "Repeat until stopped". (Comments from Ryan)
    • I have reached out Feifan Wang  to take a look.
  • FLINK-35009 - Getting issue details... STATUS
    • Can anyone reach out Martijn Visser and ask him about this? I'm inclined to think we can downgrade it.
  • FLINK-34997 - Getting issue details... STATUS
    • Robert Metzger Any thoughts/updates about this?
  • FLINK-35215 - Getting issue details... STATUS
    • It seems FLINK-34954 - Getting issue details... STATUS causes it.

05/07/24

• Blockers of 1.20
  •   FLINK-35040 - Getting issue details... STATUS
    
    • TODO: Try generate the flame graph on benchmark server Rui Fan 
  • FLINK-35041 - Getting issue details... STATUS
    • I can't reproduce this by running IncrementalRemoteKeyedStateHandleTest itself, but I can reliably reproduce it (within about a minute) by running the entire org.apache.flink.runtime.state package in the mode "Repeat until stopped". (Comments from Ryan)
    • I have reached out Feifan Wang  to take a look.
    • Rui Fan reach out Feifan Wang 
  • FLINK-35215 - Getting issue details... STATUS
    • It seems FLINK-34954 - Getting issue details... STATUS causes it.
    • https://github.com/apache/flink/pull/24717/files is an alternative change that does not have a performance regression, but we need someone to review
      • Ufuk Celebi to ping runtime contributors for feedback
• Rui Fan to send a reminder about:
  • The feature freeze date in 6 weeks
  • Check whether the Public or Public Evolving api related to flink 2.0 deprecation has been done

05/21/24

• Blockers of 1.20
  •   FLINK-35040 - Getting issue details... STATUS
    
    • Generated the flame graph on benchmark server Zakelly Lan  Rui Fan 
    • But we didn't see any common-ios package code is called. And it doesn't influence other dependency.
    • Update it to dev mail list to collect some valuable suggestion.
    • We will accept this regression if we don't have any suggestion.
      • It only happens on benchmark server with jdk11
  • FLINK-35041 - Getting issue details... STATUS
    • it's solved.
  • FLINK-35215 - Getting issue details... STATUS
    • It seems FLINK-34954 - Getting issue details... STATUS causes it.
    • https://github.com/apache/flink/pull/24717/files is an alternative change that does not have a performance regression, but we need someone to review
      • Ufuk Celebi to ping runtime contributors for feedback

06/04/24

• Blockers of 1.20
  • closed since the last sync: 
    • FLINK-35040 - Getting issue details... STATUS
      
      • We accepted this regression
    • FLINK-35215 - Getting issue details... STATUS
      • The performance has been recovered.
  • unclosed:
    • FLINK-35423 - Getting issue details... STATUS
      • Have reached out Sergey for updating the progress.
• The development progress
  • Only left 2 weeks for feature freeze, do we need to remind developers in dev SUMMARY mail list?
  • And ask developers whether we need to extend it because of some awesome feature?
  • Update the percentage of progress.
• CI pipeline triggered by pull request seems sort of unstable. https://github.com/apache/flink/pull/24883
• According to the last discussion, this call will be scheduled once a week.

06/11/24

• No blockers of 1.20
• June 15 is the planned feature freeze day, total 16, done 8, will make it 6, very unlikely 2;
  • There are only three days left to feature freeze
  • We need to check all the features one by one
  • Highlight in the SUMMARY mail that it is very close to the feature freeze date.
  • Whether to delay feature freeze day
• When to cut release-1.20 branch? 
  • 1～2 weeks after feature freeze.
• Start cross team release testing after feature freeze Weijie Guo 

06/18/24

• Blockers:
  • FLINK-35629 - Getting issue details... STATUS
    • It just happened in the past few days, not sure it's caused by flink side or benchmark side?
  • FLINK-35587 - Getting issue details... STATUS
    • 10 TB TPC-DS blocking shuffle sometimes raise due to unexpected credit. Maybe we should upgrade it to blocker.
• Feature Freeze
  • We have 12 completed feature before feature freeze, and 3 features were moved to the next release.
  • Zakelly Lan want to merge  FLINK-32086 - Getting issue details... STATUS and replied in the feature freeze thread. Any objections?
  • Matthias Pohl wants to merge non-crucial (but rather cosmetic) FLINK-35539 - Getting issue details... STATUS proposed by zhen wang . Any objections?
• Release Testing
  • Weijie Guo and Rui Fan created all cross-team testing instructions issue: FLINK-35602 - Getting issue details... STATUS
    • We should remind feature contributors to complete the documentation and testing instruction.
  • We'll start release testing after release-1.20 branch cut.
• Cut Branch
  • Cut 1.20 branch this Friday?
  • If we have unresolved blockers, should it be delayed?
• Release Note
  • We should check all related jira/ticket to see if it need a release note. In particular, the deprecation stuff for 2.0.
  • Would anyone like to help with this? → Ufuk Celebi  to take a look
• Release Sync Schedule
  • We decide to change the schedule of release sync to 9.30am (UTC+2) and 3.30pm (UTC+8).

06/25/24

• Blockers
  • Congrats, no known blocker for now.
• Cut Branch
  • release-1.20 branch has been cut, and also updated the version of master branch to 2.0-SNAPSHOT(confirmed with all RMs of 2.0)
  • Two PR in flink-docker repo to be reviewed:
    • https://github.com/apache/flink-docker/pull/198
    • https://github.com/apache/flink-docker/pull/199
  • We should also update AZure CI to cover new release-1.20 branch.
    • It seems this matter can only be handled by Ververica employees since they are owning the AzureCI setup. Reached out Jing for helping.
• Cross-team release testing FLINK-34285 - [Umbrella] Test Flink Release 1.20  
  • total 13 flip/features, 8 confirmed,  5 wait for response
  • testing instructions ready: 8  (1 assigned)
• RC0
  • Any suggestions for the time, maybe this Friday or next Monday?

07/02/24

• Blockers
  • Congrats, no known blocker for now.
• Release Candidate #0
  • It was announced last Friday and receive no feedback.
• Documentation
  • We should check all documentation for the features listed in this wiki page and mark it as `Documented`.
• Cross-team release testing FLINK-34285 - [Umbrella] Test Flink Release 1.20 Rui Fan  
  • total 13 flip/features, all of them are confirmed
    
  • 10 testing instructions ready, 8 assigned, 2 wait for volunteer
    
    • FLINK-35697 - Getting issue details... STATUS
    • FLINK-35738 - Getting issue details... STATUS
  • Weijie Guo  plans to prepare RC1 the Monday after next, so it is better to finish all the testing work by the end of next week.
• Release Note Ufuk Celebi 
  • We should go through all features, because some feature don't have FLIP.
  • We'd better complete all the release notes by the end of this week.
• RC1
  • Weijie Guo  plan to vote on RC1 the Monday after next

07/09/24

• Blockers
  • FLINK-35784 - Getting issue details... STATUS Founded via release testing(good catch).
    • Zakelly has filed a PR to fix this, it will be merged after CI green.
• Release Candidate #0
  • There are no known issues for now.
  • Thanks to the people from Confluent for helping with the validation!
• Documentation
  • Weijie Guo has checked all documentation for the features listed in this wiki page and updated the status.
• Cross-team release testing FLINK-34285 - [Umbrella] Test Flink Release 1.20 Rui Fan  
  
  • We only left two testing to be done:
    
    • FLINK-35624 - Getting issue details... STATUS
    • FLINK-35738 - Getting issue details... STATUS
  • We expect finish all the testing work by the end of this week.
• Release Note Weijie Guo Rui  
  • We have gone through all features, because some feature don't have FLIP.
  • All release notes have been added now and Weijie Guo will draft a release note documentation(i.e. flink-1.20.md) for the 1.20 release.
• RC1
  • Weijie Guo plan to vote on RC1 next Monday if we don't have any new blocker.

07/16/24

• Blockers
  • Congrats, no known blocker for now.
  • Known issue: Javadocs are broken. Working on it: FLINK-35154 - Getting issue details... STATUS
• Cross-team release testing
  • All done!
• RC1
  • Weijie Guo has started the voting on RC1 in the dev mail list.
  • Encourage PMCs to test and vote.
• Release
  • Once this candidate is accepted, Weijie Guo  will do the rest of the work as soon as possible. If necessary, we can talk about the subsequent work divisions in the next meeting.

07/23/24

• Blockers
  • Congrats, no known blocker for now.
  • FLINK-35872 - Getting issue details... STATUS is solved
  • Known issue:
    • Javadocs are broken. Working on it: FLINK-35154 - Getting issue details... STATUS
      • We have updated the `maven-javadoc-plugin` to 3.7.0 and Samrat verified it locally.
      • But the Documentation Build CI Pipeline is broken:    FLINK-35863 - Getting issue details... STATUS (It's not clear that this has anything to do with plugin updating)
• RC1 is canceled : FLINK-35872 - Getting issue details... STATUS  
• Create RC2 recently Weijie Guo 
• Release
  • Once this candidate is accepted, Weijie Guo  will do the rest of the work as soon as possible. If necessary, we can talk about the subsequent work divisions in the next meeting.

07/30/24

• Blockers
  • FLINK-35925 - Getting issue details... STATUS requests merge in 1.20 in rc2 mail list
  • Known issue:
    • Javadocs are broken. Working on it: FLINK-35154 - Getting issue details... STATUS
      • We have updated the `maven-javadoc-plugin` to 3.7.0 and Samrat verified it locally.
      • But the Documentation Build CI Pipeline is broken:    FLINK-35863 - Getting issue details... STATUS (It's not clear that this has anything to do with plugin updating)
• Voting on RC2: Involve more PMC members in validating RC2
• Release
  • Once this candidate is accepted, Weijie Guo  will do the rest of the work as soon as possible.
  • FLINK-35675 - Getting issue details... STATUS