Table of contents

Jira Boards

Flink 1.18 Burndown: 1.18 Burndown - Agile Board - ASF JIRA (apache.org)

Sync meeting

• The release sync will start from April 4th, 2023, at 10am (UTC+2) and 4pm (UTC+8). The release sync happens bi-weekly at first, and will be adjusted to weekly as we approaching the feature freeze date. 
• Feel free to join on Zoom meeting (with passcode eSj55R if required). iCalendar file can be downloaded via this link. 
• Due to the limit of Zoom, a meeting session is limited to 40 mins. We set up another Zoom meeting (with passcode jhS5w7 if required) as a backup if we cannot finish discussions within 40 mins in the first meeting session. 

Timeline

• Feature Freeze
  • July 11, 2023, end of business CEST
  • July 24, 2023, 00:00 CEST(UTC+2) (announced in dev ML)
• Release
  • End of September 2023

Highlight features

please feel free to add/suggest.

Features

List of features announced by contributors and committers that are likely to be ready for the feature freeze:n

NOTICE: It's preferred if only new features end up there and not all bugs/tasks separately, so that the page is not over bloated. Of course, unless fixing a bug is a really big or important one equivalent to implementing a completely new feature. A good rule of thumb would be that each entry in the page could (but does not have to) be later on included in a release blog post.

Legend

State

symbol	meaning	comment
	validated	through cross team testing
	done	well documented with a complete test coverage
	will make it	there is no reason this effort should not go into 1.18
	in danger	there are some concerns the effort could be ready for the feature freeze of 1.18
	very unlikely	there are severe concerns the effort could make it to 1.18
	won't make it	it was decided against adding this for the 1.18 release. working on the effort has been stopped.
	state unclear
	independent	as the artifact could be released independent of Apache Flink

X-Team verification

symbol	meaning
	done
	not required

Feature Stage

Please align with the list on the Apache Flink Roadmap (https://flink.apache.org/roadmap.html).

• MVP: Have a look, consider whether this can help you in the future.
• Beta: You can benefit from this, but you should carefully evaluate the feature.
• Ready and Evolving: Ready to use in production, but be aware you may need to make some adjustments to your application and setup in the future, when you upgrade Flink.
• Stable: Unrestricted use in production
• Reaching End-of-Life: Stable, still feel free to use, but think about alternatives. Not a good match for new long-lived projects.
• Deprecated: Start looking for alternatives now 

Summary

Numbers are based on the items in the list below, not on the tickets 

									∑	Remaining weeks
05/02/2023			11						32	11
05/16/2023			11						32	8
06/13/2023		3	23	7		2	2	1	35	4
06/27/2023		3	22	5		4		1	35	2
07/25/2023		18	14	1		11	1		45	0

List

Feel free to add categories.

Runtime
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
Shuffle
	Yuxin Tan  Wencong Liu	Xintong Song		done		100%	24-07-2023		see https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/#hybrid-shuffle
Quantify metrics of shuffle(i.e. network) memory	Weijie Guo	Xintong Song					13-06-2023
Coordination
	Lijie Wang	Zhu Zhu				100%	17-07-2023		No documentation needed
Supports deciding the parallelism of sources according to runtime information (e.g. DPP result)	Lijie Wang	Zhu Zhu					19-06-2023
Supports JobMaster status recovery after JobMaster failover	Lijie Wang	Zhu Zhu					19-06-2023
	Chesnay Schepler	Matthias Pohl		Pekko 1.0.0 RC2 passed our tests successfully Ok to merge after the feature freeze in the next days		100%	26-07-2023		No documentation needed
State & Checkpoint
	yue ma	Yun Tang / ？					17-07-2023
FLIP-306: Unified File Merging Mechanism for Checkpoints	Zakelly Lan  @yinhan Yanfei Lei	Yuan Mei					25-07-2023
FLIP-263: schema compatibility	Hangxiang Yu	Yuan Mei				80%	25-07-2023
	@Rui xia	Yuan Mei					13-06-2023
	Yanfei Lei	Yuan Mei					30-05-2023
	Yanfei Lei	Yuan Mei					25-07-2023		Documented as Metric
	Stefan Richter	Stefan Richter					25-07-2023		No documentation needed
	Stefan Richter	Stefan Richter					25-07-2023		No documentation needed
REST
Support CPU profiler on taskmanager	Yu Chen	Yun Tang					13-06-2023
SQL
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
Calcite Update
	Sergey Nuyanzin	Timo Walther		done		100%	25-07-2023		No documentation needed
	Sergey Nuyanzin	Timo Walther		done		100%	25-07-2023		No documentation needed
	Sergey Nuyanzin	Timo Walther		done		100%	25-07-2023		No documentation needed
	ron liu	Godfrey He				50%	25-07-2023		Unclear if it needed. I asked on the ticket.
	Sergey Nuyanzin	Shengkai Fang		done		100%	25-07-2023		No documentation needed
FLIP-300	lincoln lee			done		100%	25-07-2023
FLIP-292: Enhance COMPILED PLAN to support operator-level state TTL configuration	Jane Chan			done		100%	25-07-2023
FLIP-302: Support TRUNCATE TABLE statement in batch mode	luoyuxia			done, docs missing		90%	01-08-2023
FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement	Mang Zhang	luoyuxia		done, docs missing		90%	01-08-2023
FLIP-303: Support REPLACE TABLE AS SELECT statement	luoyuxia			done		100%	01-08-2023		See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-or-replace-table
FLIP-311: Support Call Stored Procedure	luoyuxia			done, docs missing		90%	01-08-2023		See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/show/#show-procedures
FLIP-308: Support Time Travel	Feng Jin	luoyuxia		done, docs missing		90%	01-08-2023
	ron liu	luoyuxia		done		100%	25-07-2023		See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#add & https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#drop &https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/show/#show-partitions
FLIP-296	Kui Yuan	Yun Tang		done		100%	25-07-2023		See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/time_attributes/#advanced-watermark-features
FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway	Fang Yong	Benchao Li		done		100%	25-07-2023		https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/
FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode	Jing Ge	Jark Wu		done		100%	25-07-2023		No documentation needed
Connectors
FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations	Feng Jin Hang Ruan	Leonard Xu		done, some follow ups next release, docs missing		20%	25-07-2023
FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source	Hongshun Wang	Leonard Xu		done		100%	25-07-2023		https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/ & https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/
FLIP-294: Support Customized Catalog Modification Listener	Fang Yong	Leonard Xu		done		100%	25-07-2023		https://issues.apache.org/jira/browse/FLINK-32676
FLIP-314: Support Customized Job Lineage Listener	Fang Yong	Leonard Xu		under discussion		10%	25-07-2023
	luoyuxia	lincoln lee		done		100%	25-07-2023		No documentation needed
	Cai Liuyang  Rui Fan	Piotr Nowojski  Rui Fan		done		100%	25-07-2023
	Rui Fan	Piotr Nowojski Stefan Richter		done		100%	25-07-2023		see row above
Resource Management
	David Morávek	David Morávek		done, documentation missing		80%	25-07-2023
	Chesnay Schepler	Chesnay Schepler				90%	25-07-2023		No documentation needed
	Chesnay Schepler	Chesnay Schepler				60%	25-07-2023
Runtime
	Panagiotis Garefalakis	Piotr Nowojski		done, some follow ups next release, docs missing		90%	25-07-2023
/FLIP-285: Leader Election refactoring	Matthias Pohl	Chesnay Schepler				100%	25-07-2023		No documentation needed
Misc
	Chesnay Schepler	Chesnay Schepler				100%	25-07-2023		Checking with the contributor
	Martijn Visser	Matthias Pohl				100%	25-07-2023		No documentation needed

Status / Follow-ups

04/04/2023

• Is Zoom OK for everyone? 
  • Looks good for now, except the 40 min limit. One possible solution is to create two meetings as a backup. Usually release syncs can finish within 40 mins.
• Retrospective of 1.17
  • Has been summarized in 1.17 Release, at the end of the page.
• Daily work divisions
  
  • JIRA ticket (Mainly focus on new blockers, progress of existing blockers each week) (Qingsheng Ren )
  • CI (Failures reported in #build channel and not reported) (Sergey Nuyanzin )
  • Benchmark (Check if there's regression in the #flink-dev-benchmarks channel) Konstantin Knauf 
  • Mailing list monitoring (Jing Ge )
  • This division doesn't mean that other RMs are now allowed to do the work. All RMs should take responsibilities on all of these daily issues if they are available. 
• Feature freezing date is set to July 11, 2023. 

04/18/2023

• CI instabilities
  •  - PR is under review
  •  - Lijie Wang will take a look 
  • Konstantin Knauf will ping David again
  • - wait if there are more occurrences
  • Konstantin Knauf will ping David, Chesnay
  •  - Xingbo will take a look. Seems like a CI environment issue
  •   - wait if this happens more often ( Martijn Visser will check)
  •  - Hangxiang Yu will take a look
  • Konstantin Knauf will ping Ferenc
  •  - Pinged Francesco on JIRA
  • Azure CI failing with no space left on device → Robert Metzger will check if cleanup script needs to be udpated →
• Version management
  • Qingsheng Ren will remove 1.15 related resources within this week. 

05/30/2023

• Remind in the mailing list about updating the release wiki page.

06/13/2023

• Progress checking
  
  • yue ma Yun Tang 
  • Support CPU profiler on taskmanager Yu Chen Yun Tang 
  • Kui Yuan Yun Tang 
  • David Morávek 
  • Panagiotis Garefalakis Piotr Nowojski
  •   Matthias Pohl Chesnay Schepler 
• Blockers
  • Konstantin Knauf to ping Tzu-Li (Gordon) Tai 

06/27/2023

• Feature freezing date
  • We are still collecting progress of features, and will decide whether to postpone the feature freezing date in the next week. Currently we will stick to and push the original date.
  • Akka 2.6 will not receive any critical security updates after Sep 2023, so we might need to get rid of it in 1.18. Chesnay Schepler is working on it and might affect the feature freezing date. 
• Blockers and critical issues
  • Sergey Nuyanzin to ping Matthias Pohl 
  • Konstantin Knauf to ping Stefan Richter
    • PR by Stefan is already opened.
  • Konstantin Knauf to ping Roman Khachatryan or Anton Kalashnikov 
    • I've pinged Roman. 
  • Konstantin Knauf 
    • I've pinged Ferenc. 
  • Qingsheng Ren 
  • Qingsheng Ren to ping Fang Yong 
  • Qingsheng Ren to ping Junhan Yang 
  • Yunhong Zheng is working on it
  • Qingsheng Ren to ping liudalong
  • Lijie Wang will try to fix it before feature freeze
  • Qingsheng Ren to ping Yanfei Lei 
  • Konstantin Knauf will ask Chesnay Schepler if he could take it over
    • Chesnay will do another round of reviews today. It was actually updated by the contributor.
  • Qingsheng Ren 

07/11/2023

• Feature freeze date is extended until July 24, 2023, which is a Monday
• Blocker / Critical issues

07/25/2023

• Deprecate APIs in 1.18
  • Purely adding annotations / JavaDocs: go directly because they are not new features
  • Works requiring changes to code base
    • Create a thread in ML for receiving applications including: 
      • FLIP ID and current status (in discussion / vote passed / developing)
      • Why this deprecation must be done in 1.18
    • All 2.0 unrelated requests will be denied
    • To be discussed: deadline of these deprecation works. 1.17 release testing lasted 3 weeks, so what about 2 weeks for feature freeze of these deprecations?
    • Get 2.0 RMs involved in evaluating cases
  • Decided not to give more extension, as we couldn't see a lot of benefits to 2.0 giving another 2 weeks
• Cross-team testing
  • Goes in parallel with additional deprecation works above
  • will go normally, for about 3 weeks
  • Add a reminder about documentation before x-team test
• Cutting branch will be delayed until at least 2 weeks later we will target at the next Tuesday. 
• Blocker / Critical issues
  • 

08/01/2023

• ical event doesn't work anymore
• Requests to merge into master :
  • 
  •  Deprecating SourceFunction API
• Blockers/Critical (last 7 days)
  • Performance regression, PR has been approved and waiting for CI
  • Performance regression, Weijie Guo is looking into it
  • 1.18 API deprecations, maybe downgrade?
  • Release testing issue
  • Commit merged to master after feature freeze
  •  
  • : PR provided (test-code issue)
  • : PR provided (test-code issue)
  •  
  • : Test-code issue. Lijie will try to fix it
  • : Looks like a test instability caused by the underlying OS (Stefan Richter is looking into it)

08/08/2023

• Blockers / Critical issues (last 7 days)
  • blocker, Junrui is working on it
  • blocker, Dalong is working on it
  • blocker, PR is ready and Chesnay is reviewing it.
  • critical, looks like an old one
• Benchmark: TPC-DS result? Qingsheng Ren 
• Documentation: Remind contributors to finish docs
• Release announcement
• Cutting the branch: we'll wait for the status of blockers

08/15/2023

• Blockers / Critical issues (last 7 days)
  
  • blocker, Dalong is working on it, waiting for Lincoln Lee to review
  • blocker, PR is ready and Chesnay is reviewing it.
• Benchmark: TPC-DS result? Qingsheng Ren 
• Cutting the branch: we'll wait for the status of blockers

08/22/2023

• Blocker / Critical issues (last 7 days)
  • new CI instability. Qingsheng Ren to ping the author
• Cutting 1.18 branch
  • The branch will be cut today in order to unblock the master branch Jing Ge 
• Release testing
  • Issues are mistakenly assigned to the original author of new features. Qingsheng Ren will correct them. The original author should provide instruction / docs to help volunteers to do the verification
  • We target to finish all tests in 2 weeks. We'll make sure that all tickets are assigned on next Tuesday, and close them all before the next Tuesday after. 
• Release Announcement
  • All contributors with the link can edit / comment now
• Release Managing
  • Sergey Nuyanzin will clone the umbrella issue we used in 1.17 for tracking task assignees and the entire progress of promoting the release.

08/29/2023

• Release Testing
  • 2 unassigned issues: 
    • No descriptions
    •  
  • Progress 50%, with 15 open / 30 total
  • Try to close all before next Tuesday
• Blocker / Critical issues (last 7 days)
  • Blocker
    • 
    • 
    • 
    • 
  • Critical
    • 
    • 
• CI status
  • release-1.18 cron build?

09/05/2023

• Release Testing:
• Blocker / Critical issues (last 7 days)
  • Blocker
  • Critical
    • 
  • Major
    • 
      
• CI status

last ci with nightly (20230903.4) is green https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=52961&view=results

09/12/2023

• Release Testing:
  
  • 
    
• Blocker / Critical issues (last 7 days)
  • Blocker
    
• CI status
  • last green was on 3rd of September, after that blocked because of blocker above

09/19/2023

• Blocker / Critical issues (last 7 days)
  • Critical
    • 
• CI status
  • last green nightly was on 16th of September https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53296&view=results