Release Managers

Jiangjie Qin Xintong Song Martijn Visser Jark Wu 

Timeline & Branch Management

  • August 1, 2023: 2.0 Status Update
    • We are expecting a 1.19 → 1.20 → 2.0 release sequence, with a normal 4-5 months release cycle. We expect to deliver the 2.0 release by the end of 2024, as long as not compromising the quality.
    • Public APIs need to be deprecated in 1.19, and PublicEvolving APIs in 1.20, in order to be removed in 2.0. Please try to reach out to the release managers early if you see a risk of planned work items not completed in time.
    • The 2.0 branch will be cut around the time of releasing 1.20, from the master.

Work Items

Priority

  • Must-have Items
    • Things that must be done in release 2.0.
    • Typically, this should include only API breaking changes.
    • If not finished, the release will be blocked.
  • Nice-to-have Items
    • Everything else that is planned for the 2.0 release.
    • If not finished, the release will not be blocked.

If there's a resource conflict (e.g., requiring efforts from the same group of people, interference due to actively working on the same pieces of codes, etc.), must-have items should be prioritized over nice-to-have items

Legend


SymbolMeaning

Priority

(lightbulb)

Must-have Item

(grey lightbulb)

Nice-to-have Item

(question)

TBD

State


(tick)Done
(green star)Will make it
(star)In danger
(red star)Very unlikely
(minus)Won't make it
(question)State unclear

List

Modification of the list would require community decision. Please reach out to the release managers and do not modify the list directly.

Remove deprecated APIs
Name / JIRA / FLIPPriorityContributorReviewer

Milestones

State%Last UpdateDescription / Note
DataSet(lightbulb)
  • Old API deprecated: 1.18 (tick)
  • Old API removed: 2.0
(green star)

2023-08-01


Deprecated methods/fields/classes in DataStream

(lightbulb)
  • Old API removed: 2.0
(green star)
2023-08-01

FLINK-29741 - Getting issue details... STATUS

(lightbulb)
  • Old API deprecated: 1.17 & 1.18(tick)
  • Old API removed: 2.0


2023-12-01

FLINK-33747 - Getting issue details... STATUS

(lightbulb)


  • Old API deprecated: 1.15 (tick)
  • Old API removed: 2.0
(green star)
2023-08-01

FLINK-33748 - Getting issue details... STATUS

(lightbulb)
  • New API functionality ready: 1.11(tick)
  • Old API deprecated: 1.12 (tick)
  • Old API removed: 2.0




Deprecated methods in Table API(lightbulb)
  • Old API removed: 2.0




Legacy SQL function stack and operators(lightbulb)
Jark Wu 
  • Old API removed: 2.0




Queryable state(grey lightbulb)Martijn Visser Queryable State will not be removed until we have a new substitution. Its timeline depends on the progress of the disaggrated state management  



FLINK-33750 - Getting issue details... STATUS

(lightbulb)Junrui Lee 
  • Old API removed: 2.0
(green star)
2023-08-22Note: This includes removing config options marked @Deprecated and removing config keys placed withDeprecatedKeys().
Imperative APIs (DataStream / DataSet / ProcessFunction)
Name / JIRA / FLIPPriorityContributorReviewer

Milestones

State%Last UpdateDescription / Note
Introduce a MVP for a new ProcessFunction API(grey lightbulb)
  • Design ready: 1.19
  • New API functionality ready: 2.0
  • New API marked as stable: 2.4
  • Old API deprecated: 2.4
  • Old API removed: 3.0

(green star)


2023-08-01

This is an either in-place or smooth replacement of DataStream API. It differs from the current DataStream API mainly in the following aspects.

  • Users define the data processing only through implementing a series of generic process functions
  • No exposes of the operator internals (mailbox thread model, barrier alignment, etc.)
  • No dependencies on internal implementations and/or 3rd party dependencies.

FLINK-14068 - Getting issue details... STATUS

FLINK-32570 - Getting issue details... STATUS

(lightbulb)Rui Fan 
  • Mark org.apache.flink.api.common.time.Time and 
     org.apache.flink.streaming.api.windowing.time.Time as @Deprecated in 1.19 (Must do in 1.19)

  • Refactor all usages of them to Java's Duration (Nice do in 1.19, must do in 1.20)

  • Remove them in 2.0

(green star)


2024-01-08

Subsume Flink Time class with Java Duration.

Refactor context-like APIs

(grey lightbulb)
  • New API functionality ready: 1.19
  • New API marked as stable: 1.19
    • The changes are expected to be minor and straightforward
  • Old API deprecated: 1.19
  • Old API removed: 2.0




Refactor classes like RuntimeContext to return a concrete class containing various bits of metadata, that can be shared in various context implementations.

We currently extend various context-like APIs to often, exposing one additional item at a time.

Table API / SQL
Name / JIRA / FLIPPriorityContributorReviewer

Milestones

State%Last UpdateDescription / Note

State compatibility for SQL jobs

(grey lightbulb)


Jark Wu 
  • New API functionality ready: 2.0




State compatibility of upgrading Flink version for SQL jobs. This can be a good API story and milestone for Flink SQL since 2.0.0. 

Move Calcite rules from Scala to Java
FLINK-34156 - Getting issue details... STATUS

(grey lightbulb)
  • MVP with several moved rules: 1.19
  • Finalize movement: 2.0




Move Flink Calcite rules from Scala to Java.
The reason is that since 1.28.0( CALCITE-4787 - Getting issue details... STATUS ) Calcite started to use Immutables (https://immutables.github.io/) and since 1.29.0 removed ImmutableBeans ( CALCITE-4839 - Getting issue details... STATUS ). All rule configuration related api which is not Immutables based is marked as deprecated. Since Immutables implies code generation while java compilation it is seems impossible to use for rules in Scala code.

States
Name / JIRA / FLIPPriorityContributorReviewer

Milestones

State%Last UpdateDescription / Note
Disaggregated State Management(lightbulb)
  1. 2023 Nov - Dec: Design discussion 
  2. Dec. 31: PoC ready and get initial performance numbers
  3. Flink 1.19 (2024 Jan): New API and Framework discussion finalized
  4. 2024 March: API change and Framework change complete
  5. 2024 May: MVP end-2-end Agg/Join Job working with the new state backend




Disaggregated State Backend/Management in the cloud-native era.

  • able to support very large states with a reasonable TPS
  • able to support state Sharing and Querying without interfering with normal data processing
  • Support fast recovering/rescaling in the cloud-native era

Note: Only the public API related part is must-have for release 2.0.

Eager state declaration.(grey lightbulb)
Part of the new queryable state API?


This is basically a restart of discussion around FLIP-22: Eager State Declaration.
(lightbulb)
  • change package in 2.0




Update RocksDB statebackend package

Serialization
Name / JIRA / FLIPPriorityContributorReviewer

Milestones

State%Last UpdateDescription / Note


Plugin serializers

(lightbulb)


Fang Yong 






2023-12-18

It is currently too difficult to introduce custom serializers, requiring modifications to the actual program sources, which in some cases isn't feasible. The general idea is that instead of defaulting to Kryo we first ask all found serializers whether they support the given type.

This would allow us to get rid of the Kryo-related APIs in DataStream.

Make Kryo usage opt-in

(lightbulb)




2023-12-18

It is currently too easy to accidentally use Kryo, causing problems down the line with schema evolution.

Do not eagerly register kryo avro serializer

(grey lightbulb)




2023-12-18

Currently the avro serializers is registered with Kryo if flink-avro is on the classpath. This also happens if avro isn't even used by the job, be it due to a mistake in the dependencies setup, branching or flink-avro being in lib.

This forces users to always provide flink-avro going forward for affected jobs, because on recovery Flink complains if any of the Kryo serializers cant be loaded.

Enable built-in Map/List/Set serializers by default

(grey lightbulb)


Fang Yong 



2023-12-18

These are basic data-structures that are commonly used but currently required Kryo to be used.

Configuration FLINK-32383 - Getting issue details... STATUS

Name / JIRA / FLIPPriorityContributorReviewer

Milestones

State%Last UpdateDescription / Note
Introduce a clean configuration layer

FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
FLIP-391: Deprecate RuntimeContext#getExecutionConfig
(lightbulb)
  • Deprecate unwanted configuration interfaces: 1.19
  • Feature development: 1.19
  • Remove above deprecated items: 2.0

(green star)


2023-08-22

Goals:
1. All configuration can be set using key-value pairs
2. Ensure key-value job configuration to take effect in JobGraph generation stage
3. Show all the job configuration in effect to users via web UI
4. Avoid exposing unnecessary configuration (i.e. the entire ExecutionConfig) to user functions

Clarify the scopes of configuration options(grey lightbulb)
  • Add interface to specify the scopes of ConfigOption: 1.19
  • Label the scopes of each existing ConfigOption: 1.20
  • Leverage the scopes to do usage check or access control: 2.0

(green star)


2023-08-22

Clarify the scopes of configuration options, i.e. job config, cluster config or both, to avoid misuses of configuration. The scopes should be displayed in Flink configuration documentation.

Add a mechanism to prevent setting configuration from wrong scopes, e.g. setting cluster configuration when submitting a job to an existing Flink session, or setting job configuration in cluster components.

FLINK-23620 - Getting issue details... STATUS

(lightbulb)
  • Introduce standard YAML parsing process: 1.19
  • Remove legacy parsing logic: 2.0

(green star)


2023-08-22

Use a YAML compliant configuration parser.

Note: While this can be introduced as an optional feature in 1.x (i.e., first parse with standard YAML parser, fall back to custom parsing as a backup), the goal for 2.0 would be to remove the custom parsing logic.

Revisit all config option types and default values

(lightbulb)


  • Revisit all config option and find out all the config option that requires modification
  • Replace and deprecate the 'Public' config option: 1.19
  • Replace and deprecate all config options: 1.20



2023-10-17

Revisit all config option types and default values to determine whether any modifications are required for Flink 2.0. If any changes are necessary for configuration options marked as 'Public', they should be replaced with a new config option and deprecated in version 1.19; for other options, it should take place in version 1.20.

FLINK-32684 - Getting issue details... STATUS

(grey lightbulb)


  • Introducing PekkoOptions (or RpcOptions?) in 1.19
  • Removal of AkkaOptions in 2.0

(green star)


2023-08-22

It's a plain renaming of the @PublicEvolving class AkkaOptions into PekkoOptions (using a more generic name here can be discussed as there are plans to migrate to gRPC as part of the FLIP).

FLINK-21126 - Getting issue details... STATUS

(grey lightbulb)Zhu Zhu 

Decide to not do it in Flink 2.0.

Go to FLINK-21126 for more details or further discussions.

(minus)


2023-08-22

Drop FLINK_PROPERTIES from Flink docker image

REST API FLINK-32377 - Getting issue details... STATUS

Name / JIRA / FLIPPriorityContributorReviewer

Milestones

State%Last UpdateDescription / Note
Review and refactor the REST API(lightbulb)




  • Clarify the capability guarantees and modification process
  • revisit existing REST API and improve/remove incorrect interfaces
  • unify conventions for naming fields
  • passing parameters and HTTP methods
FLIP-336: Remove "now" timestamp field from REST responses(lightbulb)
  • remove in 2.0




Remove "now" field from various REST responses. This field contains the current timestamp, which is problematic w.r.t. caching and testing, while providing no value.

FLIP-337: Remove JarRequestBody#programArgs(lightbulb)
  • remove in 2.0




Remove JarRequestBody#programArgs in favor of #programArgsList.

FLIP-338: Remove terminationMode query parameter from job cancellation REST endpoint(lightbulb)
  • remove in 2.0




Remove terminationMode parameter from job cancellation endpoint.

Effectively it only supports "cancel" as "stop" was moved into a dedicated endpoint.

(lightbulb)
  • change all fields in 2.0



Use a consistent naming convention for all REST API fields (camelCase)
(lightbulb)
  • remove in 2.0



Remove deprecated query parameters that were subsumed by request body parameters, like the jar handlers.
(lightbulb)
  • remove in 2.0




remove rescale REST endpoint. This endpoint hasn't been working for years. We can add it back later if necessary.

(lightbulb)




REST API returns only numeric values that conform to the JSON spec. Essentially, never return NaN/Inf as these are serious headaches for generated clients.


(lightbulb)
  • Break API in 2.0



Refactor certain REST return values to be less special, like ConfigurationInfo which is returned as a list of key-value pairs, when it should just be a map.

FLINK-32688 - Getting issue details... STATUS

(lightbulb)
  • API is already deprecated (related FLIP will be opened in 1.19)
  • Removal of deprecated fields: 2.0
(green star)
2023-08-22Remove deprecated fields from ExceptionHistory API (all-exceptions, timestamp, root-exception, truncated)

FLINK-32198 - Getting issue details... STATUS

(lightbulb)





Enforce single maxExceptions query parameter

Note: Could be generalized to change all applicable query parameters.

FLINK-32597 - Getting issue details... STATUS

(lightbulb)





Drop YARN-specific mutating GET REST endpoints (yarn-cancel, yarn-stop)

We shouldn't continue having such hacks in our APIs to work around YARN deficiencies.

Metrics FLINK-32378 - Getting issue details... STATUS

Name / JIRA / FLIPPriorityContributorReviewer

Milestones

State%Last UpdateDescription / Note

(lightbulb)




Review and refactor the metrics implementation (revisit and improve/remove incorrect metrics)

(lightbulb)


  • Removal in 2.0



Drop deprecated Availability/Network metrics: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/metrics/#network

FLINK-14350 - Getting issue details... STATUS

(lightbulb)
  • Deprecate MG scope-related methods in 1.19
  • Rework interfaces in 2.0



Split MetricGroup interface to target different audiences (reporters that retrieve metadata, users that register metrics/groups)


(lightbulb)

1.19:

  • Deprecate MG#getScopeComponents
  • Deprecate MG#getMetrticIdentifier(String, CharacterFilter)

2.0:

  • Remove methods




Rework MetricGroup scope APIs.

  • Drop MetricGroup#getScopeComponents because it is unused and puts limitations on internal representations
  • remove character filter / delimiter arguments from MetricGroup#getMetricIdentifier and similar methods
    • instead, let reporters provide these up-front via the factory as proposed in FLINK-7958
(lightbulb)
  • Deprecate int methods in 1.19
  • Remove in 2.0




Drop MetricGroup methods accepting an int as a metric/group name. These turned out to not be useful, and typically encourage bad design since a key-value pair should be used instead to attach some semantic to then number.

(lightbulb)

1.19:

  • Deprecate getAllVariables
  • Add new default getScopeVariables method that filters brackets

2.0:

  • Remove getAllVariables
  • Change getScopeVariables to not have a default impl




Remove brackets around keys returned by MetricGroup#getAllVariables.

This has turned out to just be a bother to anyone using it.


(lightbulb)

1.19: deprecate method

2.0: remove method




Drop MetricReporter#open

Enforce a single configuration path via the constructor. The factory continues to have access to the configuration.


(grey lightbulb)
  • change API in 2.0




Pass configuration instead of Properties into ReporterFactory.

Flinks configuration classes are really useful but not available to reporters.

Blocked on splitting flink-core.


(lightbulb)





Attach semantics to Gauges (ratio/count/rate/something else); refactor Counter/Meter to be Gauges with syntactic sugar on top.

The goal is to generalize metrics from the perspective of reporters, and allow them to better map gauges to the external metrics backends supported types.

This implies subsuming Metric#getMetricType.

FLINK-6053 - Getting issue details... STATUS

(lightbulb)

1.19: introduce warning if non-numeric gauge is used

2.0: force type to Number




Enforce that gauges are numeric. String gauges are exceedingly rare and push a lot of complexity into reporter implementations.

FLINK-7957 - Getting issue details... STATUS

(lightbulb)
  • deprecate LogicalScopeProvider
  • add new APIs in 2.0




Properly expose the logical scope of a metric group. This is currently exposed via a crude casting code path, and is primarily blocked on rework of the scope APIs as the current deficiencies push down to many requirements into the APIs (support for per-call delimiters/filters).

Misc
Name / JIRA / FLIPPriorityContributorReviewer

Milestones

State%Last UpdateDescription / Note

FLINK-15736 - Getting issue details... STATUS
FLINK-33163 - Getting issue details... STATUS

(lightbulb)
  • Make Java 17 default in 2.0



Add support for Java 17 and make this the default. 
Nice-to-have: also add support for Java 21

Drop support for Java 8

(lightbulb)
  • Remove support in 2.0




Drop support for Java 11

(grey lightbulb)
  • Remove support in 2.0




Restructure modules

(grey lightbulb)





Restructure modules (maybe overlapping with All user APIs should be in a separate module)

  • Split flink-core such that it doesn't contain data-processing related APIs
  • Merge flink-java/flink-streaming-java
  • Maybe break up flink-runtime (JM/TM/TM runtime/HA, security, metrics)

Note: These can  be done in 1.x or 2.1+, it's just easier to do in 2.0 if you don't need to worry about backwards-compatibility.

Upgrade checkstyle

(lightbulb)Martijn Visser 







  • No labels