Programming

API

Use a YAML compliant configuration parser.

Note: While this can be introduced as an optional feature in 1.x (i.e., first parse with standard YAML parser, fall back to custom parsing as a backup), the goal for 2.0 would be to remove the custom parsing logic.

Revisit all config option types and default values to determine whether any modifications are required for Flink 2.0. If any changes are necessary for configuration options marked as 'Public', they should be replaced with a new config option and deprecated in version 1.19; for other options, it should take place in version 1.20.

Update RocksDB statebackend package

Subsume Flink Time class with Java Duration.

• Mark org.apache.flink.api.common.time.Time and 
   org.apache.flink.streaming.api.windowing.time.Time as @Deprecated in 1.19 (Must do in 1.19)

• Refactor all usages of them to Java's Duration (Nice do in 1.19, must do in 1.20)

• Remove them in 2.0

This is an either in-place or smooth replacement of DataStream API. It differs from the current DataStream API mainly in the following aspects.

• Users define the data processing only through implementing a series of generic process functions
• No exposes of the operator internals (mailbox thread model, barrier alignment, etc.)
• No dependencies on internal implementations and/or 3rd party dependencies.

Disaggregated State Backend/Management in the cloud-native era.

• able to support very large states with a reasonable TPS
• able to support state Sharing and Querying without interfering with normal data processing
• Support fast recovering/rescaling in the cloud-native era

Note: Only the public API related part is must-have for release 2.0.

FLIP-468: Introducing StreamGraph-Based Job Submission

FLIP-469: Supports Adaptive Optimization of StreamGraph

• Clarify the capability guarantees and modification process
• revisit existing REST API and improve/remove incorrect interfaces
• unify conventions for naming fields
• passing parameters and HTTP methods

• remove in 2.0

Remove "now" field from various REST responses. This field contains the current timestamp, which is problematic w.r.t. caching and testing, while providing no value.

• remove in 2.0

Remove JarRequestBody#programArgs in favor of #programArgsList.

• remove in 2.0

Remove terminationMode parameter from job cancellation endpoint.

Effectively it only supports "cancel" as "stop" was moved into a dedicated endpoint.

• change all fields in 2.0

• remove in 2.0

• remove in 2.0

remove rescale REST endpoint. This endpoint hasn't been working for years. We can add it back later if necessary.

• Break API in 2.0

Enforce single maxExceptions query parameter

Note: Could be generalized to change all applicable query parameters.

Drop YARN-specific mutating GET REST endpoints (yarn-cancel, yarn-stop)

We shouldn't continue having such hacks in our APIs to work around YARN deficiencies.

• Removal in 2.0

• Deprecate MG scope-related methods in 1.19
• Rework interfaces in 2.0

1.19:

• Deprecate MG#getScopeComponents
• Deprecate MG#getMetrticIdentifier(String, CharacterFilter)

2.0:

• Remove methods

Rework MetricGroup scope APIs.

• Drop MetricGroup#getScopeComponents because it is unused and puts limitations on internal representations

• remove character filter / delimiter arguments from MetricGroup#getMetricIdentifier and similar methods
  
  • instead, let reporters provide these up-front via the factory as proposed in FLINK-7958

• Deprecate int methods in 1.19
• Remove in 2.0

Drop MetricGroup methods accepting an int as a metric/group name. These turned out to not be useful, and typically encourage bad design since a key-value pair should be used instead to attach some semantic to then number.

1.19:

• Deprecate getAllVariables
• Add new default getScopeVariables method that filters brackets

2.0:

• Remove getAllVariables
• Change getScopeVariables to not have a default impl

Remove brackets around keys returned by MetricGroup#getAllVariables.

This has turned out to just be a bother to anyone using it.

1.19: deprecate method

2.0: remove method

Drop MetricReporter#open

Enforce a single configuration path via the constructor. The factory continues to have access to the configuration.

• change API in 2.0

Pass configuration instead of Properties into ReporterFactory.

Flinks configuration classes are really useful but not available to reporters.

Blocked on splitting flink-core.

Attach semantics to Gauges (ratio/count/rate/something else); refactor Counter/Meter to be Gauges with syntactic sugar on top.

The goal is to generalize metrics from the perspective of reporters, and allow them to better map gauges to the external metrics backends supported types.

This implies subsuming Metric#getMetricType.

1.19: introduce warning if non-numeric gauge is used

2.0: force type to Number

Enforce that gauges are numeric. String gauges are exceedingly rare and push a lot of complexity into reporter implementations.

• deprecate LogicalScopeProvider
• add new APIs in 2.0

Properly expose the logical scope of a metric group. This is currently exposed via a crude casting code path, and is primarily blocked on rework of the scope APIs as the current deficiencies push down to many requirements into the APIs (support for per-call delimiters/filters).

• Make Java 17 default in 2.0

• Remove support in 2.0

• Remove support in 2.0

Restructure modules (maybe overlapping with All user APIs should be in a separate module)

• Split flink-core such that it doesn't contain data-processing related APIs
• Merge flink-java/flink-streaming-java
• Maybe break up flink-runtime (JM/TM/TM runtime/HA, security, metrics)

Note: These can  be done in 1.x or 2.1+, it's just easier to do in 2.0 if you don't need to worry about backwards-compatibility.