This is an either in-place or smooth replacement of DataStream API. It differs from the current DataStream API mainly in the following aspects.

• Users define the data processing only through implementing a series of generic process functions
• No exposes of the operator internals (mailbox thread model, barrier alignment, etc.)
• No dependencies on internal implementations and/or 3rd party dependencies.

Progress: 

Basic stateless & stateful ds v2 framework: Done

Generalized Watermark: FLIP in preparation

Disaggregated State Backend/Management in the cloud-native era.

• able to support very large states with a reasonable TPS
• able to support state Sharing and Querying without interfering with normal data processing
• Support fast recovering/rescaling in the cloud-native era

Note: Only the public API related part is must-have for release 2.0.

Progress: The new db and async state API & implementation are done. Local disk cache is done. Now working on checkpoint & recovery part.

FLIP-478 Introduce Config Option Scope

• The slot level balancing strategy is completed
• Progress in pending: 
  • The failover strategy for balancing tasks scheduling is in discussing.

In development:

• Parametriezed type extractor to allow switching between Kryo-backed serializors (for backwards-compatibility) and dedicated ones for collection types
• Dedicated serializer support for common Java collection types

FLIP-468: Introducing StreamGraph-Based Job Submission

FLIP-469: Supports Adaptive Optimization of StreamGraph

FLIP-468 is done

FLIP-469 is in development, target to finish in 2024-12