Status
...
Page properties | |
---|---|
|
...
JIRA: will create umbrella JIRA once the proposal is accepted
...
|
Motivation
With the current state registration interface, managed state (both operator and keyed state) can be lazily declared any time during processing. This poses problems in several dimensions for Flink’s state management capabilities:
...
All operators, as well as rich UDFs can be stateful. Previously, state can be lazily declared with a state descriptor any time during processing by using getRuntimeContext().getState(descriptor)
. Both concerns of state access as well as state registration is achieved addressed through the same call.
We propose to decompose this into 2 steps: (a) pre-flight (i.e. before the job graph is submitted) state descriptor registration, and (b) runtime access to registered state using the state ID only.
...
redistributeMode
: the redistribution scheme of this operator list state. For the initial version, we would not allow changing the used mode for previous state. Ideally, this can be changed freely depending on what redistribution scheme the user wishes to use for the current restore.
These annotations can be used at the operator-level or in rich UDFs. Briefly speaking, users' custom operators / UDFs should declare fields typed with the user-facing State
types, and annotate them as either KeyedState
or OperatorState
.
...
Annotated state fields will also be automatically initialized during the StreamOperator#initializeState
phase using Java reflection. The user is not required to access state via the RuntimeContext
if the state was declared using state annotations. This should also enhance the perceived user experience, such that programming against Flink managed state is similar to manipulating a local variable.
(3) Deprecate existing
...
lazy state declaration interfaces
All existing lazy state declaration interfaces will be deprecated, and eventually removed (Flink version for removal TBD). This includes:
...