Document the state by adding a label to the FLIP page with one of "discussion", "accepted", "released", "rejected".

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Reading and analyzing the state of stateful Flink pipelines have become an increasing requirement over the years. Accessing the state of production applications allows developers to debug and understand how their pipelines work exactly without having to code around the black-box nature of it.

Flink had a state processor API for a long time, initially only in DataStream API but it has been extended with a keyed state table connector and some easier metadata reading capabilities. Unfortunately it still requires quite tedious setup, with defining the table schema, custom classes etc which makes it very impractical.

This FLIP introduces a StateCatalog: a read-only Flink SQL catalog that automatically discovers checkpoints and savepoints on any configured filesystem path, infers the keyed state schema and exposes them as SQL tables — all without requiring the original POJO classes on the classpath, without a running Flink cluster, and without writing any Flink DataStream code.


Public Interfaces

The StateCatalog builds a three-level SQL hierarchy on top of Flink snapshots:

Catalog  (StateCatalog name, e.g. "state")
└── Database (one per discovered snapshot, e.g. "prod/savepoint-a1b2c3")
├── View: metadata (operator inventory via savepoint_metadata())
└── Tables:
// Keyed state
└── uid_<uid>_keyed (one per operator with keyed state)
└── uid_<uid>_<statename>_keyed_flat (one per keyed map/list state in flat format for large states)
// Non keyed states
└── uid_<uid>_<statename>_broadcast/list/union (one per non-keyed operator state)
// Window state
         └── uid_<uid>_windowed_content_flat (one per keyed window operator flat format window content)
          └── uid_<uid>_windowed (one per keyed window operator for per window states)
// Operator level metadata
          └── uid_<uid>_metadata (operator level state metadata)





The entire system is read-only. There are no background threads, no shared caches between queries, and no cluster requirements — the catalog runs entirely in the SQL client or local mini-cluster.

CREATE CATALOG state WITH (
'type' = 'state',
'directory.prod' = 's3://my-bucket/flink/checkpoints/prod',
'directory.staging' = 's3://my-bucket/flink/savepoints/staging'
);


All mutating operations (createDatabase, createTable, alterTable, etc.) throw UnsupportedOperationException. The catalog is purely read-only.

Proposed Changes

Discovering savepoints and mapping to CatalogDatabase

The catalog scans one or more user-provided root directories to build the set of databases exposed to SQL. Discovery is configured through one or more directory.<label> options, each pointing to a filesystem path (local, HDFS, S3, GCS, etc.).


OptionType

Default

Description

directory.<label>String
One or more filesystem paths to scan. <label> becomes a prefix in database names.
listing-parallelism Integer10Max concurrent directory listing requests (for high-latency filesystems such as S3).



The <label> is a short human-readable name for the root directory. Additional options controlling discovery:

Snapshot discovery algorithm. The catalog performs a breadth-first search starting at each configured root. It descends into subdirectories until it finds a _metadata file — the universal marker written by both savepoints and checkpoints. Once a _metadata file is found the subtree rooted there is not descended further, so nested snapshots do not shadow each other.

Configured:  directory.prod = s3://bucket/flink/checkpoints/prod/

Filesystem:
s3://bucket/flink/checkpoints/prod/
savepoint-a1b2c3/_metadata → database: prod/savepoint-a1b2c3
00000000000000000000/chk-5/_metadata → database: prod/00000000000000000000/chk-5

SQL:
USE `prod/savepoint-a1b2c3`;
USE `prod/00000000000000000000/chk-5`;


The BFS issues concurrent listStatus calls (up to listing-parallelism, default 10) to tolerate the high round-trip latency of object stores.

Database naming. Every discovered snapshot becomes one CatalogDatabase. Its name is <label>/<relative-path-from-root>, using / as a hierarchy separator so that related snapshots group naturally:

SHOW DATABASES;
-- prod/savepoint-a1b2c3
-- prod/savepoint-d4e5f6
-- prod/00000000000000000000/chk-5
-- staging/savepoint-x9y8z7

Savepoint To Tables and Views

Each snapshot database exposes its contents as a fixed set of tables and views derived automatically from the snapshot's metadata file. No user configuration is needed beyond registering the catalog.

Tables and views per database. Every database contains:

Metadata

  1. global metadata view — a view over savepoint_metadata() that lists every operator in the snapshot along with its UID, name, parallelism, and state-size statistics. Useful for identifying which operators hold the most state and for understanding the snapshot's operator topology before querying individual tables.
  2. One operator level metadata table (per-operator) - includes operator/keyed state level metadata such as tate name, state type, serializer snapshot/serializer class, TTL etc where applicable

Keyed States

  1. One keyed-state table per operator — one table per operator that carries keyed state, named uid<uid>keyed when the operator has an explicit UID assigned, or id<uidhash>_keyed otherwise (where <uidhash> is a hex digest of the internal operator ID). This table contains one row per key with columns for all the different keyed states
  2. One flattened table per keyed list/map state - to allow querying large collection states that cannot fit into a single row
  3. One flattened table per window operator with the window content / records
  4. One table per window operator with window state as columns

Non keyed

  1. One table per non-keyed operator state - one table per operator list/broadcast/union state that has an explicit type declaration so we can represent it.
  2. One table per non-keyed window content
  3. One table per non-keyed window state


Column/Schema inference

Schema extraction from serializer snapshots. The schema of each state table is derived entirely from the TypeSerializerSnapshot tree embedded inside the _metadata file. Flink's serialization framework persists a full description of every serializer — field names, field types, subclass registrations — as part of the snapshot. The catalog includes utilities for converting different type serializer snapshots Flink SQL LogicalType without instantiating any user class.

This also means that only state that has explicit serializer snapshot/metadata in the state itself can be represented by the catalog. Fully custom states implemented through some legacy interfaces are naturally out of scope here.

Keyed States

Types supported by automatic inference:

  • All Flink primitive types and their wrappers
  • String, BigDecimal, Date, Time, Instant
  • POJO types (via PojoSerializerSnapshot)
  • Avro SpecificRecord and GenericRecord types
  • Flink TupleN types
  • List<T> and Map<K,V> wrappers
  • RowData types

Mapping individual keyed states to columns. Each keyed state entry in an operator becomes one column in the table. The three Flink state types map to SQL column types as follows:

VALUE state — a single value per key. Maps to a scalar SQL column of the value's logical type.
LIST state — a list of values per key. Maps to an ARRAY<T> column.
MAP state — a map of key-value pairs per key. Maps to a MAP<K,V> column.


Multiple state entries — most real operators maintain several state entries. All entries become separate columns in one table:


-- Operator with: ValueState<Long> "balance", ListState<Order> "open_orders",
-- MapState<String, String> "flags"
CREATE TABLE `uid_order_manager_keyed` (
state_key STRING NOT NULL,
balance BIGINT,
open_orders ARRAY<ROW<
order_id STRING,
amount DECIMAL(19, 2),
status STRING
>>,
flags MAP<STRING, STRING>,
PRIMARY KEY (state_key) NOT ENFORCED
) WITH ( 'connector' = 'savepoint', ... );

End-to-end example. Consider a running Flink job that maintains customer orders. A checkpoint has been taken at s3://prod-bucket/checkpoints/order-svc/00000.../chk-42:

-- Register the catalog once (e.g. in sql-client-defaults.yaml)
CREATE CATALOG state WITH (
'type' = 'state',
'directory.prod' = 's3://prod-bucket/checkpoints/order-svc'
);

-- Browse available snapshots
USE CATALOG state;
SHOW DATABASES;
-- 00000000000000000000/chk-40
-- 00000000000000000000/chk-41
-- 00000000000000000000/chk-42

-- Query state directly — schema was inferred automatically, no JAR required
SELECT
state_key AS customer_id,
balance AS wallet_balance,
CARDINALITY(open_orders) AS pending_order_count
FROM `uid_order_processor_keyed`
WHERE balance < 0
ORDER BY balance ASC
LIMIT 20;

Representing collection states as separate tables

There are some cases where the single table representation is not suitable for collection type keyed states, such as large list/map states. Simple example is a window operator but the general design/use-cases of these collection type states makes this very common. In most of these cases it's impossible to read the whole state and map to a single row (due to size constraints even). 

For these cases we should allow an option to map collection states to individual (flattened) tables where each row represents an element/entry in the collection.

For List states we would have (state_key, index, value), for Map states, (state_key, map_key, value).

We could control the extra flattened table and removal of the non-flattened columns from the shared keyed state table through a catalog config map. From a user perspective the projection pushdown would allow reading other state columns without problem even if there exists a non-flattened collection state column but may be simplest to allow a global config for turning these off.

- flattened list state table:
  uid_<operator_uid>_<state_name>_keyed_list
 (state_key, index, value)

- flattened map state table:
  uid_<operator_uid>_<state_name>_keyed_map
  (state_key, map_key, value)

Non-Keyed States

A single operator can have several logically unrelated non-keyed state entries. The natural mapping is therefore one table per named state entry, not one table per operator:

```
uid_[uid]_[state_name]_[type]
```

Where `[type]` is one of `broadcast`, `list`, or `union`.

There is currently no SQL connector/input format for non-keyed states so this has to be added as part of the implementation, but overall the schema inference and general flow will be similar to keyed states.

Initially we can support operator states that have a well defined serializer/type information stored in the snapshot. Custom states or where type info is not readily accessible from the metadata need a separate pluggable design, this could be relevant to some connector specific states etc. A follow up ticket/flip is needed here.

Window States

While out of scope for the initial implementation, it is worth considering window states as part of this FLIP to aid the implementation effort and subsequent contributions.

The main difference with window operators is the multi layered keyed state due to the window grouping itself which is represented by the Namespace on the runtime layer.

Window operators have different types of state that have to be represented as individual tables:

  1. Regular keyed ("global keyed state"), this is exactly like the existing keyed states and should be represented by same keyed state table structure
  2. The window contents: this is basically the "internal" state of the window operators, the records that are currently in the state.
    uid_<uid>_<statename>_windowed_flat (one per keyed window operator flat format window content)
  3. Per window state: actual state kept on a per window basis. 
    uid_<uid>_windowed (one per keyed window operator for per window states)


Reading state without user classes

A key challenge in reading savepoint state is that user-defined types — POJOs, Avro generated classes — are typically not available on the classpath of the SQL client or the Flink Table API runtime. This section describes the new mechanisms introduced to make state readable without those classes.

PojoToRowDataDeserializer. The PojoSerializerSnapshot embedded in a savepoint records the complete structural description of a POJO: the ordered list of field names, each field's individual TypeSerializerSnapshot, and the list of registered subclass snapshots in tag order. PojoToRowDataDeserializer uses this description to mirror the exact binary layout of PojoSerializer and reconstruct field values as GenericRowData — without loading the user's POJO class.

At runtime the deserializer reads the one-byte flag written by PojoSerializer to choose between the null, no-subclass, tagged-subclass, and unregistered-subclass cases, then reads fields in declaration order. POJO fields that are themselves POJOs are handled recursively; Avro-typed fields use the generic-record fallback described below; primitive and standard Flink types are deserialized using serializers restored from their own snapshots.

The result is that a deeply nested POJO hierarchy is fully queryable in SQL without shipping any user JAR to the SQL client.

Avro generic-record fallback. When a state column's serializer is AvroSerializer<T> and T is not available on the classpath, the deserializer falls back to GenericRecord-based deserialization using the Avro schema embedded in the AvroSerializerSnapshot. The resulting GenericRecord is converted to a GenericRowData using the field mapping derived from AvroSchemaConverter. This makes Avro-typed state columns fully readable without the generated Avro class.

Operator / Connector specific views to represent high level behavior

The initial scope of the State Catalog mainly focuses on the raw keyed, non-keyed, window states and metadata.

There are however certain operators and connectors where more high level structured view is necessary to make good use of the state. Some examples that come to mind:

  • High level representation of connector state, such as Kafka partition → offsets, File reading offsets etc.
  • Well structured view of some internal operator state, such as SQL joins and other SQL operators

To serve these use-cases we should introduce a pluggable mechanism to allow exposing views on certain operator states, which would

  • detect a certain type of operator/connector in the checkpoint
  • create a view representing the high-level connector/operator specific state view

Restricted Access Modes

There are certain environments where it would be desirable to limit the StateCatalog access to either metadata or the states themselves.

We could introduce catalog level configs to manage the types of tables that would be exposed / accessible through the catalog on a granular basis. 

Compatibility, Deprecation, and Migration Plan

All runtime changes are fully backward compatible. The catalog itself is completely new, no migration necessary.
Changes/improvements to the savepoint connector will be necessary and some will be breaking to simplify the logic but this is an experimental connector with the main purpose of enabling the ultimate catalog use case.

The new StateCatalog interface is experimental, and we need to highlight that we provide no guarantees for the catalog or the state of Flink internal operators:
 1. The physical state representation of SQL and other built in operators is a Flink internal and not part of the public api. Therefore the structure / schema of these tables are subject to change without notice across Flink versions.
 2. The StateCatalog itself including configuration, operator/state to table mapping, metadata views, etc. are all initially experimental and provide no backward compatibility guarantees and can change without notice initially.

Test Plan

Comprehensive unit and integration test coverage should be included for the new functionality including type extraction, deserialization without original type pojo classes, and different state backends

Rejected Alternatives