DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Flink tasks today report a single scalar numRecordsOut counter that counts every record emitted by the task's last operator, irrespective of how many distinct downstream consumers that output goes to. When a task has more than one network output, there is no way for any consumer (REST API, metric reporter, Web UI, Flink Kubernetes Operator’s Autoscaler) to distinguish how many records went to each downstream.
This becomes a correctness problem for capacity planning and autoscaling in three common topology patterns:
- Side outputs - one operator splits records into a main stream and one or more side outputs, each consumed by a different downstream vertex with a potentially different cost profile.
- Multi-sink fan-out - the same source/operator feeds several independent sinks in parallel.
- Broadcast - one upstream broadcasts every record to N downstream vertices; the aggregate "records written" counts one emit per record, not N, so the absolute outbound load on each downstream is invisible at the source level.
The Flink Kubernetes Operator's Autoscaler computes per-vertex target parallelism from upstream throughput. Without per-edge numbers it can only divide aggregate throughput by a uniform assumption (for multi-output upstream edges the last resort fallback is the total numRecordsOut), which either over-provisions low-traffic branches or under-provisions high-traffic branches.
To overcome this we need a first-class, per-downstream-target breakdown of numRecordsOut, exposed both as a live metric (for reporters) and in the archived REST snapshot (for the Web UI and the autoscaler).
Proposed Changes
Draft PR: https://github.com/apache/flink/pull/28014
The change has four layers: planner, runtime metric registration, archival snapshot, and REST exposure.
1. Planner: carry the downstream JobVertexID on every network output
The REST API identifies every vertex in the job graph by its JobVertexID (hex-encoded). /jobs/:jobid returns a list of vertices keyed by this id; /jobs/:jobid/vertices/:vertexid uses it as the path parameter. So, every cross-reference between vertices in the REST model (inputs, outputs, plan edges) is expressed in JobVertexID space. Keying the per-target map by JobVertexID therefore makes the map self-joining against the rest of the REST payload: a consumer that reads write-records-per-target on vertex A can look up each key directly in the same /jobs/:jobid response to resolve the downstream vertex's name, parallelism, state, or metrics. This means no extra translation table, no extra round trip.
Any other candidate key (StreamNode id, OperatorID, IntermediateDataSetID) would require the consumer (including the autoscaler, which is the primary motivation of this FLIP) to maintain or fetch a secondary mapping back to JobVertexID before it could correlate per-target counts with per-vertex scaling decisions. Stamping the id at the planner is therefore not just a convenience: it is what makes the metric directly consumable at the same level of abstraction the rest of the REST view already operates on.
The approach that can be taken here resides in NonChainedOutput. NonChainedOutput (the serialized description of a task's network output edge) today captures the IntermediateDataSetID and the downstream operator's StreamNode id, but does not carry the downstream JobVertexID. StreamingJobGraphGenerator#createOrReuseOutput will be updated to stamp each NonChainedOutput with the JobVertexID of the vertex that consumes the edge:
```
new NonChainedOutput(
consumerEdge.supportsUnalignedCheckpoints(),
consumerEdge.getSourceId(),
jobVertexBuildContext.getJobVertex(consumerEdge.getTargetId()).getID(), // new
consumerParallelism,
...);
```
Resolution happens at job-graph generation time when all JobVertexes for the graph are already registered in JobVertexBuildContext, so the lookup is always defined.
Adaptive Execution Note: Support depends on whether the downstream JobVertexID is known by the time each upstream task starts:
- Schedulers that build the full JobGraph upfront (default scheduler, streaming Adaptive scheduler) always have every downstream id available at planning time, so the per-target metric is fully populated end-to-end.
- Schedulers that build the JobGraph incrementally (AdaptiveBatch / AdaptiveExecutionHandler) create downstream vertices only after their upstream finishes. The id the planner would stamp on the upstream's network output is not yet known when that upstream task is deployed, and the runtime binds per-target counters once at task startup. The aggregate numRecordsOut stays fully correct, but the per-target breakdown is unavailable for edges whose downstream was not yet known at upstream-task start.
This is a scheduler-timing limitation rather than a failure mode. Consumers are expected to treat an empty per-target map as "breakdown unavailable for this edge" and fall back to the aggregate. Closing the gap would require a runtime capability to rebind per-target counters after task startup, which is out of scope for this FLIP.
2. Runtime: install a per-target counter per network output
Every network output of a task has exactly one downstream target JobVertex, known at planning time (see step 1). We want to count records per (task, downstream_target) pair, and simultaneously preserve the existing aggregate numRecordsOut as "one count per logical emit from this task", independent of how many downstream targets that emit fans out to.
The subtlety is that side-output / multi-sink topologies and broadcast topologies impose different relationships between the per-target counters and the aggregate:
Topology | Emit semantics | Aggregate behavior | Per-target behavior |
Single network output | 1 emit → 1 target | aggregate == per-target value | per-target == aggregate |
Routed multi-output (side output, multi-sink) | 1 emit → 1 of N targets | aggregate == sum(per-target) | partitions the aggregate |
Broadcast fan-out | 1 emit → all N targets | aggregate == 1 per emit | each per-target == aggregate |
This yields two registration modes on TaskIOMetricGroup, and the invariant we must hold is:
- Shared mode (reuseRecordsOutputCounter(counter, jobVertexId)): one Counter instance is simultaneously the aggregate contributor and the per-target metric. Used whenever each logical emit crosses exactly one edge (i.e. the single-output and routed multi-output cases). One inc() on the record-writer output correctly advances both.
- Target-only mode (registerNumRecordsOutPerTarget(counter, jobVertexId)): the Counter is exposed as numRecordsOut.<jobVertexId> but does not contribute to the aggregate SumCounter. Used for the broadcast branch, where each logical emit crosses N edges but the aggregate must still reflect 1. The BroadcastingOutputCollector owns the aggregate counter directly and increments it once per logical emit; the per-target counters on the N wrapped RecordWriterOutputs each advance on their own edge crossing without double-counting into the sum.
The invariant we must hold is: The aggregate numRecordsOut is incremented exactly once per logical emit from the task. Per-target counters are incremented once per edge crossing.
To address the above the OperatorChain#createOutputCollector is updated with the new logic:
- Build a side map outputToTargetNodeId: IdentityHashMap<Output, JobVertexID> while iterating operatorConfig.getOperatorNonChainedOutputs(...), populated from NonChainedOutput#getTargetNodeId().
- Single-output branch (allOutputs.size() == 1): call taskIOMetricGroup.reuseRecordsOutputCounter(counter, targetId.toHexString()). The counter is both the aggregate and the per-target metric (they are numerically equal in this branch).
- Multi-output / broadcast branch (allOutputs.size() > 1): for each RecordWriterOutput, call taskIOMetricGroup.registerNumRecordsOutPerTarget(counter, targetId.toHexString()) and set it via RecordWriterOutput#setNumRecordsOut(counter). The aggregate is owned separately by the BroadcastingOutputCollector wrapping them, so per-target counters must not contribute to the sum (otherwise broadcast would double-count: N per-target + 1 from the collector).
- Defensive fallback: If a target id is missing for any reason (planner invariant violation, future refactor, etc.), log a WARN once per affected output and fall back to the aggregate-only counter. The per-target breakdown for that output is lost, but the aggregate numRecordsOut still works and the task never fails because of metric wiring.
3. Archival: preserve the map on terminal state
Execution#updateAccumulatorsAndMetrics today rebuilds a lean IOMetrics when archiving a finished execution (it intentionally drops resultPartitionBytes to bound memory). It will be extended to preserve numRecordsOutPerTarget, which is small (N entries per task, where N is the task's out-degree) and is the only channel through which REST consumers ever see this data.
This is the subtle part of the change: without it, live metric reporters would see the per-target breakdown but REST consumers (including the autoscaler) would always see an empty map. Unit tests on Execution alone cannot catch this because they do not exercise the full path. For this, an ITCase that verifies the map reaches REST closes the gap.
4. REST: aggregate per-subtask and expose
- IOMetrics gains the map field and constructor parameter.
- MutableIOMetrics (the REST-handler aggregator that folds subtask IOMetrics into a vertex-level view) gains a per-key sum merge. This is essential: a vertex with N parallel subtasks each emitting to downstream A must show sum_i(subtask_i.toA) for the vertex-level A entry, not just one subtask's value.
- IOMetricsInfo gains the write-records-per-target field with Jackson annotations.
- JobDetailsHandler, JobVertexTaskManagersHandler, and SubtaskExecutionAttemptDetailsInfo populate the new field.
- The generated REST OpenAPI YAML (rest_v1_dispatcher.yml) and the docs shortcode (rest_v1_dispatcher.html) are regenerated.
Public Interfaces
1. REST API
Additive field on the existing IOMetricsInfo DTO, which is embedded in several REST responses:
GET /jobs/:jobid
GET /jobs/:jobid/vertices/:vertexid
GET /jobs/:jobid/vertices/:vertexid/taskmanagers
GET /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex/attempts/:attempt
New field:
```
{
"write-records": 1234,
"write-records-complete": true,
"write-records-per-target": {
"<downstream-job-vertex-id-hex>": 800,
"<another-downstream-job-vertex-id-hex>": 434
}
}
```
- Key: hex-encoded JobVertexID of the downstream consumer.
- Value: long, count of records emitted from this task/vertex toward that specific downstream.
- Semantics: For side outputs and multi-sink, the values partition the aggregate (sum(values) == write-records). For broadcast fan-out, each value equals the aggregate (max(values) == write-records) because every record went to every target.
- Compatibility: Existing fields (write-records, write-records-complete, etc.) are unchanged. Consumers that do not know about the new field continue to work.
- Empty-map case: When there are no network outputs (e.g. a sink task) the map is empty, not absent. Old-state restores where the map was not yet populated also surface as empty.
2. Metrics (live reporters)
New individual metric per network output:
numRecordsOut.<downstream-job-vertex-id-hex>
Registered under the existing TaskIOMetricGroup scope, visible to all reporters (Prometheus, JMX, OTel, StatsD, Datadog, SLF4J, Graphite, InfluxDB, Dropwizard). The aggregate numRecordsOut metric keeps its current meaning and value.
3. Internal Java API (@Internal)
New overloads on TaskIOMetricGroup:
```
/**
* Registers {@code counter} as the aggregate numRecordsOut counter and simultaneously
* exposes it as an individual metric under {@code numRecordsOut.<jobVertexId>}.
* The counter contributes to the aggregate SumCounter.
*/
public void reuseRecordsOutputCounter(Counter counter, String jobVertexId);
/**
* Registers {@code counter} as a target-only metric under
* {@code numRecordsOut.<jobVertexId>}. The counter does NOT contribute to the aggregate
* SumCounter: use this when the aggregate is incremented elsewhere (e.g. broadcast).
*/
public void registerNumRecordsOutPerTarget(Counter counter, String jobVertexId);
/**
* Snapshot of per-target counts for inclusion in {@link IOMetrics}.
*/
public Map<String, Long> getNumRecordsOutPerTarget();
IOMetrics gains an immutable Map<String, Long> numRecordsOutPerTarget field with a corresponding constructor overload. IOMetricsInfo gains Map<String, Long> getRecordsWrittenPerTarget().
```
No user-facing (@Public, @PublicEvolving, @Experimental) class gains a new method or field that users would program against; all API changes are @Internal. Users interact with this feature via REST or via metric reporters.
Compatibility, Deprecation, and Migration Plan
Backward Compatibility
This FLIP is fully backward compatible:
- Wire compatibility: The IOMetrics RPC message sent from TaskManager to JobManager adds one optional field (Map<String, Long>).
- Old TaskManagers sending to a new JobManager deserialize the absent field as an empty map.
- New TaskManagers sending to an old JobManager - Flink does not support this direction of version skew in general, so no special handling is needed.
- State compatibility: The new field is not persisted in checkpoints or savepoints. IOMetrics is a transient, per-execution snapshot, not durable state. No TypeSerializerSnapshot migration is required.
- REST & Metrics compatibility: Additive JSON field. Old clients ignore unknown fields. New clients that assume the field is present must still handle it being empty.
Migration
None On upgrade, the new REST field and new individual metrics appear automatically; the Kubernetes autoscaler can opt in on its side by a capability check ("is write-records-per-target non-empty on any vertex?") and fall back to the current aggregate-only heuristic when running against older Flink versions.
Deprecation
No APIs or configuration keys are deprecated.
Test Plan
Unit tests
- TaskIOMetricGroupTest: aggregate-contributing vs target-only registration semantics; per-target snapshot returns live values; registering the same target id twice is idempotent; per-target metrics are visible to reporters via the standard Counter surface.
- StreamingJobGraphGeneratorTest: every NonChainedOutput carries a resolved JobVertexID for single-output, side-output, and broadcast-fan-out pipelines.
- JobDetailsHandlerTest: per-subtask IOMetrics with disjoint and overlapping target keys are merged per-key; empty maps pass through; the vertex-level scalar write-records remains consistent with sum(values) for non-broadcast topologies.
- SubtaskExecutionAttemptDetailsInfoTest: Jackson round-trip for empty and populated per-target maps; field name stability.
Integration tests
- PerTargetNumRecordsOutITCase (new, in flink-tests) runs two real jobs on MiniClusterExtension and asserts via RestClusterClient#getJobDetails that the full pipeline (runtime counter → IOMetrics snapshot → MutableIOMetrics aggregation → IOMetricsInfo DTO → JSON) reports the right values:
- Side output topology: source → process (split by i % 3) → {main-sink, side-sink} with 20 records; asserts aggregate write-records == 20, per-target map equals {mainSink: 13, sideSink: 7}, and sum(values) == aggregate.
- Broadcast topology: source.broadcast() → 3 × (map → DiscardingSink) with 20 records; asserts aggregate write-records == 20 (no double-count) and each of three per-target entries equals 20. This directly validates that the target-only registration mode does not pollute the aggregate.
Architecture tests
ArchUnit (flink-architecture-tests-production, flink-architecture-tests-test) passes with no new violations. No new archunit-violations/ store entries are added.
Rejected Alternatives
Currently, there are no rejected alternatives in this area.