Motivation

We saw quite some examples when customers were confused about run​, and run-application​ in the Flink CLI and what happens when they are deploying via the former or latter. When Application Mode (AM) was introduced, run-application served an explicit and dedicated way to differentiate the new deployment mode. But since then, Per-Job mode was deprecated in Flink 1.15 and will be removed when all the blockers are tackled [1].

Considering these, it could simplify the CLI experience to merge the two different run options and simply validate the deployment mode via the --target argument in Flink 2.0 and after that it will be possible to drop run-application completely with Per-Job mode.

[1] https://issues.apache.org/jira/browse/FLINK-26000

Public Interfaces

CLI

Add deprecated warning to run-application print help.

Update the run CLI help accordingly.

Configuration

Update the execution.target config option.

Proposed Changes

Validate deployment target in CLI frontend

Currently, deployment mode validations are happening during cluster deployment in their specific ClusterDescriptor​ implementation [2]. If we validate the deployment target in CliFrontend, we can have one single method that can decide what to do. There is a small PoC, which contains the mentioned validation and a common method [3].

[2] https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206

[3] https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0

Example

A complete K8s AM deployment command example with the introduced changes:

./bin/flink run \

  --target kubernetes-application \

  -Dkubernetes.cluster-id=flink-cluster \

  -Dkubernetes.container.image.ref=flink:1.19 \

  -Duser.artifacts.raw-http-enabled=true \

  http://host.minikube.internal:9999/data/TopSpeedWindowing.jar

A complete YARN AM deployment command example with the introduced changes:

./bin/flink run --target yarn-application /tmp/WordCount.jar

Compatibility, Deprecation, and Migration Plan

With implementing the mentioned changes, run-application can be deprecated in Flink 1.20.

When Per-Job mode is dropped in a future Flink 2.x version, run-application should be removed at that point as well.

Test Plan

• All existing tests should pass.
• Add new unit/integration tests that cover the added flink run AM mode capabilities.

Rejected Alternatives

n/a