1. Motivation

Currently, the jobType has 2 types in Flink: STREAMING and BATCH. They work on completely different principles, such as: scheduler, shuffle, join, etc. These differences lead to different troubleshooting processes, so when users are maintaining a job or troubleshooting, it's needed to know whether the current job is a STREAMING or BATCH job. Unfortunately, Flink WebUI doesn't expose it to the users so far.

Also, ExecutionMode is related to DataSet api, it has been marked as @Deprecated in FLINK-32558 - Getting issue details... STATUS (1.18), but it's still shown in Flink WebUI.

Clarification on Execution mode.

• Flink 1.19 website still mentions the Execution mode, but it actually matches the JobType in the Flink code. Both of them have 2 types: STREAMING and BATCH.
• execution.runtime-mode can be set to 3 types: STREAMING, BATCH and AUTOMATIC.
  • But the jobType will be inferred as STREAMING or BATCH when execution.runtime-mode is set to AUTOMATIC.
• The ExecutionMode I describe is: code link , as we can see, ExecutionMode has 4 enums: PIPELINED, PIPELINED_FORCED, BATCH and BATCH_FORCED.  And we can see a flink streaming job from Flink WebUI, the Execution mode is PIPELINE instead of STREAMING.
• What this proposal wants to do is to remove the ExecutionMode with four enumerations on Flink WebUI and introduce the JobType with two enumerations (STREAMING or BATCH).
  • STREAMING or BATCH is clearer and more accurate for users.

I propose 4 public changes in this FLIP:

• Add jobType in rest api
• Show the JobType on Flink WebUI properly
• Remove Execution mode in Flink WebUI
• Deprecate execution-mode in ExecutionConfigInfo related rest api(Remove it in flink 2.x)

POC branch: https://github.com/1996fanrui/flink/tree/flip-441/poc (Latest 2 commits)

2. Public Interfaces

2.1 Add jobType in rest api

As Zhuzhu suggested in this comment: jobType is an automatically derived property instead of a user setting. Therefore, I would not put it in ExecutionConfig.

We can add jobType in JobDetailsInfo instead of ExecutionConfig.

The corresponding rest api is /jobs/:jobid with GET request, the json key is "job-type".

2.2 Show JobType on Flink Web UI

2.3 Remove Execution mode in Flink WebUI

2.4 Deprecate execution-mode in ExecutionConfigInfo related rest api.

Mark execution-mode as @Deprecate in ExecutionConfigInfo. ( flink 1.20)

Remove it in flink 2.0 or 2.1

3. Proposed Changes

• Adding getJobType for AccessExecutionGraph interface:
  • DefaultExecutionGraph and ArchivedExecutionGraph need to add jobType field.

4. Compatibility, Deprecation, and Migration Plan

All WebUI changes take effect for History server, the history server might show the job overview with old flink version. These jobs don't have jobType in the archived files of History server, we can hide the JobType on WebUI when json doesn't have job-type.

*ngIf="jobDetail['job-type'] != null" is easy to hide it.

5. Test Plan

• Unit test
• Test running job manually, check if the JobType is expected when execution.runtime-mode is STREAMING, BATCH and AUTOMATIC
• Test history server manually
  • Check if JobType is expected when execution.runtime-mode is STREAMING, BATCH and AUTOMATIC
  • Check if JobType is hidden when job version before flink-1.20

6. Rejected Alternatives

• Show the jobType in Execution Configuration
  
  • jobType is an automatically derived property instead of a user setting.