Motivation
As part of the Executor effort (FLIP-73), we would like to introduce a number of ConfigOptions , so that all user-defined options can be mapped to configuration options. These reflect options that were previously configurable through the CLI (e.g. FlinkYarnSessionCli and ProgramOptions ), or programatically (e.g. the RemoteEnvironment ).
Given that this FLIP does not introduce any new feature, but rather tries to map already existing options to configuration options, we would like one major point of the discussion to be the naming conventions used here, so please comment also on the names used.
Proposed Changes
| Suggested name | Comment | Value Type | Pre-existing option |
|---|---|---|---|
| execution.target | The deployment target for the execution, e.g. "local" for local execution. | String - no default | CLI: -m |
| execution.attached | Submitting the job in attached or detached mode. | Boolean - default false | CLI: -d |
| execution.shutdown-on-attached-exit | If the job is submitted in attached mode, perform a best-effort cluster shutdown when the CLI is terminated abruptly, e.g. with CTRL-C. | Boolean - default false | CLI: -sae |
| parallelism.default (existing) | The parallelism for the execution. | Integer - default 1 | CLI: -p |
| execution.savepoint.path | The location of the savepoint to use to bootstrap the state when starting from a savepoint. | String - no default | CLI: -s |
| execution.savepoint.ignore-unclaimed-state | Allow to skip savepoint state that cannot be restored, e.g. if an operator was removed from the pipeline after the savepoint was taken. | Boolean - false | CLI: -n |
| pipeline.jars | A semicolon-separated list of the jars to package with the job jars to be sent to the cluster. | String - no default | ProgramOptions.jarPath and RemoteExecutor.jarfiles |
| pipeline.classpaths | A semicolon-separated list of the classpaths to package with the job jars to be sent to the cluster. | String - no default | ProgramOptions.classpaths and RemoteExecutor.globalClasspaths |
| yarn.log-config-file | The location of the log config file, e.g. the path to your log4j.properties for log4j. | String - no default | It was silently discovered in the YarnClusterDescriptor.startAppMaster() |
| yarn.ship-directories | A semicolon-separated list of directories to be shipped to the YARN cluster. | String - no default | YarnCLI: -yt |
| yarn.flink-dist-jar | The location of the Flink dist jar | String - no default | YarnCLI: -yj |
| yarn.application.id | The application id of the running yarn cluster. | String - no default | YarnCLI: -yid |
| yarn.application.queue | The YARN queue on which to put the current pipeline. | String - no default | YarnCLI: -yqu |
| yarn.application.name | A custom name for your YARN application | String - no default | YarnCLI: -ynm |
| yarn.application.type | A custom type for your YARN application | String - no default | YarnCLI: -yat |
| yarn.application.node-label | Specify YARN node label for the YARN application | String - no default | YarnCLI: -ynl |
Compatibility, Deprecation, and Migration Plan
- There are no compatibility concerns as this FLIP does not change any existing behaviour but it simply maps CLI and programatic options to Configuration Options.
Future Work
Our intention is, in a future FLIP, to remove the following options:
1) execution.attached : as, conceptually, the JobClient API (FLIP-74) will allow the user to decide on-the-fly if he/she wants to wait for the result of a job.
2) execution.shutdown-on-attached-exit