Page tree
Skip to end of metadata
Go to start of metadata

Status

Current state: Accepted

Discussion threadhttps://lists.apache.org/thread.html/a4dd8e0c7b79350c59f5afefc1bc583dac99abcf94caaa8c22017974@%3Cdev.flink.apache.org%3E

JIRAhere (<- link to https://issues.apache.org/jira/browse/FLINK-XXXX)

Released: <Flink Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As part of the Executor effort (FLIP-73), we would like to introduce a number of ConfigOptions , so that all user-defined options can be mapped to configuration options. These reflect options that were previously configurable through the CLI (e.g. FlinkYarnSessionCli and ProgramOptions ), or programatically (e.g. the RemoteEnvironment ).

Given that this FLIP does not introduce any new feature, but rather tries to map already existing options to configuration options, we would like one major point of the discussion to be the naming conventions used here, so please comment also on the names used.

Proposed Changes

Suggested nameCommentValue TypePre-existing option
execution.targetThe deployment target for the execution, e.g. "local" for local execution.String - no defaultCLI: -m
execution.attachedSubmitting the job in attached  or detached  mode.Boolean - default falseCLI: -d
execution.shutdown-on-attached-exitIf the job is submitted in attached  mode, perform a best-effort cluster shutdown when the CLI is terminated abruptly, e.g. with CTRL-C.Boolean - default falseCLI: -sae
parallelism.default (existing)The parallelism for the execution.Integer - default 1CLI: -p
execution.savepoint.pathThe location of the savepoint to use to bootstrap the state when starting from a savepoint.String - no defaultCLI: -s
execution.savepoint.ignore-unclaimed-stateAllow to skip savepoint state that cannot be restored, e.g. if an operator was removed from the pipeline after the savepoint was taken.Boolean - falseCLI: -n
pipeline.jarsA semicolon-separated list of the jars to package with the job jars to be sent to the cluster.String - no defaultProgramOptions.jarPath and RemoteExecutor.jarfiles 
pipeline.classpathsA semicolon-separated list of the classpaths to package with the job jars to be sent to the cluster.String - no defaultProgramOptions.classpaths and RemoteExecutor.globalClasspaths
yarn.log-config-fileThe location of the log config file, e.g. the path to your log4j.properties for log4j.String - no defaultIt was silently discovered in the YarnClusterDescriptor.startAppMaster() 
yarn.ship-directoriesA semicolon-separated list of directories to be shipped to the YARN cluster.String - no defaultYarnCLI: -yt
yarn.flink-dist-jarThe location of the Flink dist jarString - no defaultYarnCLI: -yj
yarn.application.idThe application id of the running yarn cluster.String - no defaultYarnCLI: -yid
yarn.application.queueThe YARN queue on which to put the current pipeline.String - no defaultYarnCLI: -yqu
yarn.application.nameA custom name for your YARN applicationString - no defaultYarnCLI: -ynm
yarn.application.typeA custom type for your YARN applicationString - no defaultYarnCLI: -yat
yarn.application.node-labelSpecify YARN node label for the YARN applicationString - no defaultYarnCLI: -ynl

Compatibility, Deprecation, and Migration Plan

  • There are no compatibility concerns as this FLIP does not change any existing behaviour but it simply maps CLI and programatic options to Configuration Options.

Future Work

Our intention is, in a future FLIP, to remove the following options:

1) execution.attached : as, conceptually, the JobClient API (FLIP-74) will allow the user to decide on-the-fly if he/she wants to wait for the result of a job.
2) execution.shutdown-on-attached-exit

  • No labels