...
Public Interfaces
New Config Option
To facilitate a smooth transition, it is planned to introduce a new configuration optionTo facilitate a smooth transition, it is planned to introduce a new configuration option `table.exec.hive.infer-source-parallelism.mode`, and the default value will be 'InferMode.DYNAMIC':
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public static final ConfigOption<InferMode> TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM_MODE =
ConfigOptions.key("table.exec.hive.infer-source-parallelism.mode")
.enumType(InferMode.class)
.defaultValue(InferMode.DYNAMIC)
.withDescription(
"An option for selecting the hive source parallelism
inference mode: 'static' denotes static inference, where
flink will decide source parallelism at job creation stage;
'dynamic' represents dynamic inference, which will decide
parallelism at job execution stage. In terms of functionality,
the static mode is a subset of the dynamic mode.
Note: it only works when the value of option
'table.exec.hive.infer-source-parallelism.enabled' is
true."); |
...
In addition, the current configuration option `table.exec.hive.infer-source-parallelism` acts not only as a switch to enable or disable hive source parallelism inference but also as a prefix for other configuration options (e.g. `table.exec.hive.infer-source-parallelism.max`). This practice does not conform to the standards of YAML specifications. Therefore, we plan to address this issue in this FLIP by introducing a new configuration item, `table.exec.hive.infer-source-parallelism.enabled`, as a replacement. The old configuration item `table.exec.hive.infer-source-parallelism` will continue to be effective but will be marked as deprecated. By the way, the default value for the new configurations is the same as the old ones, which is true
, meaning that the automatic parallelism inference is enabled by default.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public static final ConfigOption<Boolean> TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM_ENABLED = key("table.exec.hive.infer-source-parallelism.enabled") .booleanType() .defaultValue(true) .withDeprecatedKeys("table.exec.hive.infer-source-parallelism") .withDescription( "If is false, parallelism of source are set by config.\n" + "If is true, source parallelism is inferred according to splits number.\n"); |
Behavior Change
After introducing this FLIP, the behavior changes are as follows:
...