Document the state by adding a label to the FLIP page with one of "discussion", "accepted", "released", "rejected".
Discussion thread | https://lists.apache.org/thread/xxx4k1qx6lodhbkknkhjyl0lq9bx8fcpjvn | ||||||||
---|---|---|---|---|---|---|---|---|---|
Vote thread | https://lists.apache.org/thread/lktnb162l2z3042m76to6xfbsdndy4r7 | ||||||||
JIRA |
| ||||||||
Release | 1.20 |
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public static final ConfigOption<InferMode> TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM_MODE = .key("table.exec.hive.infer-source-parallelism.mode") .enumType(InferMode.class) .defaultValue(InferMode.DYNAMIC) .withDescription( "An option for selecting the hive source parallelism inference mode: 'static' denotes static inference, where flink will decide source parallelism at job creation stage; 'dynamic' represents dynamic inference, which will decide parallelism at job execution stage. In terms of functionality, the static mode is a subset of the dynamic mode; 'none' represents disabling inference. Note: at the current stage, it only works when the value of option 'table.exec.hive.infer-source-parallelism.enabled' is true, until it trueis removed."); |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public enum InferMode implements DescribedEnum { STATIC("static", text("Static parallelism inference mode for hive source.")), DYNAMIC("dynamic", text("Dynamic parallelism inference mode for hive source.")); NONE("none", text("Disable parallelism inference for hive source.")); private final String value; private final InlineElement description; InferMode(String value, InlineElement description) { this.value = value; this.description = description; } @Override public String toString() { return value; } @Override public InlineElement getDescription() { return description; } } |
In addition, the current configuration option `table.exec.hive.infer-source-parallelism` acts not only as a switch to enable or disable hive source parallelism inference but also as a prefix for other configuration options (e.g. `table.exec.hive.infer-source-parallelism.max`). This practice does not conform to the standards of YAML specifications. Besides, with the introduction of `table.exec.hive.infer-source-parallelism.mode`
, we can disable parallelism inference by specifying InferMode.NONE, thus substituting the function of `table.exec.hive.infer-source-parallelism`
. Therefore, we plan to mark `table.exec.hive.infer-source-parallelism`
as Deprecated in this FLIP. Before this configuration option is completely phased out, `table.exec.hive.infer-source-parallelism`
will still serve as the main switch to enable or disable parallelism inference.
The temporary coexistence logic of the two configuration options is as follows:
Inference Mode Relationship Table | |||
---|---|---|---|
`table.exec.hive.infer-source-parallelism` | `table.exec.hive.infer-source-parallelism.mode` | ||
DYNAMIC | STATIC | NONE | |
true | dynamic parallelsim inference | static parallelism inference | disabled |
false | disabled | disabled | disabled |
Behavior Change
After introducing this FLIP, the behavior changes are as follows:
...
A new configuration option `table.exec.hive.infer-source-parallelism.mode`
will be introduced to distinguish between the static and dynamic modes. After this FLIP, users can still revert to static inference by setting `table.exec.hive.infer-source-parallelism.mode`
to InferMode.STATIC
, or they can disable parallelism inference altogether by setting `table.exec.hive.infer-source-parallelism.mode`
to InferMode.NONE or setting `table.exec.hive.infer-source-parallelism.enabled` to false
.
Due to the non-compliance of the configuration `table.exec.hive.infer-source-parallelism`
with YAML standard specifications, it will be marked as deprecated.
...