Page History

Document the state by adding a label to the FLIP page with one of "discussion", "accepted", "released", "rejected".

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

public static final ConfigOption<InferMode> TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM_MODE =
           .key("table.exec.hive.infer-source-parallelism.mode")
                   .enumType(InferMode.class)
                   .defaultValue(InferMode.DYNAMIC)
                   .withDescription(
                     "An option for selecting the hive source parallelism
                     inference mode: 'static' denotes static inference, where 
                     flink will decide source parallelism at job creation stage; 
                     'dynamic' represents dynamic inference, which will decide 
                     parallelism at job execution stage. In terms of functionality, 
                     the static mode is a subset of the dynamic mode;
                     'none' represents disabling inference.
                     Note: at the current stage, it only works when the value of option 
                     'table.exec.hive.infer-source-parallelism.enabled' is 
true, until it                   trueis removed.");

public enum InferMode implements DescribedEnum {
        STATIC("static", text("Static parallelism inference mode for hive source.")),
        DYNAMIC("dynamic", text("Dynamic parallelism inference mode for hive source."));
        NONE("none", text("Disable parallelism inference for hive source."));

        private final String value;

        private final InlineElement description;

        InferMode(String value, InlineElement description) {
            this.value = value;
            this.description = description;
        }

        @Override
        public String toString() {
            return value;
        }

        @Override
        public InlineElement getDescription() {
            return description;
        }
    }

In addition, the current configuration option `table.exec.hive.infer-source-parallelism` acts not only as a switch to enable or disable hive source parallelism inference but also as a prefix for other configuration options (e.g. `table.exec.hive.infer-source-parallelism.max`). This practice does not conform to the standards of YAML specifications. Besides, with the introduction of `table.exec.hive.infer-source-parallelism.mode`, we can disable parallelism inference by specifying InferMode.NONE, thus substituting the function of `table.exec.hive.infer-source-parallelism`. Therefore, we plan to mark `table.exec.hive.infer-source-parallelism` as Deprecated in this FLIP. Before this configuration option is completely phased out, `table.exec.hive.infer-source-parallelism` will still serve as the master main switch to enable or disable parallelism inference. 

The temporary coexistence logic of the two configuration options is as follows:

Behavior Change

After introducing this FLIP, the behavior changes are as follows:

...

A new configuration option `table.exec.hive.infer-source-parallelism.mode` will be introduced to distinguish between the static and dynamic modes. After this FLIP, users can still revert to static inference by setting `table.exec.hive.infer-source-parallelism.mode` to InferMode.STATIC, or they can disable parallelism inference altogether by setting `table.exec.hive.infer-source-parallelism.mode` to InferMode.NONE or setting `table.exec.hive.infer-source-parallelism.enabled` to false.

Due to the non-compliance of the configuration `table.exec.hive.infer-source-parallelism` with YAML standard specifications, it will be marked as deprecated.

...