...

Public Interfaces

New Config Option

To facilitate a smooth transition, it is planned to introduce a new configuration optionTo facilitate a smooth transition, it is planned to introduce a new configuration option `table.exec.hive.infer-source-parallelism.mode`, and the default value will be 'InferMode.DYNAMIC':

public static final ConfigOption<InferMode> TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM_MODE =
           ConfigOptions.key("table.exec.hive.infer-source-parallelism.mode")
                   .enumType(InferMode.class)
                   .defaultValue(InferMode.DYNAMIC)
                   .withDescription(
                     "An option for selecting the hive source parallelism
                     inference mode: 'static' denotes static inference, where 
                     flink will decide source parallelism at job creation stage; 
                     'dynamic' represents dynamic inference, which will decide 
                     parallelism at job execution stage. In terms of functionality, 
                     the static mode is a subset of the dynamic mode.
                     Note: it only works when the value of option 
                     'table.exec.hive.infer-source-parallelism.enabled' is 
                     true.");

...

In addition, the current configuration option `table.exec.hive.infer-source-parallelism` acts not only as a switch to enable or disable hive source parallelism inference but also as a prefix for other configuration options (e.g. `table.exec.hive.infer-source-parallelism.max`). This practice does not conform to the standards of YAML specifications. Therefore, we plan to address this issue in this FLIP by introducing a new configuration item, `table.exec.hive.infer-source-parallelism.enabled`, as a replacement. The old configuration item `table.exec.hive.infer-source-parallelism` will continue to be effective but will be marked as deprecated. By the way, the default value for the new configurations is the same as the old ones, which is true, meaning that the automatic parallelism inference is enabled by default.

public static final ConfigOption<Boolean> TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM_ENABLED =
            key("table.exec.hive.infer-source-parallelism.enabled")
                    .booleanType()
                    .defaultValue(true)
                    .withDeprecatedKeys("table.exec.hive.infer-source-parallelism")
                    .withDescription(
                            "If is false, parallelism of source are set by config.\n"
                                    + "If is true, source parallelism is inferred according to splits number.\n");

Behavior Change

After introducing this FLIP, the behavior changes are as follows:

...