Status
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The goal of those changes is to improve the TableConfig so it possible to configure current planner settings through ConfigOptions. It also aims to improve the configuration user experience.
Public Interfaces
New ConfigOptions
key | setting | type | Class to place it in |
---|---|---|---|
table.exec.min-idle-state-retention | TableConfig#setIdleStateRetentionTime | Duration | ExecutionConfigOptions |
table.exec.max-idle-state-retention | TableConfig#setIdleStateRetentionTime | Duration | ExecutionConfigOptions |
table.planner.sql-dialect | TableConfig#setSqlDialect | Enum<SqlDialect> | PlannerConfigOptions |
table.planner.time-zone | TableConfig#setLocalTimeZone | String (ZoneId.of()) | PlannerConfigOptions |
table.planner.max-generated-code | TableConfig#setMaxGeneratedCodeLength | Integer | PlannerConfigOptions |
table.planner.math-context.precision | TableConfig#setDecimalContext | Integer (precision of MathContext) | PlannerConfigOptions |
table.planner.math-context.rounding-mode | TableConfig#setDecimalContext | Enum<MatchContext.RoundingMode> | PlannerConfigOptions |
Changes to TableConfig
- change the class to be an interface.
- Remove getConfiguration method
- Remove getDefault method
- Make TableConfig extend from ReadableConfig & WritableConfig
- Deprecate {{setNullCheck}}
@PublicEvolving public interface TableConfig extends ReadableConfig, WritableConfig { /* Old setters/getters that will be stored in a Configuration in the TableConfigImpl. */ ..... /** * Returns the current SQL dialect. */ SqlDialect getSqlDialect(); /** * Sets the current SQL dialect to parse a SQL query. Flink's SQL behavior by default. */ void setSqlDialect(SqlDialect sqlDialect); ...... /* Methods inherited from ReadableConfig/WritableConfig <T> TableConfig set(ConfigOption<T> option, T value); <T> Optional<T> getOptional(ConfigOption<T> option); <T> T get(ConfigOption<T> option); */ /* Methods to be dropped: public static TableConfig getDefault(); public Configuration getConfiguration(); */ }
Changes to (Stream)TableEnvironment
- drop deprecated
create()
methods that createTableEnvironment
with a givenTableConfig
Proposed Changes
First of the suggestion is to change the TableConfig to be an interface. This makes future changes to the TableConfig easier. It also moves the control over the lifecycle of the TableConfig to the {{TableEnvironment}}. Along with this change we will remove deprecated methods that create StreamTableEnvironment with a TableConfig
We suggest to remove the getConfiguration
method as it exposed too much of internal implementation. We might want to change the underlying way how we persist/store options, which might be hard to do with the current contract of getConfiguration
which gives a read/write access. The write access should happen either through addConfiguration
or TableConfig#set
Both approaches are easier from user perspective than the current solution tEnv.getConfig().getConfiguration().set(CoreOptions.DEFAULT_PARALLELISM, 128)
For setting a batch of properties (or string properties) users can use addConfiguration
method:
tEnv.getConfig().addConfiguration( new Configuration() .set(CoreOptions.DEFAULT_PARALLELISM, 128) .set(PipelineOptions.AUTO_WATERMARK_INTERVAL, Duration.ofMillis(800)) .set(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(30)) ); // or string - string options Configuration config = new Configuration(); config.setString("default.parallelism", "128"); config.setString("pipeline.auto-watermark-interval", "800 ms"); config.setString("execution.checkpointing.interval", "30 s"); tEnv.getConfig().addConfiguration(config);
The other option is to set options on TableConfig
explicitly:
tEnv.getConfig() .set(CoreOptions.DEFAULT_PARALLELISM, 128) .set(PipelineOptions.AUTO_WATERMARK_INTERVAL, Duration.ofMillis(800)) .set(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(30));
Compatibility, Deprecation, and Migration Plan
This flip suggests two breaking changes:
- TableConfig can no longer be instantiated. → This was already discouraged.
- Remove
getConfiguration
method. There are alternatives to this method. Either throughaddConfiguration
orTableConfig#set