Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
When doing the planning phase of a Flink Table job, among other things, we execute an expression reducer, which aims to do constant folding. One of these optimizations is to invoke deterministic functions if all of their arguments are constants.
This is fine for local functions, but becomes more troublesome for UDFs which invoke RPCs or have other side effects, which you either don’t want to occur during planning time, or where it’s important that it happens on a per result row basis.
Currently, FunctionDefinition.isDeterministic()
exists, where you can override a definition to indicate that a function call is not deterministic, and this will avoid invoking the expression reducer. Unfortunately, this also indicates that the results are not deterministic and will have a material effect on the planning of the job.
A new mechanism is required to differentiate the scenario where an expression with a function call is deterministic, but still should not be reduced anyway.
Public Interfaces
The proposal is to add a method call to FunctionDefinition
:
public interface FunctionDefinition { ... /** * If the constant-folding should be run during planning time on calls to * this function. If not, the expression will be left as-is and the call * will be made during runtime. * * @return Whether invocations can be reduced during planning time */ default boolean supportsConstantFolding() { return true; } }
Proposed Changes
ExpressionReducer
already supports avoiding constant-folding of certain expressions, so this needs to be extended to support function calls. To do this, we need to be able to identify an expression with a function call which has supportsConstantFolding()
returning false. Such an visitor could be written as below.
private static class SupportsConstantFoldingExpressionVisitor extends RexDefaultVisitor<Boolean> { @Override public Boolean visitNode(RexNode rexNode) { return true; } private boolean supportsConstantFolding(RexCall call) { FunctionDefinition definition = ShortcutUtils.unwrapFunctionDefinition(call); return definition == null || definition.supportsConstantFolding(); } @Override public Boolean visitCall(RexCall call) { boolean supportsConstantFolding = supportsConstantFolding(call); return supportsConstantFolding && (call.getOperands().stream().allMatch(node -> node.accept(this))); } }
Compatibility, Deprecation, and Migration Plan
Since all existing FunctionDefinition
s will have a default supportsConstantFolding
()
returning true, the behavior will default to what it is today, that the ExpressionReducer
can do constant-folding and invocation.
Test Plan
Change will be covered by unit tests which inspect the plans for function calls which both have supportsConstantFolding
()
return both possibilities, and validate the use of constant-folding.
Rejected Alternatives
None