Created on behalf of Mario Fiore Vitale as we currently can't create new Confluence accounts.

Status

Current state: Accepted

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-581: Value of optional null field which has default value introduced the configuration property replace.null.with.default to JsonConverter to choose whether to replace fields that have a default value and that are null to the default value. When set to true (default), the field default value is used, otherwise null is used. However if a user sets the replace.null.with.default to false on JsonConverter and then uses some specific types of transformation the behavior is not maintained.

Sometimes this is directly related to the behavior of the SMT (for example, the ExtractField SMT may extract the default value of the configured field, instead of null), but other times, it's orthogonal to the purpose of the SMT (for example, the SetSchemaMetadata SMT creates a copy of the original Struct in order to utilize a new schema, and that copying process accidentally replaces null fields in the original Struct with default values in the returned Struct).

With this proposal we want to add the same configuration to the relevant transformations.

Public Interfaces

Add a new configuration to InsertField, ExtractField, HeaderFrom, Cast, SetSchemaMetadata, TimestampConverter, MaskField, ValueToKey and ReplaceField:

Name: replace.null.with.default
Description: Whether to replace fields that have a default value and that are null to the default value. When set to true, the default value is used, otherwise null is used.
Type: Boolean
Default: true

Proposed Changes

The proposed changes is to add the replace.with.default configuration with true as default value to maintain the old behavior as per default as done for the JsonConverter.

Compatibility, Deprecation, and Migration Plan

The changes are backwards compatible since the old behavior in case the new configuration is not set will be the default. 

Test Plan

All affected SMTs currently have their own unit test suites, each of which will be augmented to cover this new logic.

Rejected Alternatives

None