Current stateUnder Vote

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]


Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).


A previous contribution I made to was suggested as by a member of confluent as being a nice addition to the out of the box Kafka Connect SMTs. The discussion is here The proposed change would add a new Kafka Connect SMT which would allow for keys or values to be hashed using the configured algorithm. The addition of this would allow for sensitive fields to be obfuscated to prevent private information such as ssn or other identifiable information from flowing.

Currently there exists a MaskField SMT but that would completely remove the value by setting it to an equivalent null value. One problem with this would be that you’d not be able to know in the case of say a password going through the mask transform it would become "" which could mean that no password was present in the message, or it was removed. However this hash transformer would remove this ambiguity if that makes sense. The proposed hash functions would be MD5, SHA1, SHA256. which are all supported via MessageDigest.

Public Interfaces

One new class connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ and a helper class connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util/ are proposed additions. No modifications required to existing interfaces.

Proposed Changes

The proposed change can be viewed here it would allow for hashing specific fields within a kafka connect message value, or the entire value, additionally the key could be hashed if desired. The configuration would look something like the folllowing. where type is Either Key or Value.


Based on feedback from Gunnar Morling ( I think that this should also support
1) an optional salt, which would be set via transforms.HashEmail.salt
2) a comma separated list of fields where a period is used to denote nested fields
Given these suggestions


Compatibility, Deprecation, and Migration Plan

  • NA

Rejected Alternatives


  • No labels