This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • KIP 172: Add regular-expression topic support for sink connector
Skip to end of metadata
Go to start of metadata

Status

Current state: Discarded, and superseded by KIP-215: Add topic regex support for Connect sinks

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

For now we provide topic names in a properties file as following:

topics=test1,test2

If there are a few topics to deal with, It's ok to list all the topics in a comma-seperated way. But when you have to deal with hundreds or thousands of topics, it doesn't help. In those situations, we would like to use a wildcard expression to specify topics.

Public Interfaces

  • client configuration file

Proposed Changes

  • add an aditional option "topics.regex"
  • a user can only specify one of "topics" or "topics.regex"
  • if a user specifies "topics" in their config file, it works the same way as the current version. 
  • if a user specifies "topics.regex", it can handle all the topics matching the regular expressions.

Compatibility, Deprecation, and Migration Plan

  • What impact (if any) will there be on existing users?
    • N/A
  • If we are changing behavior how will we phase out the older behavior?
    • N/A
  • If we need special migration tools, describe them here.
    • Not needed
  • When will we remove the existing behavior?
    • N/A

Rejected Alternatives

It is possible to update the interpretation of the existing option "topics" in a way that our users can specify reglar expressions there. But since there are some special characters in the regular expression that can be used in topic names, eg. "." and "-",  we need some workaround. Also this aproach has some effects on existing users. 

 

  • No labels