This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • KIP-154 Add Kafka Connect configuration properties for creating internal topics

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

 

Status

Current stateUnder DiscussionAdopted

Discussion thread: here

Vote thread: here

JIRA: KAFKA-4667

Released: 0.11.0.0

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

  • config.storage.replication.factor will specify the replication factor for the topic used to store connector configurations. The default value will be ‘3’. The configuration importance is 'low'.

  • offset.storage.replication.factor will specify the replication factor for the topic used to store connector offset information. The default value will be ‘3’. The configuration importance is 'low'.

  • offset.storage.partitions will specify the number of partitions for the topic used to store connector offset information. The default value will be ‘25’. The configuration importance is 'low'.

  • status.storage.replication.factor will specify the replication factor for the topic used to store connector and task status information. The default value will ‘3’. The configuration importance is 'low'.

  • status.storage.partitions will specify the number of partitions for the topic used to store connector and task status information. The default value will be ‘5’. The configuration importance is 'low'.

The names of these configuration properties intentionally mirror the existing config.storage.topic, offset.storage.topic, and status.storage.topic configurations.

Note that the replication factor may not be larger than the number of available Kafka brokers in the cluster. In such cases, Kafka Connect will be unable to create the topics and will fail with an error stating this limitation, and the user will need to explicitly set the aforementioned configurations.

It is not a mistake that a config.storage.partitions configuration property is not mentioned above, since such a configuration property is not useful as the configuration storage mechanism requires a single partition.

Compatibility, Deprecation, and Migration Plan

...