You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 19 Next »

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: "Under Discussion" (Draft)

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently Mirrormaker uses deprecated alterConfigs API when syncing topic configurations for broker compatibility to 0.11.0. In addition, Mirromaker has a configurable ConfigPropertyFilter class to select the topic configuration properties to replicate and uses `org.apache.kafka.connect.mirror.DefaultConfigPropertyFilter|

by default. The default class excludes certain topic configurations from replication such as follower.replication.throttled.replicas and leader.replication.throttled.replicas. 

However, the deprecated alterConfigs() API replaces any existing configuration with the new configuration. Due to this, additional configurations set on a remote topic get cleared up when MirrorMaker syncs topic configurations whether they are excluded from replication or not. For example this prevents running Cruise Control on the target cluster as it may set follower.replication.throttled.replicas and leader.replication.throttled.replicas.

MirrorMaker should not clear topic configurations that are excluded from replication.

Public Interfaces

This KIP proposes to migrate to incrementalAlterConfigs API for syncing topic configurations in MirrorMaker. The incrementalAlterConfigs API has been around for several years since it was introduced in Kafka 2.3.0 and addresses the shortcoming of the deprecated alterConfigs  API. In order not to break the compatibility, we will introduce a new setting to MirrorMaker to give the option to enable incrementalAlterConfigs API for syncing topic configuraMirrorMaker syncs topic configurations whether they are filter or not. This new setting is expected to serve as a temporary measure until the next major release when the API is always used. 

Proposed Changes

1) Add a new method updateTopicConfigsUsingIncrementalAlterConfigs() to MirrorSourceConnector.java which takes a list topic configurations as a parameter and calls incrementalAlterConfigs() to the target cluster. 


2) Add a new configuration setting to MirrorMaker:


Name: use.incremental.alter.configs

Description: Deprecated. Whether to automatically use incrementalAlterConfigs API for syncing topic configurations. This configuration will be removed in Kafka 4.0 and the incrementalAlterConfigs API will be used by default. Users should make sure that the target cluster is running at least Kafka 2.3.0 or later. 

Type: Boolean

Default: false

When set to false, MirrorMaker will keep using deprecated alterConfigs API for syncing topic configurations.

When explicitly set to true, MirrorMaker will use incrementalAlterConfigs API for syncing topic configurations. If it receives an error from an incompatible broker, the MirrorMaker will report this to the user and fail the connector. 


3) Update syncTopicConfigs()  in MirrorSourceConnector.java to call the updateTopicConfigsUsingIncrementalAlterConfigs() instead of updateTopicConfigs() if enable.incremental.alter.configs is set to true.


4) From Kafka 4.0, syncTopicConfigs() will only call  updateTopicConfigsUsingIncrementalAlterConfigs() by default. The existing updateTopicConfigs() method will be removed.

Compatibility, Deprecation, and Migration Plan

By default, the new setting will be set to false so it does not change the current behaviour. This setting will be marked as deprecated immediately. 

In the next major version:

  • use.incremental.alter.configs will be removed 
  • MirrorMaker will use the incrementalAlterConfigs API to sync topic configurations. 

Users running the latest MirrorMaker connector against a target cluster with brokers older than Kafka 2.3.0, will have to upgrade.

Test Plan

Currently, the integration tests for MirrorMaker do not check if the target topic has the expected configurations after applying configuration filters, hence this bug was not caught. A new integration test that checks target topic configurations once they have been synced will be added. 

Rejected Alternatives

  • We could have changed the current updateConfigs() method to call describeConfigs API before alterConfigs  so that it had an incremental mode. This would have avoided the migration, however, the topics configurations could change between describeConfigs  and alterConfigs  calls so we could still end up syncing incorrect topic configurations. We would need to stop using the deprecated API at some point anyway. 
  • No labels