Status

Current state: Under Discussion

Discussion thread: here

JIRA: here


Motivation

Currently, ConnectorUtils in Kafka Connect API provides functionality for assigning continuous elements in groups. However, there are scenarios where this approach falls short, particularly when a more balanced distribution of elements across groups is desired. In such cases, a round-robin assignment strategy can offer a more equitable allocation of elements.

Consider a scenario where a Kafka Connect connector needs to distribute data across multiple destinations or partitions in a balanced manner. The existing functionality of ConnectorUtils may result in suboptimal assignments, leading to uneven distribution and potential performance bottlenecks. Introducing a round-robin assignment strategy would enable connectors to achieve a more balanced distribution of elements, thereby enhancing overall system efficiency and performance.

For example, suppose we have 5 elements {a, b, c, d, e} and want to assign them into 3 groups. In current assignment strategy, the result will be {{a, b}, {c, d}, {e}}, we keep current strategy and add a round-robin strategy, the result will be {{a, d}, {b, e}, {c}}. In this example, elements "a", "b", and "c" are evenly distributed among three groups using the round-robin assignment strategy.

Public Interfaces

Will add a new static method to connect/api/src/main/java/org/apache/kafka/connect/util/ConnectorUtils.java

public static <T> List<List<T>> groupElementsRoundRobin(List<T> elements, int numGroups)

Proposed Changes

Extend ConnectorUtils with a round-robin assignment strategy to facilitate more balanced distribution of continuous elements across different groups. This strategy would evenly distribute elements among the available groups, ensuring a more equitable allocation and minimizing potential imbalances.

Compatibility, Deprecation, and Migration Plan

This proposal does not introduce any compatibility issues or deprecations. Existing functionality in ConnectorUtils will remain unchanged, and the round-robin assignment strategy will be introduced as an additional option.

Test Plan

No other classes depend on this util. The change only adds a new static method and won't break existing method. So, unit tests are enough to cover all cases.

Rejected Alternatives

Define a Strategy interface, then user can pass different strategy implementation. This solution is incompatible with current ConnectorUtils. Current strategy and round-robin strategy are enough for most scenarios, no other strategies are needed, so there is no need to make extension on strategy.



  • No labels