Status

Current state: Voting

Discussion thread: here

Voting thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-932: Queues for Kafka introduced share groups and included the kafka-share-groups.sh  tool for initializing and resetting the share-partition start offsets. This was inspired by the equivalent kafka-consumer-groups.sh  tool for consumer groups, but it did not include all of the options for resetting options which are available for consumer groups.

When users are migrating from a consumer group to a share group, it is natural that they would want to initialize the share group with the committed offsets from the consumer group. Programmatically, this is easy, but there is no way using kafka-share-groups.sh  to achieve this. This KIP adds extra options to that tool to make this kind of migration easy.

Proposed Changes

Additional options are added to kafka-share-groups.sh  to give more flexibility for initialization of share group offsets. These options match those which exist in kafka-consumer-groups.sh .

First, --to-offset lets the user specify a specific offset. For example, the following command will set the SPSO for share group SG1 topic T1 partition 0 to offset 1234.

$ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group SG1 --topic T1:0 --to-offset 1234 --execute 

Second, --from-file  lets the user provide a CSV file containing lines topic,partition,offset  to specify the topic names, partitions and offsets to reset. This makes the following migration from a consumer group to a share group possible.

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group CG1 --all-topics --to-current --export --dry-run > FILENAME 

$ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group SG1 --from-file FILENAME --execute 

In order to facilitate migration in the other direction, or from one share group to another, the --export  option is provided, along with --to-current . The purpose of --to-current  is to export the current offsets, as opposed to the earliest or latest offsets.

$ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group SG1 --all-topics --to-current --export --dry-run > FILENAME

$ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group SG2 --from-file FILENAME --execute 

Public Interfaces

Command-line tools

kafka-share-groups.sh

The following new options are added to this tool.

Compatibility, Deprecation, and Migration Plan

New options are added to an existing tool. The underlying admin APIs are unchanged and the options will work with Apache Kafka 4.2 and later.

Test Plan

Unit tests will be written to test the new options.

Rejected Alternatives

None