...
Code Block |
---|
Key => GroupId TransactionalId GroupId => String TransactionalId => String Value => GenerationId AssignedPartitions GenerationId => Int32 AssignedPartitions => [Topic [Partition]] Topic => String Partition => Int32 |
To be able to upgrade Kafka Streams application to leverage this new feature, a new config shall be introduced to control the producer upgrade decision:
Code Block | ||||
---|---|---|---|---|
| ||||
Boolean CONSUMER_GROUP_AWARED_EOS_PRODUCER = "consumer.group.awared.eos.producer"; // default to false |
Example
Below we provide an example of a simple read-process-write loop with consumer group-aware EOS processing.
...
This is a server-client integrated change, and it's required to upgrade the broker first with `inter.broker.protocol.version` to the latest. Any produce request with higher version will automatically get fenced, because we will be use the same transaction id for different topic partitions at the same time (breaking EOS), while losing transaction status tracking once we switched coordinator location. Once
Rejected Alternatives
- Producer Pooling:
- Producer support multiple transactional ids:
- Tricky rebalance synchronization:
...