...
The state transitions look like this:
| Code Block | ||
|---|---|---|
| ||
+--------------+
| Available |<------------------+
+--------------+ |
| |
| acquired for consumer | - if (delivery count < share.delivery.attempt.limit)
| for consumer(delivery count++) | - released by consumer
| - released by consumer
| (delivery count++) | - acquisition lock elapsed
V |
+--------------+ |
| Acquired |-------------------+
+--------------+ |
| |
| accepted | - if (delivery count == share.delivery.attempt.limit)
| accepted by consumer | - released by consumer
| | - acquisition lock elapsed
V | OR
+--------------+ | - rejected by consumer as unprocessable
| Acknowledged | |
+--------------+ |
| |
| SPSO moves |
| past record|
| SPSO moves past record |
| |
V |
+--------------+ |
| Archived |<------------------+
+--------------+ |
The share-partition leader persists the states and delivery counts. These updates are not performed with exactly-once semantics, so the delivery count cannot be relied upon to be precise in all situations. It is intended as a way to protect against poison messages, rather than a precise count of the number of times a record is delivered to a consumer.
When records are fetched for a consumer, the share-partition leader starts at the SPSO and finds Available records. For each record it finds, it moves it into Acquired state, bumps its delivery count and adds it to a batch of acquired records to return to the consumer. The consumer then processes the records and acknowledges their consumption. The delivery attempt completes successfully and the records move into Acknowledged state.
...
| Configuration | Description | Values |
|---|---|---|
group.type | Type of the group: "consumer" or "share" . | Default "consumer" |
record.lock.duration.ms | Record acquisition lock duration in milliseconds. | null, which uses the cluster configuration share.record.lock.duration.ms, minimum 1000, maximum limited by the cluster configuration share.record.lock.duration.max.ms |
Kafka protocol changes
Further details to follow as the design progresses.
Metrics
Further details to follow as the design progresses.
...