This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Voting

Discussion thread: here 

JIRA: [KAFKA-19883]( KAFKA-19883 - Getting issue details... STATUS )

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

1 Motivation

Today, the Producer's sendOffsetsToTransaction(offsets, consumerGroupMetadata) allows EOS in read-process-write topologies that consume from

regular consumer groups. With KIP-932 introducing share groups, the equivalent capability is missing for share-group consumers.

This blocks share-group adoption in:

 1.  MirrorMaker and other Kafka-to-Kafka mirroring/forwarding pipelines.
 2.  Kafka Streams stateless topologies that want to use share groups for parallelism beyond partition count.

 3.  Atomic DLQ write in different connectors


The Goal: Atomic "Read-Process-Write"

Enable Exactly-Once Semantics (EOS) by making acknowledgments part of a Kafka transaction.

Either the output is produced AND the source record is acknowledged, or neither happens. This KIP is scopred for Kafka producer write AND consumes from a share group.

1.1 Background: Share-partition record states

AVAILABLE → ACQUIRED → (ACKNOWLEDGED | ARCHIVING → ARCHIVED) | back to AVAILABLE.
  - ACKNOWLEDGED / ARCHIVED are terminal. ARCHIVING is a non-terminal step toward DLQ/ARCHIVED.
  - (correction: RENEW is an AcknowledgeType, not a state.)
  - Ack modes: implicit (auto-ack on next poll()) and explicit (acknowledge(record, type)).

  This KIP adds one state: TX_PENDING (RecordState, byte id 5, server module).


2 Public Interfaces (clients module)

Public API additions:

  • Producer.sendShareAcknowledgementsToTransaction(Map<TopicIdPartition, List<AcknowledgementBatch>>, ShareGroupMetadata).
  • ShareGroupMetadata class (groupId, memberId, memberEpoch, optional groupInstanceId).
  • ShareConsumer.shareGroupMetadata() to obtain the above.


Producer (clients module)

public interface Producer<K, V> {

    /**
   * Stages share-group acknowledgements for atomic commit with the records produced in this
   * transaction. On the broker the records move to TX_PENDING until the transaction resolves:
   *   COMMIT: ACCEPT -> ACKNOWLEDGED; REJECT -> ARCHIVING (DLQ) or ARCHIVED (no DLQ).
   *   ABORT : all staged records -> AVAILABLE (lock released, memberId cleared, redeliverable
   *           to ANY member on next fetch).
   * Only ACCEPT and REJECT are valid in a transaction; RELEASE/RENEW/GAP -> InvalidRecordStateException.
   *
   * Threading: enqueues the request for the producer Sender thread and returns immediately;
   *   the broker has not processed it yet. Errors surface at commitTransaction()/abortTransaction().
   *
   * @throws IllegalStateException      no transaction in progress / non-transactional producer
   * @throws ProducerFencedException / InvalidProducerEpochException  stale producer
   * @throws UnsupportedVersionException  cluster does not advertise apiKey 94 (TxnShareAcknowledge)
   * @throws GroupAuthorizationException  principal lacks READ on the share group   // correction: READ, not WRITE
   * @throws KafkaException             other abortable errors (abort + retry the CTP loop)
   */
  void sendShareAcknowledgementsToTransaction(ShareAcknowledgements acknowledgements,
                                              ShareGroupMetadata groupMetadata) throws ProducerFencedException;
}


ShareConsumer (clients module)

public interface ShareConsumer<K, V> {

  /** Immutable snapshot of group identity (groupId, memberId, memberEpoch). Thread-safe. */
  ShareGroupMetadata shareGroupMetadata();
  
  /** Returns AND CLEARS the acks staged during processing, for the producer txn.
   *  Explicit acknowledgement mode only. */
  ShareAcknowledgements acknowledgementsForTransaction();

}


ShareGroupMetadata (new class in clients module, package o.a.k.clients.consumer)

/* Thread-safe, Immutable, Concurrent with poll/acknowledge */
public final class ShareGroupMetadata {
    public ShareGroupMetadata(String groupId, String memberId, int memberEpoch);
    public final String groupId();
    public final String memberId();
    public final int memberEpoch();
    @Override public boolean equals(Object other);
    @Override public int hashCode();
    @Override public String toString();
}


New public classes (org.apache.kafka.clients.consumer) 

  - ShareAcknowledgements — final/immutable wrapper over Map<TopicIdPartition, List<ShareAcknowledgementBatch>>; empty(), isEmpty(), acknowledgements().

  - ShareAcknowledgementBatch — firstOffset, lastOffset, List<Byte> acknowledgeTypes.

Wire protocol additions:

New RPC TxnShareAcknowledge — apiKey 94, Listeners: broker.

Request — TransactionalId, GroupId, ProducerId, ProducerEpoch, MemberId, MemberEpoch, then Topics[] → Partitions[] → AcknowledgementBatches[] {FirstOffset, LastOffset, AcknowledgeTypes[]}. Only
  ACCEPT(1)/REJECT(3) valid.

Response — ThrottleTimeMs, top-level ErrorCode, Responses[] → Partitions[] {PartitionIndex, ErrorCode, ErrorMessage}. (correction: no CurrentLeader/NodeEndpoints fields — the schema does not carry 
  leader-redirect fields.)
  - Top-level errors: GROUP/TOPIC/TRANSACTIONAL_ID_AUTHORIZATION_FAILED, TRANSACTIONAL_ID_NOT_FOUND, INVALID_PRODUCER_EPOCH/PRODUCER_FENCED, INVALID_PRODUCER_ID_MAPPING, INVALID_TXN_STATE, UNKNOWN_MEMBER_ID, STALE_MEMBER_EPOCH, TRANSACTION_ABORTABLE.
  - Per-partition: UNKNOWN_TOPIC_OR_PARTITION/UNKNOWN_TOPIC_ID, INVALID_RECORD_STATE, INVALID_REQUEST, KAFKA_STORAGE_ERROR, NOT_LEADER_OR_FOLLOWER.

Existing RPCs — behavior:
  - AddPartitionsToTxn (24): no schema change, but used broker-internally (correction: the KIP earlier claimed share partitions are NOT registered — they are). The TxnShareAcknowledge handler calls
  AddPartitionsToTxnManager.addOrVerifyTransaction(..., verifyOnly=false) to register __share_group_state-N as a real transaction participant.
  - WriteTxnMarkers (27): no schema change. On the __share_group_state partition the broker calls shareCoordinator.completeTransaction(...), which returns the affected Set<SharePartitionKey>; the broker then
  invalidates only those caches. (correction: it does not broadcast to all SharePartitions with per-record fencing — a SharePartitionManager.applyTxnMarker broadcast method exists but is unused on this path; 
  tests assert never().) 
  - TxnOffsetCommit (28), ApiVersions (18): no schema change. Broker advertises apiKey 94 only when share.version >= 3 (below).

New RPC: TxnShareAcknowledgeResponse (apiKey 94, v0)

Top-level supported errors:
- GROUP_AUTHORIZATION_FAILED
- TOPIC_AUTHORIZATION_FAILED
- TRANSACTIONAL_ID_AUTHORIZATION_FAILED
- TRANSACTIONAL_ID_NOT_FOUND
- INVALID_PRODUCER_EPOCH / PRODUCER_FENCED
- INVALID_PRODUCER_ID_MAPPING
- INVALID_TXN_STATE
- UNKNOWN_MEMBER_ID
- STALE_MEMBER_EPOCH
- TRANSACTION_ABORTABLE 
- UNKNOWN_SERVER_ERROR

Per-partition supported errors:
- UNKNOWN_TOPIC_OR_PARTITION / UNKNOWN_TOPIC_ID
- NOT_LEADER_OR_FOLLOWER (with CurrentLeader populated)
- INVALID_RECORD_STATE
- INVALID_REQUEST
- KAFKA_STORAGE_ERROR


  TxnShareAcknowledgeRequest.json

  {
    "apiKey": 94,
    "type": "request",
    "listeners": ["broker"],
    "name": "TxnShareAcknowledgeRequest",
    "validVersions": "0",
    "flexibleVersions": "0+",
    "fields": [
      { "name": "TransactionalId", "type": "string", "versions": "0+",
        "nullableVersions": "0+", "entityType": "transactionalId",
        "about": "The transactional ID of the producer." },
      { "name": "GroupId", "type": "string", "versions": "0+",
        "entityType": "groupId", "about": "The share group ID." },
      { "name": "ProducerId", "type": "int64", "versions": "0+",
        "entityType": "producerId", "about": "The producer ID." },
      { "name": "ProducerEpoch", "type": "int16", "versions": "0+",
        "about": "The producer epoch." },
      { "name": "MemberId", "type": "string", "versions": "0+",
        "about": "The share group member ID." },
      { "name": "MemberEpoch", "type": "int32", "versions": "0+",
        "about": "The share group member epoch for fencing." },
      { "name": "Topics", "type": "[]TxnShareAcknowledgeTopic", "versions": "0+",
        "about": "The topics and partitions to acknowledge.", "fields": [
        { "name": "TopicId", "type": "uuid", "versions": "0+",
          "about": "The unique topic ID." },
        { "name": "Partitions", "type": "[]TxnShareAcknowledgePartition", "versions": "0+",
          "fields": [
          { "name": "PartitionIndex", "type": "int32", "versions": "0+",
            "about": "The partition index." },
          { "name": "AcknowledgementBatches", "type": "[]TxnShareAcknowledgeBatch", "versions": "0+",
            "about": "Record ranges to acknowledge. Only ACCEPT (1) and REJECT (3) are valid inside a transaction.",
            "fields": [
            { "name": "FirstOffset", "type": "int64", "versions": "0+",
              "about": "First offset of the batch (inclusive)." },
            { "name": "LastOffset", "type": "int64", "versions": "0+",
              "about": "Last offset of the batch (inclusive)." },
            { "name": "AcknowledgeTypes", "type": "[]int8", "versions": "0+",
              "about": "Per-offset acknowledge types: 1=ACCEPT, 3=REJECT. Size 1 = uniform type for whole range." }
          ]}
        ]}
      ]}
    ]
  }


TxnShareAcknowledgeResponse.json

  {
    "apiKey": 94,
    "type": "response",
    "name": "TxnShareAcknowledgeResponse",
    "validVersions": "0",
    "flexibleVersions": "0+",
    "fields": [
      { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
        "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
      { "name": "ErrorCode", "type": "int16", "versions": "0+",
        "about": "The top-level error code, or 0 if there was no error." },
      { "name": "Responses", "type": "[]TxnShareAcknowledgeTopicResponse", "versions": "0+",
        "about": "The results for each topic.", "fields": [
        { "name": "TopicId", "type": "uuid", "versions": "0+",
          "about": "The unique topic ID." },
        { "name": "Partitions", "type": "[]TxnShareAcknowledgePartitionResponse", "versions": "0+",
          "fields": [
          { "name": "PartitionIndex", "type": "int32", "versions": "0+",
            "about": "The partition index." },
          { "name": "ErrorCode", "type": "int16", "versions": "0+",
            "about": "The error code, or 0 if there was no error." },
          { "name": "ErrorMessage", "type": "string", "versions": "0+",
            "nullableVersions": "0+", "default": "null",
            "about": "The error message, or null if there was no error." }
        ]}
      ]}
    ]
  }


Internal schema changes


  - ShareSnapshotValue.json / ShareUpdateValue.json: StateBatch gains 4 tagged fields — StagedProducerId, StagedProducerEpoch, StagedAckType, StagedDeliveryState (default −1, so old records are compatible).
  - WriteShareGroupStateRequest / ReadShareGroupStateResponse and PersisterStateBatch: same 4 fields.
  - ShareCoordinatorShard: adds completeTransaction / replayEndTransactionMarker / isMatchingPendingBatch; restores staged fields on replay.
  - __transaction_state: unchanged (share-state partitions tracked via existing TransactionMetadata.topicPartitions).

  Consequence: TX_PENDING is durably persisted to the share-state log and restored on replay/leader change — so broker crash and share-coordinator leader change during staging no longer lose staged state
  (previously corner cases 1 & 12, now covered).

  State machine additions

  - New TX_PENDING, entered only from ACQUIRED (on transactional ACCEPT/REJECT). Staging cancels the acquisition-lock timer — the transaction now governs the hold. 
  - COMMIT: ACCEPT → ACKNOWLEDGED; REJECT → ARCHIVING (→ DLQ, cause CLIENT_REJECT) or ARCHIVED if DLQ disabled. 
  - ABORT: TX_PENDING → AVAILABLE (lock released, redeliverable).

Execution path & threading

StepSide / ModuleComponent (class)Thread
send(output) + acknowledge(rec) + acknowledgementsForTransaction()client / clientsKafkaProducer, ShareConsumerImplapp thread (consumer posts a ShareGroupMetadataEvent to the consumer background thread, which reads ShareMembershipManager for member id/epoch)
enqueue sendShareAcknowledgementsInTransactionclient / clientsTransactionManagerapp thread enqueues; producer sender thread sends TxnShareAcknowledge (apiKey 94, TV2-required)
handle requestserver / coreKafkaApis.handleTxnShareAcknowledgeRequestbroker request-handler (io) thread - authorizes (WRITE txn-id, READ group, READ topic), validates member via GroupCoordinator.validateShareGroupMember
register participantserver / coreAddPartitionsToTxnManager -> TransactionCoordinatorinter-broker send thread; registers __share_group_state-N in __transaction_state
stage acks (on success)server / core + share-coordinatorSharePartitionManager.acknowledgeTransactional -> SharePartition.stageTxnAcknowledge -> persister WriteShareGroupStaterequest-handler stages in-memory TX_PENDING; share-coordinator event-loop thread persists staged batch
commitTransaction()client -> serverTransactionManager -> TransactionCoordinatorapp thread blocks; Sender thread sends EndTxn; TxnCoord resolves participant leaders, dispatches WriteTxnMarkers
apply markerserver / core + share-coordinatorKafkaApis.handleWriteTxnMarkersRequest -> shareCoordinator.completeTransaction -> invalidate cachesrequest-handler thread; share-coordinator event-loop thread finalizes TX_PENDING and persists final state



Flow — Share Group (THIS KIP)
  1. producer.send("output-A", rec) → TransactionManager sends AddPartitionsToTxn(["output-A-0"]); participants = {output-A-0}.
  2. producer.sendShareAcknowledgementsToTransaction(acks, meta) → TxnShareAcknowledge to the SharePartition-leader broker. Broker: resolves __share_group_state-N, registers it as a participant

      (verifyOnly=false), then only on success stages the acks to TX_PENDING (in memory + persisted via WriteShareGroupState). Participants = {output-A-0, __share_group_state-N}.

  3. producer.commitTransaction() → EndTxn(COMMIT). TxnCoord resolves leaders and sends WriteTxnMarkers to both.
  4. Output-leader appends COMMIT to output-A-0. Share-leader appends COMMIT to __share_group_state-N, then shareCoordinator.completeTransaction(COMMIT) finalizes matching TX_PENDING batches

      (ACCEPT→ACKNOWLEDGED, REJECT→ARCHIVING/ARCHIVED) and returns affected keys → caches invalidated.

  5. TxnCoord → COMPLETE_COMMIT; the blocking commitTransaction() returns.

 (Note: TV2-only — broker-side auto-registration + epoch fencing; TV2 is default since 4.0.)


Pseudo code


shareConsumer.subscribe(List.of("source-topic"));   // explicit acknowledgement mode
producer.initTransactions();
  while (running) {
    var records = shareConsumer.poll(Duration.ofSeconds(5));
    if (records.isEmpty()) continue;
    try {
      producer.beginTransaction();
      for (var rec : records) {
        producer.send(new ProducerRecord<>("destination-topic", process(rec)));
        shareConsumer.acknowledge(rec, AcknowledgeType.ACCEPT);          // accumulate acks
      }
      producer.sendShareAcknowledgementsToTransaction(
          shareConsumer.acknowledgementsForTransaction(),                // harvests + clears
          shareConsumer.shareGroupMetadata());
      producer.commitTransaction();                                     // atomic: produced AND acked
    } catch (ProducerFencedException | UnsupportedVersionException fatal) {
      throw fatal;
    } catch (KafkaException abortable) {
      producer.abortTransaction();                                      // staged records -> AVAILABLE, redelivered
    }
  }


Corner cases 

  1. Broker crash / share-coordinator leader change during staging - TX_PENDING is persisted in __share_group_state; the new leader replays it and the pending marker still resolves it. 
  2. Participant registration fails (COORDINATOR_NOT_AVAILABLE/NOT_COORDINATOR) — no staging; abortable; retry.
  3. Producer fenced mid-stage — registration rejects (PRODUCER_FENCED); staged-owner (producerId, epoch) filter rejects any late marker; idempotent.
  4. Duplicate WriteTxnMarkers — second marker finds state != TX_PENDING → no-op.
  5. Transaction timeout while TX_PENDING — TxnCoord auto-sends WriteTxnMarkers(ABORT); records → AVAILABLE and redeliver. This (transaction.timeout.ms), not lock expiry, is the abandoned-txn safety net — the
  acquisition-lock timer is cancelled at staging. (correction to old cases 6/13.)
  6. Stale member epoch — broker compares memberEpoch; mismatch → STALE_MEMBER_EPOCH; abortable, retry with fresh snapshot.
  7. Mixed-version cluster — apiKey 94 absent → UnsupportedVersionException synchronously, no bytes on wire; abort to clean up the output-side registration.
  8. Invalid ack type in txn — RELEASE/RENEW/GAP → INVALID_RECORD_STATE; no staging.
  9. Partial multi-partition stage failure — per-partition rollback reverts already-staged records to ACQUIRED before the response; all-or-nothing.
  10. Concurrent txns, different epochs — TxnCoord rejects the stale one (PRODUCER_FENCED); state-machine owner filter blocks stale markers.

Metrics

All new metrics follow the existing Kafka metric conventions (Yammer for broker JMX, KafkaMetric for client-side).

Metric names mirror the established kafka.server:type=group-coordinator-metrics,name=... and kafka.server:type=share-coordinator-metrics,name=... patterns.

Backward-compatible: no existing metrics renamed or removed.

ModuleMetricTypePurpose
Broker — SharePartitionManagerTxnPendingRecordsCountGaugeCurrent count of records in TX_PENDING (primary health signal)
Broker — SharePartitionManagerTxnShareAcknowledgeRequestLatencyMsHistogramp99 latency of staging requests
Broker — SharePartitionManagerTxnPendingLockExpiredCountCounterAny non-zero = abandoned txns or missing markers (critical alert)
Broker — TransactionCoordinatorTransactionPartitionsCount (existing)GaugeReused; now includes __share_group_state-N entries
Producershare-ack-txn-send-rateMeterEOS-call throughput
Producershare-ack-txn-send-error-rateMeterStage-failure rate (drives retry loops)
Consumershare-group-metadata-fetch-rateMeterConfirms read-process-write loop is active
Total: 6 new + 1 reused = 7 metrics. Primary alert: TxnPendingRecordsCount > 0 sustained for > 60s.


Compatibility, enablement, downgrade

Enablement and Rollout Plan

  - Gating: share.version >= 3 (SV_3) advertises apiKey 94 (correction: SV_2 is the DLQ feature, KIP-1191); transaction.version >= 2 (TV2-only). Uses existing kafka-features.sh; no new flag.
  - Rolling upgrade: upgrade binaries (dormant at SV<3) → soak → kafka-features.sh upgrade --feature share.version --version 3 (online, no restart) → onboard clients.
  - Cross-version: new client + old broker → UnsupportedVersionException; old clients unaffected.
  - Downgrade: halt txn-ack producers; wait transaction.timeout.ms to drain; --feature share.version --version 2. (correction: there is no broker guard that rejects downgrade on in-memory TX_PENDING; because 
  staging is persisted, drain-then-downgrade is the safe procedure.)


Cross-Version Client/Broker Matrix

ClientBroker, share.version=2Broker, share.version=1Old broker (no binary)
NewWorksUnsupportedVersionException (synchronous, no wire bytes)Same
OldUnaffectedUnaffectedUnaffected

Downgrade

1. Drain in-flight TX_PENDING:
   - Halt producers calling sendShareAcknowledgementsToTransaction
   - Wait for transaction.timeout.ms (default 60s) for any abandoned txns to clear
2. kafka-features.sh downgrade --feature share.version --version 1
Broker rejects the downgrade RPC if any in-memory TX_PENDING exists, preventing data inconsistency.
Software downgrade (binary): must follow feature downgrade; standard rolling restart.

Test Plan

Rough implementation [The actual implementation will be phasewise with multiple smaller PRs]:
    https://github.com/apache/kafka/pull/22357

Testing in Flink connector kafka: https://github.com/apache/flink-connector-kafka/pull/271/ 

Unit (state transitions ACCEPT/REJECT commit vs abort→AVAILABLE, idempotency, TV2), integration (end-to-end commit/abort, coordinator recovery, multi-consumer, network partitions), system/perf (txn vs non-txn,  EOS verification), chaos (broker/coordinator crash during commit).

Rejected alternatives

  - Extend TxnOffsetCommit to carry acks — rejected; share acks aren't offsets and reuse would overload the consumer-offset path.

  - Broadcast markers to all SharePartitions with per-record fencing (no participant tracking) — rejected in favor of registering __share_group_state-N and routing to shareCoordinator.completeTransaction, which

  is targeted and reuses the standard participant/marker machinery.

  - Support TV1 — rejected; TV2 gives broker-side auto-registration and epoch fencing and is default since 4.0.

  - First-class share partitions as txn participants (vs. the owning __share_group_state partition) — rejected; the state partition is the durable source of truth.


Follow-up 

  - Per-share-partition TX_PENDING residency histogram (diagnostics).
  - Metrics implementation (above).
  - External processing engines / databases as 2PC participants for sink writes.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.

2 Comments

  1. Shekhar Rajak

    Ref -   producer side 2PC (Kafka as participant)     KAFKA-15370 - Getting issue details... STATUS   KIP: KIP-939: Support Participation in 2PC#2PCRefresher