Current state: "Under Discussion"
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Kafka Connect Source currently has a "safe producer" approach to producing data. This means currently:
// These settings are designed to ensure there is no data loss. They *may* be overridden via configs passed to the
// worker, but this may compromise the delivery guarantees of Kafka Connect.
Idempotent producers have been introduced in KIP-98 - Exactly Once Delivery and Transactional Messaging and allow produces to be idempotent and therefore:
- Prevent duplicates on network failure (idempotence)
- Increase performance of safe producers by using MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION=5. See: - KAFKA-5494Getting issue details... STATUS
Currently Kafka Connect does not benefit from these two improvements which could be very valuable to existing connectors in order to improve both the performance and safety
We will add the following code to Worker.java:
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
Users running Kafka Connect against older clusters (< 1.0) may have a compatibility issue due to API changes for idempotent producers.