This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • KIP-318: Make Kafka Connect Source idempotent

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

// These settings are designed to ensure there is no data loss. They *may* be overridden via configs passed to the
// worker, but this may compromise the delivery guarantees of Kafka Connect.
...
producerProps.put(ProducerConfig.ACKS_CONFIG, "all");
producerProps.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "1");


Idempotent Idempotent producers have been introduced in KIP-98 - Exactly Once Delivery and Transactional Messaging and allow produces to be idempotent and therefore:

  • Prevent duplicates on network failure (idempotence)
  • Increase performance of safe producers by using MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION=5.  See: 
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyKAFKA-5494
     


Currently Kafka Connect does not benefit from these two improvements which could be very valuable to existing connectors in order to improve both the performance and safety

Public Interfaces

We will add the following code to Worker.java:  

...