Current state: Under Discussion
Discussion thread: https://firstname.lastname@example.org/msg100621.html
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
As part of KIP-82, the headers are added to Kafka ProducerRecord and ConsumerRecord.As discussed in that KIP the headers provide lot of benefits especially metadata association with the record.
Currently the RecordMetadata contains details like topic, partition, offset, timestamp etc. But there is no information on for which record this RecordMetadata belongs to. Having headers information as part of RecordMetaData will bridge this gap.
It will make the context of the record available to Interceptors and open up possibility for building more features.
As the Kafka usage is growing, we see Kafka being used for Payment transactional, Settlement and reconciliation use cases. These kind of use cases are very sensitive to message loss.Current dataloss auditing systems like chaperone uses time window based aggregations, which is not sufficient when we need message level traceability. In order to build message level traceability, we need the capability to track message production and consumption.If we choose to track message production status based on acknowledgment the producer receives, we will need ability to relate acknowledgment to the record. Having this ability in Interceptors can help us to link the record to its acknowledgement.
We will change RecordMetaData and FutureRecordMetaData to have headers information.
Changes to RecordMetaData
Changes to FutureRecordMetaData
In ProduceBatch add headers to FutureRecordMetaData while appending record to the batch.
- Add headers to RecordMetaData for send error scenarios also.
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
There is no impact. Since we are passing same reference to the headers there is no memory impact.
- If we are changing behavior how will we phase out the older behavior?
Older behavior does not change so no need to phase it out.
- If we need special migration tools, describe them here.
No special migration tools needed.
- When will we remove the existing behavior?
- Producer callback provides a way to link record to its RecordMetaData.
In this case the context linking on client side and depends on client implementation where as interceptors having similar RecordMetaData but no access to headers onAcknowledgement(...).