Current state: Discussion
Discussion thread: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
As KIP-82 introduced Headers into the core Kafka Product, it would be advantageous to expose them in the Kafka Connect Framework.
Connectors that replicate data between Kafka cluster or between other messaging products and Kafka would want to replicate the headers.
This KIP has the following public interface changes:
Add new constructor to Source Record
Add new constructor to Sink Record
modify parent ConnectRecord constructor , and expose headers accessor method.
newRecordabstract method to
ConnectRecordand override implementations in
SinkRecordto take headers parameter, with existing signature delegating to new signature, but left in for compatibility.
Add a headers field Headers to shared ConnectRecord, in turn would be available in Source/SinkRecords
- Accessor methods of Headers headers() added ConnectRecord
- Add constructor(s) of Connect.Source/SinkRecord to allow passing in of an existing/pre-constructed headers via Iterable<Header>
- use case is Replication connectors able to copy headers.
- See above.
- Update convertMessages method, to take from the ConsumerRecord the headers and pass them into the created SinkRecord
- Update sendRecords method, to take from the SourceRecord the headers and pass them into the created ProducerRecord
Compatibility, Deprecation, and Migration Plan
- Current client users should not be affected, this is new api methods being added to connect source/sink apis
- This builds on KIP-82 - Add Record Headers
Out of Scope