Child pages
  • KIP-145 - Expose Record Headers in Kafka Connect
Skip to end of metadata
Go to start of metadata

 

Status

Current state:  Discussion

Discussion threadhere

JIRAKAFKA-5142 - KIP-145 - Expose Record Headers in Kafka Connect

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As KIP-82 introduced Headers into the core Kafka Product, it would be advantageous to expose them in the Kafka Connect Framework.

Connectors that replicate data between Kafka cluster or between other messaging products and Kafka would want to replicate the headers.

 

Public Interfaces

This KIP has the following public interface changes:

  1. Add new constructor to Source Record

  2. Add new constructor to Sink Record

  3. modify parent ConnectRecord constructor , and expose headers accessor method.

  4. Add newRecord abstract method to ConnectRecord and override implementations in SourceRecord and SinkRecord to take headers parameter, with existing signature delegating to new signature, but left in for compatibility.

Proposed Changes

Add a headers field Headers to shared ConnectRecord, in turn would be available in Source/SinkRecords

  • Accessor methods of Headers headers() added ConnectRecord
  • Add constructor(s) of Connect.Source/SinkRecord to allow passing in of an existing/pre-constructed headers via Iterable<Header> 
    1. use case is Replication connectors able to copy headers.
  • See above.

WorkerSinkTask

  • Update convertMessages method, to take from the ConsumerRecord the headers and pass them into the created SinkRecord

 

WorkerSourceTask

  • Update sendRecords method, to take from the SourceRecord the headers and pass them into the created ProducerRecord

 

Compatibility, Deprecation, and Migration Plan

  • Current client users should not be affected, this is new api methods being added to connect source/sink apis
  • This builds on KIP-82 - Add Record Headers

Out of Scope

Rejected Alternatives

  • No labels