Child pages
  • KIP 141 - ProducerRecord & SourceRecord: Add timestamp constructors
Skip to end of metadata
Go to start of metadata

Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: KAFKA-5092

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka 0.10 added the Timestamp as part of messages, which is particularly important for KStreams application. Currently that timestamp can be set by the broker, or by the producer. This addresses concerns with timestamps set at the producer level.
Currently, if the producer application does not join a timestamp to the ProducerRecord, such as in public ProducerRecord(String topic, K key, V value) , then the timestamp is inferred at send time being System.currentTimeMillis(). If a producer wants to explicitly set a timestamp for the record, it has got to use public ProducerRecord(String topic, Integer partition, Long timestamp, K key, V value). 

This interface is particularly dangerous as it indicates to the user that they should provide an Integer partition. They would have to dive into the code to know that partition can be null, and then will be inferred using the key. I also believe offering partition as an argument goes against the current belief that "the same keys always goes to the same partition". 

This KIP adds the missing constructor to add a Timestamp to a ProducerRecord or a SourceRecord. 

Public Interfaces

New interface:

public SourceRecord(Map<String, ?> sourcePartition, Map<String, ?> sourceOffset,
 String topic, Schema valueSchema, Object value, Long timestamp)
public SourceRecord(Map<String, ?> sourcePartition, Map<String, ?> sourceOffset,
 String topic, Schema keySchema, Object key, Schema valueSchema, Object value,
 Long timestamp)
public ProducerRecord(String topic, V value, Long timestamp)
public ProducerRecord(String topic, K key, V value, Long timestamp)

Proposed Changes

See KAFKA-5092

Compatibility, Deprecation, and Migration Plan

  • No deprecation
  • No migration needed for existing programs

Rejected Alternatives

  •  public ProducerRecord(String topic, Long timestamp, K key, V value)

Adding this constructor is quite dangerous because it's almost the same as the one that takes partition (the only difference is that one is a Long and the other is an Integer)
Therefore the ordering of arguments should explicitly prevent mistakes

 

 

  •  Adding a builder interface

Although more newbie friendly, it would imply deprecating the existing constructors, which could take years, because of maintaining code "backward compatibility". There are too many drawbacks and not enough advantages (see mailing list discussion). 

  • No labels