This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • KIP-397: Add methods to override fetch offsets based on timestamp

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
Currently, if I wanted to subscribe to a topic at the offset closest to a specified timestamp, I'd need to construct a set of TopicPartition objects using information gathered from the KafkaConsumer#partitionsFor method, construct a Map<TopicPartition, Long> with the timestamps, and then call KafkaConsumer#offsetsForTime. Once I do that, id need to call KafkaConsumer#seek for each

entry in the mapThis proposes adding a method that would determine the offsets for a list of topic partitions based on offset and update the consumer assignment to seek to those offsets.

Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Describe the problems you are trying to solve.Currently, if I wanted to subscribe to a topic at the offset closest to a specified timestamp, I'd need to construct a set of TopicPartition objects using information gathered from the KafkaConsumer#partitionsFor method, construct a Map<TopicPartition, Long> with the timestamps, and then call KafkaConsumer#offsetsForTime. Once I do that, id need to call KafkaConsumer#seek for each entry in the map.

Public Interfaces

  • Add method seekToOffsetsForTime(List<TopicPartition> partitions, Long timestampToStart)

...