This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • KIP-451: Make TopologyTestDriver output iterable
Skip to end of metadata
Go to start of metadata

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Discarded in favor of KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: https://issues.apache.org/jira/browse/KAFKA-8200

PR: https://github.com/apache/kafka/pull/6556

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

When using the TopologyTestDriver, one examines the output on a topic with the readOutput method. This method returns one record at a time, until no more records can be found, at which point in returns null.

Many times, the usage pattern around readOutput will involve writing a loop to extract a number of records from the topic, building up a list of records, until it returns null.

It would be helpful to offer an iterable signature of readOutput, which returns either an iterator or list over the records that are currently available in the topic. This would effectively remove the loop that a user needs to write for him/herself each time.

Public Interfaces

This KIP would propose to add methods which return an Iterable of the output records as byte arrays and with key/value deserializers:

public Iterable<ProducerRecord<byte[], byte[]>> iterableOutput(final String topic)

public <K, V> Iterable<ProducerRecord<K, V>> iterableOutput(final String topic,
                                                               final Deserializer<K> keyDeserializer,
                                                               final Deserializer<V> valueDeserializer)

Proposed Changes

As above.

Compatibility, Deprecation, and Migration Plan

  • As these are new methods there should be no impact on existing users

Rejected Alternatives

  • No labels