Child pages
  • KIP-120: Cleanup Kafka Streams builder API
Skip to end of metadata
Go to start of metadata

Status

Current state: Accepted [VOTE] KIP-120: Cleanup Kafka Streams builder API

Discussion thread[DISCUSS] KIP-120: Cleanup Kafka Streams builder API

JIRA: KAFKA-3856

Released: 1.0.0

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, Kafka Streams public API leaks a bunch of internal methods that should not be public. Furthermore, DSL and PAPI abstraction are not completely separated at the moment and TopologyBuilder offers methods that belong to DSL only.

Public Interfaces

In order to get a clean refactoring, we will deprecate classes 

  • org.apache.kafka.streams.processor.TopologyBuilder
  • org.apache.kafka.streams.kstream.KStreamBuilder

and add them again with new name and different package

  • org.apache.kafka.streams.Topology
  • org.apache.kafka.streams.StreamsBuilder

Furthermore, we add a new interface to get a full description the a topology (to compensate for some internal methods that get removed)

  • org.apache.kafka.streams.TopologyDescription

    • TopologyDescription will have public interfaces:

      • Subtopology
      • GlobalStores
      • Node
      • Source
      • Sink
      • Processor

The following methods will not be available in the newly added classes:

  • TopologyBuilder -> Topology: setApplicationId, connectSourceStoreAndTopicconnectProcessorsaddInternalTopiccopartitionSources, nodeGroups, build, buildGlobalStateTopology, globalStateStores, topicGroups, earliestResetTopicPattern, latestResetTopicPattern, stateStoreNamesToSourceTopics, copartitioneGroups, sourceTopicPattern, updateSubscriptions
  • KStreamBuilder -> StreamsBuilder: all methods from TopologyBuilder (as KStreamBuilder does not inherit from TopologyBuilder anymore) except addStateStore and addGlobalStore (which are both added explicitly to StreamsBuilder) plus newName
  • methods highlighted in TopologyBuilder list, are methods that actually belong to DSL abstraction

 

New org.apache.kafka.streams.Topology class:

 

New org.apache.kafka.streams.StreamsBuilder class:

 

New org.apache.kafka.streams.TopologyDescription class:

 

Proposed Changes

We will add two new internal classes

  • org.apache.kafka.streams.processor.internals.InternalTopologyBuilder
  • org.apache.kafka.streams.kstream.internals.InternalStreamsBuilder 

that offer the methods remove from current API. Thus, both classes are the actual implementation. Old  TopologyBuilder and KStreamBuilder are only proxy classes to both classes respectively, for backward compatibility.

The newly added Topology uses InternalTopologyBuilder as member.

The newly added StreamsBuilder uses the new Topology as a member (no class hierarchy anymore – using it as member gives a clear separation between PAPI and DSL).

Because the new StreamsBuilder does not inherit from new Topology we need to add StreamsBuilder#build() that returns the actual Topology to be passed into KafkaStreams client.

Note: because of backward compatibility, removed DSL specific classes offered by old TopologyBuilder must be offered by InternalTopologyBuilder for now. However, after both deprecated classes got removed, this cleanup can be done (and does not require a KIP anymore, because it's internal refactoring -- we just need to create a JIRA for this). Thus, this KIP falls short of separating PAPI and DSL completely. But it's a necessary first step to do the separation in a backward compatible way (backward compatibility requires a two step approach).

Compatibility, Deprecation, and Migration Plan

  • Because no classes/method will be removed but only deprecated, this change will be fully backward compatible
  • We intend to remove all deprecated classes/methods in 0.11.1, but we can keep them longer on user request

Test Plan

Tests need to be rewritten but no new tests are required. All tests for public API need to be updated to use new Topology and StreamsBuilder. All tests using internal API, need to be rewritten to use InternalTopologyBuilder and InternalStreamsBuilder.

Rejected Alternatives

None.


  • No labels