Current state: Accepted [VOTE] KIP-120: Cleanup Kafka Streams builder API
Discussion thread: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Currently, Kafka Streams public API leaks a bunch of internal methods that should not be public. Furthermore, DSL and PAPI abstraction are not completely separated at the moment and
TopologyBuilder offers methods that belong to DSL only.
In order to get a clean refactoring, we will deprecate classes
and add them again with new name and different package
Furthermore, we add a new class to get a full description the a topology (to compensate for some internal methods that get removed)
TopologyDescriptionwill have public subclasses
The following methods will not be available in the newly added classes:
TopologyBuilder -> Topology: setApplicationId, connectSourceStoreAndTopic, connectProcessors, addInternalTopic, copartitionSources, nodeGroups, build, buildGlobalStateTopology, globalStateStores, topicGroups, earliestResetTopicPattern, latestResetTopicPattern, stateStoreNamesToSourceTopics, copartitioneGroups, sourceTopicPattern, updateSubscriptions
KStreamBuilder -> StreamsBuilder: all methods from TopologyBuilder (as KStreamBuilder does not inherit from TopologyBuilder anymore) except addStateStore and addGlobalStore (which are both added explicitly to
StreamsBuilder) plus newName
- methods highlighted in
TopologyBuilderlist, are methods that actually belong to DSL abstraction
We will add two new internal classes
that offer the methods remove from current API. Thus, both classes are the actual implementation. Old
KStreamBuilder are only proxy classes to both classes respectively, for backward compatibility.
The newly added
InternalTopologyBuilder as member.
The newly added
StreamsBuilder uses the new
Topology as a member (no class hierarchy anymore – using it as member gives a clear separation between PAPI and DSL).
Because the new
StreamsBuilder does not inherit from new
Topology we need to add
StreamsBuilder#build() that returns the actual
Topology to be passed into
Note: because of backward compatibility, removed DSL specific classes offered by old
TopologyBuilder must be offered by
InternalTopologyBuilder for now. However, after both deprecated classes got removed, this cleanup can be done (and does not require a KIP anymore, because it's internal refactoring -- we just need to create a JIRA for this). Thus, this KIP falls short of separating PAPI and DSL completely. But it's a necessary first step to do the separation in a backward compatible way (backward compatibility requires a two step approach).
Compatibility, Deprecation, and Migration Plan
- Because no classes/method will be removed but only deprecated, this change will be fully backward compatible
- We intend to remove all deprecated classes/methods in 0.11.1, but we can keep them longer on user request
Tests need to be rewritten but no new tests are required. All tests for public API need to be updated to use new
StreamsBuilder. All tests using internal API, need to be rewritten to use