Check out the docs at Apache Kafka web site: http://kafka.apache.org/documentation/streams/
Sub-pages:
- Using Kafka Streams:
- Development:
- Internals:
Find more links about Kafka Streams at Kafka Ecosystem page.
Kafka Streams related KIPs:
Below is a list of KIPs that are not release yet. Go to Kafka Streams KIP Overview for KIPs by release (including discarded KIPs).
Under discussion
- KIP-655: Windowed Distinct Operation for Kafka Streams API
- KIP-816: Topology changes without local state reset
- KIP-948: Allow custom prefix for internal topic names in Kafka Streams
- KIP-969: Support range Interactive Queries (IQv2) for Versioned State Stores
- KIP-997: update WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery
- KIP-1027: Add MockFixedKeyProcessorContext and TestFixedKeyRecordFactory
- KIP-1071: Streams Rebalance Protocol
- KIP-1072: Add FunctionalInterface annotation to Kafka Streams SAM methods
- KIP-1088: Replace KafkaClientSupplier with KafkaClientInterceptor
- KIP-1111: Enforcing Explicit Naming for Kafka Streams Internal Topics
Adopted
- inactive, not (fully) implemented (feel free to pick up):
- KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner (partially implemented in v1.1)
- KIP-216: IQ should throw different exceptions for different errors
- KIP-258: Allow to Store Record Timestamps in RocksDB (partially implemented in v2.3)
- KIP-300: Add Windowed KTable API in StreamsBuilder (accepted but postponed because current code base does not allow to implement it as proposed)
- KIP-328: Ability to suppress updates for KTables (partially implemented in v2.1)
- KIP-557: Add emit on change support for Kafka Streams (
partially implemented in v2.6reverted again in 2.8.0, 2.7.1, and 2.6.2 due to potential data loss) - KIP-698: Add Explicit User Initialization of Broker-side State to Kafka Streams
- KIP-796: Interactive Query v2 (split up into multiple sub KIPs)
- KIP-878: Internal Topic Autoscaling for Kafka Streams
- WIP (open PR / in review):
- KIP-759: Unneeded repartition canceling
- KIP-770: Replace "buffered.records.per.partition" & "cache.max.bytes.buffering" with "{statestore.cache}/{input.buffer}.max.bytes" (partially implemented in 3.4)
- KIP-892: Transactional Semantics for StateStores (blocked – waiting for KIP-1035 to be finished first)
- KIP-869: Improve Streams State Restoration Visibility (partially implemented in 3.5)
- KIP-1020: Move `window.size.ms` and `windowed.inner.serde.class` from `StreamsConfig` to TimeWindowedDe/Serializer class
- KIP-1034: Dead letter queue in Kafka Streams
- KIP-1035: StateStore managed changelog offsets
- KIP-1106: Add duration based offset reset option for consumer clients
- KIP-1112: allow custom processor wrapping
- next release Kafka 4.0 (merged):
- KIP-1056: Remove `default.` prefix for exception handler StreamsConfig
- KIP-1065: Add "retry" return-option to ProductionExceptionHandler
- KIP-1070: deprecate MockProcessorContext
- KIP-1076: Metrics for client applications KIP-714 extension
- KIP-1077: Deprecate `ForeachProcessor` and move to internal package
- KIP-1078: Remove Leaking Getter Methods in Joined Helper Class
- KIP-1085: Fix leaking *_DOC variables in StreamsConfig
- KIP-1087: Removing intermediateTopicsOption from StreamsResetter
- KIP-1091: Improved Kafka Streams operator metrics
- KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins
Inactive (feel free to pick up and resume the discussion):
- KIP-95: Incremental Batch Processing for Kafka Streams
- KIP-159: Introducing Rich functions to Streams
- KIP-260: add primary join operation for Stream-Stream join (WIP) (draft)
- KIP-311: Async processing with dynamic scheduling in Kafka Streams
- KIP-314: KTable to GlobalKTable Bi-directional Join
- KIP-326: Schedulable KTable as Graph source
- KIP-335: Consider configurations for KafkaStreams
- KIP-362: Support dynamic gap session window
- KIP-406: GlobalStreamThread should honor custom reset policy
- KIP-408: Add Asynchronous Processing To Kafka Streams
- KIP-424: Allow suppression of intermediate events based on wall clock time
- KIP-457: Add DISCONNECTED status to Kafka Streams
- KIP-459: Improve KafkaStreams#close
- KIP-463: Auto-configure non-default Serdes passed alongside the TopologyBuilder
- KIP-508: Make Suppression State Queriable
- KIP-513: Distinguish between Key and Value serdes in scala wrapper library for kafka streams
- KIP-540: Implement per key stream time tracking
- KIP-560: Auto infer external topic partitions in stream reset tool
- KIP-598: Augment TopologyDescription with store and source / sink serde information
- KIP-634: Complementary support for headers and record metadata in Kafka Streams DSL
- KIP-645: Replace Windows with a proper interface
- KIP-647: Add ability to handle late messages in streams-aggregation
- KIP-667: Remove deprecated methods from ReadOnlyWindowStore (no KIP needed)
- KIP-669: Preserve Source Partition in Kafka Streams from context
- KIP-674: Metric Reporter to Aggregate Metrics in Kafka Streams
- KIP-675: Convert KTable to a KStream using the previous value
- KIP-677: Migrating Topology methods to the Builder pattern
- KIP-718: Make KTable Join on Foreign key unopinionated
- KIP-747 Add support for basic aggregation APIs
- KIP-807: Refactor KafkaStreams exposed metadata hierarchy
- KIP-819: Merge multiple KStreams in one operation
- KIP-839: Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams
- KIP-857: Streaming recursion in Kafka Streams
- KIP-955: Add stream-table join on foreign key
- KIP-XXX: [Streams] Support multiple topic resolution in TopicNameExtractor (draft)