Status
Current state: Under Discussion Adopted (1.0)
Discussion thread: [DISCUS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engineshere [Change the link from the KIP proposal email archive to your own email thread]
JIRA: KAFKA-5651
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Code Block |
---|
|
void print(final Printed<K, V> printed);
KStream<K, V> through(final String topic, final Produced<K, V> partitioned);
void to(final String topic, final Produced<V, V> partitioned);
KGroupedStream<K, V> groupByKey(final Serialized<K, V> serialized);
<KR> KGroupedStream<KR, V> groupBy(final KeyValueMapper<? super K, ? super V, KR> selector, Serialized<KR, V> serialized);
<VO, VR> KStream<K, VR> join(final KStream<K, VO> other, final ValueJoiner<? super V, ? super VO, ? extends VR> joiner, final JoinWindows windows, final Joined<K, V, VO> options);
<VT, VR> KStream<K, VR> join(final KTable<K, VT> other, final ValueJoiner<? super V, ? super VT, ? extends VR> joiner, final Joined<K, V, VT> options);
<VO, VR> KStream<K, VR> leftJoin(final KStream<K, VO> other, final ValueJoiner<? super V, ? super VO, ? extends VR> joiner, final JoinWindows windows, final Joined<K, V, VO> options);
<VT, VR> KStream<K, VR> leftJoin(final KTable<K, VT> other, final ValueJoiner<? super V, ? super VT, ? extends VR> joiner, final Joined<K, V, VT> options);
<VO, VR> KStream<K, VR> outerJoin(final KStream<K, VO> other, final ValueJoiner<? super V, ? super VO, ? extends VR> joiner, final JoinWindows windows, final Joined<K, V, VO> options);
<VT |
Code Block |
---|
|
<KR, VR> KStream<KKGroupedTable<KR, VR> outerJoingroupBy(final KTable<K,KeyValueMapper<? VT>super otherK, final ValueJoiner<? super V, ?KeyValue<KR, superVR>> VTselector, ?Serialized<KR, extends VR> joiner, final Joined<K, V, VT> options); |
Code Block |
---|
|
<KR, VR> KGroupedTable<KR, VR> groupBy(final KeyValueMapper<serialized);
KTable<K, V> filter(final Predicate<? super K, ? super VV> predicate, final KeyValue<KRMaterialized<K, VR>> selectorV, Serialized<KRKeyValueStore<Bytes, VR> serializedbyte[]>> materialized);
KTable<K, V> filterfilterNot(final Predicate<? super K, ? super V> predicate, final Materialized<K, V, KeyValueStore<KKeyValueStore<Bytes[], V>>byte[]>> materialized);
<VR> KTable<K, V>VR> filterNotmapValues(final Predicate<ValueMapper<? super KV, ? superextends V>VR> predicatemapper, final Materialized<K, V, KeyValueStore<KKeyValueStore<Bytes[], V>>byte[]>> materialized);
<VR><VO, VR> KTable<K, VR> mapValuesjoin(final ValueMapper<?KTable<K, superVO> Vother,
? extends VR> mapper, final Materialized<K, V, KeyValueStore<K, V>> materialized);
void to(final String topic, final Produced<V, V> options);
KTable<K, V> through(final String topic, final Materialized<K, V> options);
<VO, VR> KTable<K, VR> join(final KTable<K, VO> other,
final ValueJoiner<? super V, ? super VO, ? extends VR> joiner,
final Materialized<K, VR, KeyValueStore<KKeyValueStore<Bytes, VR>>byte[]>> materialized);
<VO, VR> KTable<K, VR> leftJoin(final KTable<K, VO> other,
final ValueJoiner<? super V, ? super VO, ? extends VR> joiner,
final Materialized<K, VR, KeyValueStore<KKeyValueStore<Bytes, VR>>byte[]>> materialized);
<VO, VR> KTable<K, VR> outerJoin(final KTable<K, VO> other,
final ValueJoiner<? super V, ? super VO, ? extends VR> joiner,
final Materialized<K, VR, KeyValueStore<KKeyValueStore<Bytes, VR>>byte[]>> materialized);
|
We add some new helper methods to Stores so people can conveniently and quickly create basic StateStoreSuppliers for use in the DSL or PAPI. We will also deprecate the existing Stores.create(...)
Code Block |
---|
|
public static <K, V> BytesStoreSupplier<KeyValueStore<Bytes, byte[]>>KeyValueBytesStoreSupplier persistentKeyValueStore(final String name)
public static <K, V> BytesStoreSupplier<KeyValueStore<Bytes, byte[]>> inMemoryKeyValueStore(KeyValueBytesStoreSupplier inMemoryKeyValueStore(final String name)
public static <K, V> BytesStoreSupplier<KeyValueStore<Bytes, byte[]>> KeyValueBytesStoreSupplier lruMap(final String name)
public static <K, V> BytesStoreSupplier<WindowStore<Bytes, byte[]>>WindowBytesStoreSupplier persistentWindowStore(final String name,
final Windows windows)
public static <K, V> BytesStoreSupplier<SessionStore<Bytes, byte[]>> persistentSessionStore(final String name,
long retentionPeriod,
final int numSegments,
final long windowSize,
final boolean retainDuplicates)
public static <K, V> SessionBytesStoreSupplier persistentSessionStore(final String name, final long retentionPeriod)
/**
* The following methods are for use with the PAPI. They allow building of StateStores that can be wrapped with
* caching, logging, and any other convenient wrappers provided by the KafkaStreams library
*/
public <K, V> StateStoreBuilder<WindowStore<K, V>> windowStoreBuilder(final WindowBytesStoreSupplier supplier,
final Serde<K> keySerde,
final Serde<V> valueSerde)
public <K, V> StateStoreBuilder<KeyValueStore<K, V>> keyValueStoreBuilder(final KeyValueBytesStoreSupplier supplier,
final SessionWindowsSerde<K> windows)
/**
* The following methods are for use with the PAPI. They allow building of StateStores that can be wrapped with
* caching, logging, and any other convenient wrappers provided by the KafkaStreams library
*/
keySerde,
final Serde<V> valueSerde)
public <K, V> StateStoreBuilder<WindowStore<KStateStoreBuilder<SessionStore<K, V>> windowStoreBuildersessionStoreBuilder(final BytesStoreSupplier<WindowStore<Bytes, byte[]>> SessionBytesStoreSupplier supplier,
final Serde<K> keySerde,
final Serde<V> valueSerde,
final String metricsScope)
public ) |
Code Block |
---|
language | java |
---|
title | Topology |
---|
|
public synchronized <K, V> StateStoreBuilder<KeyValueStore<K, V>> keyValueStoreBuilderTopology addGlobalStore(final BytesStoreSupplier<KeyValueStore<Bytes, byte[]>> supplier,
final Serde<K> keySerde,
StateStoreBuilder storeSupplier,
final Serde<V> valueSerde,
final String metricsScope)
public <K, V> StateStoreBuilder<SessionStore<K, V>> sessionStoreBuilder(final BytesStoreSupplier<SessionStore<Bytes, byte[]>> supplier,
final Serde<K> keySerde,
final Serde<V> valueSerde,
final String metricsScope) final String sourceName,
final TimestampExtractor timestampExtractor,
final Deserializer keyDeserializer,
final Deserializer valueDeserializer,
final String topic,
final String processorName,
final ProcessorSupplier stateUpdateSupplier)
public synchronized final Topology addStateStore(final StateStoreBuilder supplier, final String... processorNames)
|
Code Block |
---|
language | java |
---|
title | Topology | KGroupedStream |
---|
|
<W extends Window> TimeWindowedKStream<K, V> windowedBy(Windows<W> timeWindows);
SessionWindowedKStream<K, V> windowedBy(SessionWindows sessionWindows);
KTable<K, Long> count(final Materialized<K, Long> materialized);
KTable<K, V> reduce(final Reducer<V> reducer, final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);
<VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
public synchronized <K, V> Topology addGlobalStore(final StateStoreBuilder storeSupplier,
final Aggregator<? super K, ? super V, VR> aggregator,
final String sourceName,
final Materialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized); |
Code Block |
---|
language | java |
---|
title | KGroupedTable |
---|
|
KTable<K, Long> count(final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);
KTable<K, V> reduce(final Reducer<V> adder, final Reducer<V> subtractor, final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);
<VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
final TimestampExtractor timestampExtractor,
final Aggregator<? super K, ? super V, VR> aggregator,
final Aggregator<? Deserializersuper keyDeserializerK,
? super V, VR> subtractor,
final Materialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized); |
For StreamsBuilder we remove all stream, table, and globalTable overloads that take more than a single argument and replace them with:
Code Block |
---|
language | java |
---|
title | StreamsBuilder |
---|
|
public synchronized <K, V> KStream<K, V> stream(final String topic)
public synchronized <K, V> KStream<K, V> stream(final String topic, final Deserializer valueDeserializerConsumed<K,
V> options)
public synchronized <K, V> KStream<K, V> stream(final Collection<String> topic, final Consumed<K, V> options)
public synchronized <K, V> KStream<K, V> stream(final Collection<String> topic)
public synchronized <K, V> KStream<K, V> stream(final Pattern pattern, final Consumed<K, V> options)
public synchronized <K, V> KTable<K, V> table(final String topic, final Consumed<K, V> consumed)
public synchronized <K, V> KTable<K, V> table(final String topic,
final Consumed<K, V> consumed, final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized)
public synchronized <K, V> KTable<K, V> table(final String topic, final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized)
public synchronized <K, V> GlobalKTable<K, V> globalTable(final String topic, final Consumed<K, V> consumed)
public synchronized <K, V> GlobalKTable<K, V> globalTable(final String topic, final Consumed<K, V> consumed, final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized)
public synchronized <K, V> GlobalKTable<K, V> globalTable(final String processorNametopic,
final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized) |
New classes and interfaces:
Code Block |
---|
language | java |
---|
title | WindowedKStream |
---|
|
public interface TimeWindowedKStream<K, V> {
KTable<Windowed<K>, Long> count();
KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materializedAs);
<VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
final ProcessorSupplier stateUpdateSupplier)
public synchronized final Topology addStateStore(final StateStoreBuilder supplier, final String... processorNames)
|
Code Block |
---|
language | java |
---|
title | KGroupedStream |
---|
|
<W extends Window> WindowedKStream<K, V> windowedBy(Windows<W> timeWindows);
SessionWindowedKStream<K, V> windowedBy(SessionWindows sessionWindows);
KTable<K, Long> count(final Materialized<K, Long> materialized);
KTable<K, V> reduce(final Reducer<V> reducer, final Materialized<K, V, KeyValueStore<K, V>> materialized);
<VR> KTable<K final Aggregator<? super K, ? super V, VR> aggregator);
<VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
final Aggregator<? super K, ? super V, VR> aggregator,
final Aggregator<? super K, ? super V, VR> aggregator,
final Serde<VR> aggValueSerde,
final Materialized<K, VR, KeyValueStore<KWindowStore<Bytes, VR>> materialized); |
Code Block |
---|
language | java |
---|
title | KGroupedTable |
---|
|
KTable<K, Long> count(final Materialized<K, V, KeyValueStore<K, V>> materialized);
KTable<K, byte[]>> materializedAs);
KTable<Windowed<K>, V> reduce(final Reducer<V> adder, final Reducer<V> subtractor, final Materialized<K, V, KeyValueStore<K, V>> materializedreducer);
<VR> KTable<KKTable<Windowed<K>, VR>V> aggregatereduce(final Initializer<VR>Reducer<V> initializerreducer,
final Aggregator<? super K, ?final superMaterialized<K, V, VR> aggregatorWindowStore<Bytes,
byte[]>> materializedAs);
} |
Code Block |
---|
language | java |
---|
title | SessionWindowedKStream |
---|
|
public interface SessionWindowedKStream<K, V> {
KTable<Windowed<K>, Long> count();
KTable<Windowed<K>, Long> count(final final Aggregator<? super K, ? super V, VR> subtractorMaterialized<K, Long, SessionStore<Bytes, byte[]>> materializedAs);
<VR, T> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
final Materialized<K, VR, KeyValueStore<K, VR>> materialized); |
For StreamsBuilder we remove all stream, table, and globalTable overloads that take more than a single argument and replace them with:
Code Block |
---|
language | java |
---|
title | StreamsBuilder |
---|
|
public synchronized <K, V> KStream<K, V> stream(final String topic)
public synchronized <K, V> KStream<K, V> stream(final String topicAggregator<? super K, final? super Consumed<KV, V>VR> options)
public synchronized <K, V> KStream<K, V> stream(final Collection<String> topic, final Consumed<K, V> options)
public synchronized <K, V> KStream<K, V> stream(final Pattern pattern, final Consumed<K, V> options)
public synchronized <K, V> KTable<K, V> table(final String topic, final Consumed<K, V> consumed)
public synchronized <K, V> KTable<K, V> table(final String topic, final Consumed<K, V> consumed, final Materialized<K, V> materialized)
public synchronized <K, V> GlobalKTable<K, V> globalTable(final String topic, final Consumed<K, V> consumed)
public synchronized <K, V> GlobalKTable<K, V> globalTable(final String topic, final Consumed<K, V> consumed, final Materialized<K, V> materialized) |
New classes and interfaces:
Code Block |
---|
language | java |
---|
title | WindowedKStream |
---|
|
public interface WindowedKStream<K, V> {
KTable<Windowed<K>, Long> count();
KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<K, Long>> materializedAs);
<VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
aggregator,
final Merger<? super K, T> sessionMerger);
<VR, T> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
final Aggregator<? super K, ? super V, VR> aggregator,
final Merger<? super K, T> sessionMerger,
final Aggregator<? super K, ? super V, VR> aggregator);
<VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
final Materialized<K, VR, SessionStore<Bytes, byte[]>> materializedAs);
KTable<Windowed<K>, V> reduce(final Reducer<V> reducer);
KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
final Aggregator<? super K, ? super V, VR> aggregator,
final Materialized<K, V, SessionStore<Bytes, byte[]>> materializedAs);
} |
Code Block |
---|
language | java |
---|
title | Materialized |
---|
|
/**
* Used when materializing a state store, i.e, during an aggregation operation or KTable operations
*/
public class final Materialized<K, VRV, WindowStore<K, VR>> materializedAs);
KTable<Windowed<K>, V> reduce(final Reducer<V> reducer);
KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,S extends StateStore> {
public static <K, V, S extends StateStore> Materialized<K, V, S> as(final String storeName)
public static <K, V> Materialized<K, V, WindowStore<Bytes, byte[]>> as(final WindowBytesStoreSupplier supplier);
public static <K, V> Materialized<K, V, SessionStore<Bytes, byte[]>> as(final SessionBytesStoreSupplier supplier);
finalpublic static Materialized<K<K, V,V> WindowStore<K, V>> materializedAs);
} |
Code Block |
---|
language | java |
---|
title | SessionWindowedKStream |
---|
|
public interface SessionWindowedKStream<K, V> {
KTable<Windowed<K>, Long> count();Materialized<K, V, KeyValueStore<Bytes, byte[]>> as(final KeyValueBytesStoreSupplier supplier)
public Materialized<K, V, S> withValueSerde(final Serde<V> valueSerde)
KTable<Windowed<K>, Long> count(final public Materialized<K, LongV, SessionStore<K, Long>> materializedAs);S> withKeySerde(final Serde<K> valueSerde)
<VR, T> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
public Materialized<K, V, S> withLoggingEnabled(final Map<String, String> topicConfig)
public Materialized<K, V, S> withLoggingDisabled()
public Materialized<K, V, S> withCachingEnabled()
public Materialized<K, V, S> withCachingDisabled()
public String storeName();
public StoreSupplier<S> storeSupplier()
public Serde<K> keySerde()
public Serde<V> valueSerde()
public boolean loggingEnabled()
public Map<String, String> logConfig()
public boolean cachingEnabled()
} |
Code Block |
---|
language | java |
---|
title | Serialized |
---|
|
/**
* Optional params that can be passed to groupBy finaland Aggregator<?groupByKey super K, ? super V, VR> aggregator,operations
*/
public class Serialized<K, V> {
public static <K, V> Serialized<K, V> with(final Serde<K> keySerde, final Serde<V> valueSerde)
public Serialized<K, V> withKeySerde(final Serde<K> keySerde)
public Serialized<K, V> withValueSerde(final Serde valueSerde)
public Serde<K> keySerde()
public Serde<V> final Merger<? super K, T> sessionMerger);
<VR, T> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
final Aggregator<? super K, ? super V, VR> aggregator,
final Merger<? super K, T> sessionMerger,
final Materialized<K, VR, SessionStore<K, VR>> materializedAs);
KTable<Windowed<K>, V> reduce(final Reducer<V> reducer);
KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
final Materialized<K, V, SessionStore<K, V>> materializedAs);
}valueSerde()
} |
Code Block |
---|
|
/**
* Optional params that can be passed to join, leftJoin, outerJoin operations
*/
public class Joined<K, V, VO> {
public static <K, V, VO> Joined<K, V, VO> with(final Serde<K> keySerde, final Serde <V> valueSerde, final Serde<VO> otherValueSerde)
public static <K, V, VO> Joined<K, V, VO> keySerde(final Serde<K> keySerde)
public static <K, V, VO> Joined<K, V, VO> valueSerde(final Serde<V> valueSerde)
public static <K, V, VO> Joined<K, V, VO> otherValueSerde(final Serde<V> valueSerde)
public Joined<K, V, VO> withKeySerde(final Serde<K> keySerde)
public Joined<K, V, VO> withValueSerde(final Serde<V> valueSerde)
public Joined<K, V, VO> withOtherValueSerde(final Serde<VO> otherValueSerde)
public Serde<K> keySerde()
public Serde<V> valueSerde()
public Serde<VO> otherValueSerde()
} |
Code Block |
---|
language | java |
---|
title | Produced |
---|
|
/**
* Optional arguments that can be specified when doing to and through operations
*/
public static <K, V> Produced<K, V> with(final Serde<K> keySerde, final Serde<V> valueSerde)
public static <K, V> Produced<K, V> with(final Serde<K> keySerde, final Serde<V> valueSerde, final StreamPartitioner<K, V> partitioner)
public static <K, V> Produced<K, V> keySerde(final Serde<K> keySerde)
public static <K, V> Produced<K, V> valueSerde(final Serde<V> valueSerde)
public static <K, V> Produced<K, V> streamPartitioner(final StreamPartitioner<K, V> partitioner)
public Produced<K, V> withStreamPartitioner(final StreamPartitioner<K, V> partitioner)
public Produced<K, V> withValueSerde(final Serde<V> valueSerde)
public Produced<K, V> withKeySerde(final Serde<K> keySerde)
public Serde<K> keySerde()
public Serde<K> valueSerde()
public StreamPartitioner<K, V> streamPartitioner() |
Code Block |
---|
language | java |
---|
title | MaterializedPrinted |
---|
|
/**
* Usedoptions whenthat materializingcan abe stateused store, i.e, during an aggregation operation or KTable operationswhen printing to stdout our writing to a file
*/
public class Materialized<K, V, S extends StateStore> {Printed<K, V> {
public static <K, V> Printed<K, V> toFile(final String filepath)
public static <K, V, S extends StateStore> Materialized<K, V, S> asV> Printed<K, V> toSysOut()
public Printed<K, V> withLabel(final String storeNamelabel)
public static <K, V, S extends StateStore> Materialized<K, V, S> as(final BytesStoreSupplier<S> supplier)
public Materialized<K, V, S> withValueSerde(final Serde<V> valueSerde)
Printed<K, V> withKeyValueMapper(final KeyValueMapper<? super K, ? super V, String> mapper)
public ProcessorSupplier<K, V> build(final String processorName)
}
|
Code Block |
---|
language | java |
---|
title | Consumed |
---|
|
/**
* Options for consuming a topic as a KStream or KTable
*/
public class Consumed<K, V> {
public Materialized<Kstatic <K, VV> Consumed<K, S>V> withKeySerdewith(final Serde<K> valueSerde)
public Materialized<K, V, S> withLoggingEnabled(final Map<String, String> topicConfig)
keySerde, final Serde<V> valueSerde, final TimestampExtractor extractor, final Topology.AutoOffsetReset resetPolicy)
public Materialized<K,static V<K, S>V> withLoggingDisabled()
Consumed<K, V> with(final publicSerde<K> Materialized<KkeySerde, V,final S>Serde<V> withCachingEnabled(valueSerde)
public Materialized<K, Vstatic <K, V> Consumed<K, S>V> withCachingDisabledwith()
} |
Code Block |
---|
language | java |
---|
title | Serialized |
---|
|
/**
* Optional params that can be passed to groupBy and groupByKey operations
*/
public class Serialized<K, V> {
final TimestampExtractor extractor)
public static <K, V> Consumed<K, V> with(final Topology.AutoOffsetReset resetPolicy)
public static <KConsumed<K, V> Serialized<K, V> withwithKeySerde(final Serde<K> keySerde)
public Consumed<K, V> withValueSerde(final Serde<V> valueSerde)
public Serialized<KConsumed<K, V> withKeySerdewithTimestampExtractor(final Serde<K>TimestampExtractor keySerdetimestampExtractor)
public Serialized<K, V> withValueSerde(final Serde valueSerdeConsumed<K, V> withOffsetResetPolicy(final Topology.AutoOffsetReset resetPolicy)
public Serde<K> keySerde()
public Serde<V> valueSerde()
public TimestampExtractor timestampExtractor()
public Toploogy.AutoOffsetReset offsetResetPolicy()
} |
Code Block |
---|
language | java |
---|
title | JoinedStateStoreBuilder |
---|
|
/**
* Implementations Optionalof paramsthis thatwill canprovide bethe passedability to join, leftJoin, outerJoin operations wrap a given StateStore
* with or without caching/loggging etc.
*/
public class Joined<K, V, VO> {
interface StoreBuilder<T extends StateStore> {
StoreBuilder<T> withCachingEnabled();
public static <K, V, VO> Joined<K, V, VO> with(final Serde<K> keySerde, final Serde <V> valueSerde, final Serde<VO> otherValueSerde)
public static <K, V, VO> Joined<K, V, VO> keySerde(final Serde<K> keySerde)StoreBuilder<T> withLoggingEnabled(Map<String, String> config);
StoreBuilder<T> withLoggingDisabled()
T build();
Map<String, String> logConfig();
boolean loggingEnabled();
} |
Code Block |
---|
language | java |
---|
title | StoreSupplier |
---|
|
public interface StoreSupplier<T extends StateStore> {
public/**
static <K, V, VO> Joined<K,* V,Return VO>the valueSerde(finalname Serde<V> valueSerde)
public static <K, V, VO> Joined<K, V, VO> otherValueSerde(final Serde<V> valueSerde)
public Joined<K, V, VO> withKeySerde(final Serde<K> keySerde)
public Joined<K, V, VO> withValueSerde(final Serde<V> valueSerde)
public Joined<K, V, VO> withOtherValueSerde(final Serde<VO> otherValueSerde)
} |
Code Block |
---|
language | java |
---|
title | Produced |
---|
|
/**
* Optional arguments that can be specified when doing to and through operations
*/
public static <K, V> Produced<K, V> with(final Serde<K> keySerde, final Serde<V> valueSerde)
public static <K, V> Produced<K, V> with(final Serde<K> keySerde, final Serde<V> valueSerde, final StreamPartitioner<K, V> partitioner)
public static <K, V> Produced<K, V> keySerde(final Serde<K> keySerde)
public static <K, V> Produced<K, V> valueSerde(final Serde<V> valueSerde)
public static <K, V> Produced<K, V> streamPartitioner(final StreamPartitioner<K, V> partitioner)
public Produced<K, V> withStreamPartitioner(final StreamPartitioner<K, V> partitioner)
public Produced<K, V> withValueSerde(final Serde<V> valueSerde)
public Produced<K, V> withKeySerde(final Serde<K> keySerde)of this state store supplier.
* This must be a valid Kafka topic name; valid characters are ASCII alphanumerics, '.', '_' and '-'
*
* @return the name of this state store supplier
*/
String name();
/**
* Return a new {@link StateStore} instance.
*
* @return a new {@link StateStore} instance of type T
*/
T get();
/**
* Return a String that is used as the scope for metrics recorded by Metered stores
* @return metricsScope
*/
String metricsScope();
} |
Code Block |
---|
language | java |
---|
title | PrintedWindowBytesStoreSupplier |
---|
|
/**
* A store optionssupplier that can be used when printing to stdout our writing to a file can be used to create one or more {@link WindowStore} instances of type <Byte, byte[]>
*/
public interface WindowBytesStoreSupplier classextends Printed<KStoreSupplier<WindowStore<Bytes, V>byte[]>> {
public static <K, V> Printed<K, V> toFile(final String filepath)
public static <K, V> Printed<K, V> toSysOut()
public Printed<K, V> withLabel(final String label)
public Printed<K, V> withKeyValueMapper(final KeyValueMapper<? super K, ? super V, String> mapper)
}
|
Code Block |
---|
language | java |
---|
title | Consumed |
---|
|
/**
* Options for consuming a topic as a KStream or KTable
*/
public class Consumed<K, V> {
public static <K, V> Consumed<K, V> with(final Serde<K> keySerde, final Serde<V> valueSerde, final TimestampExtractor extractor, final Topology.AutoOffsetReset resetPolicy)
public static <K, V> Consumed<K, V> with(final Serde<K> keySerde, final Serde<V> valueSerde)
public static <K, V> Consumed<K, V> with(final TimestampExtractor extractor)
public static <K, V> Consumed<K, V> with(final Topology.AutoOffsetReset resetPolicy)
public Consumed<K, V> withKeySerde(final Serde<K> keySerde)
public Consumed<K, V> withValueSerde(final Serde<V> valueSerde)
public Consumed<K, V> withTimestampExtractor(final TimestampExtractor timestampExtractor)
public Consumed<K, V> withOffsetResetPolicy(final Topology.AutoOffsetReset resetPolicy)
} |
Code Block |
---|
language | java |
---|
title | StateStoreBuilder |
---|
|
/**
* Implementations of this will provide the ability to wrap a given StateStore
* with or without caching/loggging etc.
*/
public interface StateStoreBuilder<T extends StateStore> {
StateStoreBuilder<T> withCachingEnabled();
StateStoreBuilder<T> withLoggingEnabled(Map<String, String> config);
T build();
Map<String, String> logConfig();
boolean loggingEnabled();
} |
/**
* The number of segments the store has. If your store is segmented then this should be the number of segments
* in the underlying store. It is also used to reduce the amount of data that is scanned when caching is enabled
*
* @return number of segments
*/
int segments();
/**
* The size of the windows any store created from this supplier is creating
* @return window size
*/
long windowSize();
/**
* Whether or not this store is retaining duplicate keys. Usually only true if the store is being used
* for joins. Note this should return false if caching is enabled
* @return true if duplicates should be retained
*/
boolean retainDuplicates();
/**
* The time period for which the {@link WindowStore} will retain historic data
* @return retentionPeriod
*/
long retentionPeriod();
} |
Code Block |
---|
language | java |
---|
title | KeyValueBytesStoreSupplier |
---|
|
/**
* A store supplier that can be used to create one or more {@link KeyValueStore} instances of type <Byte, byte[]>
*/
public interface KeyValueBytesStoreSupplier extends StoreSupplier<KeyValueStore<Bytes, byte[]>> {
} |
Code Block |
---|
/**
* A store supplier that can be used to create one or more {@link SessionStore} instances of type <Byte, byte[]>
*/
public interface SessionBytesStoreSupplier extends StoreSupplier<SessionStore<Bytes, byte[]>> {
/* |
Code Block |
---|
language | java |
---|
title | BytesStoreSupplier |
---|
|
public interface BytesStoreSupplier<T extends StateStore> {
/**
* Return the name of this state store supplier.
* This must be a valid Kafka topic name; valid characters are ASCII alphanumerics, '.', '_' and '-'
*
* @returnThe thesize name of thisa statesegment, storein supplier
milliseconds. Used when caching is */
enabled to segment String name();
the cache
/**
and reduce the amount *of Returndata athat newneeds {@linkto StateStore} instance.
*be scanned when performing range queries
*
@return a new {@link StateStore}* instance@return ofsegmentInterval typein Tmilliseconds
*/
Tlong getsegmentIntervalMs();
} |
Proposed Changes
Add the above methods, interfaces, classes to the DSL. Deprecate existing overloads on KStream, KTable, and KGroupedStream that take more than the required parameters, for example, KTable#filter(Predicate, String) and KTable#filter(Predicate, StateStoreSupplier) will be deprecated. StateStoreSupplier will also be deprecated deprecated. All versions of KTable#through and KTable#to will be deprecated in favour of using KTable#toStream()#through and KTable#toStream()#to
The new Interface BytesStoreSupplier supersedes the existing StateStoreSupplier (which will remain untouched). This so we can provide a convenient way for users creating custom state stores to wrap them with caching/logging etc if they chose. In order to do this we need to force the inner most store, i.e, the custom store, to be a store of type `<Bytes, byte[]>`.
...