...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Table of Contents |
---|
Motivation
This FLIP aims to solve several problems/shortcomings in the current streaming source interface (SourceFunction
) and simultaneously to unify the source interfaces between the batch and streaming APIs. The shortcomings or points that we want to address are:
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public interface SourceOutput<E> extends WatermarkOutput { void emitRecord(E record, String splitId); void emitRecord(E record, String splitId, long timestamp); } |
SourceReaderContext
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
/** * Emit a record to the downstream. * * @param <E> the type of the record emitted by the {@link SplitReader} * @param <T> the type of records that are eventually emitted to the {@link SourceOutput}. * @param <SplitStateT> the mutable type of split state. */ public interface RecordEmitter<E, T, SplitStateT> { /** * Process and emit the records to the {@link SourceOutputRecordEmitter.Output}. A few recommendations to the implementation * are following: * * <ul> * <li>The method maybe interrupted in the middle. In that case, the same set of records will be passed * to the record emitter again later. The implementation needs to make sure it reades * <li> * </ul> * * @param element The intermediate element read by the SplitReader. * @param splitId The ID of the split the element was consumed from. * @param output The output to which the final records are emit to. * @param splitState The state of the split. */ void emitRecord(E element, SourceOutput<T> String splitId, Output<T> output, SplitStateT splitState) throws Exception; /** * The interface for RecordEmitter output. * * @param <T> the type of the record emit */ interface Output<T> { /** * Emit an element without a timestamp. Equivalent to {@link #collect(Object, Long) collect(timestamp, null)}; * * @param element */ void collect(T element) throws Exception; /** * Emit an element with timestamp. * * @param element * @param timestamp */ void collect(T element, Long timestamp) throws Exception; } } |
Public interface from RPC gateway
...