Current state: Accepted
Discussion thread: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Many stream processing applications need to maintain state across a period of time. Online services, for example, like to understand the behaviour of a typical user. In order to understand this behaviour, the interactions a user has with their website, or streaming service, are grouped together into sessions.
We will provide a way for developers using the DSL to specify that they want an aggregation to be aggregated into SessionWindows. Three overloaded methods will be added to KGroupedStream:
A typical aggregation might look like:
In order to process SessionWindows we’ll need to add a new Processor. This will be responsible for creating sessions, merging existing sessions into sessions with larger windows, and producing aggregates from a session’s values.
On each incoming record the process method will:
Find any adjacent sessions that either start or end within the inactivity gap, i,e., where the end time of the session is > now - inactivity gap, or the start time is < now + inactivity gap.
Merge any existing sessions into a new larger session using the SessionMerger to merge the aggregates of the existing sessions.
Aggregate the value record being processed with the merged session.
Store the new merged session in the SessionStore.
Remove any merged sessions from the SessionStore.
Late arriving data
Late arriving data is mostly treated the same as non-late arriving data, i.e., it can create a new session or be merged into an existing one. The only difference is that if the data has arrived after the retention period, defined by SessionWindows.until(..), a new session will be created and aggregated, but it will not be persisted to the store.
We propose to add a new class SessionWindows. SessionWindows will be able to be used with new overloaded operations on KGroupedStream, i.e, aggregate(…), count(..), reduce(…). A SessionWindows will have a defined gap, that represents the period of inactivity. It will also provide a method, until(...), to specify how long the data is retained for, i.e., to allow for late arriving data.
We propose to add a new type of StateStore, SessionStore. A SessionStore, is a segmented store, similar to a WindowStore, but the segments are indexed by session end time. We index by end time so that we can expire (remove) the Segments containing sessions where session endTime < stream-time - retention-period.
The records in the SessionStore will be stored by a Windowed key. The Windowed key is a composite of the record key and a TimeWindow. The start and end times of the TimeWindow are driven by the data. If the Session only has a single value then start == end. The segment a Session is stored in is determined by TimeWindow.end. Fetch requests against the SessionStore use both the TimeWindow.start and TimeWindow.end to find sessions to merge.
Each Segment is for a particular interval of time. To work out which Segment a session belongs in we simply divide TimeWindow.end by the segment interval. The segment interval is calculated as Math.max(retentionPeriod / (numSegments - 1), MIN_SEGMENT_INTERVAL).
|TimeWindow End||Segment Index|
As session aggregates arrive, i.e., on put, the implementation of SessionStore will:
use TimeWindow.end to get an existing segment or create a new Segment to store the aggregate in. A new Segment will only be created if the TimeWindow.end is within the retention period.
If the Segment is non-null, we add the aggregate to the Segment.
If the Segment was null, this signals that the record is late and has arrived after the retention period. This record is not added to the store.
When SessionStore.findSessionsToMerge(...) is called we find all the aggregates for the record key where TimeWindow.end >= earliestEndTime && TimeWindow.start <= latestStartTime. In order to do this:
Find the Segments to search by getting all Segments starting from earliestEndTime
Define the range query as:
from = (record-key, end=earliestEndTime, start=0)
to = (record-key, end=Long.MAX_VALUE, start=latestStartTime)
For example, if for an arbitrary record key we had the following sessions in the store:
|Session Start||Session End|
This is primarily provided for InteractiveQueries
Compatibility, Deprecation, and Migration Plan
None required as we are introducing new APIs only
The majority of this feature can be tested with unit tests, however we will write integration and/or system tests to cover the end-to-end scenarios.
Using the KeyValueStore to store the sessions. It would result in excessive IOs as we’d need to store a List per sessionId. So every lookup for a given sessionId would need to fetch and deserialize the list. Also, punctuate would need to retrieve every session every time it runs in order to determine those that need to be aggregated and forwarded.
Keeping the list of values in the store. Whilst this would result in us being able to use the existing Windows based API methods on KGroupedStream, it could result in excessive IO operations and possibly excessive storage use.
- Using punctuate to perform the aggregations and forward downstream. This would result in potentially fewer records being forwarded downstream as they would only be forwarded when a session is ‘closed’. However, it doesn’t align with the current approach in Kafka Streams, i.e., to use the cache for de-duplication and forward on commit, flush, or eviction (KIP-63). Further, punctuate currently depends on stream time and is will not be called in the absence of records.