Status

Current state: Accepted

Discussion thread: here

JIRA: KAFKA-17895

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The number of keys in a state store is a useful metric to track, and Kafka Streams currently only emits this metric for RocksDB as an estimated value. This KIP adds a new metric to track the number of entries for in-memory state stores.

Public Interfaces

New Metric:

  • Metric name: num-keys
  • Telemetry name: org.apache.kafka.stream.num.keys
  • Level: INFO
  • Group: stream-state-metrics
  • Tags: thread-id, task-id
  • Description:
    • number of keys in the in-memory state store
  • Unit: long
  • MBean: kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)

Proposed Changes

This metric will be implemented in the following classes starting under streams/state/internals, where the gauge will record data only when the underlying state store is not persistent:

  • MeteredKeyValueStore
  • MeteredSessionStore
  • MeteredWindowStore

Compatibility, Deprecation, and Migration Plan

The addition of this metric will have no effect on existing users, and the performance impact of adding this metric is trivial. This metric is recorded at the INFO level to match the behavior of the equivalent RocksDB metric. 

Test Plan

Unit tests for this new metric will be added to ensure that it correctly tracks the size of the in memory state store.

Rejected Alternatives

Combining this metric with estimate-num-keys from RocksDB was considered, but a new metric was decided on to avoid altering the behavior of an existing metric. 

A slightly different name was chosen (num-keys rather than to estimate-num-keys in RocksDB) because tracking the size of an in-memory state store is more accurate than tracking the size of a persistent state store. 

  • No labels