...
The following metrics will be exposed in the Kafka Streams' metrics
- num-immutable-mem-table
- mem-table-flush-pending
- compaction-pending
- background-errors
- cur-size-active-mem-table
- cur-size-all-mem-tables
- size-all-mem-tables
- num-entries-active-mem-table
- num-entries-imm-mem-tables
- num-deletes-active-mem-table
- num-deletes-imm-mem-tables
- estimate-num-keys
- estimate-table-readers-mem
- num-live-versions
- estimate-live-data-size
- total-sst-files-size
- live-sst-files-size
- estimate-pending-compaction-bytes
- num-running-compactions
- num-running-flushes
- actual-delayed-write-rate
- estimate-oldest-key-time
- block-cache-capacity
- block-cache-usage
- block-cache-pinned-usage
- memtables-size [bytes]
- caches-size [bytes]
- table-readers-memory-size [bytes]
The recording level for all metrics will be INFO.Note that the recorded values are approximations of the memory usages.
Proposed Changes
In this section, I will explain the meaning of the metrics listed in the previous section. To better understand the metrics, some basic concepts of RocksDB need to be explained.
- Memtable: Memtables are in-memory write buffers. Each new key-value pair is first written to a memtable and each read looks first into the memtable before it looks on disk. Once a memtable is full it becomes immutable and it is replaced by a new memtable. A background thread flushes a memtable asynchronously to disk. Additionally, memtables can also be flushed manually. RocksDB keeps in memory the currently active memtables, full but not yet flushed memtables, and flushed memtables that are kept around to maintain write history in memory.
- Compaction: From time to time RocksDB needs to clean up the data it stores on disk and bring is LSM tree into a good shape (see https://github.com/facebook/rocksdb/wiki/Compaction). Compactions might block writes and flushes. Additionally, RocksDB offers different compaction algorithms with different properties. Thus, it is a good practise to monitor compactions in RocksDB.
- SST files: SST files are the files in which RocksDB stores the data on disk. SST stands for Sorted Sequence Table.
num-immutable-mem-table
number Number of immutable memtables that have not yet been flushed.
mem-table-flush-pending
returns 1 if a memtable flush is pending; otherwhise, returns 0
compaction-pending
returns 1 if at least one compaction is pending; otherwise, return 0
background-errors
returns the accumulated number of background errorsFor segmented state stores, the sum of the number of immutable memtables over all segments is reported.
cur-size-active-mem-table
returns approximate Approximate size of active memtable in bytes. For segmented state stores, the sum of the sizes over all segments is reported.
cur-size-all-mem-tables
returns approximate Approximate size of active and unflushed immutable memtablesmemtable in bytes. For segmented state stores, the sum of sizes over all segments is reported.
size-all-mem-tables
returns approximate Approximate size of active, unflushed immutable, and pinned immutable memtables (bytes)in bytes. For segmented state stores, the sum of sizes over all segments is reported.
num-entries-active-mem-table
returns total Total number of entries in the active memtable. sum for segmented stores
num-entries-imm-mem-tables
returns total Total number of entries in the unflushed immutable memtables. sum for segmented stores
num-deletes-active-mem-table
returns total Total number of delete entries in the active memtable. sum for segmented stores
num-deletes-imm-mem-tables
returns total Total number of delete entries in the unflushed immutable memtables. sum for segmented stores
mem-table-flush-pending
This metric returns 1 if a memtable flush is pending; otherwhise it returns 0. sum for segmented stores
num-running-flushes
Number of currently running flushes. sum for segmented stores
num-running-compactions
Number of currently running compactions. For segmented state stores, the sum of the number of currently running compactions over all segments is reported.
estimate-pending-compaction-bytes
Estimated total number of bytes a compaction needs to rewrite to get all levels down to under target size. This metric is not valid for compactions other than level-based. For segmented state stores, the sum of the estimated total number of bytes over all segments is reported.
compaction-pending
This metric 1 if at least one compaction is pending; otherwise, the metric reports 0. For segmented state stores, the sum of ones and zeros over all segments is reported.
total-sst-files-size
Total size in bytes of all SST files. For segmented state stores, the sum of the sizes of SST files over all segments is reported.
live-sst-files-size
returns total size (bytes) of all SST files belong to the latest LSM tree. sum for segmented stores
background-errors
returns the accumulated number of background errors. sum for segmented stores
estimate-num-keys
returns estimated number of total keys in the active and unflushed immutable memtables and storage. sum for segmented stores
estimate-table-readers-mem
returns estimated memory used for reading SST tables, excluding memory used in block cache (e.g., filter and index blocks). sum for segmented stores
num-live-versions
returns number of live versions. `Version` is an internal data structure. See version_set.h for details. More live versions often mean more SST files are held from being deleted, by iterators or unfinished compactions. sum for segmented stores
estimate-live-data-size
returns an estimate of the amount of live data in bytes.
min-log-number-to-keep
min-obsolete-sst-number-to-keep
total-sst-files-size
live-sst-files-size
base-level
estimate-pending-compaction-bytes
num-running-compactions
num-running-flushes
sum for segmented stores. sum for segmented stores
actual-delayed-write-rate
is-write-stoppedreturns the current actual delayed write rate. 0 means no delay. sum for segmented stores
estimate-oldest-key-time
returns an estimation of oldest key timestamp in the DB. Currently only available for FIFO compaction with compaction_options_fifo.allow_compaction = false. min for segmented stores
block-cache-capacity
returns block cache capacity. sum for segmented stores if separate caches are used, otherwise any.
block-cache-usage
returns the memory size for the entries residing in block cache. sum for segmented stores if separate caches are used, otherwise any.
block-cache-pinned-usage
returns the memory size for the entries being pinned. sum for segmented stores if separate caches are used, otherwise any.
memtables-size
Memory usage of all mem-tables. Mem-tables are in-memory write buffers. Each new key-value pair is first written to a mem-table and each read looks first into the mem-table before it looks on disk. Once a mem-table is full it becomes immutable and it is replaced by a new mem-table. A background thread flushes a mem-table asynchronously to disk. Additionally, mem-tables can also be flushed manually. RocksDB keeps in memory the currently active mem-tables, full but not yet flushed mem-tables, and flushed mem-tables that are kept around to maintain write history in memory.
...