Motivation

Currently Paimon runs without monitoring and metrics out of box. In the production environment, users need to know how their Paimon Table behaves like what is the commit duration, how many files each commit added or deleted, the status of compaction operation, the duration of scan query, lag of streaming reading and so on. So we need to introduce a metrics system and more metrics for Paimon.  

Public Interfaces

Measurable metric interfaces

Metric

Metric is the base measurable metric interface, which indicate a class is a metric.

public interface Metric {}

Gauge

Gauge is a type of metric interface provides a value of any type at a point in time.

/** Gauge calculates a specific value at a point in time. */
@Public
public interface Gauge<T> {

    /**
     * Calculates and returns the measured value.
     *
     * @return calculated value
     */
    T getValue();
}

Counter

Counter is a type of metric interface which is used to count values by incrementing and decrementing.

/** A Counter is a metric measured by incrementing and decrementing. */
@Public
public interface Counter extends Metric {

    /** Increment the current count by 1. */
    void inc();

    /**
     * Increment the current count by the given value.
     *
     * @param n value to increment the current count by
     */
    void inc(long n);

    /** Decrement the current count by 1. */
    void dec();

    /**
     * Decrement the current count by the given value.
     *
     * @param n value to decrement the current count by
     */
    void dec(long n);

    /**
     * Returns the current count.
     *
     * @return current count
     */
    long getCount();
}

Histogram

Histogram is a type of metric interface measure the statistical distribution of a set of values including the min, max, mean, standard deviation and percentile.

/** The histogram allows to record values, get the current count of recorded values and create
 * histogram statistics for the currently seen elements.*/
@Public
public interface Histogram extends Metric {

    /**
     * Update the histogram with the given value.
     *
     * @param value Value to update the histogram with
     */
    void update(long value);

    /**
     * Get the count of seen elements.
     *
     * @return Count of seen elements
     */
    long getCount();

    /**
     * Create statistics for the currently recorded elements.
     *
     * @return Statistics about the currently recorded elements
     */
    HistogramStatistics getStatistics();

	interface HistogramStatistics {
    	/**
     	* Return the number of values that the statistics computation is based on.
     	*/
    	int size();

    	/** Returns the mean value of the histogram observations. */
    	double mean();

    	/** Returns the standard deviation of the histogram distribution. */
    	double stdDev();

    	/** Returns the maximum value of the histogram observations. */
    	long max();

    	/** Returns the minimum value of the histogram observations. */
    	long min();

    	/**
     	* Returns the percentile value based on the histogram statistics.
     	*
     	* @param percentile percentile point in double. E.g., 0.75 means 75 percentile. It is up to the
     	*     implementation to decide what valid percentile points are supported.
     	* @return Value for the given percentile
     	*/
    	long percentile(double percentile);
  } 
}


Metrics

Class Metrics  is the core of metrics system, there are  MetricGroup  and  MetricsReporter  container in it. When the Metrics  instance is initiating, the MetricGroup  is instantiated and metrics reporters are started. 

Metrics reporters are configurable, users can use custom reporters, Paimon will provide a default metrics reporter of JMX metrics reporter. 

public class Metrics {

    /** The metrics reporters container. */
  	private final List<MetricsReporter> reporters = new ArrayList<>();

	/** The metrics group. */  
	private final List<MetricGroup> metricGroups = new ArrayList<>();   

	/** Add a metrics reporter. */
	public void addReporter(MetricsReporter reporter) {};

	/** Add a metric group. */
	public void addGroup(MetricGroup group) {};

	/** Get metrics reporters. */
	public List<MetricsReporter> getReporters() {};

	/** Get metric groups. */
	public List<MetricGroup> getMetricGroups() {};
}

MetricGroup

MetricGroup is a class responsible for metrics registering and tagging, there is a metrics container in it. It provides register method for each type of measurable metric, registering metrics will put metrics to the metrics container.

public interface MetricGroup {
   /** Register gauge metric. */
 	public void gauge(String name, Gauge gauge) {}

	/** Register counter metric. */
	public void counter(String name, Counter counter) {}

	/** Register histogram metric. */
	public void histogram(String name, Histogram counter) {}
}

Paimon support to add tags for metric groups, tags is likely to scopes in Flink's metric group. The tagged metrics group will report metric value with tags as a part of prefix of the metric name. 

Table partition and bucket can be added as tag for some metric groups. Like the number of files monitoring in committing / scan / compaction, users can monitor the files of different buckets to check the data skew, and by checking the compaction duration of different buckets to see which unexpected bucket caused the long compaction duration.

MetricsReporter

MetricsReporter  is used to report metrics to external backend, Paimon will implement an out of box reporter as JMX MetricsReporter .

public interface MetricsReporter {
	/** Configure reporter after instantiating it.*/
     void open();

    /** Closes this reporter. */
    void close();

	/** Report the current measurements. This method is called periodically by the Metrics. */
	void report();
}

Proposed Changes

Architecture

Metrics Registering

Take CommitMetrics as example, the CommitMetrics will be instantiated by FileStoreCommitImpl, then commit related metrics will be registered by the corresponding MetricGroup in the Metrics instance

The Metrics has instance of MetricGroups and MetricsReporters set. MetricGroup maintains metrics map containers. Metrics registering is a process of putting metrics instances into the metric (gauge, counter) map container. 

Update metrics value

The CommitMetrics values will be updated around commit() operation, for example the commit starting time will be recorded before commit operation, CommitDuration value will be recorded after commit completing.

CompactionMetrics values will be updated around compaction operation, and ScanMetrics will be recorded through the scan planning process.

Report metrics to external backend

Each reporter instance has an timer task fetching metrics from the metrics containers periodically and report them out to the external backends. Paimon will has a default reporter backend with JMX, users can define their own MetricsReporter by implement MetricsReporter  interface. Here we can introduce a core option metrics.reporter  to specify a metrics backend.

Metrics list

We introduce CommitMetrics , CompactionMetrics , ScanMetrics as metrics set to measure the stats of Paimon table committing, compaction and scanning.

Common metrics

CommitMetrics

public class CommitMetrics {
	private final String LAST_COMMIT_DURATION_METRIC = "lastCommitDuration";
	...

    private final Map<Integer, BucketMetricGroup> bucketMetricGroups = new HashMap<>();
    private final Map<BinaryRow, PartitionMetricGroup> partitionMetricGroups = new HashMap<>();
    private final MetricGroup genericMetricGroup;

	private void registerGenericCommitMetrics() {   
		group.gauge(LAST_COMMIT_DURATION_METRIC, new CommitDurationGauge());
		...
	}

 	private void registerTaggedCommitMetrics() {
		if(isNewBucket()) {
			registerTaggedMetrics();
		}

	 	if(isNewBucket()) {
			registerTaggedMetrics();
		}
	}
}

CommitMetrics list includes commit duration, counter of files / records etc.

Metric Name

Description

Type

Tagged

Unit

Update at

lastCommitDuration

The time it took to complete the last commit.

Gauge

none

Ms

Timer starts before commit starting, update commit duration after the last commit finished.

commitDuration

Distributions of the time taken by the last few commits.

Histogram

none

Ms

Timer starts before commit starting, update commit duration after each commit finished.

lastCommitAttempts

The number of attempts the last commit made.

Gauge

none

Number

Increment by 1 when trying to commit once, clear the counter after the last commit finished.

lastTableFilesAdded

Number of added table files in last commit, including newly created data files and compacted after.

Gauge

none

Number

Collecting changes from committables

lastTableFilesDeleted

Number of deleted table files in last commit, which comes from compacted before.

Gauge

none

Number

Collecting changes from committables

lastTableFilesAppended

Number of appended table files in last commit, which means the newly created data files.

Gauge

none

Number

Collecting changes from committables

lastTableFilesCommitCompated

Number of compacted table files in last commit, including compacted before and after.

Gauge

none

Number

Collecting changes from committables

lastChangelogFilesAppended

Number of appended changelog files in last commit

Gauge

none

Number

Collecting changes from committables

lastChangelogFileCommitCompacted

Number of compacted changelog files in last commit

Gauge

none

Number

Collecting changes from committables

lastGeneratedSnapshots

Number of snapshot files generated in last commit, maybe 1 snapshot or 2 snapshots.

Gauge

none

Number

After collecting changes from committables

lastDeltaRecordsAppended

Delta records count in last commit with APPEND commit kind

Gauge

none

Number

Preparing snapshot file with APPEND commit kind

lastChangelogRecordsAppended

Changelog records count in last commit with APPEND commit kind

Gauge

none

Number

Preparing snapshot file with APPEND commit kind

lastDeltaRecordsCommitCompated

Delta records count in last commit with COMPACT commit kind

Gauge

none

Number

Preparing snapshot file with COMPACT commit kind

lastChangelogRecordsCommitCompated

Changelog records count in last commit with COMPACT commit kind

Gauge

none

Number

Preparing snapshot file with COMPACT commit kind

lastPartitionsWritten

Number of partitions written in last commit

Gauge

none

Number

After collecting changes from committables

lastBucketsWritten

Number of buckets written in last commit

Gauge

none

Number

After collecting changes from committables

ScanMetrics

public class ScanMetrics {
	private final String LAST_SCAN_SKIPPED_MANIFESTS_METRIC = "lastScanSkippedManifests";
	...
 	
	private final Map<Integer, BucketMetricGroup> bucketMetricGroups = new HashMap<>();
    private final Map<BinaryRow, PartitionMetricGroup> partitionMetricGroups = new HashMap<>();

 	private void registerTaggedScanMetrics() {
		if(isNewBucket()) {
			registerTaggedMetrics();
		}

	 	if(isNewPartition()) {
			registerTaggedMetrics();
		}
	}       
}

ScanMetrics list includes duration, data files and manifest files counter.

Metric Name

Description

Type

Tagged

Unit

Update at

lastScanDuration

The time it took to complete the last scan planning.

Gauge

none

Ms

Timer starts before the scan planning starts, update after the last scan planning operation finished

scanDuration

Distributions of the time taken by the last few scan planning operations.

Histogram

none

Ms

Timer starts before the scan planning starts, update after each planning finished

lastScannedManifests

Number of scanned manifests files in the last scan planning.

Gauge

none

Number

Scan planning

lastScanSkippedManifests

Number of skipped manifests files in the last scan planning.

Gauge

none

Number

Scan planning

lastScanResultTableFiles

Number of result table files in the last scan planning.

Gauge

none

Number

Scan planning

CompactionMetrics

public class CompactionMetrics {

	private final String LAST_TABLE_FILES_COMPACTED_BEFORE_METRIC = "lastTableFilesCompactedBefore";
 	...
 	
	private final Map<Integer, BucketMetricGroup> bucketMetricGroups = new HashMap<>();
    private final Map<BinaryRow, PartitionMetricGroup> partitionMetricGroups = new HashMap<>();

 	private void registerTaggedCompactionMetrics() {
		if(isNewBucket()) {
			registerTaggedMetrics();
		}

	 	if(isNewPartition()) {
			registerTaggedMetrics();
		}
	}
}

CompactionMetrics list includes duration, and counter of files, sorted runs etc.

Metric Name

Description

Type

Tagged

Unit

Update at

lastCompactionDuration

The time it took to complete the last compaction.

Gauge

none

Ms

Timer starts before compaction, update after the last compaction finished

compactionDuration

Distributions of the time taken by the last few compactions.

Histogram

none

Ms

Timer starts before compaction, update after each compaction finished

lastTableFilesCompactedBefore

Number of deleted files in last compaction

Gauge

none

Number

After getting compaction result.

lastTableFilesCompactedAfter

Number of added files in last compaction

Gauge

none

Number

After getting compaction result.

lastChangelogFilesCompacted

Number of changelog files compacted in last compaction

Gauge

none

Number

After getting compaction result.

lastCompactOutputLevel

Number of total sorted runs of all levels at last compaction

Gauge

none

Number

Updating levels after getting compaction result.

lastCompactFileSize

Sum of file size compacted in last compaction

Gauge

none

Number

Updating levels after getting compaction result.

Flink connector metrics

Implement important source metrics in FLIP-33.

Metric name

Description

Type

Unit

Update at

numBytesIn

The total number of input bytes since the source started.

Counter

Number

FileStoreSourceSplitReader fetching.

pendingRecords

The number of records that have not been fetched by the source.

Gauge

Number

FileStoreSourceSplitReader fetching.

Compatibility, Deprecation, and Migration Plan

There are no changes to the public interface and no impact to existing users.

Test Plan


Rejected Alternatives

  • No labels