Status

Current state: "Under Discussion" Released

Discussion thread: -

JIRA:

Released: -Flink 1.2

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

The MetricRegistry will contain a MetricDumperMetricQueryService, which acts like an unscheduled reporter.
The Dumper can be used by the Job-/TaskManager to create service is a separate actor that creates and returns a Key-Value representation of the entire metric space when queried.

The keys represent the name of the metric; formatted according to the following scope format strings:

...

As there aren't any details as to how the separation will work, specifically whether a TaskManager -> WebInterface heartbeat will exist, i will assume that there is no message that we can piggyback on.

As such the WebInterface will regularly query The WebRuntimeMonitor will contain a MetricFetcher which queries the JobManager for all available TaskManagers, and then query each of them for a metric dump.

This will be done in a separate Thread inside the WebRuntimeMonitor, which also has the responsibility to merge the returned dumps.

Metrics are only fetched if they actually accessed via REST calls, with a minimum time period (10 seconds) between updates.

The fetched metrics are merged and The merged dump is kept in a central location inside the WebRuntimeMonitorMetricFetcher, available to different handlers.

...

MetricStore {
	void addMetric(String name, Object value);

	JobManagerMetricStore  jobManager;jobMan

	class JobManagerMetricStore {
		Map<String, Object> metrics;
	}

	Map<String, TaskManager>TaskManagerMetricStore> taskmanagers;

	class TaskManagerMetricStore {
		Map<String, Object> metrics;
	}

	Map<String, Job>JobMetricStore> jobs;

	class JobMetricStore {
		Map<String, Object> metrics;
		Map<String, Task>TaskMetricStore > tasks;
	}

	class TaskMetricStore {
		Map<String, Object> metrics;
		Map<String, Subtask>SubtaskMetricStore>;
	}

	class SubtaskMetricStore  {
		Map<String, Object> metrics;
	}
}

...

• JobManager metrics are kept indefinitely, as they have a limited size
• TaskManager metrics are kept as long as the given TaskManager is registered on the JobManager
• Job and Task metrics are kept as long as they are running or archived. In other words, if the job is listed in the WebInterface as either running or completed, the metrics are still available.

...

Everything can be tested with unit tests.

Rejected Alternatives

-