...
Access to the structure should be guarded by a lock, so that not every map must a concurrent one.There aren't any plans for data retirement right now
Data Retirement
- JobManager metrics are kept indefinitely, as they have a limited size
- TaskManager metrics kept as long as the given TaskManager is registered on the JobManager
- Job and Task metrics are kept as long as they are running or archived. In other words, if the job is listed in the WebInterface as either running or completed, the metrics are still available.
Access from the WebInterface
Several new Handlers will be added (one for each category) that will access the central MetricStore.
The REST call calls for a the list of all available metrics of a task could look like this:
- JobManager: "/jobmanager/metrics"
- TaskManager: "/taskmanagers/:taskmanagerid/metrics"
- Job: "/jobs/:jobid/metrics"
- Task: "/jobs/:jobid/vertices/:vertexid/metrics"
This will return a JSON array containing the names of all available metrics.
The values for a list of metrics can be requested by appending "?get=[<metric_name1>[,<metric_nameX>]]"
This will return a JSON array containing "id":"values" pairs.jobs/JOB_ID/vertices/TASK_ID/metricsThe REST call for specific metrics of a task could look like this: /jobs/JOB_ID/vertices/TASK_ID/metrics?get=elements-in,elements-out,bytes-per-second-out
Prototype
A working prototype that follows this FLIP can be found here.
It allows the WebInterface to display task/operator metrics. Credits to Piotr Godek who provided the code to display of task metrics in the WebInterface.
...