Metrics are collected from task when metrics tick tuple is read by task.
It’s easy way to integrate metrics in thread-safe way, but it also means when task collects the metrics is undetermined (especially the task has lots of pending input tuples or execute latency is too high), which basically makes metrics flawed (no fixed interval between datapoint) and very hard to aggregate.
Metrics tick tuple is emitted based on each worker’s timer.
It can make the big difference for timestamp of metrics between workers. Suppose metric interval secs is 60, and worker B launches after 30 secs worker A is launched. Even though we assume the happy case of above case (it guarantees all tasks can provide their metrics in time), there’re 30 seconds difference between metrics from worker A and metrics from worker B.
IMetric has a method “Object getValueAndReset()” which is too flexible and also no guarantee of thread-safe.
Built-in metrics utilize it via having Map on value. While it’s easier way to expand a static metric to contain dynamic metrics, interpreting the value is completely depending on metrics consumers. Moreover we didn’t document it, so there’s only implicit contract, no explicit contract.
Assume we want to aggregate values of specific metric into one (by worker-component or component or so). Which function (Sum, Avg, Min, Max, etc) do we apply to values of specific metric? Could we determine that?
Implementations of IMetric are not thread-safe (for example, CountMetric) so the ways of collecting metrics are restricted. In order to collect task level metrics from worker thread or so, IMetric should ensure thread-safety. But how to ensure thread-safety between “get the value and reset the value” and “update the value”? I feel that there should be lock but introducing lock would heavily affect the performance.
Metrics are sent to MetricsConsumer via metrics tuples, and it can affect the performance of whole topology when MetricsConsumer can’t keep up processing the metrics tuples for given input rate.
RX queue is handled from single worker receive thread, and it contains user-level tuples and also metrics tuples if MetricsConsumerBolt(s) is/are running on that worker. Amount of pending metrics tuples can affect the worker performance and it also affects whole topology performance.
Built-in metrics are also used as task heartbeat, which are stored to ZK incurring heavy writes.
While we don’t need to collect metrics for every 3 seconds, we do this since it’s also used as task heartbeat. In other word, we write built-in metrics to ZK every 3 seconds, and many tasks in topology gives ZK lots of pressure. We introduce Pacemaker for handling this, but it’s not by default and I think pacemaker is not needed if we can move out metrics to other than task heartbeat. We can even change the role of Pacemaker to metrics daemon and make it mandatory.
- Built-in metrics are aggregated by several time-window and stored to each executor.
- When worker dies, metrics for executors in worker are all lost, and aggregated value goes really odd to users.
- Not possible to apply interpolation.
- The thing which makes issue even worse is, all actions including worker restart, rebalance trigger this, even though they're valid actions.
AFAIK, redesigned JStorm metrics feature resolves all of the above. (I’d like to confirm this from JStorm guys.) That’s why I’m waiting for JStorm merger phase instead of working on designing our own. There are some of dropped features like custom interval of user-defined metrics, and non-topology specific consumer (since Nimbus finally handles the metrics) but I think it’s not a big deal.