This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • Troubleshooting Guide
Skip to end of metadata
Go to start of metadata

The following page documents common problems discovered with Ambari Metrics Service and provides a guide for things to look out for and already solved problems.

Important facts to collect from the system:

Problems with Metric Collector host
  • Output of "rpm -qa | grep ambari" on the collector host. 
  • Total available System memory, output of : "free -g"
  • Total available disk space and available partitions, output of : "df -h "
  • Total number of hosts in the cluster
  • Configs: /etc/ams-hbase/conf/hbase-env.sh, /etc/ams-hbase/conf/hbase-site.xml, /etc/ambari-metrics-collector/conf/ams-env.sh, /etc/ambari-metrics-collector/conf/ams-site.xml
  • Collector logs: /var/log/ambari-metrics-collector/ambari-metrics-collector.log, /var/log/ambari-metrics-collector/hbase-ams-master-<host>.log, /var/log/ambari-metrics-collector/hbase-ams-master-<host>.out
    Note: Additionally, If distributed mode is enabled, /var/log/ambari-metrics-collector/hbase-ams-zookeeper-<host>.log, /var/log/ambari-metrics-collector/hbase-ams-regionserver-<host>.log
  • Response to the following URLs -
    http://<ams-host>:6188/ws/v1/timeline/metrics/metadata
    http://<ams-host>:6188/ws/v1/timeline/metrics/hosts 
    The response will be JSON and can be attached as a file.
  • From AMS HBase Master UI - http://<METRICS_COLLECTOR_HOST>:61310
    • Region Count
    • StoreFile Count
    • JMX Snapshot - http://<METRICS_COLLECTOR_HOST>:61310/jmx


Problems with Metric Monitor host

  • Monitor log file: /etc/ambari-metrics-monitor/ambari-metrics-monitor.out

Check out Configurations - Tuning for scale issue troubleshooting.

Issue 1: AMS HBase process slow disk writes

The symptoms and resolutions below address the embedded mode of AMS only.

Symptoms:

BehaviorHow to detect
High CPU usageHBase process on Collector host taking up close to 100% of every core
HBase Log: Compaction times

grep hbase-ams-master-<host>.log | grep "Finished memstore flush"

This yields MB written in X milliseconds, generally 128 MBps and above is average speed unless the disk is contended.

Also this search reveals how many times compaction ran per minute. A value greater than 6 or 8 is a warning that write volume is far greater than what HBase can hold in memory

HBase Log: ZK timeout

HBase crashes saying zookeeper session timed out. This happens because in embedded mode the zk session timeout is limited to max of 30 seconds (HBase issue: fix planned for 2.1.3).

The cause is again slow disk reads.

Collector Log : "waiting for some tasks to finish"ambari-metric-collector log shows messages where AsyncProcess writes are queued up

Resolutions:

Configuration ChangeDescription
ams-hbase-site :: hbase.rootdirChange this path to a disk mount that is not heavily contended.
ams-hbase-ste :: hbase.tmp.dirChange this path to a location different from hbase.rootdir

ams-hbase-env :: hbase_master_heapsize

ams-hbase-site :: hbase.hregion.memstore.flush.size

Bump this value up so more data is held in memory to address I/O speeds.

If heap size is increased and resident memory usage does not go up, this parameter can be changed to address how much data can be stored in a memstore per Region. Default is set to 128 MB. The size is in bytes.

Be careful with modifying this value, generally limit the setting between 64 MB (small heap with fast disk write), to 512 MB (large heap > 8 GB, and average write speed), since more data held in memory means longer time to write it to disk during a Flush operation.

Issue 2: Ambari Metrics take a long time to load

Symptoms:

BehaviorHow to detect

Graphs: Loading time too long

Graphs: No data available

Check out service pages / host pages for metric graphs
Socket read timeoutsambari-server.log shows: Error message saying socket timeout for metrics
Ambari UI slowing down

Host page loading time is high, heatmaps do not show data

Dashboard loading time is too high

Multiple sessions result in slowness

Resolutions:

Upgrade to 2.1.2+ is highly recommended.

Following is a list of fixes in 2.1.2 release that should greatly help to alleviate the slow loading and timeouts:

https://issues.apache.org/jira/browse/AMBARI-12654

https://issues.apache.org/jira/browse/AMBARI-12983

https://issues.apache.org/jira/browse/AMBARI-13108

 


  • No labels