After upgrading to HDP2.2.6 in our test cluster, I’ve started receiving “No Data Available” on a number of metrics services. (used to be Ganglia, and was replaced with Ambari Metrics).
I’m seeing the following in the log:
05 Jul 2015 18:39:09,094 ERROR [qtp-client-953] MetricsPropertyProvider:201 – Error getting timeline metrics. Can not connect to collector, socket error.
It appears, as per tcpdump, that the metrics service is getting:
{“exception”:”WebApplicationException”,”message”:”org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: METRIC_RECORD,master.FileSystem.MetaHlogSplitSize_95th_percentile\\x00myservername.com\\x00\\x00\\x00\\x01NSe\\xE5\\xCCams-hbase,1435986466106.4b43f4743abef911273fdc6e5d594d32.: Requested memory of 5771520 bytes could not be allocated from remaining memory of 153795468 bytes from global pool of 159055872 bytes after waiting for 10000ms.\n\tat org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:83)\n\tat org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:51)\n\tat org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:162)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1931)\n\tat org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3173)\n\tat org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)\n\tat org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
This also shows up in:
/var/log/ambari-metrics-collector/ambari-metrics-collector.log.
I tried the following (no change)
– restarting YARN
– restarting Ambari metrics
– Increasing Ambari metrics “metrics_collector_heapsize” from 0.5G to 1G
– increased hbase_master_heapsize, hbase_regionserver_heapsize, and hbase_regionserver_xmn_max by 0.5G
The last change did increase the “remaining memory” in the error, but it’s still happening…
Caused by: org.apache.phoenix.memory.InsufficientMemoryException: Requested memory of 5771520 bytes could not be allocated from remaining memory of 237466659 bytes from global pool of 237699072 bytes after waiting for 10000ms.
This is a small 4-node test cluster that’s mostly idle.