HBASE 监控指标

原网址：http://hbase.apache.org/book.html#hbase_metrics

HBase emits metrics which adhere to the Hadoop metrics API. Starting with HBase 0.95[3], HBase is configured to emit a default set of metrics with a default sampling period of every 10 seconds. You can use HBase metrics in conjunction with Ganglia. You can also filter which metrics are emitted and extend the metrics framework to capture custom metrics appropriate for your environment.

17.4.1. Metric Setup

For HBase 0.95 and newer, HBase ships with a default metrics configuration, or sink. This includes a wide variety of individual metrics, and emits them every 10 seconds by default. To configure metrics for a given region server, edit the conf/hadoop-metrics2-hbase.properties file. Restart the region server for the changes to take effect.

To change the sampling rate for the default sink, edit the line beginning with *.period. To filter which metrics are emitted or to extend the metrics framework, see http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html

HBase Metrics and Ganglia

By default, HBase emits a large number of metrics per region server. Ganglia may have difficulty processing all these metrics. Consider increasing the capacity of the Ganglia server or reducing the number of metrics emitted by HBase. See Metrics Filtering.

17.4.2. Disabling Metrics

To disable metrics for a region server, edit the conf/hadoop-metrics2-hbase.properties file and comment out any uncommented lines. Restart the region server for the changes to take effect.

17.4.3. Discovering Available Metrics

Rather than listing each metric which HBase emits by default, you can browse through the available metrics, either as a JSON output or via JMX. Different metrics are exposed for the Master process and each region server process.

Procedure 17.1. Access a JSON Output of Available Metrics

After starting HBase, access the region server's web UI, at http://REGIONSERVER_HOSTNAME:60030 by default (or port 16030 in HBase 1.0+).

Click the Metrics Dump link near the top. The metrics for the region server are presented as a dump of the JMX bean in JSON format. This will dump out all metrics names and their values. To include metrics descriptions in the listing — this can be useful when you are exploring what is available — add a query string of ?description=true so your URL becomes http://REGIONSERVER_HOSTNAME:60030/jmx?description=true. Not all beans and attributes have descriptions.

To view metrics for the Master, connect to the Master's web UI instead (defaults to http://localhost:60010 or port 16010 in HBase 1.0+) and click its Metrics Dump link. To include metrics descriptions in the listing — this can be useful when you are exploring what is available — add a query string of ?description=true so your URL becomes http://REGIONSERVER_HOSTNAME:60010/jmx?description=true. Not all beans and attributes have descriptions.

Procedure 17.2. Browse the JMX Output of Available Metrics

You can use many different tools to view JMX content by browsing MBeans. This procedure uses jvisualvm, which is an application usually available in the JDK.

Start HBase, if it is not already running.

Run the command jvisualvm command on a host with a GUI display. You can launch it from the command line or another method appropriate for your operating system.

Be sure the VisualVM-MBeans plugin is installed. Browse to Tools → Plugins. Click Installed and check whether the plugin is listed. If not, click Available Plugins, select it, and click Install. When finished, click Close.

To view details for a given HBase process, double-click the process in the Local sub-tree in the left-hand panel. A detailed view opens in the right-hand panel. Click the MBeans tab which appears as a tab in the top of the right-hand panel.

To access the HBase metrics, navigate to the appropriate sub-bean:

Master: Hadoop → HBase → Master → Server

RegionServer: Hadoop → HBase → RegionServer → Server

The name of each metric and its current value is displayed in the Attributes tab. For a view which includes more details, including the description of each attribute, click the Metadata tab.

17.4.4. Units of Measure for Metrics

Different metrics are expressed in different units, as appropriate. Often, the unit of measure is in the name (as in the metric shippedKBs). Otherwise, use the following guidelines. When in doubt, you may need to examine the source for a given metric.

Metrics that refer to a point in time are usually expressed as a timestamp.

Metrics that refer to an age (such as ageOfLastShippedOp) are usually expressed in milliseconds.

Metrics that refer to memory sizes are in bytes.

Sizes of queues (such as sizeOfLogQueue) are expressed as the number of items in the queue. Determine the size by multiplying by the block size (default is 64 MB in HDFS).

Metrics that refer to things like the number of a given type of operations (such as logEditsRead) are expressed as an integer.

17.4.5. Most Important Master Metrics

Note: Counts are usually over the last metrics reporting interval.

hbase.master.numRegionServers
Number of live regionservers

hbase.master.numDeadRegionServers
Number of dead regionservers

hbase.master.ritCount
The number of regions in transition

hbase.master.ritCountOverThreshold
The number of regions that have been in transition longer than a threshold time (default: 60 seconds)

hbase.master.ritOldestAge
The age of the longest region in transition, in milliseconds

17.4.6. Most Important RegionServer Metrics

Note: Counts are usually over the last metrics reporting interval.

hbase.regionserver.regionCount
The number of regions hosted by the regionserver

hbase.regionserver.storeFileCount
The number of store files on disk currently managed by the regionserver

hbase.regionserver.storeFileSize
Aggregate size of the store files on disk

hbase.regionserver.hlogFileCount
The number of write ahead logs not yet archived

hbase.regionserver.totalRequestCount
The total number of requests received

hbase.regionserver.readRequestCount
The number of read requests received

hbase.regionserver.writeRequestCount
The number of write requests received

hbase.regionserver.numOpenConnections
The number of open connections at the RPC layer

hbase.regionserver.numActiveHandler
The number of RPC handlers actively servicing requests

hbase.regionserver.numCallsInGeneralQueue
The number of currently enqueued user requests

hbase.regionserver.numCallsInReplicationQueue
The number of currently enqueued operations received from replication

hbase.regionserver.numCallsInPriorityQueue
The number of currently enqueued priority (internal housekeeping) requests

hbase.regionserver.flushQueueLength
Current depth of the memstore flush queue. If increasing, we are falling behind with clearing memstores out to HDFS.

hbase.regionserver.updatesBlockedTime
Number of milliseconds updates have been blocked so the memstore can be flushed

hbase.regionserver.compactionQueueLength
Current depth of the compaction request queue. If increasing, we are falling behind with storefile compaction.

hbase.regionserver.blockCacheHitCount
The number of block cache hits

hbase.regionserver.blockCacheMissCount
The number of block cache misses

hbase.regionserver.blockCacheExpressHitPercent
The percent of the time that requests with the cache turned on hit the cache

hbase.regionserver.percentFilesLocal
Percent of store file data that can be read from the local DataNode, 0-100

hbase.regionserver.<op>_<measure>
Operation latencies, where <op> is one of Append, Delete, Mutate, Get, Replay, Increment; and where <measure> is one of min, max, mean, median, 75th_percentile, 95th_percentile, 99th_percentile

hbase.regionserver.slow<op>Count
The number of operations we thought were slow, where <op> is one of the list above

hbase.regionserver.GcTimeMillis
Time spent in garbage collection, in milliseconds

hbase.regionserver.GcTimeMillisParNew
Time spent in garbage collection of the young generation, in milliseconds

hbase.regionserver.GcTimeMillisConcurrentMarkSweep
Time spent in garbage collection of the old generation, in milliseconds

hbase.regionserver.authenticationSuccesses
Number of client connections where authentication succeeded

hbase.regionserver.authenticationFailures
Number of client connection authentication failures

hbase.regionserver.mutationsWithoutWALCount
Count of writes submitted with a flag indicating they should bypass the write ahead log