Region Server意外退出之后...

早晨上班好好的，突然nagios报出一台regionserver挂了。顿时忙碌起来。

上去一看，从log中看到这样一条信息

2011-04-08 04:02:22,083 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expired

之后， regionserver就理直气壮地退出了。

于是查了下代码，看到了在org.apache.hadoop.hbase.regionserver.HRegionSever.java下这样一段代码。

/**

* We register ourselves as a watcher on the master address ZNode. This is

* called by ZooKeeper when we get an event on that ZNode. When this method

* is called it means either our master has died, or a new one has come up.

* Either way we need to update our knowledge of the master.

* @param event WatchedEvent from ZooKeeper.

*/

public void process(WatchedEvent event) {

EventType type = event.getType();

KeeperState state = event.getState();

LOG.info("Got ZooKeeper event, state: " + state + ", type: " +

type + ", path: " + event.getPath());

// Ignore events if we're shutting down.

if (stopRequested.get()) {

LOG.debug("Ignoring ZooKeeper event while shutting down");

return;

}

if (state == KeeperState.Expired) {

LOG.error("ZooKeeper session expired");

boolean restart =

this.conf.getBoolean("hbase.regionserver.restart.on.zk.expire", false);

if (restart) {

restart();

} else {

abort();

}

} else if (type == EventType.NodeDeleted) {

watchMasterAddress();

} else if (type == EventType.NodeCreated) {

getMaster();

// ZooKeeper watches are one time only, so we need to re-register our watch.

watchMasterAddress();

}

}

这段注释写的很清楚了。对于一个reigonserver， 他需要将自己注册到Zookeeper上master的Znode上。这样的目的，是当master 宕机或者新的master启动的时候，能及时收到通知。对于regionserver来说，维持和Zookeeper的联系是非常重要的。因为regionserver需要定期的将心跳包发给master server。如果regionserver不能及时的知道master的改变，就会导致regionserver和master失去联系，而成为一个僵死的进程。

于是，在默认情况下，regionserver遇到这种情况，就选择退出。

为什么regionserver 和Zookeeper的session expired? 可能的原因有

1. 网络不好。

2. Java full GC， 这会block所有的线程。如果时间比较长，也会导致session expired.

怎么办？

1. 将Zookeeper的timeout时间加长。

2. 配置“hbase.regionserver.restart.on.zk.expire” 为true。 这样子，遇到ZooKeeper session expired ， regionserver将选择 restart 而不是 abort

具体的配置是，在hbase-site.xml中加入

 

<property>

<name>zookeeper.session.timeout</name>

<value>90000</value>

<description>ZooKeeper session timeout.

HBase passes this to the zk quorum as suggested maximum time for a

session.  See http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions

"The client sends a requested timeout, the server responds with the

timeout that it can give the client. The current implementation

requires that the timeout be a minimum of 2 times the tickTime

(as set in the server configuration) and a maximum of 20 times

the tickTime." Set the zk ticktime with hbase.zookeeper.property.tickTime.

In milliseconds.

</description>

</property>

<property>

<name>hbase.regionserver.restart.on.zk.expire</name>

<value>true</value>

<description>

Zookeeper session expired will force regionserver exit.

Enable this will make the regionserver restart.

</description>

</property>

为了避免java full GC suspend thread 对Zookeeper heartbeat的影响，我们还需要对hbase-env.sh进行配置。

将

 

export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError \
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"

修改成

export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError \
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled \
-XX:+CMSInitiatingOccupancyFraction=70 \
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseParNewGC -Xmn256m"

更多关于Hbase performance tuning 的信息，可以参考

http://wiki.apache.org/hadoop/PerformanceTuning