[HBase]abort引发的死锁

iwinit

浏览: 455055 次
性别:
来自: 杭州

最近访客更多访客>>

lingxiajiudu

plisking

rochine123

尼欧-张

博主相关

博客

微博

相册

留言

关于我

文章分类

全部博客 (45)

社区版块

存档分类

hbase abort dead lock

今天在HBase群里碰到一个客户端死锁问题，堆栈如下

Java stack information for the threads listed above:
===================================================
"KVQueueService-Handler-2-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000760ca2680> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:605)
        - locked <0x0000000760c88e00> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1698)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookee
per.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
"TableDataQueueService-Handler-4":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:600)
        - waiting to lock <0x0000000760c88e00> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1698)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:132)
        - locked <0x0000000760ca2680> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.
apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:83)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:830)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:942)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.j
ava:942)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:845)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1520)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1376)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:936)
        at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:792)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:775)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
        at com.alibaba.alimonitor.store.hbase.TableDataDaoService.writeData(TableDataDaoService.java:111)
        at com.alibaba.alimonitor.store.service.Tab
leDataQueueService.handle(TableDataQueueService.java:28)
        at com.alibaba.alimonitor.store.service.TableDataQueueService.handle(TableDataQueueService.java:14)
        at com.alibaba.alimonitor.store.service.AbstractQueueService$HandleThread.run(AbstractQueueService.java:115)

Found 1 deadlock.

可见发生死锁的业务线程和zookeeper的eventThread线程。EventThread是zookeeper用来通知客户端watcher的线程，它会回调使用者注册的watcher。

在这里，首先是eventThread处理connectionEvent的Expired事件

 case Expired:
        if (ZKUtil.isSecureZooKeeper(this.conf)) {
          // We consider Expired equivalent to AuthFailed for this
          // connection. Authentication is never going to complete. The
          // client should proceed to do cleanup.
          saslLatch.countDown();
        }
        String msg = prefix(this.identifier + " received expired from " +
          "ZooKeeper, aborting");
        // TODO: One thought is to add call to ZooKeeperListener so say,
        // ZooKeeperNodeTracker can zero out its data values.
        if (this.abortable != null) this.abortable.abort(msg,
            new KeeperException.SessionExpiredException());
        break;
    }

这里会abort调用，其实是调用的HConnectionImplementation的abort方法

 public void abort(final String msg, Throwable t) {
      if (t instanceof KeeperException) {
        LOG.info("This client just lost it's session with ZooKeeper, will"
            + " automatically reconnect when needed.");
        if (t instanceof KeeperException.SessionExpiredException) {
          LOG.info("ZK session expired. This disconnect could have been" +
              " caused by a network partition or a long-running GC pause," +
              " either way it's recommended that you verify your environment.");
          resetZooKeeperTrackers();
        }
        return;
      }
      if (t != null) LOG.fatal(msg, t);
      else LOG.fatal(msg);
      this.aborted = true;
      close();
    }

再看resetZooKeeperTrackers方法，是一个同步方法，EventThread在这里拿到HConnectionImplementation对象锁

    private synchronized void resetZooKeeperTrackers() {
      if (masterAddressTracker != null) {
        masterAddressTracker.stop();
        masterAddressTracker = null;
      }
      if (rootRegionTracker != null) {
        rootRegionTracker.stop();
        rootRegionTracker = null;
      }
      clusterId = null;
      if (zooKeeper != null) {
        zooKeeper.close();
        zooKeeper = null;
      }
    }

这个时候业务线程开始region location，进入RootRegionTrack的waitRootRegionLocation

  public ServerName waitRootRegionLocation(long timeout)
  throws InterruptedException {
    if (false == checkIfBaseNodeAvailable()) {
      String errorMsg = "Check the value configured in 'zookeeper.znode.parent'. "
          + "There could be a mismatch with the one configured in the master.";
      LOG.error(errorMsg);
      throw new IllegalArgumentException(errorMsg);
    }
    return dataToServerName(super.blockUntilAvailable(timeout, true));
  }

业务线程进入blockUntilAvailable，拿到父类ZookeeperNodeTracker的对象锁

public synchronized byte [] blockUntilAvailable(long timeout, boolean refresh)
  throws InterruptedException {
    if (timeout < 0) throw new IllegalArgumentException();
    boolean notimeout = timeout == 0;
    long startTime = System.currentTimeMillis();
    long remaining = timeout;
    if (refresh) {
      try {
	//这里正好抛出异常
        this.data = ZKUtil.getDataAndWatch(watcher, node);
      } catch(KeeperException e) {
        abortable.abort("Unexpected exception handling blockUntilAvailable", e);
      }
    }
    while (!this.stopped && (notimeout || remaining > 0) && this.data == null) {
      // We expect a notification; but we wait with a
      //  a timeout to lower the impact of a race condition if any
      wait(100);
      remaining = timeout - (System.currentTimeMillis() - startTime);
    }
    return this.data;
  }

由于这里zookeeper调用正好抛出异常导致业务线程去abort，这个时候因为EventThread已经先abort了，所以业务线程需要等待HConnectionImplementation对象锁的释放。而这个时候EventThread继续执行，resetZooKeeperTrackers中继续调用 rootRegionTracker.stop()方法

 public synchronized void stop() {
    this.stopped = true;
    notifyAll();
  }

这个方法需要拿到父类ZookeeperNodeTracker的对象锁，而此时这个锁已经被业务线程拿到了，这就发生了死锁。根本原因abort操作没有做并发控制。最新版本的hbase已经修复了这个bug，修复后的abort方法，增加了resetting同步状态，后续abort请求直接return了。

public void abort(final String msg, Throwable t) {
      if (t instanceof KeeperException) {
        LOG.info("This client just lost it's session with ZooKeeper, will"
            + " automatically reconnect when needed.");
        if (t instanceof KeeperException.SessionExpiredException) {
          LOG.info("ZK session expired. This disconnect could have been" +
              " caused by a network partition or a long-running GC pause," +
              " either way it's recommended that you verify your environment.");
          //同步状态
          synchronized (resetLock) {
            if (resetting) return;
            this.resetting = true;
          }
          resetZooKeeperTrackers();
          this.resetting = false;
        }
        return;
      }
      if (t != null) LOG.fatal(msg, t);
      else LOG.fatal(msg);
      this.aborted = true;
      close();
    }

官方issue地址 https://issues.apache.org/jira/browse/HBASE-7259

分享到：