HBase MSLAB和MemStoreChunkPool源码

  MSLAB工作原理，举个例子HStore在add的时候的例子，其他操作也差不多，很简单

MSLAB是解決menstorm的內存碎片，  MemStoreChunkPool是解決full gc頻繁，自己管理chunk数据，避免gc

 

 MemStoreChunkPool使用的是memstorm的limit * chuckpoolpercent：是memstorm的百分比

 hbase.regionserver.global.memstore.upperLimit * hbase.hregion.memstore.chunkpool.initialsize

 

  MemStore

    /**
   * Write an update
   * @param kv
   * @return approximate size of the passed key and value.
   */
  long add(final KeyValue kv) {
    KeyValue toAdd = maybeCloneWithAllocator(kv);//mslab将kv放到大chunk下
    return internalAdd(toAdd);//添加到memkvset中
  }
  
  
  private KeyValue maybeCloneWithAllocator(KeyValue kv) {
    if (allocator == null) {
      return kv;
    }
    //使用mslsb,获得当前chunk或放不下返回一个空chunk
    int len = kv.getLength();
    Allocation alloc = allocator.allocateBytes(len);
    if (alloc == null) {
      // The allocation was too large, allocator decided
      // not to do anything with it.
      return kv;
    }
    assert alloc.getData() != null;
    //将kv拷贝到chunk里
    System.arraycopy(kv.getBuffer(), kv.getOffset(), alloc.getData(), alloc.getOffset(), len);
    KeyValue newKv = new KeyValue(alloc.getData(), alloc.getOffset(), len);
    newKv.setMvccVersion(kv.getMvccVersion());
    return newKv;
  }

 

MemStoreLAB的方法allocateBytes，获得当前chunk或放不下返回一个空chunk

  public Allocation allocateBytes(int size) {
    Preconditions.checkArgument(size >= 0, "negative size");

    // Callers should satisfy large allocations directly from JVM since they
    // don't cause fragmentation as badly.
    if (size > maxAlloc) {//太大的kv（超过256K）不会造成内存碎片，所以不用mslab，
      return null;
    }

    while (true) {
      Chunk c = getOrMakeChunk();//获得当前chunk

      // Try to allocate from this chunk
      int allocOffset = c.alloc(size);
      if (allocOffset != -1) {//当前chunck可以容下
        // We succeeded - this is the common case - small alloc
        // from a big buffer
        return new Allocation(c.data, allocOffset);
      }

      // not enough space!
      // try to retire this chunk
      tryRetireChunk(c);//当前chunck装不下kv，设置当前chunk为空，等待下次循环生成chunk
    }
  }

 

   其中MemStoreChunkPool类

  pool put方法

  //在没有compareandset成功后，将刚取来的chunk放回pool中，

  可能有bug，当pool中没有，会新建一个，但是没有初始化byte[],也就是chunk里的data为空，没申请资源，就放到pool里给别人用了

    void putbackChunk(Chunk chunk) {
    if (reclaimedChunks.size() >= this.maxCount) {
      return;
    }
    reclaimedChunks.add(chunk);
   }

  memstore在将snapshot flush之后，将涉及到的chunk放回pool中

    /**
   * Add the chunks to the pool, when the pool achieves the max size, it will
   * skip the remaining chunks
   * @param chunks
   */
  void putbackChunks(BlockingQueue<Chunk> chunks) {
    int maxNumToPutback = this.maxCount - reclaimedChunks.size();
    if (maxNumToPutback <= 0) {
      return;
    }
    chunks.drainTo(reclaimedChunks, maxNumToPutback);
  }

 

  pool get方法

    /**
   * Poll a chunk from the pool, reset it if not null, else create a new chunk
   * to return
   * @return a chunk
   */
  Chunk getChunk() {
    Chunk chunk = reclaimedChunks.poll();
    if (chunk == null) {
      chunk = new Chunk(chunkSize);
      createdChunkCount.incrementAndGet();
    } else {
      chunk.reset();
      reusedChunkCount.incrementAndGet();
    }
    return chunk;
  }