Replica Selection/Propogate Strategy

Choose sequence: local node -> local rack -> remote rack

a selected replica node should be a good node:
1> if the node is (being) decommissed
2> the remaining capacity of the target machine
3> the communication traffic of the target machine
 based on current connection of that machine, if current connection number > average connection num(total connections / total machine size) node.getXceiverCount() > (2.0 * avgLoad)
4> if the target rack has chosen too many nodes

Data replication topology:
first replica machine -> second replica -> ...

After a list of replicas are selected, replica list should be sorted in pipe.
The aim of this sort is to find the shortest path, since data replication will be travelling from the first replica machine to the final replica machine, so, we should reduce round trip.
This is basically a traveling salesman problem:

java 代码

int index=0;
for( ;index
  DatanodeDescriptor shortestNode = null;
  int shortestDistance = Integer.MAX_VALUE;
  int shortestIndex = index;
  for( int i=index; i
    DatanodeDescriptor currentNode = nodes.get(i);
    int currentDistance = clusterMap.getDistance( writer, currentNode );
    if(shortestDistance>currentDistance ) {
      shortestDistance = currentDistance;
      shortestNode = currentNode;
      shortestIndex = i;
    }
  }
  //switch position index & shortestIndex
  if( index != shortestIndex ) {
    nodes.set(shortestIndex, nodes.get(index));
    nodes.set(index, shortestNode);
  }
  writer = shortestNode;
}

分享到：

Functional Programming For The Rest of U ... | Requirements

• 2007-03-22 18:11
• 浏览 939
• 评论(0)
• 查看更多