Hadoop World 大会上的文章选摘（一）

Compression

LZOor Snappy(20% better than LZO)——Block(default)

Serialization

AVROdidn’t work well——deserialization issue

Developedconfigurable serialization mechanism that uses JSON except Data type

Secondary Indexes

Wereusing ITHBase and IHBase from contrib——doesn’t work well

Redesignedschema without need for index

Westill need it though

Performance

Severaltunable parameters

Hardware(Hadoop+HBase)

DataNode -24GB RAM,8Cores,4*1TB(64GB,24Cores,8*2TB)

6mappers and 6 reducers per node (16 mappers,4reducers)

Memoryallocation by process

DataNode——1GB(2GB)

TaskTracker——1GB(2GB)

MapTasks——6*1GB(16*1.5GB)

ReduceTasks——6*1GB(4*1.5GB)

RegionServer——8GB(24GB)

TotalAllocation——24GB(64GB)

Deployment

Donot run ZK instances on DN,have a separate ZK quorum(3 minimum)

Donot run HMaster on NN

AvoidSPOF for HMaster(run additional master(s))