浏览 3074 次
|
精华帖 (0) :: 良好帖 (0) :: 灌水帖 (0) :: 隐藏帖 (0)
|
|
|---|---|
| 作者 | 正文 |
|
发表时间：2014-07-21
继上次升级hadoop完毕后，集群启动正常，但是在访问Namenode的50070的界面上，发现了如下截图的警告信息：
如上异常，是什么意思呢？看了下官方的FAQ,大致意思就是，有几个块的数据，在现有的DataNode节点上，没有一个存储的，但是在NameNode的元数据里却存在。 怎么解决这个问题呢，下面介绍一个hadoop的健康监测命令fsck， fsck工具来检验HDFS中的文件是否正常可用。这个工具可以检测文件块是否在DataNode中丢失，是否低于或高于文件副本。 fack命令用法如下： 注：此处必须是启动hadoop hdfs的账号才有权查看 Usage: DFSck &lt;path&gt; [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]] &lt;path&gt; 检查的起始目录 -move 将损坏的文件移到到/lost+found -delete 删除损坏的文件 -files 打印出所有被检查的文件 -openforwrite 打印出正在写的文件 -list-corruptfileblocks print out list of missing blocks and files they belong to -blocks 打印出block报告 -locations 打印出每个block的位置 -racks 打印出data-node的网络拓扑结构 默认情况下，fsck会忽略正在写的文件，使用-openforwrite可以汇报这种文件 下面是是一个运行的命令例子： <pre name="code" class="java">[search@fse01 hadoop]$ bin/hadoop fsck / -files DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. Connecting to namenode via http://hadoop1:50070 FSCK started by search (auth:SIMPLE) from /127.0.0.1 for path / at Mon Jul 21 15:10:06 CST 2014 / &lt;dir&gt; /tmp &lt;dir&gt; /tmp/hadoop-search &lt;dir&gt; /tmp/hadoop-search/mapred &lt;dir&gt; /tmp/hadoop-search/mapred/staging &lt;dir&gt; /tmp/hadoop-search/mapred/staging/search &lt;dir&gt; /tmp/hadoop-search/mapred/staging/search/.staging &lt;dir&gt; /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0001 &lt;dir&gt; /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006 &lt;dir&gt; /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar 13393150 bytes, 1 block(s): /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-2026656432152987415 MISSING 1 blocks of total size 13393150 B /tmp/hadoop-search/mapred/staging/zhouming &lt;dir&gt; /tmp/hadoop-search/mapred/staging/zhouming/.staging &lt;dir&gt; /tmp/hadoop-search/mapred/staging/zhouming/.staging/job_201403111456_0028 &lt;dir&gt; /tmp/hadoop-search/mapred/system &lt;dir&gt; /tmp/hadoop-search/mapred/system/jobtracker.info 4 bytes, 1 block(s): /tmp/hadoop-search/mapred/system/jobtracker.info: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-7234718182174847600 MISSING 1 blocks of total size 4 B /tmp/hadoop-yarn &lt;dir&gt; /tmp/hadoop-yarn/staging &lt;dir&gt; /tmp/hadoop-yarn/staging/history &lt;dir&gt; /tmp/hadoop-yarn/staging/history/done_intermediate &lt;dir&gt; /tmp/hadoop-yarn/staging/history/done_intermediate/search &lt;dir&gt; /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0001-1405911635075-search-random%2Dwriter-1405912101075-40-0-SUCCEEDED-default.jhist 340102 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0001.summary 342 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0001_conf.xml 79756 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0003-1405921787298-search-com.dhgate.search.fse.proinfo.dataload.NewLoadData-1405921809468-1-1-SUCCEEDED-default.jhist 32989 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0003.summary 390 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0003_conf.xml 83089 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0004-1405922486108-search-Rebuild+Index%3Afsesearch%2D%3Ecollection1%2D%3Eshard1-1405922501203-1-0-SUCCEEDED-default.jhist 20173 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0004.summary 369 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0004_conf.xml 86592 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/search &lt;dir&gt; /tmp/hadoop-yarn/staging/search/.staging &lt;dir&gt; /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002 &lt;dir&gt; /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.jar 30472539 bytes, 1 block(s): Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742242_1099511720037. Target Replicas is 10 but found 4 replica(s). /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.split 116 bytes, 1 block(s): Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742243_1099511720038. Target Replicas is 10 but found 4 replica(s). /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.splitmetainfo 31 bytes, 1 block(s): OK /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.xml 72772 bytes, 1 block(s): OK /user &lt;dir&gt; /user/search &lt;dir&gt; /user/search/fse &lt;dir&gt; /user/search/fse/kms &lt;dir&gt; /user/search/fse/kms/in &lt;dir&gt; /user/search/fse/kms/in/0 3 bytes, 1 block(s): OK /user/search/fse/kms/index &lt;dir&gt; /user/search/fse/kms/index/index_shard_00_5e62b650 &lt;dir&gt; /user/search/fse/kms/index/index_shard_00_5e62b650/_0.fdt 11069 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/_0.fdx 47 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/_0.fnm 992 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/_0.nvd 683 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/_0.nvm 68 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/_0.si 376 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.doc 50029 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.pos 101757 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.tim 62843 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.tip 2319 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/segments.gen 20 bytes, 1 block(s): OK /user/search/fse/kms/index/index_shard_00_5e62b650/segments_1 69 bytes, 1 block(s): OK /user/search/fse/kms/out &lt;dir&gt; /user/search/fse/kms/out/0 &lt;dir&gt; /user/search/fse/kms/out/0/00 425334 bytes, 1 block(s): OK /user/search/fse/proinfo &lt;dir&gt; /user/search/fse/proinfo/schema.xml 32194 bytes, 1 block(s): OK /user/search/rand &lt;dir&gt; /user/search/rand/_SUCCESS 0 bytes, 0 block(s): OK /user/search/rand/part-m-00000 1077280794 bytes, 9 block(s): OK /user/search/rand/part-m-00001 1077294346 bytes, 9 block(s): OK /user/search/rand/part-m-00002 1077276960 bytes, 9 block(s): OK /user/search/rand/part-m-00003 1077280905 bytes, 9 block(s): OK /user/search/rand/part-m-00004 1077288199 bytes, 9 block(s): OK /user/search/rand/part-m-00005 1077290933 bytes, 9 block(s): OK /user/search/rand/part-m-00006 1077270249 bytes, 9 block(s): OK /user/search/rand/part-m-00007 1077290630 bytes, 9 block(s): OK /user/search/rand/part-m-00008 1077279315 bytes, 9 block(s): OK /user/search/rand/part-m-00009 1077294441 bytes, 9 block(s): OK /user/search/rand/part-m-00010 1077288193 bytes, 9 block(s): OK /user/search/rand/part-m-00011 1077299978 bytes, 9 block(s): OK /user/search/rand/part-m-00012 1077274992 bytes, 9 block(s): OK /user/search/rand/part-m-00013 1077295278 bytes, 9 block(s): OK /user/search/rand/part-m-00014 1077280338 bytes, 9 block(s): OK /user/search/rand/part-m-00015 1077285350 bytes, 9 block(s): OK /user/search/rand/part-m-00016 1077276847 bytes, 9 block(s): OK /user/search/rand/part-m-00017 1077285962 bytes, 9 block(s): OK /user/search/rand/part-m-00018 1077280969 bytes, 9 block(s): OK /user/search/rand/part-m-00019 1077285691 bytes, 9 block(s): OK /user/search/rand/part-m-00020 1077292174 bytes, 9 block(s): OK /user/search/rand/part-m-00021 1077292141 bytes, 9 block(s): OK /user/search/rand/part-m-00022 1077285419 bytes, 9 block(s): OK /user/search/rand/part-m-00023 1077285751 bytes, 9 block(s): OK /user/search/rand/part-m-00024 1077282762 bytes, 9 block(s): OK /user/search/rand/part-m-00025 1077282468 bytes, 9 block(s): OK /user/search/rand/part-m-00026 1077283184 bytes, 9 block(s): OK /user/search/rand/part-m-00027 1077281056 bytes, 9 block(s): OK /user/search/rand/part-m-00028 1077292387 bytes, 9 block(s): OK /user/search/rand/part-m-00029 1077283341 bytes, 9 block(s): OK /user/search/rand/part-m-00030 1077288015 bytes, 9 block(s): OK /user/search/rand/part-m-00031 1077276923 bytes, 9 block(s): OK /user/search/rand/part-m-00032 1077296469 bytes, 9 block(s): OK /user/search/rand/part-m-00033 1077288224 bytes, 9 block(s): OK /user/search/rand/part-m-00034 1077278284 bytes, 9 block(s): OK /user/search/rand/part-m-00035 1077297217 bytes, 9 block(s): OK /user/search/rand/part-m-00036 1077289846 bytes, 9 block(s): OK /user/search/rand/part-m-00037 1077284545 bytes, 9 block(s): OK /user/search/rand/part-m-00038 1077290084 bytes, 9 block(s): OK /user/search/rand/part-m-00039 1077282121 bytes, 9 block(s): OK Status: CORRUPT Total size: 43136702998 B Total dirs: 32 Total files: 71 Total symlinks: 0 Total blocks (validated): 390 (avg. block size 110606930 B) ******************************** CORRUPT FILES: 2 MISSING BLOCKS: 2 MISSING SIZE: 13393154 B CORRUPT BLOCKS: 2 ******************************** Minimally replicated blocks: 388 (99.48718 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 2 (0.51282054 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9897435 Corrupt blocks: 2 Missing replicas: 12 (1.0075567 %) Number of data-nodes: 4 Number of racks: 1 FSCK ended at Mon Jul 21 15:10:06 CST 2014 in 48 milliseconds The filesystem under path '/' is CORRUPT</pre> 通过上面这个命令我们就可以监测出文件的存储状况，监测出有无效的或者缺失的block，下面我们就可以执行-delete命令，来清楚无效的信息块。 <pre name="code" class="java"> [search@fse01 hadoop]$ bin/hadoop fsck / -delete DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. Connecting to namenode via http://hadoop1:50070 FSCK started by search (auth:SIMPLE) from /127.0.0.1 for path / at Mon Jul 21 15:11:39 CST 2014 . /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-2026656432152987415 /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar: MISSING 1 blocks of total size 13393150 B.. /tmp/hadoop-search/mapred/system/jobtracker.info: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-7234718182174847600 /tmp/hadoop-search/mapred/system/jobtracker.info: MISSING 1 blocks of total size 4 B........... /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.jar: Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742242_1099511720037. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.split: Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742243_1099511720038. Target Replicas is 10 but found 4 replica(s). ..........................................................Status: CORRUPT Total size: 43136702998 B Total dirs: 32 Total files: 71 Total symlinks: 0 Total blocks (validated): 390 (avg. block size 110606930 B) ******************************** CORRUPT FILES: 2 MISSING BLOCKS: 2 MISSING SIZE: 13393154 B CORRUPT BLOCKS: 2 ******************************** Minimally replicated blocks: 388 (99.48718 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 2 (0.51282054 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9897435 Corrupt blocks: 2 Missing replicas: 12 (1.0075567 %) Number of data-nodes: 4 Number of racks: 1 FSCK ended at Mon Jul 21 15:11:39 CST 2014 in 45 milliseconds The filesystem under path '/' is CORRUPT</pre> 最后一点，需要注意的是，这个命令在namenode文件信息较大的时候，会比较影响hadoop性能，所以应该慎用，通常可以在集群空闲的时间段，执行一次，查看整体的HDFS副本健康状况！ 声明：ITeye文章版权属于作者，受法律保护。没有作者书面许可不得转载。
推荐链接
|
| 返回顶楼 | |
