`

(转)hadoop 根据SecondaryNameNode恢复Namenode

 
阅读更多

 制造namenode宕机的情况 
1) kill 掉namenode的进程

[hadoop@hadoop bin]$ kill -9 13481

2)删除dfs.name.dir所指向的文件夹,这里是/home/hadoop/hdfs/name.

 

current  image  in_use.lock  previous.checkpoint
[hadoop@hadoop name]$ rm -rf *

 

删除name目录下的所有内容,但是必须保证name这个目录是存在的
3)从secondarynamenode元数据namesecondary目录下复制到namenode name目录下

   我的secodarynamenode 元数据目录

/home/hadoop/tmp/dfs/namesecondary

   复制过程

[hadoop@hadoop name]$ cp -R /home/hadoop/tmp/dfs/namesecondary/* .

4)启动namenode

[hadoop@hadoop bin]$ ./hadoop-daemon.sh start namenode

 5)检查
使用hadoop fsck /user命令检查文件Block的完整性

<font size="2">[hadoop@hadoop bin]$ hadoop fsck /</font>
Warning: $HADOOP_HOME is deprecated.

FSCK started by hadoop from /192.168.0.101 for path / at Sun Dec 22 23:04:31 CST 2013
...................................
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310222119_0001/job.jar:  Under replicated blk_-8571652065964704775_1020. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310222119_0002/job.jar:  Under replicated blk_-5947701456602696019_1021. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310222119_0003/job.jar:  Under replicated blk_8214183112681524571_1022. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310222142_0001/job.jar:  Under replicated blk_4805420250921446015_1024. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310222142_0002/job.jar:  Under replicated blk_7913185784171356584_1027. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310222142_0004/job.jar:  Under replicated blk_-8411847042533891069_1035. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310222142_0005/job.jar:  Under replicated blk_2163772543235273521_1036. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310222142_0007/job.jar:  Under replicated blk_-3491660194168043022_1044. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242056_0002/job.jar:  Under replicated blk_5280511346594851641_1270. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242056_0003/job.jar:  Under replicated blk_5588149584508213931_1271. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242056_0004/job.jar:  Under replicated blk_-1846184614352398688_1272. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242056_0005/job.jar:  Under replicated blk_8253537375261552577_1273. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242100_0001/job.jar:  Under replicated blk_-6858089306760733073_1275. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242100_0002/job.jar:  Under replicated blk_-630176777256891004_1276. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242100_0003/job.jar:  Under replicated blk_3453389521553623867_1277. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242100_0004/job.jar:  Under replicated blk_-4262000880964323956_1278. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242105_0001/job.jar:  Under replicated blk_-5324801167724976561_1280. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242105_0002/job.jar:  Under replicated blk_3284342834321881345_1281. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242105_0004/job.jar:  Under replicated blk_5174401550469241860_1295. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242105_0009/job.jar:  Under replicated blk_6390129220783606015_1327. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201310242105_0010/job.jar:  Under replicated blk_8995477665353821346_1328. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201311292212_0007/job.jar:  Under replicated blk_-6447241034801532571_1699. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201312082210_0001/job.jar:  Under replicated blk_-187920261151639503_1741. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201312082210_0002/job.jar:  Under replicated blk_1912732980088631445_1742. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/staging/hadoop/.staging/job_201312092348_0001/job.jar:  Under replicated blk_448639237400606735_1953. Target Replicas is 10 but found 1 replica(s).
.
/home/hadoop/tmp/mapred/system/jobtracker.info: CORRUPT block blk_-4973841422235657473

/home/hadoop/tmp/mapred/system/jobtracker.info: MISSING 1 blocks of total size 4 B.Status: CORRUPT
 Total size:        367257 B
 Total dirs:        83
 Total files:        60
 Total blocks (validated):        57 (avg. block size 6443 B)
  ********************************
  CORRUPT FILES:        1
  MISSING BLOCKS:        1
  MISSING SIZE:                4 B
  CORRUPT BLOCKS:         1
  ********************************
 Minimally replicated blocks:        56 (98.24561 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:        25 (43.85965 %)
 Mis-replicated blocks:                0 (0.0 %)
 Default replication factor:        1
 Average block replication:        0.98245615
 Corrupt blocks:                1
 Missing replicas:                225 (401.7857 %)
 Number of data-nodes:                1
 Number of racks:                1
FSCK ended at Sun Dec 22 23:04:31 CST 2013 in 89 milliseconds

恢复工作完成,检查hdfs的数据

 

原文链接:hadoop 根据SecondaryNameNode恢复Namenode
http://www.aboutyun.com/thread-6196-1-1.html
(出处: about云开发)

原作者:lzw

分享到:
评论

相关推荐

    Hadoop Namenode恢复

    Hadoop Namenode 恢复 Hadoop Namenode 是 Hadoop 分布式文件系统的核心组件之一,负责管理文件系统的命名空间。然而,在生产环境中,namenode 的崩溃可能会导致整个集群的不可用。因此,namenode 的恢复是非常重要...

    Hadoop中namenode和secondarynamenode工作机制讲解

    secondarynamenode并非namenode的热备份,它不会直接处理客户端的请求,其主要功能是定期合并namenode的编辑日志(edits)和命名空间镜像(FSImage),以防止编辑日志过大,减轻namenode的工作压力,并提供一种恢复...

    大数据与云计算技术 Hadoop概论和快速入门 共40页.ppt

    SecondaryNamenode,NameNode的镜像备份节点 ==Map Reduce== JobTracker,hadoop的Map/Reduce调度器,负责与TackTracker通信分配计算任务并跟踪任务进度。 TaskTracker,启动和管理Map和Reduce子任务的节点。

    最新大数据Hadoop面试题!(附答案解析).pdf

    SecondaryNameNode是NameNode的热备节点,负责合并编辑日志,减少NameNode启动时间。 2. HDFS Block大小 HDFS默认Block大小为64MB,用户可以根据需要进行调整。Block是HDFS的基本存储单元,一个Block可以被分割成...

    HDFS中NameNode节点的配置、备份和恢复.doc

    为了恢复 NameNode 节点,需要使用 SecondaryNameNode 节点上的备份数据,并将其恢复到 NameNode 节点上。 HDFS 中 NameNode 节点的配置、备份和恢复是非常重要的, SecondaryNameNode 节点的配置和备份策略可以...

    搭建hadoop伪分布式.docx

    DataNode,JobTracker,TaskTracker,SecondaryNameNode),请注意分布式运行中的这几个结点的区别:从分布式存储的角度来说,集群中的结点由一个NameNode和若干个DataNode组成,另有一个SecondaryNameNode作为NameNode的...

    Hadoop试题试题库.doc

    NameNode 是 HDFS 的主节点,SecondaryNameNode 是 NameNode 的热备,帮助 NameNode 合并编辑日志,减少 NameNode 启动时间。SecondaryNameNode 应与 NameNode 部署到不同的节点。 HBase 和 MapReduce 关系 ...

    SecondaryNameNode职责.pptx

    如果NameNode出现故障,可以使用SecondaryNameNode上的最新fsimage和edits来恢复NameNode的状态,确保服务的连续性。 5. **优化NameNode启动**: - 合并后的fsimage文件使得NameNode在重启时能快速加载整个文件...

    Hadoop期末考试题总结.doc

    本资源摘要信息主要介绍了Hadoop的相关知识点,涵盖了Hadoop的版本、4V特征、大数据存储、HDFS、MapReduce、SecondaryNameNode、Hadoop shell命令、集群管理工具等方面。 1. Hadoop的最高版本是Hadoop 3.x,当前...

    大数据技术Hadoop面试题.pdf

    SecondaryNameNode并非NameNode的热备份,而是协助NameNode合并编辑日志,减少NameNode重启时的恢复时间。 4. Hadoop作者:Hadoop的作者是Doug Cutting。 5. HDFS默认Block Size:默认大小为128MB。 6. 集群瓶颈...

    hadoop-3.1.3.tar.gz

    Hadoop的服务包括NameNode、DataNode、SecondaryNameNode以及ResourceManager、NodeManager等。启动Hadoop集群,可以依次执行以下命令: ``` start-dfs.sh start-yarn.sh ``` 七、验证Hadoop安装 启动成功后,可以...

    大数据技术之Hadoop知识分享 Hadoop面试题目及其答案 Hadoop面试题目整理 Hadoop使用经验 共9页.pdf

    6. **SecondaryNameNode**:SecondaryNameNode并非NameNode的热备份,而是辅助NameNode定期合并编辑日志,以减少NameNode重启时的恢复时间。它通常部署在独立的硬件上。 7. **集群管理工具**:Puppet、Pdsh和...

    Hadoop大数据平台构建、规划大数据平台集群教学课件.pptx

    在这种模式下,NameNode、DataNode、JobTracker(在Hadoop 2.x中被ResourceManger替代)、TaskTracker(被NodeManager取代)以及SecondaryNameNode分别分布在不同的节点上,确保了系统的可扩展性和容错性。...

    Hadoop集群架构搭建分析

    环境搭建是指在搭建 Hadoop 集群环境的过程中需要进行的步骤,包括配置 NameNode、DataNode、JobTracker、TaskTracker、HMaster、HRegionServer、SecondaryNameNode、HQuorumPeer 等组件。 配置 NameNode 需要设置 ...

    Hadoop学习总结之五:Hadoop的运行痕迹

    - 位于`/var/log/hadoop/hadoop-namenode-namenode.out` (具体路径可能因版本而异) - 记录了HDFS元数据操作的日志信息 - **DataNode日志**: - 位于`/var/log/hadoop/hadoop-datanode-datanode.out` (具体路径可能...

    11_尚硅谷大数据之HDFS_NameNode和SecondaryNameNode1

    在Hadoop大数据存储系统中,HDFS(Hadoop Distributed File System)是核心组件之一,用于分布式存储大量数据。NameNode是HDFS的核心节点,负责管理文件系统的元数据,包括文件和目录的命名空间以及文件的块映射信息...

    hadoop-2.7.5 windows环境

    2. **启动Hadoop服务**: 先启动DataNode和SecondaryNameNode,再启动NameNode,最后启动ResourceManager和NodeManager。使用以下命令: ``` hadoop dfsadmin -safemode leave start-dfs.cmd start-yarn.cmd ```...

    《大数据技术原理与操作应用》第6章习题答案.docx

    【大数据技术原理与操作应用】第6章习题答案涉及Hadoop分布式文件系统(HDFS)、Hadoop2.x架构、高可用性(HA)配置、NameNode与SecondaryNameNode的角色、资源管理、集群性能瓶颈等多个知识点。 1. Hadoop2.0集群服务...

    hadoop3.x笔记.docx

    2. NameNode 根据元数据找到文件的数据块位置,选择最近的 DataNode 节点。 3. 客户端通过 DataNode 读取数据,并将数据缓存在本地。 4. 客户端将数据写入目标文件。 三、HDFS 写数据流程 HDFS 写数据流程包括以下...

    大数据之hadoop伪分布模式启动.docx

    在启动Hadoop伪分布模式之前,需要先对NameNode进行格式化。格式化HDFS的命令为`hdfs namenode -format`。如果格式化成功,将显示"successfully formatted"的信息。 启动Hadoop伪分布模式 在格式化HDFS后,可以...

Global site tag (gtag.js) - Google Analytics