- 浏览: 219488 次
- 性别:
- 来自: 北京
文章分类
- 全部博客 (114)
- hbase (3)
- akka (7)
- hdfs (6)
- mapreduce (1)
- hive (0)
- zookeeper (8)
- storm (0)
- geese (0)
- leaf (0)
- stormbase (0)
- scala (2)
- oozie (11)
- zeromq (1)
- netty (3)
- mongodb (0)
- sqoop (2)
- flume (3)
- mahout (1)
- redis (0)
- lucene (1)
- solr (1)
- ganglia (3)
- 分布式理论 (2)
- hadoop (42)
- others (14)
- mq (1)
- clojure (3)
- flume ng (1)
- linux (1)
- esper (0)
最新评论
-
javalogo:
[b][i][u]引用[list]
[*][*][flash= ...
什么是Flume -
leibnitz:
what are they meanings
Hadoop Ganglia Metric Item -
di1984HIT:
没用过啊。
akka 介绍-Actor 基础 -
di1984HIT:
写的不错。
Hadoop管理-集群维护 -
developerinit:
很好,基本上介绍了
什么是Flume
根据日志:
StandBy NN启动过程
1.获得Active NN Checkpoints信息
2.在内存中,注册Live Nodes
3.SB NN 进入Safe Mode
4.从Datanod获取包信息
5.离开Safe Mode
Checkpointing active NN at bigdata-4:50070
Serving checkpoints at bigdata-3/172.16.206.206:50070
2012-08-02 11:07:24,761 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from DatanodeRegistration(172.16.206.209, storageID=DS-1124007630-172.16.206.209-50010-1343796008920, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0) storage DS-1124007630-172.16.206.209-50010-1343796008920
2012-08-02 11:07:24,763 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/172.16.206.209:50010
2012-08-02 11:07:24,763 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from DatanodeRegistration(172.16.206.206, storageID=DS-1236477555-172.16.206.206-50010-1343796211132, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0) storage DS-1236477555-172.16.206.206-50010-1343796211132
2012-08-02 11:07:24,764 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/172.16.206.206:50010
2012-08-02 11:07:24,995 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode extension entered.
The reported blocks 8854 has reached the threshold 0.9990 of total blocks 8863. Safe mode will be turned off automatically in 29 seconds.
2012-08-02 11:07:24,995 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 172.16.206.209:50010 after becoming active. Its block contents are no longer considered stale.
2012-08-02 11:07:24,996 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* processReport: from DatanodeRegistration(172.16.206.209, storageID=DS-1124007630-172.16.206.209-50010-1343796008920, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0), blocks: 12908, processing time: 84 msecs
2012-08-02 11:07:25,041 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 172.16.206.206:50010 after becoming active. Its block contents are no longer considered stale.
2012-08-02 11:07:25,041 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* processReport: from DatanodeRegistration(172.16.206.206, storageID=DS-1236477555-172.16.206.206-50010-1343796211132, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0), blocks: 12908, processing time: 45 msecs
2012-08-02 11:07:44,998 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 8863 has reached the threshold 0.9990 of total blocks 8863. Safe mode will be turned off automatically in 9 seconds.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 32 secs.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode is OFF.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 1 racks and 2 datanodes
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
StandBy NN启动过程
1.获得Active NN Checkpoints信息
2.在内存中,注册Live Nodes
3.SB NN 进入Safe Mode
4.从Datanod获取包信息
5.离开Safe Mode
Checkpointing active NN at bigdata-4:50070
Serving checkpoints at bigdata-3/172.16.206.206:50070
2012-08-02 11:07:24,761 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from DatanodeRegistration(172.16.206.209, storageID=DS-1124007630-172.16.206.209-50010-1343796008920, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0) storage DS-1124007630-172.16.206.209-50010-1343796008920
2012-08-02 11:07:24,763 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/172.16.206.209:50010
2012-08-02 11:07:24,763 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from DatanodeRegistration(172.16.206.206, storageID=DS-1236477555-172.16.206.206-50010-1343796211132, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0) storage DS-1236477555-172.16.206.206-50010-1343796211132
2012-08-02 11:07:24,764 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/172.16.206.206:50010
2012-08-02 11:07:24,995 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode extension entered.
The reported blocks 8854 has reached the threshold 0.9990 of total blocks 8863. Safe mode will be turned off automatically in 29 seconds.
2012-08-02 11:07:24,995 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 172.16.206.209:50010 after becoming active. Its block contents are no longer considered stale.
2012-08-02 11:07:24,996 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* processReport: from DatanodeRegistration(172.16.206.209, storageID=DS-1124007630-172.16.206.209-50010-1343796008920, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0), blocks: 12908, processing time: 84 msecs
2012-08-02 11:07:25,041 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 172.16.206.206:50010 after becoming active. Its block contents are no longer considered stale.
2012-08-02 11:07:25,041 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* processReport: from DatanodeRegistration(172.16.206.206, storageID=DS-1236477555-172.16.206.206-50010-1343796211132, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0), blocks: 12908, processing time: 45 msecs
2012-08-02 11:07:44,998 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 8863 has reached the threshold 0.9990 of total blocks 8863. Safe mode will be turned off automatically in 9 seconds.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 32 secs.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode is OFF.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 1 racks and 2 datanodes
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
发表评论
-
Hadoop TestDFSIO
2013-04-21 21:02 2439@VM [bigdata@bigdata hadoo ... -
Hadoop NNBENCH
2013-04-21 20:46 1632@VM [bigdata@bigdata hadoop]$ ... -
Hadoop 安装手册
2013-04-08 15:47 1200Hadoop 安装手册 软件准备 ... -
What do real life hadoop workloads look like
2012-09-10 15:52 834http://www.cloudera.com/blog/20 ... -
CDH4 HA 切换时间
2012-09-05 15:15 4384blocksize:35M filesize 96M zk-s ... -
CDH4 HA 切换
2012-09-05 10:51 1398HA 切换问题 切换时间太长。。。 copy 0 ... ... -
CDH4 HA test
2012-08-01 14:55 2654场景: NN HA 设置成功,HA切换客户端出现异 ... -
Hadoop TextOutput
2012-07-29 21:08 912TextOutputFormat 分隔符参数: mapredu ... -
Hadoop SteamXMLRecordReader
2012-07-28 23:59 707StreamXmlRecordReader 设置属性 str ... -
Hadoop NLineInputFormat
2012-07-28 23:52 1654NLineInputFormat 重写了splits 设置 ... -
KeyValueTextInputFormat
2012-07-28 23:40 957key/value 分割符 mapreduce.input. ... -
Hadoop 控制split尺寸
2012-07-28 23:08 1342三个参数决定Map的Split尺寸 1.mapred.min ... -
Setting up Disks for Hadoop
2012-07-22 12:13 878Setting up Disks for Hadoop He ... -
Upgrade hadoop need think about it
2012-07-21 17:17 885Compatibility When movin ... -
Hadoop 0.23 config differ from 0.20.205
2012-07-21 17:14 926http://hadoop.apache.org/common ... -
Hadoop hdfs block 状态
2012-07-15 13:37 7271.In Service -
Hadoop 配置不当引起集群不稳
2012-07-05 15:35 1028配置不当内容 资源配置不当:内存、文件句柄数量、磁盘空间 ... -
Hadoop管理-集群维护
2012-07-03 15:27 50071.检查HDFS状态 fsck命令 1)f ... -
Hadoop Ganglia Metric Item
2012-06-27 11:13 2031dfs.FSDirectory.files_delete ... -
Hadoop 参数
2012-06-27 10:05 1021转发自:http://www.cnblogs.com/g ...
相关推荐
标题中的“hadoop cdh5 centos 64位系统本地库编译文件”涉及到的是在基于64位CentOS操作系统上编译Hadoop CDH5(Cloudera's Distribution Including Apache Hadoop)的本地库的过程。CDH5是Cloudera公司提供的一个...
在本文中,我们将深入探讨如何在Linux环境中安装Hadoop CDH5,这是一个广泛使用的Hadoop分发版,包含了多个开源大数据处理组件。CDH5提供了高效的数据存储、处理和分析功能,适合大规模数据处理场景。 首先,安装...
【标题】"hadoop-cdh4-0.3.zip" 提供的是针对Presto优化的CDH4 Hadoop发行版,这个版本是专为Presto大数据查询引擎设计的,旨在提高其在CDH4(Cloudera Distribution Including Apache Hadoop)环境中的性能和兼容性...
在Hadoop CDH的源码编译过程中,FindBugs起到了关键的角色。它能够帮助开发者在编译阶段发现潜在的问题,比如空指针异常、资源泄漏等,从而提高代码质量和可维护性。使用FindBugs进行源码检查,可以确保Hadoop CDH的...
根据提供的标题、描述以及部分内容,本文将详细解析Hadoop CDH5.7.0的离线安装过程,并针对其中可能出现的问题进行深入分析。此外,还会介绍如何进行暴力卸载的操作,帮助读者解决在安装和卸载过程中遇到的各种“坑...
在CDH5.5.0中,HDFS(Hadoop Distributed File System)和YARN(Yet Another Resource Negotiator)是两个核心组件,它们在高可用性(HA)模式下的配置尤为重要。HDFS HA允许数据节点和名称节点的冗余,以确保即使单...
HadoopCDH发行版架构
对于已有Hadoop使用经验的用户而言,了解这些变化有助于他们更好地评估升级至CDH4的必要性和价值。 #### 安装CDH4之前的准备工作 在这一章节中,文档强调了安装CDH4前所需进行的一系列准备步骤。这些步骤包括但不...
本文将详细介绍如何在单台机器上搭建Hadoop CDH版本的伪分布式集群,并通过图文形式进行指导,帮助初学者快速掌握搭建过程。 #### 二、准备工作 在开始搭建伪分布式集群之前,需要完成以下准备工作: 1. **下载所需...
### Hadoop之CDH:基于Cloudera的HA部署指南 #### 关于本指南 本文档旨在提供关于如何在Cloudera Distribution Including Hadoop (CDH)上配置高可用性的详细指南。CDH是由Cloudera公司提供的一个企业级Hadoop发行...
主要是因为hadoop的cdh5官网收费,项目下载不了了,上传我下载的到csdn方便各位下载
hadoop cdh3u6的版本在eclipse3.6.2下的插件,将jar包放入eclipse安装路径下的plugins文件夹,重启即可。
本文将详细介绍CDH HA部署的过程,包括前期准备、配置hosts文件、设置主机名、关闭防火墙和SELinux等步骤,并深入探讨NFS的安装与配置过程。 #### 二、前期准备 在进行CDH HA部署之前,需要完成一系列的基础配置...
在本文中,我们将深入探讨如何在CentOS 6.2环境下安装Cloudera's Hadoop Distribution (CDH) 4.3版本。CDH是Cloudera公司提供的一款开源大数据处理平台,它集成了Hadoop生态系统中的多个组件,如Hadoop、Zookeeper和...
Hadoop 2.6.0是Hadoop发展过程中的一个重要版本,它在Hadoop 2.x系列中引入了许多改进和优化,包括提升系统性能、增强容错性和稳定性,以及提供了更丰富的功能。CDH(Cloudera Distribution Including Apache Hadoop...
这个过程包括安装CDH4的各个组件、配置集群参数、初始化HDFS和YARN、启动服务以及进行基本的健康检查。安装过程中需要注意的是,配置文件通常位于`/etc/cassandra`目录下,而Hadoop相关服务则可以通过`service`命令...
HadoopHA集群 批量启动脚本HadoopHA集群 批量启动脚本HadoopHA集群 批量启动脚本HadoopHA集群 批量启动脚本
4. **CDH**: CDH是Cloudera对Hadoop生态的商业发行版,它包含了经过测试和优化的Hadoop组件,包括HDFS、MapReduce、YARN等,同时也包含了其他如Hive(SQL查询工具)、Pig(数据流编程工具)、Oozie(工作流调度器)...
hadoop-cdh-4.5.0 eclipse plugin hadoop eclipse 插件 hadoop-eclipse-plugin-2.0.0-cdh4.5.0.jar
CDH(Cloudera Distribution Including Apache Hadoop)是Cloudera公司提供的一个全面、经过优化且企业级的Hadoop发行版,包含了多个相关项目,如Hive、HBase、Impala和Spark等。CDH5是该发行版的一个版本,它提供了...