hadoop+hbase

fengqinyun
浏览: 568297 次
性别:
来自: 北京
最近访客更多访客>>

donlianli
Veromca-源代码
laifu901
lxy31092
博主相关

博客
微博
相册
留言
关于我
文章分类

社区版块

存档分类

131做master ，91,133做salve
131操作

192.168.0.131    chupiao
192.168.0.91     chupiao2
192.168.0.133    rycdb

useradd hadoop

一.ssh免密码登陆
ssh-keygen  -t  rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
b0:b0:84:bb:ad:27:15:c4:0f:56:4c:1e:68:cb:73:53 hadoop@chupiao

 [hadoop@chupiao ~]$ cd .ssh/
 [hadoop@chupiao ~]$ll
 [hadoop@chupiao .ssh]$ cp id_rsa.pub  authorized_keys
 [hadoop@chupiao .ssh]$ ssh chupiao
The authenticity of host 'chupiao (192.168.0.131)' can't be established.
RSA key fingerprint is 3a:a1:ae:c8:1c:a0:15:ef:9e:3d:87:dd:89:9f:f1:93.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'chupiao,192.168.0.131' (RSA) to the list of known hosts.
Last login: Thu Jan 10 10:03:41 2013 from localhost.localdomain

输入 yes 来继续。这会把该服务器添加到你的已知主机的列表中
发现链接成功，并且无需密码。

在 91，133机器上分别操作
[root@chupiao2 ~]# useradd hadoop
[root@chupiao2 hadoop]# su - hadoop
[hadoop@chupiao2 ~]$ ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
06:88:7e:7c:c8:64:25:4c:09:b9:04:b2:03:86:ab:df hadoop@chupiao2
cd  .ssh
[hadoop@chupiao .ssh]$ chmod 644 authorized_keys
在131机器上执行，将authorized_keys传输到91,133

[hadoop@chupiao .ssh]$ scp authorized_keys 192.168.0.133:/home/hadoop/.ssh
测试：ssh 91或者ssh 133（第一次需要输入yes）。
 如果不须要输入密码则配置成功，如果还须要请检查上面的配置能不能正确。 

在三台机器上，下载hadoop并解压

vi hadoop-env.sh
export  JAVA_HOME=/usr/local/jdk1.6.0_32
export  HADOOP_HOME=/home/hadoop/hadoop-2.0.2-alpha
vi /home/hadoop/hadoop-2.0.2-alpha/etc/hadoop/core-site.xml 



<configuration>
<property>
   <!-- 用于dfs命令模块中指定默认的文件系统协议 -->
  <name>fs.default.name</name>
  <value>hdfs://192.168.0.131:9000</value>
   </property>
   <property>
   <name>hadoop.tmp.dir</name>
  <value>/home/hadoop/tmp/var</value>
  <!-- hadoop错误INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1  -->

 </property>
</configuration>

1）fs.default.name是NameNode的URI。hdfs://主机名:端口/
2）hadoop.tmp.dir ：Hadoop的默认临时路径，这个最好配置，
如果在新增节点或者其他情况下莫名其妙的DataNode启动不了，
就删除此文件中的tmp目录即可。不过如果删除了NameNode机器的此目录，
那么就需要重新执行NameNode格式化的命令。 

conf/mapred-site.xml
 
<?xmlversion="1.0"?>
 <?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
 <configuration>
 <property>
   <name>mapred.job.tracker</name>
   <value>192.168.0.131:9001</value>
 </property>
 <property>
   <name>mapred.local.dir</name>
  <value>/home/hadoop/tmp/var</value>
 </property>
 </configuration> 

1）mapred.job.tracker是JobTracker的主机（或者IP）和端口。主机:端口。
conf/hdfs-site.xml
 
<?xmlversion="1.0"?>
 <?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
 <configuration>
 <property>
 <name>dfs.name.dir</name>
 <value>/home/hadoop/name1, /home/hadoop/name2</value> #hadoop的name目录路径
 </property>
 <property>
 <name>dfs.data.dir</name>
 <value>/home/hadoop/data1, /home/hadoop/data2</value>
 </property>
 <property>
   <name>dfs.replication</name>
   <!-- 我们的集群又两个结点，所以rep两份 -->
   <value>2</value>
 </property>
 </configuration> 

1）   dfs.name.dir是NameNode持久存储名字空间及事务日志的本地文件系统路径。 
   当这个值是一个逗号分割的目录列表时，nametable数据将会被复制到所有目录中做冗余备份。
 2）   dfs.data.dir是DataNode存放块数据的本地文件系统路径，逗号分割的列表。 当这个值是逗号分割的目录列表时，数据将被存储在所有目录下，通常分布在不同设备上。
 3）dfs.replication是数据需要备份的数量，默认是3，如果此数大于集群的机器数会出错。 

注意：此处的name1、name2、data1、data2目录不能预先创建，hadoop格式化时会自动创建，如果预先创建反而会有问题。

vi Masters

192.168.0.131
vi
Slaves

192.168.0.133
192.168.0.91


启动

sbin/
start-all.sh

查看是否启动

/bin/
 ./hadoop dfs -ls
　　如果没有错误提示或者出现文件列表，那么恭喜你，Hadoop成功启动了

http://www.cnblogs.com/aniuer/archive/2012/07/16/2594448.html
最新版本的安装文档

[hadoop@hadoop ~]$ vi /home/hadoop/HadoopInstall/hadoop-conf/hadoop-env.sh
指定JAVA_HOME和HADOOP_HOME
export JAVA_HOME=/usr/local/jdk1.6.0_32
export HADOOP_HOME=/home/hadoop/hadoop-2.0.2-alpha

export PATH=$PATH:/home/hadoop/hadoop-0.22.0/bin

由于Hadoop用ssh 通信，因此先进行免密码登录设定， 
$ apt-get install ssh 
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub>> ~/.ssh/authorized_keys
 chmod 600 authorized_keys
 或者 cp id_rsa.pub authorized_keys

$ ssh localhost
完成后请登入确认不用输入密码，（第一次登入需按enter键，第二次就可以直接登入到系统。
~$ ssh localhost
~$ exit
~$ ssh localhost
~$ exit
如果mapred启动不起来 用 start-mapred.sh试一下，请查看start-all 里面的各环境变量

HADOOP_MAPRED_HOME

http://blog.csdn.net/hguisu/article/details/7237395

http://www.cnblogs.com/maczpc/archive/2010/01/17/1650191.htm

      ./hadoop  dfs -mkdir /input
    ./hadoop fs -ls /
    ./hadoop dfs -put ~/test-txt /input

     ./hadoop  dfs -ls /input/test-txt/file2.txt
      ./hadoop fs -cat /input/test-txt/file2.txt
      ./hadoop dfs -cat /input/test-txt/file2.txt
hadoop 命令
      http://blog.csdn.net/cheungmine/article/details/7744487

zookeeper 安装
cp zoo_sample.cfg zoo.cfg
http://www.linuxidc.com/Linux/2012-07/65607.htm

vi zoo.cfg
dataDir=/home/hadoop/zookeeper-3.4.3/data
server.1=192.168.0.131:2888:3888
server.2=192.168.0.91:2888:3888
server.3=192.168.0.133:2888:3888
#dataLogDir=/home/hadoop/zookeeper/log # 日志保存路径

/home/hadoop/zookeeper-3.4.3/data
vi myid
1

在各节点上分别启动
 ./zkServer.sh start
各节点测试测试
 ./zkCli.sh  192.168.0.91:2181

http://www.linuxidc.com/Linux/2012-07/65607.htm

export ZOOKEEPER_INSTALL=/home/hadoop/zookeeper-3.4.3
export PATH=$PATH:$ZOOKEEPER_INSTALL/bin

hbase 安装
http://blog.csdn.net/shi1122/article/details/7676488


1、修改conf/hbase-env.sh
 
export JAVA_HOME=/usr/local/jdk1.6.0_32
export HBASE_CLASSPATH=/home/hadoop/hbase-0.94.0/conf
export HBASE_MANAGES_ZK=false

2、修改hbase-site.xml，增加以下内容
 
<property>
 
    <name>hbase.rootdir</name>
 
    <value>hdfs://192.168.0.131:9000/hbase</value>(注：这里须hadoop-config/core-site.xml中的fs.default.name保持一致）
 
</property>
 
<property>
 
    <name>hbase.cluster.distributed</name>
 
    <value>true</value>
 
</property>
 
    <property>
 
      <name>hbase.zookeeper.property.clientPort</name>
 
      <value>2181</value>(注：这里的端口需要和zookeeper的端口配置值一样）
 
      <description>Property from ZooKeeper's config zoo.cfg.</description>
    </property>

 vi regionservers
 192.168.0.131
192.168.0.91
192.168.0.133

hbase 启动出现版本不一样的错误：
org.apache.Hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown
将HBase解压缩包中lib的Hadoop Core jar文件替换为当前你所使用的Hadoop版本即可
2.2 版本是
/home/hadoop/hadoop-2.0.2-alpha/share/hadoop/common
hadoop-comm
2013-01-11 15:55:41,202 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster

3、把/home/frank/HadoopInstall/hadoop-config/hdfs-site.xml文件拷贝至hbase的conf文件夹下
 
4、把${ZOOKEEPER_HOME}/conf/zoo.cfg拷贝至hbase的conf文件夹下
 
5、在conf/regionservers中添加hadoop-config/conf/slaves中所有的datanode节点。
 
6、删除/hbase-0.90.2/lib/hadoop-core-0.20-append-r1056497.jar
 
     拷贝/hadoop-0.20.2/hadoop-0.20.0-core.jar到/hbase-0.90.2/lib/


java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1740)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:146)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1754)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.authentication.util.KerberosName
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:212)
        at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:275)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
        at org.apache.hadoop.hbase.security.User.call(User.java:586)
        at org.apache.hadoop.hbase.security.User.callStatic(User.java:576)
        at org.apache.hadoop.hbase.security.User.access$400(User.java:50)
        at org.apache.hadoop.hbase.security.User$SecureHadoopUser.isSecurityEnabled(User.java:564)
        at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:535)
        at org.apache.hadoop.hbase.security.User.login(User.java:198)
        at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:267)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1735)
        ... 5 more


	Server IPC version 5 cannot communicate with client version 4

	版本不同引起的，可以将 hadoop 里面的相关jar包放在hbase的lib里面
cd bin  
hbase shell  
hbase(main):001:0> help  
hbase(main):002:0> status  
hbase(main):003:0> version  

java 程序如何操作hbase
http://blog.csdn.net/shi1122/article/details/7676488

http://blog.csdn.net/karen_wang/article/details/6283729
分享到：