hadoop hive hbase 入门学习 (二)

knight_black_bob

浏览: 851057 次
性别:
来自: 北京

最近访客更多访客>>

bian1024

lli

cqh520llr

hxgdragon

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

hadoop
hive
hbase

hadoop hive hbase 大数据 nosql

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

hadoop 自学系列

hadoop hive hbase 入门学习 (一) hadoop安装、hdfs学习及mapreduce学习

hadoop 软件下载 (hadoop-1.2.1.tar.gz) 点击下载

hadoop hive hbase 入门学习 (二) hbase 安装、hbase语句学习

hbase 软件下载 (hbase-0.94.26.tar.gz) 点击下载

hadoop hive hbase 入门学习 (三) hive安装、及hive语句学习

hive软件下载 (hive-0.9.0.tar.gz)点击下载

mysql 客户端软件下载 (MySQL-client-5.5.23-1.linux2.6.i386.rpm)点击下载

mysql 服务端软件下载 (MySQL-server-5.5.23-1.linux2.6.i386.rpm)点击下载

mysql connector软件下载 (mysql-connector-java-5.1.16-bin.jar)点击下载

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

hbase 伪分布安装

1.hadoop版本：hadoop-1.1.2，HBase版本：hbase-0.94.26

最好的搭配是：

本人之前是： hadoop1.1.2 + hbase0.90.4 在csdn 下载的 hbase0.90.4 后来发现怎么也启动不了？
问题：ERROR : org.apache.Hadoop.hbase.MasterNotRunningException: null

网站上的解决方法：

1)启动hadoop后，需要等一段时间，再开启hbase

2)去掉hadoop的安全模式：

hadoop dfsadmin -safemode leave

3)把/etc/hosts里的master01的IP改为服务器当前的IP

4)确认hbase的hbase-site.xml中

<name>hbase.rootdir</name>

<value>hdfs://hadoop:9000/hbase</value>

与hadoop的core-site.xml中

<name>fs.default.name</name>
<value>hdfs://hadoop:9000</value>

红字部分保持一致

5)把 hadoop 目录下的 hadoop-core-xxx.jar 拷到 hbase/lib 替换 hadoop-core-yyY.jar

最后发现还是不行，没办法重新安装 hadoop-0.94.26 最后成功了~~~~

2.到官方网站下载好hbase-0.94.26.tar.gz，解压hbase安装包到一个可用目录如(/home/hadoop/hbase-0.94.26/)
tar -zxvf hbase-0.94.26.tar.gz

设置hbase-0.94.26所在用户为hadoop，所在组为hadoop(root用户操作)
chown -R hadoop hbase-0.94.26
chgrp -R hadoop hbase-0.94.26或者
chown -R hadoop:hadoop hbase-0.94.26
3.设置环境变量：在/etc/profile和/home/hadoop/.bashrc中配置添加(红色字体的)
#set java environment
HBASE_HOME=/home/hadoop/hbase-0.94.26

HIVE_HOME=/home/hadoop/hive-0.9.0
HADOOP_HOME=/home/hadoop/hadoop-1.1.1
JAVA_HOME=/home/hadoop/jdk1.7.0
PATH=$JAVA_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$HBASE_HOME/lib:$HIVE_HOME/lib:$JAVA_HOME/lib/tools.jar
export HBASE_HOME
export HADOOP_HOME
export JAVA_HOME
export HIVE_HOME
export PATH
export CLASSPATH

4.hbase配置：
在$HBASE_HOME/conf/hbase-env.sh中配置添加(红色字体的)

export JAVA_HOME= /usr/baoy/java

export HBASE_MANAGES_ZK=true

如果想使用HBase自带的zookeeper，要添加export HBASE_MANAGES_ZK=true

在$HBASE_HOME/conf目录中，确保hbase-site中的hbase.rootdir的主机和端口号与$HADOOP_HOME/conf目录中core-site.xml中的fs.default.name的主机和端口号一致，添加如下内容：

<configuration>
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://hadoop:9000/hbase</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.master</name>
        <value>hadoop:54311</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>hadoop</value>
    </property>
</configuration>

5.先启动hadoop，再启动hbase：
./start-hbase.sh #确认hadoop完全启动后启动hbase
jps #查看hbase启动情况，确认HQuorumPeer，HMaster，HRegionServer全部启动

第一个实例：

1、运行Eclipse，创建一个新的Java工程“HBaseClient”，右键项目根目录，选择 “Properties”->“Java Build Path”->“Library”->“Add External JARs”，将HBase解压后根目录下的hbase-0.94.1-security.jar、hbase-0.94.1-security-tests.jar和lib子目录下所有jar 包添加到本工程的Classpath下。

2. 按照步骤1中的操作，将自己所连接的HBase的配置文件hbase-site.xml添加到本工程的Classpath中，如下所示为配置文件的一个示例：

	private final static String TABLE_NAME ="t5";
	
	private static Configuration getConfiguration() {
		 final Configuration  conf =  HBaseConfiguration.create();
		conf.set("hbase.rootdir","hdfs://192.168.32.130:9000/hbase" );
		conf.set("hbase.zookeeper.quorum","192.168.32.130" );  
		return conf;
	}
	
	private static Configuration getConfiguration2() {
		Configuration  conf = new Configuration(); 
		conf.set("hbase.zookeeper.quorum", "192.168.32.130");  
		conf.set("hbase.zookeeper.property.clientPort", "2181");  
        conf = HBaseConfiguration.create(conf);
		return  conf;
	}
	
	public static void main(String[] args) throws Exception {
	//一类对表自身操作 修改 增加删除
	//一类对数据 的操作  增加 修改 删除
		final Configuration conf = getConfiguration();
		final HBaseAdmin hBaseAdmin = new HBaseAdmin(conf); 
		HTableDescriptor htableDescriptor =new HTableDescriptor(TABLE_NAME);
		htableDescriptor.addFamily(new HColumnDescriptor("f1")); 
		hBaseAdmin.createTable(htableDescriptor);
		hBaseAdmin.close();
	}

问题1：

java.net.ConnectException: Connection refused: no further information

a. zookeeper.ClientCnxn: Session 0x0 for server null,

解决: zppkeeper未启动，或无法连接，从查看各节点zookeeper启动状态、端口占用、防火墙等方面查看原因

b. getMaster attempt 4 of 10 failed; retrying after sleep of 2000

解决:查看 master log , 如果有信息org.apache.hadoop.hbase.regionserver.HRegionServer: Serving as BRDVM0240,43992,1373943529301, RPC listening on /127.0.0.1:43992, sessionid=0x13fe56a7d4b0001

则说明， HRegionServer

监听的端口是localhost 127.0.0.1, 需要修改 server端 /etc/hosts 文件， 127.0.0.1
servername localhost.localdomain localhost
去掉 servername, 然后重启hbase
第二种情况：
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/09/03 15:37:15 INFO zookeeper.ClientCnxn: Opening socket connection to server /192.168.0.118:2181
12/09/03 15:37:16 INFO zookeeper.ClientCnxn: EventThread shut down
12/09/03 15:37:16 INFO zookeeper.ZooKeeper: Session: 0x0 closed
Exception in thread "main" org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1209)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:511)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:502)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:172)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:92)
at com.biencloud.test.first_hbase.main(first_hbase.java:22)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
... 6 more

这个错误说明eclipse没有连接到zookeeper，在程序中添加zookeeper配置信息即可，具体如下：
Configuration conf=HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum","192.168.0.118, 192.168.0.186, 192.168.0.182");
conf.set("hbase.zookeeper.property.clientPort","2222");

还不行，在 windows 中

C:\Windows\System32\drivers\etc 添加你的 ip 和别名

比如 192.168.32.130 hadoop

我就死在这里，我记得我的所有配置都是用 ip 192.168.32.130 的没有用别名 hadoop,还是不行，，之后就加了一个就好了。

hbase 对数据库表的操作 & 对数据的操作

package study.hbase.demo.test;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor; 
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;

public class Test {
	
	private final static String TABLE_NAME ="t5";
	private static final byte[] ROWKEY = "rowkey".getBytes();
	private static final byte[] FAMILY = "f1".getBytes();
	private static final byte[] QULIFIER = "c1".getBytes();
	private static final byte[] VALUE = "value".getBytes();
	
	private static Configuration getConfiguration() {
		 final Configuration  conf =  HBaseConfiguration.create();
		conf.set("hbase.rootdir","hdfs://192.168.32.130:9000/hbase" );
		conf.set("hbase.zookeeper.quorum","192.168.32.130" );  
		return conf;
	}
	
	private static Configuration getConfiguration2() {
		Configuration  conf = new Configuration(); 
		conf.set("hbase.zookeeper.quorum", "192.168.32.130");  
		conf.set("hbase.zookeeper.property.clientPort", "2181");  
        conf = HBaseConfiguration.create(conf);
		return  conf;
	}
	
	public static void main(String[] args) throws Exception {
	//一类对表自身操作 修改 增加删除
	//一类对数据 的操作  增加 修改 删除
		final Configuration conf = getConfiguration();
		//createTable(conf);
		final HTable htable = new HTable(conf ,TABLE_NAME);
		//put 'f1','rowkey','f1:c1','value'
		final Put put =new Put(ROWKEY);
		put.add(FAMILY, QULIFIER, VALUE);
		htable.put(put);
		
 	
		final Get get =new Get(ROWKEY);
		final Result result = htable.get(get);
		final byte[] value2 = result.getValue(FAMILY, QULIFIER);
		System.out.println(new String (value2));
		
		System.out.println("-----------------------------------------------");
		
		final Scan scan= new Scan (); 
		final ResultScanner resultScanner = htable.getScanner(scan);
		for (Result temp : resultScanner) {
			final byte[] value3 = temp.getValue(FAMILY, QULIFIER);
			System.out.println(new String (value3));
		}
		htable.close();
		
		//deleteTable(conf);
		
	}

	@SuppressWarnings("unused")
	private static void deleteTable(final Configuration conf)
			throws MasterNotRunningException, ZooKeeperConnectionException,
			IOException {
		final HBaseAdmin hBaseAdmin = new HBaseAdmin(conf); 
		if(hBaseAdmin.tableExists(TABLE_NAME)){
		hBaseAdmin.disableTable(TABLE_NAME);
		hBaseAdmin.deleteTable(TABLE_NAME);
		hBaseAdmin.close();
		}
	}

	
	@SuppressWarnings("unused")
	private static void createTable(final Configuration conf)
			throws MasterNotRunningException, ZooKeeperConnectionException,
			IOException {
		final HBaseAdmin hBaseAdmin = new HBaseAdmin(conf); 
		if(hBaseAdmin.tableExists(TABLE_NAME)){
		HTableDescriptor htableDescriptor =new HTableDescriptor(TABLE_NAME);
		htableDescriptor.addFamily(new HColumnDescriptor("f1")); 
		hBaseAdmin.createTable(htableDescriptor);
		hBaseAdmin.close();
		}
	}

	
}