使用eclipse远程连接hbase

基础环境：
CDH 5.4.10
hadoop 2.6.0
hive  1.1.0
hbase 1.0.0
zookeeper  3.4.5
sqoop 1.4.5
jdk 1.7.0_67
os  centos6.5

由于我的hive和hbase是一起测试的，因此我就在hive连接的基础上，做hbase的连接测试，上面是hive的连接配置：
[root@master01 ~]# cd /opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/bin/
[root@master01 bin]# ./beeline
Beeline version 1.1.0-cdh5.4.10 by Apache Hive
beeline>

beeline> !connect jdbc:hive2://192.168.1.207:10000
Connecting to jdbc:hive2://192.168.1.207:10000
Enter username for jdbc:hive2://192.168.1.207:10000: root
Enter password for jdbc:hive2://192.168.1.207:10000: ****   --root/root直接连上了！
Connected to: Apache Hive (version 1.1.0-cdh5.4.10)
Driver: Hive JDBC (version 1.1.0-cdh5.4.10)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://192.168.1.207:10000>


先来创建几个表，导入点数据，方便后面操作:
[root@master01 bin]# hive
Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/jars/hive-common-1.1.0-cdh5.4.10.jar!/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.
hive>
hive> create table runningrecord_old(id int,systemno string,longitude string,latitude string,speed string,direction smallint,elevation string,acc string,islocation string,mileage string,oil string,currenttime timestamp,signalname string,currentvalue string) row format delimited fields terminated by ',';
OK
Time taken: 2.607 seconds
hive> load data local inpath '/tmp/rtest.txt' into table runningrecord_old;
Loading data to table default.runningrecord_old
Table default.runningrecord_old stats: [numFiles=1, totalSize=5480004]
OK
Time taken: 1.575 seconds


下一步，我们拷贝jar包到eclipse项目中去：
新建java project：hiveconnect
新建class：hiveconnecttest
新建folder:lib/    跟src同级

[root@master01 jars]# cd /opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/jars
[root@master01 jars]# sz hive*.jar   放到刚刚那个lib目录下：d:/workspace/hiveconnect/lib/
[root@master01 jars]# sz hadoop*.jar
[root@master01 jars]# ll hive*.jar
hive-accumulo-handler-1.1.0-cdh5.4.10.jar
hive-ant-1.1.0-cdh5.4.10.jar
hive-beeline-1.1.0-cdh5.4.10.jar
hive-cli-1.1.0-cdh5.4.10.jar
hive-common-1.1.0-cdh5.4.10.jar
hive-contrib-1.1.0-cdh5.4.10.jar
hive-exec-1.1.0-cdh5.4.10.jar
hive-hbase-handler-1.1.0-cdh5.4.10.jar
hive-hcatalog-core-1.1.0-cdh5.4.10.jar
hive-hcatalog-pig-adapter-1.1.0-cdh5.4.10.jar
hive-hcatalog-server-extensions-1.1.0-cdh5.4.10.jar
hive-hcatalog-streaming-1.1.0-cdh5.4.10.jar
hive-hwi-1.1.0-cdh5.4.10.jar
hive-jdbc-1.1.0-cdh5.4.10.jar
hive-jdbc-1.1.0-cdh5.4.10-standalone.jar
hive-metastore-1.1.0-cdh5.4.10.jar
hive-serde-1.1.0-cdh5.4.10.jar
hive-service-1.1.0-cdh5.4.10.jar
hive-shims-0.23-1.1.0-cdh5.4.10.jar
hive-shims-1.1.0-cdh5.4.10.jar
hive-shims-common-1.1.0-cdh5.4.10.jar
hive-shims-scheduler-1.1.0-cdh5.4.10.jar
hive-testutils-1.1.0-cdh5.4.10.jar
hive-webhcat-1.1.0-cdh5.4.10.jar
hive-webhcat-java-client-1.1.0-cdh5.4.10.jar
hadoop-annotations-2.6.0-cdh5.4.10.jar
hadoop-ant-2.6.0-cdh5.4.10.jar
hadoop-ant-2.6.0-mr1-cdh5.4.10.jar
hadoop-archives-2.6.0-cdh5.4.10.jar
hadoop-auth-2.6.0-cdh5.4.10.jar
hadoop-aws-2.6.0-cdh5.4.10.jar
hadoop-azure-2.6.0-cdh5.4.10.jar
hadoop-capacity-scheduler-2.6.0-mr1-cdh5.4.10.jar
hadoop-common-2.6.0-cdh5.4.10.jar
hadoop-common-2.6.0-cdh5.4.10-tests.jar
hadoop-core-2.6.0-mr1-cdh5.4.10.jar
hadoop-datajoin-2.6.0-cdh5.4.10.jar
hadoop-distcp-2.6.0-cdh5.4.10.jar
hadoop-examples-2.6.0-mr1-cdh5.4.10.jar
hadoop-examples.jar
hadoop-extras-2.6.0-cdh5.4.10.jar
hadoop-fairscheduler-2.6.0-mr1-cdh5.4.10.jar
hadoop-gridmix-2.6.0-cdh5.4.10.jar
hadoop-gridmix-2.6.0-mr1-cdh5.4.10.jar
hadoop-hdfs-2.6.0-cdh5.4.10.jar
hadoop-hdfs-2.6.0-cdh5.4.10-tests.jar
hadoop-hdfs-nfs-2.6.0-cdh5.4.10.jar
hadoop-kms-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-app-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-common-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-core-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-hs-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-jobclient-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-jobclient-2.6.0-cdh5.4.10-tests.jar
hadoop-mapreduce-client-nativetask-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-shuffle-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-examples-2.6.0-cdh5.4.10.jar
hadoop-nfs-2.6.0-cdh5.4.10.jar
hadoop-rumen-2.6.0-cdh5.4.10.jar
hadoop-sls-2.6.0-cdh5.4.10.jar
hadoop-streaming-2.6.0-cdh5.4.10.jar
hadoop-streaming-2.6.0-mr1-cdh5.4.10.jar
hadoop-test-2.6.0-mr1-cdh5.4.10.jar
hadoop-tools-2.6.0-mr1-cdh5.4.10.jar
hadoop-yarn-api-2.6.0-cdh5.4.10.jar
hadoop-yarn-applications-distributedshell-2.6.0-cdh5.4.10.jar
hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.4.10.jar
hadoop-yarn-client-2.6.0-cdh5.4.10.jar
hadoop-yarn-common-2.6.0-cdh5.4.10.jar
hadoop-yarn-registry-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-common-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-nodemanager-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-resourcemanager-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-tests-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-web-proxy-2.6.0-cdh5.4.10.jar

添加到build path

测试ok，测试内容下方：
连接：org.apache.hive.jdbc.HiveConnection@43a25848
是否有数据：true
Running:show tables 'tinatest'
执行“show tables”运行结果:
tinatest
Running:describe tinatest
执行“describe table”运行结果:
key int
value string
Running:load data local inpath '/tmp/test2.txt' into table tinatest
Running:select * from tinatest
执行“select * query”运行结果:
1 a
2 b
3 tina
====================================================================================================================

在hive连接的基础上，我们来创建一个hbase的连接实例：
新建class：hbaseconnecttest

sz hbase*.jar
hbase-annotations-1.0.0-cdh5.4.10.jar
hbase-annotations-1.0.0-cdh5.4.10-tests.jar
hbase-checkstyle-1.0.0-cdh5.4.10.jar
hbase-client-1.0.0-cdh5.4.10.jar
hbase-client-1.0.0-cdh5.4.10-tests.jar
hbase-common-1.0.0-cdh5.4.10.jar
hbase-common-1.0.0-cdh5.4.10-tests.jar
hbase-examples-1.0.0-cdh5.4.10.jar
hbase-hadoop2-compat-1.0.0-cdh5.4.10.jar
hbase-hadoop2-compat-1.0.0-cdh5.4.10-tests.jar
hbase-hadoop-compat-1.0.0-cdh5.4.10.jar
hbase-hadoop-compat-1.0.0-cdh5.4.10-tests.jar
hbase-indexer-cli-1.5-cdh5.4.10.jar
hbase-indexer-common-1.5-cdh5.4.10.jar
hbase-indexer-demo-1.5-cdh5.4.10.jar
hbase-indexer-engine-1.5-cdh5.4.10.jar
hbase-indexer-model-1.5-cdh5.4.10.jar
hbase-indexer-morphlines-1.5-cdh5.4.10.jar
hbase-indexer-mr-1.5-cdh5.4.10.jar
hbase-indexer-mr-1.5-cdh5.4.10-job.jar
hbase-indexer-server-1.5-cdh5.4.10.jar
hbase-it-1.0.0-cdh5.4.10.jar
hbase-it-1.0.0-cdh5.4.10-tests.jar
hbase-prefix-tree-1.0.0-cdh5.4.10.jar
hbase-protocol-1.0.0-cdh5.4.10.jar
hbase-rest-1.0.0-cdh5.4.10.jar
hbase-sep-api-1.5-cdh5.4.10.jar
hbase-sep-impl-1.5-hbase1.0-cdh5.4.10.jar
hbase-sep-impl-common-1.5-cdh5.4.10.jar
hbase-sep-tools-1.5-cdh5.4.10.jar
hbase-server-1.0.0-cdh5.4.10.jar
hbase-server-1.0.0-cdh5.4.10-tests.jar
hbase-shell-1.0.0-cdh5.4.10.jar
hbase-testing-util-1.0.0-cdh5.4.10.jar
hbase-thrift-1.0.0-cdh5.4.10.jar

第一次执行报错：
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/zookeeper/KeeperException
at hiveconnect.hbaseconnecttest.main(hbaseconnecttest.java:17)

[root@master01 jars]# sz zookeeper-3.4.5-cdh5.4.10.jar

第二次执行报错：
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Level
at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:348)


[root@master01 jars]# sz *log4j*.jar
apache-log4j-extras-1.1.jar
apache-log4j-extras-1.2.17.jar
flume-ng-log4jappender-1.5.0-cdh5.4.10.jar
flume-ng-log4jappender-1.5.0-cdh5.4.10-jar-with-dependencies.jar
log4j-1.2.16.jar
log4j-1.2.17.jar
slf4j-log4j12-1.7.5.jar


[root@master01 bin]# hbase shell
16/06/12 10:56:09 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.0.0-cdh5.4.10, rUnknown, Tue Apr 12 11:10:23 PDT 2016

hbase(main):001:0>

第三次执行报错：
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)

[root@master01 jars]# ll *trace*.jar
-rw-r--r-- 1 root root  117409 Apr 13 02:42 accumulo-trace-1.6.0.jar
-rw-r--r-- 1 root root   30457 Apr 13 02:40 htrace-core-2.00.jar
-rw-r--r-- 1 root root   31212 Apr 13 02:40 htrace-core-3.0.4.jar
-rw-r--r-- 1 root root 1475955 Apr 13 02:39 htrace-core-3.1.0-incubating.jar
[root@master01 jars]# sz *trace*.jar


报错：
16/06/12 13:35:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/06/12 13:35:24 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

新建目录：conf
/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/etc/hbase/conf.dist/hbase-site.xml
[root@master01 jars]# cd /opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/etc/hbase/conf.dist/

[root@master01 conf.dist]# cd /etc/hbase/conf.cloudera.hbase/
[root@master01 conf.cloudera.hbase]# ll
total 28
-rw-r--r-- 1 root root   21 Jun  8 10:07 __cloudera_generation__
-rw-r--r-- 1 root root 3538 Jun  8 10:07 core-site.xml
-rw-r--r-- 1 root root  360 Jun  8 10:07 hbase-env.sh
-rw-r--r-- 1 root root 1984 Jun  8 10:07 hbase-site.xml
-rw-r--r-- 1 root root 1610 Jun  8 10:07 hdfs-site.xml
-rw-r--r-- 1 root root    0 Jun  8 10:07 jaas.conf
-rw-r--r-- 1 root root  312 Jun  8 10:07 log4j.properties
-rw-r--r-- 1 root root  315 Jun  8 10:07 ssl-client.xml
[root@master01 conf.cloudera.hbase]# sz hbase-site.xml
*B00000000000000
[root@master01 conf.cloudera.hbase]#

选中项目名称hbaseconnect，右键选择Propertie->Java Build Path->Libraries->Add Class Folder（选中conf，将conf目录添加进这个项目）。

报错：主机不识别
16/06/12 14:11:25 WARN zookeeper.RecoverableZooKeeper: Unable to create ZooKeeper Connection
java.net.UnknownHostException: master01

编辑文件C:\WINDOWS\system32\drivers\etc\hosts
192.168.1.207 master01


还是缺少包，干脆把jars下面的全拷贝过来算了！
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:636)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:608)

sz protobuf*.jar  slf4j*.jar commons*.jar


还是存在报错：
16/06/12 15:05:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/06/12 15:05:10 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

Hadoop2.x下 Windows下eclipse开发hadoop程序会报错，原因是因为hadoop2.x没有发布winutils.exe造成的，

提供了winutils的windows的版本，项目地址是：https://codeload.github.com/amihalik/hadoop-common-2.6.0-bin/zip/master
将包解压到D:\hadoop-common-2.6.0-bin-master

设置环境变量，并重启电脑
HADOOP_HOME=D:\hadoop-common-2.6.0-bin-master

不再报错！



测试的java代码：
package hiveconnect;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.util.Bytes;

public class hbaseconnect {

        private static Configuration conf = null;
        static {
                conf = HBaseConfiguration.create();
                conf.set("hbase.zookeeper.quorum", "master01");// 使用eclipse时必须添加这个，否则无法定位master需要配置hosts
                conf.set("hbase.zookeeper.property.clientPort", "2181");
        }

        public static void main(String[] args) throws IOException {
                String[] cols = new String[1];
                String[] colsValue = new String[1];
                cols[0] = "title";
                colsValue[0] = "AboutYunArticle";

                // 创建表
                createTable();
                // 添加值
                addData("www.aboutyun.com", "blog01", cols, colsValue);
        }

        private static void createTable() throws MasterNotRunningException,
                        ZooKeeperConnectionException, IOException {

                HBaseAdmin admin = new HBaseAdmin(conf);// 新建一个数据库管理员//新api
                if (admin.tableExists(TableName.valueOf("LogTable"))) {
                        System.out.println("table is not exist!");
                        System.exit(0);
                } else {

                        HTableDescriptor desc = new HTableDescriptor(
                                        TableName.valueOf("blog01"));
                        desc.addFamily(new HColumnDescriptor("article"));
                        admin.createTable(desc);
                        admin.close();
                        System.out.println("create table Success!");
                }
        }
        private static void addData(String rowKey, String tableName,String[] column1, String[] value1) throws IOException {
                Put put = new Put(Bytes.toBytes(rowKey));// 设置rowkey
                HTable table = new HTable(conf, Bytes.toBytes(tableName));// HTabel负责跟记录相关的操作如增删改查等//
                                                                                                                                        // 获取表
                HColumnDescriptor[] columnFamilies = table.getTableDescriptor() // 获取所有的列族
                                .getColumnFamilies();

                for (int i = 0; i < columnFamilies.length; i++) {
                        String familyName = columnFamilies[i].getNameAsString(); // 获取列族名
                        if (familyName.equals("article")) { // article列族put数据
                                for (int j = 0; j < column1.length; j++) {
                                        put.add(Bytes.toBytes(familyName),
                                                        Bytes.toBytes(column1[j]), Bytes.toBytes(value1[j]));
                                }
                        }
                }
                table.put(put);
                System.out.println("add data Success!");
        }

}



测试结果：
hbase(main):003:0> scan 'blog'
ROW                                COLUMN+CELL                                                                                       
www.aboutyun.com                  column=article:title, timestamp=1465721042702, value=AboutYunArticle                              
1 row(s) in 0.1010 seconds


将项目打包导出：
选中项目名称hiveconnect--export--runnable jar file--（hiveconnecttest-hiveconnect)--C:\Users\Administrator\Desktop\hiveconnect.jar--


将jar包导入：
新建一个项目：testhive--build path--configure build path--add external jars--选中C:\Users\Administrator\Desktop\hiveconnect.jar
新建类hivetest：

package testhive;
import hiveconnect.hiveconnecttest;

public class hivetest {

public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
hiveconnect.hiveconnecttest hiveconnecttest = new hiveconnecttest();
hiveconnecttest.main(null);
}

}
就可以直接执行了！


学习阶段，不对的地方欢迎指正。
QQ:906179271
tina