Hadoop HBase Hive伪分布式环境搭建

Hadoop HBase Hive

启动：

$HADOOP_HOME/bin/start-all.sh

$HBASE_HOME/bin/start-hbase.sh

$HIVE_HOME/bin/hive start

环境配置

1、JDK安装

2、SSH配置

3、环境变量

/etc/profile

    export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_65

export JRE_HOME=${JAVA_HOME}/jre

export CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:$JAVA_HOME/lib/tools.jar:${JRE_HOME}/lib/rt.jar

export HADOOP_HOME=/usr/local/hadoop

export CLASSPATH=.:$CLASSPATH:$HADOOP_HOME/lib

export HBASE_HOME=/usr/local/hbase

export HIVE_HOME=/usr/local/hive

export CLASSPATH=$CLASSPATH:$HIVE_HOME/lib

export PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin:${HADOOP_HOME}/bin:${HBASE_HOME}/bin:$HIVE_HOME/bin:$PATH

Hadoop   

Hadoop版本：1.1.2

目的

This document describes how to set up and configure a single-node Hadoop installation so that you can quickly perform simple operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS).

Prerequisites

Required Software

Required software for Linux and Windows include:

1.    Java^TM 1.6.x, preferably from Sun, must be installed.

2.    ssh must be installed and sshd must be running to use the Hadoop scripts that manage remote Hadoop daemons.

Additional requirements for Windows include:

1.   Cygwin - Required for shell support in addition to the required software above.

Installing Software

If your cluster doesn't have the requisite software you will need to install it.

For example on Ubuntu Linux:

$ sudo apt-get install ssh 
$ sudo apt-get install rsync

On Windows, if you did not install the required software when you installed cygwin, start the cygwin installer and select the packages:

·  openssh - the Net category

下载

To get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.

Prepare to Start the Hadoop Cluster

Unpack the downloaded Hadoop distribution. In the distribution, edit the file conf/hadoop-env.sh to define at least JAVA_HOME to be the root of your Java installation.

export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_65

Try the following command:
$ bin/hadoop 
This will display the usage documentation for the hadoop script.

 

伪分布式Pseudo-Distributed Operation

Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

配置Configuration

Use the following: 

hostname myhadoop

vi /etc/hostname

myhadoop

vi /etc/hosts

ip myhadoop

conf/core-site.xml:

<configuration>

     <property>

         <name>fs.default.name</name>

         <value>hdfs://myhadoop:9000</value>

     </property>

</configuration>

conf/hdfs-site.xml:

<configuration>

     <property>

         <name>dfs.replication</name>

         <value>1</value>

     </property>

    <property>

        <name>dfs.name.dir</name>

        <value>/usr/local/hadoop/hadoopdata/dfsname</value>

    </property>

    <property>

        <name>dfs.data.dir</name>                                    
        <value>/usr/local/hadoop/hadoopdata/dfsdata</value>

    </property>

</configuration>

conf/mapred-site.xml:

<configuration>

     <property>

         <name>mapred.job.tracker</name>

         <value>myhadoop:9001</value>

     </property>

</configuration>

Setup passphraseless ssh

Now check that you can ssh to the localhost without a passphrase:
$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

执行Execution

Format a new distributed-filesystem:
$ bin/hadoop namenode -format

Start the hadoop daemons:
$ bin/start-all.sh

The hadoop daemon log output is written to the ${HADOOP_LOG_DIR} directory (defaults to ${HADOOP_HOME}/logs).

Browse the web interface for the NameNode and the JobTracker; by default they are available at:

<!--[if !supportLists]-->·         <!--[endif]-->NameNode - http://localhost:50070/

<!--[if !supportLists]-->·         <!--[endif]-->JobTracker - http://localhost:50030/

Copy the input files into the distributed filesystem:
$ bin/hadoop fs -put conf input

Run some of the examples provided:
$ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'

Examine the output files:

Copy the output files from the distributed filesystem to the local filesytem and examine them:
$ bin/hadoop fs -get output output 
$ cat output/*

or

View the output files on the distributed filesystem:
$ bin/hadoop fs -cat output/*

When you're done, stop the daemons with:
$ bin/stop-all.sh

 

HBase

HBase版本：0.94.7

hbase-site.xml

  <property>

    <name>hbase.rootdir</name>

    <value>hdfs://myhadoop:9000/hbase</value>

  </property>

  <property>

    <name>hbase.cluster.distributed</name>

    <value>true</value>

  </property>

  <property>

    <name>hbase.zookeeper.quorum</name>

    <value>myhadoop</value>

  </property>

bin/start-hbase.sh

 

Hive

一、hive配置

 cp hive-default.xml.template hive-site.xml

 cp hive-log4j.properties.template hive-log4j.properties

 cp hive-env.sh.template hive-env.sh

 

二、修改hive-env.sh

配置HADOOP_HOME路径。

三、修改hive-site.xml配置文件，把Hive的元数据存储到MySQL中

<property>

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://myhadoop:3306/hive_metadata?createDatabaseIfNotExist=true</value>

</property>

<property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

</property>

<property>

<name>javax.jdo.option.ConnectionUserName</name>

<value>root</value>

</property>

<property>

<name>javax.jdo.option.ConnectionPassword</name>

<value>root</value>

</property>

<property>

<name>hive.metastore.warehouse.dir</name>

<value>/user/hive/warehouse</value>

</property>

 

<property>

    <name>hive.aux.jars.path</name>

<value>file:///usr/local/hive/lib/hive-hbase-handler-0.9.0.jar,file:///usr/local/hive/lib/hbase-0.94.7-security.jar,file:///usr/local/hive/lib/protobuf-java-2.4.0a.jar,file:///usr/local/hive/lib/zookeeper-3.4.5.jar</value>

</property>

删除/usr/local/hive/lib下的hbase-0.92.0.jar，hbase-0.92.0-tests.jar，zookeeper-3.4.3.jar

 

从hbase拷贝hbase-0.94.7-security.jar、zookeeper-3.4.5.jar和protobuf-java-2.4.0a.jar到hive/lib下。

三、修改hive-log4j.properties

#log4j.appender.EventCounter=org.apache.hadoop.metrics.jvm.EventCounter

log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter

 

四、在hdfs上面，创建目录

$HADOOP_HOME/bin/hadoop fs -mkdrr /tmp

$HADOOP_HOME/bin/hadoop fs –mkdir /user/hive/warehouse

$HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp

$HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse

 

五、手动上传mysql的jdbc库到hive/lib

~ ls /home/cos/toolkit/hive-0.9.0/lib

mysql-connector-java-5.1.22-bin.jar

六、启动hive

方式1：

bin/hive start

方式2：

#启动metastore服务

~ bin/hive --service metastore &

Starting Hive Metastore Server

 

#启动hiveserver服务

~ bin/hive --service hiveserver &

Starting Hive Thrift Server

#启动hive客户端

~ bin/hive shell

Logging initialized using configuration in file:/root/hive-0.9.0/conf/hive-log4j.properties

Hive history file=/tmp/root/hive_job_log_root_201211141845_1864939641.txt

 

hive> show tables

OK

 

 

 

Hive函数、复杂类型访问操作

hive提供了复合数据类型：

Structs： structs内部的数据可以通过DOT（.）来存取，例如，表中一列c的类型为STRUCT{a INT; b INT}，我们可以通过c.a来访问域a

Maps（K-V对）：访问指定域可以通过["指定域名称"]进行，例如，一个Map M包含了一个group-》gid的kv对，gid的值可以通过M['group']来获取

Arrays：array中的数据为相同类型，例如，假如array A中元素['a','b','c']，则A[1]的值为'b'

 

Array：

建表：

create table class_test(name string, student_id_list array<INT>)  ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY ':';

导数据：

vim test.txt

034,1:2:3:4

035,5:6

036,7:8:9:10

 

LOAD DATA LOCAL INPATH '/opt/test.txt' INTO TABLE class_test ;

查询：

select student_id_list[3] from class_test;

 

Map：

建表：

create table employee(id string, perf map<string, int>) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':';

导数据：

vim test2.txt  

1   job:80,team:60,person:70

2   job:60,team:80

3   job:90,team:70,person:100

 

LOAD DATA LOCAL INPATH '/opt/test2.txt' INTO TABLE employee;

查询：

select perf['person'] from employee; 

select perf['person'] from employee where perf['person'] is not null;

 

Struct使用

建表：

create table student_test(id INT, info struct<name:STRING, age:INT>) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY ':'; 

'FIELDS TERMINATED BY' ：字段与字段之间的分隔符

'COLLECTION ITEMS TERMINATED BY' ：一个字段各个item的分隔符

导入数据：

cat test3.txt  

1,zhou:30 

2,yan:30 

3,chen:20 

4,li:80 

LOAD DATA LOCAL INPATH '/opt/test3.txt' INTO TABLE student_test; 

查询：

select info.age from student_test;

 

 

查询每年9.30-10.07十一期间的身份证号：

select substr(rzsj,1,4) as year, sfzmhm from jnlk where substr(rzsj,6,5)>='09-30' and substr(rzsj,6,5)<='10-07' and rzsj is not null order by year,sfzmhm;

练习

Hbase创建表：

create 'blog','article','author'

 

插入hbase数据：

put 'blog','1','article:title','Head First HBase'

put 'blog','1','article:content','HBase is the Hadoop database. Use it when you need random, realtime read/write access to your Big Data.'

put 'blog','1','article:tags','Hadoop,HBase,NoSQL'

put 'blog','1','author:name','hujinjun'

put 'blog','1','author:nickname','一叶渡江'

 

put 'blog','10','article:tags','Hadoop'

put 'blog','10','author:nickname','heyun'

 

 

put 'blog','100','article:tags','hbase,nosql'

put 'blog','100','author:nickname','shenxiu'

 

hive:

CREATE EXTERNAL TABLE blog(key int,title string,content string,tags string,name string,nickname string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "article:title,article:content,article:tags,author:name,author:nickname") TBLPROPERTIES("hbase.table.name" = "blog");

 

 

hive> create table wyp (id int, name string, age int, tel string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE;

 

vim /opt/wyp.txt

1   wyp 25  13188888888888

2   test   30  13888888888888

3   zs  34  899314121

 

hive> load data local inpath '/opt/wyp.txt' into table wyp;

 

vim /opt/add.txt

5   wyp1   23  131212121212

6   wyp2   24  134535353535

7   wyp3   25  132453535353

8   wyp4   26  154243434355

 

$HADOOP_HOME/bin/hadoop fs -mkdir /wyp

$HADOOP_HOME/bin/hadoop fs -copyFromLocal /opt/add.txt /wyp/add.txt

 

hive> load data inpath '/wyp/add.txt' into table wyp;

 

hive> select * from wyp;

 

 

FAQ

 

hive hwi 启动错误

错误日志：

INFO hwi.HWIServer: HWI is starting up

WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml found on the CLASSPATH at /usr/local/hive/conf/hive-default.xml

FATAL hwi.HWIServer: HWI WAR file not found at /usr/local/hive/usr/local/hive/lib/hive-hwi-0.9.0.war

解决方法：

这样的错误解决办法很简单，hive-site.xml中添加：

<property>

  <name>hive.hwi.war.file</name>

  <value>lib/hive-hwi-0.9.0.war</value>

  <description>This sets the path to the HWI war file, relative to ${HIVE_HOME}. </description>

</property>

否则路径错误！