【大数据之sqoop安装】

Sqoop(发音：skup)是一款开源的工具，主要用于在Hadoop(Hive)与传统的数据库(mysql、postgresql...)间进行数据的传递，可以将一个关系型数据库（例如 ： MySQL ,Oracle ,Postgres等）中的数据导进到Hadoop的HDFS中，也可以将HDFS的数据导进到关系型数据库中。

 

一、安装

[root@hadoop0 opt]# tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz 

[root@hadoop0 opt]# clear

[root@hadoop0 opt]# ls

apache-storm-0.9.6.tar.gz  elasticsearch-2.3.3      jdk1.7.0_79             redis-3.2.0         scala-2.10.5           spark-1.3.1-bin-hadoop2.6.tgz               zookeeper-3.4.6.tar.gz   bigdata                    

elasticsearch-2.3.3.zip  kafka_2.11-0.9.0.1.tgz  redis-3.2.0.tar.gz  scala-2.10.5.tgz          

sqoop-1.4.6.bin__hadoop-2.0.4-alpha hadoop-2.7.2.tar.gz      ORCLfmap                rh                  spark-1.3.1-bin-hadoop2.6  sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

[root@hadoop0 opt]# mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop-1.4.6

[root@hadoop0 opt]# cd sqoop-1.4.6/bin/

[root@hadoop0 bin]# ./sqoop list-tables --connect jdbc:mysql://192.168.1.102:3306/taokeeper --username root --password  123456

Warning: /opt/sqoop-1.4.6/bin/../../hcatalog does not exist! HCatalog jobs will fail.

Please set $HCAT_HOME to the root of your HCatalog installation.

Warning: /opt/sqoop-1.4.6/bin/../../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

Warning: /opt/sqoop-1.4.6/bin/../../zookeeper does not exist! Accumulo imports will fail.

Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

99/04/29 18:46:52 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6

99/04/29 18:46:52 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

99/04/29 18:46:52 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

99/04/29 18:46:52 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver

java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver

        at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:856)

        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)

        at org.apache.sqoop.manager.CatalogQueryManager.listTables(CatalogQueryManager.java:102)

        at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)

        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)

        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)

        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)

        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

解决方案：拷贝数据库驱动包到sqoop-1.4.6/lib中

 

[root@hadoop0 bin]# ./sqoop list-tables --connect jdbc:mysql://192.168.1.102:3306/taokeeper --username root --password  123456

Warning: /opt/sqoop-1.4.6/bin/../../hcatalog does not exist! HCatalog jobs will fail.

Please set $HCAT_HOME to the root of your HCatalog installation.

Warning: /opt/sqoop-1.4.6/bin/../../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

Warning: /opt/sqoop-1.4.6/bin/../../zookeeper does not exist! Accumulo imports will fail.

Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

99/04/29 18:47:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6

99/04/29 18:47:56 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

99/04/29 18:47:56 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

alarm_settings

taokeeper_settings

taokeeper_stat

zookeeper_cluster

[root@hadoop0 bin]# cat ~/.bash_profile 

# .bash_profile

 

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

 

# User specific environment and startup programs

 

PATH=$PATH:$HOME/bin

 

JAVA_HOME=/opt/jdk1.7.0_79

STORM_HOME=/opt/bigdata/stomr096

HIVE_HOME=/opt/bigdata/hive2.0

HADOOP_HOME=/opt/bigdata/hadoop272

HBASE_HOME=/opt/bigdata/hbase-1.1.5

 

PATH=$PATH:$JAVA_HOME/bin:$STORM_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin

export JAVA_HOME STORM_HOME HIVE_HOME HADOOP_HOME HBASE_HOME

export PATH 

 

 

 

export SCALA_HOME=/opt/scala-2.10.5

export PATH=.:$SCALA_HOME/bin:$PATH

 

export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/bigdata/hadoop272/lib/native

export HADOOP_OPTS="-Djava.library.path=/opt/bigdata/hadoop272/lib"

[root@hadoop0 bin]# 

 

二、验证