sqoop2 安装部署、源码编译

 

 

 

1.安装准备工作：
已经装好的 hadoop 环境是cdh版本 hadoop-2.6.0 64位
下载的sqoop安装包 （不知道这个安装包支不支持64位，所以我下载的源码，然后自己编译源码）
http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.5-cdh5.5.1.tar.gz

源码下载地址：
http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.5-cdh5.5.1-src.tar.gz

编译源码

 

编译环境与hadoop的编译环境一致，详细配置请查看《Hadoop学习笔记 6 Hadoop源码编译》

 

maven编译命令：
mvn clean package -Pbinary -DskipTests

如果报内存溢出：

set MAVEN_OPTS=-XX:MaxPermSize=128M

 

编译好的安装包在 sqoop2-1.99.5-cdh5.5.1/dist/target/

2.解压文件到工作目录:
tar -xzvf sqoop-1.99.5-bin-hadoop200.tar.gz -C /usr/hadoop

3.修改环境变量：
vim /etc/profile
添加如下内容：
#sqoop
export SQOOP_HOME=/usr/hadoop/sqoop-1.99.5-bin-hadoop200
export PATH=SQOOPHOME/bin:PATH
export CATALINA_HOME=SQOOPHOME/server
LOGDIR=SQOOP_HOME/logs

保存退出即时生效：
source /etc/profile


4.修改sqoop配置：
vim /usr/hadoop/sqoop-1.99.5-bin-hadoop200/server/conf/sqoop.properties
#修改指向我的hadoop安装目录
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/hadoop/hadoop-2.6.0-cdh5.5.1
#把hadoop目录下的jar包都引进来
vim /usr/hadoop/sqoop-1.99.4-bin-hadoop200/server/conf/catalina.properties
common.loader=/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/common/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/common/lib/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/hdfs/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/hdfs/lib/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/mapreduce/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/mapreduce/lib/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/tools/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/tools/lib/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/yarn/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/yarn/lib/*.jar,
/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/httpfs/tomcat/lib/*.jar

或者
在$SQOOP_HOME中建个文件夹例如hadoop_lib，然后将这些jar包cp到此文件夹中，最后将此文件夹路径添加到common.loader属性
中，这种方法更加直观些
5.下载mysql驱动包
mysql-connector-java-5.1.32-bin.jar
并放到 /usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/httpfs/tomcat/lib/ 目录下

6.启动/停止sqoop2
/usr/hadoop/sqoop-1.99.5-bin-hadoop200/bin/sqoop.sh server start/stop
查看启动日志：
tail -500 /usr/hadoop/sqoop-1.99.5-bin-hadoop200/server/logs/catalina.out
7.进入客户端交互目录
/usr/hadoop/sqoop-1.99.5-bin-hadoop200/bin/sqoop.sh client