Sqoop 1.99.3 安装

侯上校

浏览: 228322 次
性别:
来自: 上海

最近访客更多访客>>

nothingisall

ecjtubaowp

killbrokenwolf

wulugeng-2018

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

sqoop

1.安装准备工作：
已经装好的hadoop环境是hadoop 2.2.0
下载的sqoop安装包（注意是hadoop200）
http://www.us.apache.org/dist/sqoop/1.99.3/sqoop-1.99.3-bin-hadoop200.tar.gz
2.解压文件到工作目录:
hadoop@hadoopMaster:$ sudo tar -xvf /opt/hn/hadoop_family/sqoop-1.99.3-bin-hadoop200.tar.gz 
hadoop@hadoopMaster:mv /opt/hn/hadoop_family/sqoop-1.99.3-bin-hadoop200 /usr/local/sqoop
3.修改环境变量：
hadoop@hadoopMaster:~$ vim /etc/profile
添加如下内容：
#sqoop
export SQOOP_HOME=/usr/local/sqoop
export PATH=$SQOOP_HOME/bin:$PATH
export CATALINA_HOME=$SQOOP_HOME/server
export LOGDIR=$SQOOP_HOME/logs
保存退出即时生效：
source /etc/profile
4.修改sqoop配置：
hadoop@hadoopMaster:~$ vim /usr/local/sqoop/server/conf/sqoop.properties 
#修改指向我的hadoop安装目录
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/
#把hadoop目录下的jar包都引进来
hadoop@hadoopMaster:~$ vim /usr/local/sqoop/server/conf/catalina.properties 
common.loader=/usr/local/hadoop/share/hadoop/common/*.jar,/usr/local/hadoop/share/hadoop/common/lib/*.jar,/usr/local/hadoop/share/hadoop/hdfs/*.jar,/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/share/hadoop/tools/*.jar,/usr/local/hadoop/share/hadoop/tools/lib/*.jar,/usr/local/hadoop/share/hadoop/yarn/*.jar,/usr/local/hadoop/share/hadoop/yarn/lib/*.jar,/usr/local/hadoop/share/hadoop/httpfs/tomcat/lib/*.jar
5.下载mysql驱动包
mysql-connector-java-5.1.16-bin.jar
6.启动/停止sqoop200
hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh server start/stop
查看启动日志：
hadoop@hadoopMaster:/usr/local/sqoop/server/logs$ vim catalina.out 
7.进入客户端交互目录
hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh client
+------------------------------------------+
|Sqoop home directory: /usr/local/sqoop    |
|Sqoop Shell: Type 'help' or '\h' for help.|
|sqoop:000>                                |
+------------------------------------------+
为客户端配置服务器：
+---------------------------------------------------------------------+
|sqoop:000> set server --host hadoopMaster --port 12000 --webapp sqoop|
|Server is set successfully                                           |
+---------------------------------------------------------------------+
查版本信息：
+-----------------------------------------------------------------+
|sqoop:000> show version --all                                    |
|client version:                                                  |
|  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b |
|  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013           |
|server version:                                                  |
|  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b |
|  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013           |
|Protocol version:                                                |
|  [1]                                                            |
+-----------------------------------------------------------------+
显示连接器：
+---------------------------------------------------------------------------------------------+
|sqoop:000> show connector --all                                                              |
|1 connector(s) to show:                                                                      |
|Connector with id 1:                                                                         |
|  Name: generic-jdbc-connector                                                               |
| Class: org.apache.sqoop.connector.jdbc.GenericJdbcConnector                                 |
|  Version: 1.99.3                                                                            |
|  Supported job types: [IMPORT, EXPORT]                                                      |
|    Connection form 1:                                                                       |
|      Name: connection                                                                       |
|      Label: Connection configuration                                                        |
|      Help: You must supply the information requested in order to create a connection object.|
|      Input 1:                                                                               |
|        .                                                                                    |
|        .                                                                                    |
|        .                                                                                    |
|   太长了，就拷贝这一点                                                                       |
+---------------------------------------------------------------------------------------------+
创建数据库连接：
+---------------------------------------------------------------------------------------------+
|sqoop:000> create connection --cid 1                                                         |
|Creating connection for connector with id 1                                                  |
|Please fill following values to create new connection object                                 |
|Name: My first                                                                               |
|                                                                                             |
|Connection configuration                                                                     |
|                                                                                             |
|JDBC Driver Class: com.mysql.jdbc.Driver                                                     |
|JDBC Connection String: jdbc:mysql://localhost:3306/sqoop_stu                                |
|Username: root                                                                               |
|Password: **********                                                                         |
|JDBC Connection Properties:                                                                  |
|There are currently 0 values in the map:                                                     |
|entry#                                                                                       |
|                                                                                             |
|Security related configuration options                                                       |
|                                                                                             |
|Max connections: 100                                                                         |
|New connection was successfully created with validation status FINE and persistent id 1      |
+---------------------------------------------------------------------------------------------+
创建导入任务
+------------------------------------------------------------------------------------+
|sqoop:001> create job --xid 1 --type import                                         |
|Creating job for connection with id 1                                               |
|Please fill following values to create new job object                               |
|Name: First job                                                                     |
|                                                                                    |
|Database configuration                                                              |
|                                                                                    |
|Schema name: traceweb                                                               |
|Table name: trace_web_application                                                   |
|Table SQL statement:                                                                |
|Table column names:                                                                 |
|Partition column name:                                                              |
|Nulls in partition column:                                                          |
|Boundary query:                                                                     |
|                                                                                    |
|Output configuration                                                                |
|                                                                                    |
|Storage type:                                                                       |
|  0 : HDFS                                                                          |
|Choose: 0                                                                           |
|Output format:                                                                      |
|  0 : TEXT_FILE                                                                     |
|  1 : SEQUENCE_FILE                                                                 |
|Choose: 1                                                                           |
|Compression format:                                                                 |
|  0 : NONE                                                                          |
|  1 : DEFAULT                                                                       |
|  2 : DEFLATE                                                                       |
|  3 : GZIP                                                                          |
|  4 : BZIP2                                                                         |
|  5 : LZO                                                                           |
|  6 : LZ4                                                                           |
|  7 : SNAPPY                                                                        |
|Choose: 0                                                                           |
|Output directory: /opt/sqoop_output                                                 |
|                                                                                    |
|Throttling resources                                                                |
|                                                                                    |
|Extractors:                                                                         |
|Loaders:                                                                            |
|New job was successfully created with validation status FINE  and persistent id 1   |
+------------------------------------------------------------------------------------+
启动job：
+------------------------------------------------
|sqoop:000>  start job --jid 1  
+------------------------------------------------

查看导入状态：
+------------------------------------------------
|sqoop:000>  status job --jid 1  
|Submission details
|Job ID: 1
|Server URL: http://hadoopMaster:12000/sqoop/
|Created by: hadoop
|Creation date: 2014-05-23 18:51:05 CST
|Lastly updated by: hadoop
|External ID: job_local1566994033_0001
|	http://localhost:8080/
|2014-05-23 18:51:35 CST: UNKNOWN 
+------------------------------------------------

查看输出目录：
+--------------------------------------------------------------------+
hadoop@hadoopMaster:~$ l /opt/sqoop_output/
总用量 92
drwxrwxr-x 2 hadoop hadoop 4096  5月 23 18:52 .
drwxr-xr-x 8 hadoop hadoop 4096  5月 23 18:51 ..
-rw-r--r-- 1 hadoop hadoop  209  5月 23 18:51 part-m-00000.seq
-rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00000.seq.crc
-rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00001.seq
-rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00001.seq.crc
-rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00002.seq
-rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00002.seq.crc
-rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00003.seq
-rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00003.seq.crc
-rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00004.seq
-rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00004.seq.crc
-rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00005.seq
-rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00005.seq.crc
-rw-r--r-- 1 hadoop hadoop  207  5月 23 18:51 part-m-00006.seq
-rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00006.seq.crc
-rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00007.seq
-rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00007.seq.crc
-rw-r--r-- 1 hadoop hadoop  206  5月 23 18:51 part-m-00008.seq
-rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00008.seq.crc
-rw-r--r-- 1 hadoop hadoop  682  5月 23 18:51 part-m-00009.seq
-rw-rw-r-- 1 hadoop hadoop   16  5月 23 18:51 .part-m-00009.seq.crc
-rw-r--r-- 1 hadoop hadoop    0  5月 23 18:51 _SUCCESS
-rw-rw-r-- 1 hadoop hadoop    8  5月 23 18:51 ._SUCCESS.crc
+--------------------------------------------------------------------
sqoop:000> show job
+----+------------+--------+-----------+---------+
| Id |    Name    |  Type  | Connector | Enabled |
+----+------------+--------+-----------+---------+
| 1  | First job  | IMPORT | 1         | true    |
| 2  | importHDFS | IMPORT | 1         | true    |
+----+------------+--------+-----------+---------+
sqoop:000> delete job --jid 1
sqoop:000> show job          
+----+------------+--------+-----------+---------+
| Id |    Name    |  Type  | Connector | Enabled |
+----+------------+--------+-----------+---------+
| 2  | importHDFS | IMPORT | 1         | true    |
+----+------------+--------+-----------+---------+
sqoop:000> delete job --jid 2
sqoop:000> show job
+----+------+------+-----------+---------+
| Id | Name | Type | Connector | Enabled |
+----+------+------+-----------+---------+
+----+------+------+-----------+---------+

sqoop:000> show connection

批处理模式：
sqoop.sh client /opt/sqoop/script.sqoop
hadoop@hadoopMaster:$ vim  /opt/sqoop/script.sqoop
#指定服务器信息
set server --host hadoopMaster --port 12000 --webapp sqoop
#执行JOB
start job --jid 1 
+--------------------------------------------------------------------+
hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh client /opt/hadoop/mysql/batchModel.sqoop 
Sqoop home directory: /usr/local/sqoop
sqoop:000> set server --host hadoopMaster --port 12000 --webapp sqoop
Server is set successfully
sqoop:000> start job --jid 1
Submission details
Job ID: 1
Server URL: http://hadoopMaster:12000/sqoop/
Created by: hadoop
Creation date: 2014-05-30 10:55:10 CST
Lastly updated by: hadoop
External ID: job_local945860799_0003
	http://localhost:8080/
2014-05-30 10:55:10 CST: BOOTING  - Progress is not available
+--------------------------------------------------------------------+



https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Quickstart#Sqoop2Quickstart-Fullimportdemo

================================MYSQL=======================================
hadoop@hadoopMaster:~$ mysql -uroot -pjava
mysql> create database sqoop_stu;
Query OK, 1 row affected (0.03 sec)
mysql> use sqoop_stu;
Database changed
mysql> create table student(id int(3) auto_increment not null primary key, name char(10) not null, address varchar(50));
Query OK, 0 rows affected (0.41 sec)
mysql> insert into student values(1, 'Tom','beijing'),(2, 'Joan','shanghai'), (3, 'Wang', 'shenzheng');
Query OK, 3 rows affected (0.07 sec)
Records: 3  Duplicates: 0  Warnings: 0





CREATE TABLE `demo_blog` (`id` int(11) NOT NULL AUTO_INCREMENT,  `blog` varchar(100) NOT NULL,  PRIMARY KEY (`id`)) ENGINE=MyISAM  DEFAULT CHARSET=utf8;
CREATE TABLE `demo_log` (`operator` varchar(16) NOT NULL, `log` varchar(100) NOT NULL) ENGINE=MyISAM  DEFAULT CHARSET=utf8;






https://hbase.apache.org/book/configuration.html#hadoop
http://www.tuicool.com/articles/NVfEVnn

分享到：

ubuntu使用rsync | ubuntu下创建eclipse启动项

2014-06-04 13:34
浏览 16552
评论(10)
分类:开源软件
查看更多

10 楼 chowqh 2016-09-30

#修改指向我的hadoop安装目录
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/
应该修改为hadoop配置文件的路径，在楼主那里应该是/usr/local/hadoop/etc/hadoop才行，不然导入导出任务提交会失败

9 楼 chowqh 2016-09-30

#修改指向我的hadoop安装目录
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/

8 楼 wuzhongfei 2015-04-16

sqoop1.99.3以后是不是全部取消了sqoop命令,例如:
qoop create-hive-table --connect jdbc:oracle:thin:@172.17.1.188:1521:lhorcl --username sq --password sq --table buyy --hive-table buyy
sqoop import --connect jdbc:oracle:thin:@172.17.1.188:1521:lhorcl --username sq --password sq --table buyy --hive-import
我在sqoop启动之后,发现使用sqoop指令提示:
sqoop command not found

7 楼 cyj0421129 2015-04-15

sqoop:000> show version -all
client version:
Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
Exception has occurred during processing command
Exception: com.sun.jersey.api.client.UniformInterfaceException Message: GET http://10.0.252.10:12000/sqoop/version returned a response status of 404 Not Found

我hadoop集群的主机IP是10.0.252.10 ，问题出在哪？？我看catalina.out日志，是log4j的个报错。。。
求解决方案

6 楼 mypeterhero 2014-11-03

请问，我的服务端也起来了如下：
sqoop.sh server start

用客户端也连接成功了，
sqoop.sh client
set server --host cloudmaster --port 12000 --webapp sqoop

然后新建了链接mysql
新建了job

测试把库的表的数据导到hdfs中去

start job --jid 2
如下：
sqoop:000> start job --jid 2
Submission details
Job ID: 2
Server URL: http://cloudmaster:12000/sqoop/
Created by: hadoop
Creation date: 2014-11-03 10:42:51 CST
Lastly updated by: hadoop
External ID: job_1414828812005_0005
http://cloudmaster:8088/proxy/application_1414828812005_0005/
2014-11-03 10:42:51 CST: BOOTING - Progress is not available

我点击http://cloudmaster:8088.....的链接
查看到如下报错
Application application_1414828812005_0005 failed 2 times due to AM Container for appattempt_1414828812005_0005_000002 exited with exitCode: -1000 due to: No space available in any of the local directories.
.Failing this attempt.. Failing the application.

我是一个主节点，四个子节点

请问侯上校，是什么原因啊，

5 楼 haha_hyq 2014-09-23

这篇文章写的好！！！

4 楼 q547904614 2014-08-30

我在启动show version --all时，出现GET http://localhost:12000/sqoop/version returned a response status of 404 Not Found，这个问题怎么解决？

3 楼侯上校 2014-07-18

jie8895010 写道

你好
（1）我在启动show version --all时，出现GET http://localhost:12000/sqoop/version returned a response status of 404 Not Found，这个问题怎么解决？
（2）set server --host hadoopMaster --port 12000 --webapp sqoop里面的hadoopMaster这个主机是你新添加的，还是系统自带的？可以换成localhost吗？

hadoop集群的master的hostname

2 楼 jie8895010 2014-07-15

1 楼 jie8895010 2014-07-15

你好，我在启动show version --all时，出现
Exception:com.sun.jersey.api.client.ClientHandlerException Message: java.net.ConnectException: Connection timed out，请问是什么问题？

发表评论

您还没有登录,请您登录后再发表评论

最近访客更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论