问题:
[zkkafka@yanfabu2-37 ~]$ sqoop import \ > --connect jdbc:mysql://10.156.50.36:3306/mqh \ > --username root \ > --password root \ > --table device \ > --fields-terminated-by '\t' \ > --delete-target-dir \ > --num-mappers 1 \ > --hive-import \ > --hive-database test \ > --hive-table hive_bbs_product_snappy; Warning: /home/zkkafka/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/zkkafka/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 19/05/29 12:00:27 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 19/05/29 12:00:27 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 19/05/29 12:00:27 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 19/05/29 12:00:27 INFO tool.CodeGenTool: Beginning code generation Wed May 29 12:00:27 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 19/05/29 12:00:27 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `device` AS t LIMIT 1 19/05/29 12:00:27 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `device` AS t LIMIT 1 19/05/29 12:00:27 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/zkkafka/hadoop 注: /tmp/sqoop-zkkafka/compile/0edae8bed32dad40d1fe8ca424e30387/device.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 19/05/29 12:00:31 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-zkkafka/compile/0edae8bed32dad40d1fe8ca424e30387/device.jar SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/zkkafka/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/zkkafka/hbase/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 19/05/29 12:00:32 INFO tool.ImportTool: Destination directory device deleted. 19/05/29 12:00:32 WARN manager.MySQLManager: It looks like you are importing from mysql. 19/05/29 12:00:32 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 19/05/29 12:00:32 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 19/05/29 12:00:32 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 19/05/29 12:00:32 INFO mapreduce.ImportJobBase: Beginning import of device 19/05/29 12:00:32 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 19/05/29 12:00:32 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps Wed May 29 12:00:39 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 19/05/29 12:00:39 INFO db.DBInputFormat: Using read commited transaction isolation 19/05/29 12:00:40 INFO mapreduce.JobSubmitter: number of splits:1 19/05/29 12:00:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1558676658010_0008 19/05/29 12:00:41 INFO impl.YarnClientImpl: Submitted application application_1558676658010_0008 19/05/29 12:00:41 INFO mapreduce.Job: The url to track the job: http://master1:8088/proxy/application_1558676658010_0008/ 19/05/29 12:00:41 INFO mapreduce.Job: Running job: job_1558676658010_0008 19/05/29 12:00:49 INFO mapreduce.Job: Job job_1558676658010_0008 running in uber mode : false 19/05/29 12:00:49 INFO mapreduce.Job: map 0% reduce 0% 19/05/29 12:00:57 INFO mapreduce.Job: map 100% reduce 0% 19/05/29 12:00:59 INFO mapreduce.Job: Job job_1558676658010_0008 completed successfully 19/05/29 12:01:00 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=128603 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=87 HDFS: Number of bytes written=156 HDFS: Number of read operations=4 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=5465 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=5465 Total vcore-milliseconds taken by all map tasks=5465 Total megabyte-milliseconds taken by all map tasks=5596160 Map-Reduce Framework Map input records=6 Map output records=6 Input split bytes=87 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=103 CPU time spent (ms)=1380 Physical memory (bytes) snapshot=168935424 Virtual memory (bytes) snapshot=2104311808 Total committed heap usage (bytes)=93323264 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=156 19/05/29 12:01:00 INFO mapreduce.ImportJobBase: Transferred 156 bytes in 27.7324 seconds (5.6252 bytes/sec) 19/05/29 12:01:00 INFO mapreduce.ImportJobBase: Retrieved 6 records. 19/05/29 12:01:00 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table device Wed May 29 12:01:00 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 19/05/29 12:01:00 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `device` AS t LIMIT 1 19/05/29 12:01:00 WARN hive.TableDefWriter: Column register_time had to be cast to a less precise type in Hive 19/05/29 12:01:00 INFO hive.HiveImport: Loading uploaded data into Hive 19/05/29 12:01:00 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly. 19/05/29 12:01:00 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50) at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392) at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44) ... 12 more
解决方法:
cp /home/zkkafka/hive/lib/hive-exec-**.jar /home/zkkafka/sqoop/lib/
捐助开发者
在兴趣的驱动下,写一个免费
的东西,有欣喜,也还有汗水,希望你喜欢我的作品,同时也能支持一下。 当然,有钱捧个钱场(支持支付宝和微信 以及扣扣群),没钱捧个人场,谢谢各位。
个人主页:http://knight-black-bob.iteye.com/
谢谢您的赞助,我会做的更好!