hcatalog读取hive数据并写入hive

参考
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_table_access_mapreduce.html
https://github.com/cloudera/hcatalog-examples.git
命令：
for jarfile in `ls /logdata/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib/.`; do
       export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:/logdata/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib/$jarfile"
done 
hadoop jar bigdata-mapreduce-0.0.1-SNAPSHOT.jar com.yeahmobi.bigdata.mapreduce.GroupByAge age age_group
异常
14/11/05 10:56:44 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

14/11/05 10:56:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

14/11/05 10:56:44 INFO metastore.ObjectStore: ObjectStore, initialize called

14/11/05 10:56:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored

14/11/05 10:56:47 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"

14/11/05 10:56:47 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "".

14/11/05 10:56:48 INFO metastore.ObjectStore: Initialized ObjectStore

14/11/05 10:56:48 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.12.0

14/11/05 10:56:49 INFO metastore.HiveMetaStore: 0: get_databases: NonExistentDatabaseUsedForHealthCheck

14/11/05 10:56:49 INFO HiveMetaStore.audit: ugi=root    ip=unknown-ip-addr      cmd=get_databases: NonExistentDatabaseUsedForHealthCheck

14/11/05 10:56:49 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=groups

14/11/05 10:56:49 INFO HiveMetaStore.audit: ugi=root    ip=unknown-ip-addr      cmd=get_table : db=default tbl=groups

14/11/05 10:56:49 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

14/11/05 10:56:49 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

Exception in thread "main" java.io.IOException: NoSuchObjectException(message:default.groups table not found)

        at org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:88)

        at org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:55)

        at org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:47)

        at com.cloudera.test.UseHCat.run(UseHCat.java:81)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

        at com.cloudera.test.UseHCat.main(UseHCat.java:108)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Caused by: NoSuchObjectException(message:default.groups table not found)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1377)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)

        at com.sun.proxy.$Proxy12.get_table(Unknown Source)

        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:898)

        at org.apache.hcatalog.common.HCatUtil.getTable(HCatUtil.java:194)

        at org.apache.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)

        at org.apache.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)

        at org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:86)

        ... 11 more

14/11/05 10:56:51 INFO metastore.HiveMetaStore: 1: Shutting down the object store...

14/11/05 10:56:51 INFO HiveMetaStore.audit: ugi=root    ip=unknown-ip-addr      cmd=Shutting down the object store...

14/11/05 10:56:51 INFO metastore.HiveMetaStore: 1: Metastore shutdown complete.

14/11/05 10:56:51 INFO HiveMetaStore.audit: ugi=root    ip=unknown-ip-addr      cmd=Metastore shutdown complete.

 

找不到表，将hive client配置xml加入source里，解决。

2014-11-25 06:59:33,953 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hcatalog.mapreduce.HCatOutputFormat not found
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hcatalog.mapreduce.HCatOutputFormat not found
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:467)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:368)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1477)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hcatalog.mapreduce.HCatOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2047)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:232)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:463)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hcatalog.mapreduce.HCatOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1953)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2045)
... 10 more
hive相关类找不到，mr是通过ToolRunner执行，使用-libjars参数增加hive相关包，解决。
hadoop jar bigdata-mapreduce-0.0.1-SNAPSHOT.jar com.yeahmobi.bigdata.mapreduce.GroupByAge -libjars /logdata/CDH-5.2.0-1.cdh5.2.0.p0.36/jars/hive-hcatalog-core-0.13.1-cdh5.2.0.jar,/logdata/CDH-5.2.0-1.cdh5.2.0.p0.36/jars/hive-exec-0.13.1-cdh5.2.0.jar,/logdata/CDH-5.2.0-1.cdh5.2.0.p0.36/jars/hive-metastore-0.13.1-cdh5.2.0.jar age age_group

如果字段比较多，感觉java代码停麻烦的，不如直接load。

 

评论

2 楼 lookqlp 2015-05-29  

super_a 写道

你好！找不到表这个问题是如何解决的，可以描述清楚点吗？我刚也遇到此问题，但是没找到原因，通过hive命令行是能找到并且能查询到数据的

把xml配置文件放在你工程的classpath里。

1 楼 super_a 2015-05-27  

你好！找不到表这个问题是如何解决的，可以描述清楚点吗？我刚也遇到此问题，但是没找到原因，通过hive命令行是能找到并且能查询到数据的