hive+python 入门

自己写的一个简单例子，用来做话题描述去重，表中的desc字段 “a-b-a-b-b-c”需要去重
python代码如下：
#!/usr/bin/python
import sys
reload(sys)
sys.setdefaultencoding('utf8')
def quchong(desc):
    a=desc.split('-')
    return '-'.join(set(a))
while True:
        line = sys.stdin.readline()
        if line == "":
                break
        line = line.rstrip('\n')
        # your process code here
        parts = line.split('\t')
        parts[2]=quchong(parts[2])
        print "\t".join(parts)

下面是转载过来的，比较详细
二、hive map中字段自增的写法（转）
分类： hadoop 2010-09-03 16:01 99人阅读 评论(0) 收藏 举报

1、建立表结构

hive> CREATE TABLE t3 (foo STRING, bar MAP<STRING,INT>)
    > ROW FORMAT DELIMITED
    > FIELDS TERMINATED BY '/t'
    > COLLECTION ITEMS TERMINATED BY ','
    > MAP KEYS TERMINATED BY ':'
    > STORED AS TEXTFILE;
OK

 

2、建成的效果

hive> describe t3;
OK
foo     string
bar     map<string,int>

 

3、生成test.txt

jeffgeng        click:13,uid:15

 

4、把test.txt load进来

hive> LOAD DATA LOCAL INPATH 'test.txt' OVERWRITE INTO TABLE t3;
Copying data from file:/root/src/hadoop/hadoop-0.20.2/contrib/hive-0.5.0-bin/bin/test.txt
Loading data to table t3
OK

 

load完效果如下

hive> select * from t3;
OK
jeffgeng        {"click":13,"uid":15}

 

5、可以这样查map的值

hive> select bar['click'] from t3;

...一系列的mapreduce...

OK
13

 

6、编写add_mapper

#!/usr/bin/python
import sys
import datetime

for line in sys.stdin:
    line = line.strip()
    foo, bar = line.split('/t')
    d = eval(bar)
    d['click'] += 1
    print '/t'.join([foo, str(d)])

 

7、在hive中执行

hive> CREATE TABLE t4 (foo STRING, bar MAP<STRING,INT>)
    > ROW FORMAT DELIMITED
    > FIELDS TERMINATED BY '/t'
    > COLLECTION ITEMS TERMINATED BY ','
    > MAP KEYS TERMINATED BY ':'
    > STORED AS TEXTFILE;

 

hive> add FILE add_mapper.py

 

INSERT OVERWRITE TABLE t4
    > SELECT
    >   TRANSFORM (foo, bar)
    >   USING 'python add_mapper.py'
    >   AS (foo, bar)
    > FROM t3;
FAILED: Error in semantic analysis: line 1:23 Cannot insert into target table because column number/types are different t4: Cannot convert column 1 from string to map<string,int>.

 

8、为什么会报出以上错误？貌似add_mapper.py的输出是string格式的，hive无法此这种格式的map认出。后查明，AS后边可以为字段强制指定类型

INSERT OVERWRITE TABLE t4
SELECT
  TRANSFORM (foo, bar)
  USING 'python add_mapper.py'
  AS (foo string, bar map<string,int>)
FROM t3;

 

9、同时python脚本要去除字典转换后遗留下来的空格，引号，左右花排号等

#!/usr/bin/python
import sys
import datetime

for line in sys.stdin:
    line = line.strip('/t')
    foo, bar = line.split('/t')
    d = eval(bar)
    d['click'] += 1
    d['uid'] += 1
    strmap = ''
    for x in str(d):
        if x in (' ', "'"):
            continue
        strmap += x
    print '/t'.join([foo, strmap])

 

10、执行后的结果

hive> select * from t4;
OK
jeffgeng        {"click":14,"uid":null}
Time taken: 0.146 seconds

uid没有自增，原因待查

评论

4 楼 cldtc 2014-08-13  

请问这种情况下python脚本里面怎么打counter呢？通常的打法会直接进stderr不会被解析诶

3 楼 chinesejie 2014-07-24  

print '\t'.join([foo, strmap])+ " " //此处多个空格

然后uid就出来了

2 楼 dacoolbaby 2013-03-04  

dacoolbaby 写道

请问我执行Python的时候就报错
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.

我脚本是没有问题的，测试过的。
为什么还会这样呢？？

请问，您是在分布式环境下跑的吗？

1 楼 dacoolbaby 2013-02-27  

请问我执行Python的时候就报错
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.

我脚本是没有问题的，测试过的。
为什么还会这样呢？？