`
bupt04406
  • 浏览: 351541 次
  • 性别: Icon_minigender_1
  • 来自: 杭州
社区版块
存档分类
最新评论

hive strict模式

    博客分类:
  • Hive
阅读更多
set hive.mapred.mode=nonstrict;
set hive.mapred.mode=strict;

hive> set hive.mapred.mode;
hive.mapred.mode=nonstrict
hive> set hive.mapred.mode=strict;
hive> select key, value from src order by key,value;
FAILED: Error in semantic analysis: line 1:36 In strict mode, limit must be specified if ORDER BY is present value
hive>


HiveConf:
HIVEMAPREDMODE("hive.mapred.mode", "nonstrict"),

tianzhao@ubuntu:~/hive/trunk/hive-0.6.0/conf$ grep -r "hive.mapred.mode" ../conf/
../conf/hive-default.xml:  <name>hive.mapred.mode</name>

hive-default.xml:
<property>
  <name>hive.mapred.mode</name>
  <value>nonstrict</value>
  <description>The mode in which the hive operations are being performed. In strict mode, some risky queries are not allowed to run</description>
</property>


strict:
出现的地方:
(1)
  private Operator genJoinReduceSinkChild(QB qb, QBJoinTree joinTree,
      Operator child, String srcName, int pos) throws SemanticException {

    // Use only 1 reducer in case of cartesian product
    if (reduceKeys.size() == 0) {
      numReds = 1;

      // Cartesian product is not supported in strict mode
      if (conf.getVar(HiveConf.ConfVars.HIVEMAPREDMODE).equalsIgnoreCase(
          "strict")) {
        throw new SemanticException(ErrorMsg.NO_CARTESIAN_PRODUCT.getMsg());
      }
    }

}

hive> set hive.mapred.mode=strict;
hive> EXPLAIN SELECT subq.key, tab.value FROM src subq JOIN src tab  where subq.key < 200;
FAILED: Error in semantic analysis: In strict mode, cartesian product is not allowed. If you really want to perform the operation, set hive.mapred.mode=nonstrict



(2)
  private Operator genReduceSinkPlan(String dest, QB qb, Operator input,
      int numReducers) throws SemanticException {

    if (sortExprs == null) {
      sortExprs = qb.getParseInfo().getOrderByForClause(dest);
      if (sortExprs != null) {
        assert numReducers == 1;
        // in strict mode, in the presence of order by, limit must be specified
        Integer limit = qb.getParseInfo().getDestLimit(dest);
        if (conf.getVar(HiveConf.ConfVars.HIVEMAPREDMODE).equalsIgnoreCase(
            "strict")
            && limit == null) {
          throw new SemanticException(ErrorMsg.NO_LIMIT_WITH_ORDERBY
              .getMsg(sortExprs));
        }
      }
    }

}

(3)
  public static PrunedPartitionList prune(Table tab, ExprNodeDesc prunerExpr,
      HiveConf conf, String alias,
      Map<String, PrunedPartitionList> prunedPartitionsMap) throws HiveException {
    
          // If the "strict" mode is on, we have to provide partition pruner for
          // each table.
          if ("strict".equalsIgnoreCase(HiveConf.getVar(conf,
              HiveConf.ConfVars.HIVEMAPREDMODE))) {
            if (!hasColumnExpr(prunerExpr)) {
              throw new SemanticException(ErrorMsg.NO_PARTITION_PREDICATE
                  .getMsg("for Alias \"" + alias + "\" Table \""
                  + tab.getTableName() + "\""));
            }
          }

}

strict模式在下面三种情况下有限制:
(1) partition表需要加上分区裁剪
(2) order by 只有一个reduce,需要加上limit
(3) join时,如果只有一个reduce,笛卡尔积不支持。
分享到:
评论

相关推荐

    hive参数配置说明大全

    该参数决定了Map/Reduce模式,如果设置为strict,将不允许笛卡尔积。如果设置为nonstrict,则Hive将允许笛卡尔积,默认值为'nonstrict'。 14. hive.exec.parallel 该参数决定了是否开启map/reduce job的并发提交。...

    hive常见的优化方案ppt

    6. **启用MapReduce严格模式**:`hive.exec.mapreduce.strict.mode`开启后,Hive会拒绝一些可能导致性能下降或资源浪费的查询,如未指定分区的查询和无`LIMIT`的`ORDER BY`。 7. **单个Reducer处理多组聚合**:`...

    hive配置说明

    - **含义**:定义Map/Reduce的运行模式,如果设置为`strict`,则不允许笛卡尔积出现。 - **默认值**:`nonstrict` - **建议设置**:保持默认值,除非有特别的安全需求。 13. **hive.exec.parallel** - **含义**...

    hive 优化总结

    Hive 有两种模式:非严格模式(nonstrict)和严格模式(strict)。在非严格模式下,允许跨分区表的完整扫描。而在严格模式下,如果查询涉及完整的分区表,查询会失败。这有助于避免不必要的全表扫描,确保资源的...

    05--Hive的动态分区和分桶1

    2. `hive.exec.dynamic.partition.mode=nostrict`:默认为`strict`模式,要求至少有一个分区键是静态的。如果设置为`nostrict`,则允许所有分区键都是动态的。 在插入数据时,可以使用`insert overwrite table`语句...

Global site tag (gtag.js) - Google Analytics