`
lookqlp
  • 浏览: 346861 次
  • 性别: Icon_minigender_1
  • 来自: 上海
社区版块
存档分类
最新评论

Number of dynamic partitions RemoteException LeaseExpiredException

    博客分类:
  • hive
阅读更多
hive做大数据处理时,遇到如下异常:
[Fatal Error] Operator FS_2 (id=2): Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode.
......
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /tmp/hive-maintain/hive_2012-11-28_22-39-43_810_1689858262130334284/_task_tmp.-ext-10002/part=33436268/_tmp.000004_0 File does not exist. 
Holder DFSClient_attempt_201211250925_9859_m_000004_0 does not have any open files.


问题原因:
动态分区过大,需要设置hive.exec.max.dynamic.partitions、hive.exec.max.dynamic.partitions.pernode参数

问题解决:
SET hive.exec.max.dynamic.partitions=100000;
SET hive.exec.max.dynamic.partitions.pernode=100000;

总结:
hive分区数目不是无限制的,最大分区数可以通过参数设置。
分享到:
评论

相关推荐

    Toufik Mansour - Combinatorics of Set Partitions

    ### Toufik Mansour - Combinatorics of Set Partitions #### 概述 《组合数学中的集合划分》是一本由Toufik Mansour编写的书籍,专注于近十年来非常活跃的一个数学研究领域——集合划分的组合学。本书不仅为离散...

    kernel 2.6.35移植到s3c2440

    问题:List of all partitions: 1f00 256 mtdblock0 (driver?) 1f01 64 mtdblock1 (driver?) 1f02 2048 mtdblock2 (driver?) 1f03 63152 mtdblock3 (driver?) 1f04 65536 mtdblock4 (driver?) No filesystem ...

    platform-tools_r28.0.2-linux adb fastboot

    Adds support for dynamic partitions. Updated Windows requirements The platform tools now depend on the Windows Universal C Runtime, which is usually installed by default via Windows Update. If you see...

    platform-tools-r28.0.2-windows-adb-fastboot

    Adds support for dynamic partitions. Updated Windows requirements The platform tools now depend on the Windows Universal C Runtime, which is usually installed by default via Windows Update. If you see...

    reiizy-开源

    `dynamic_partitions_op_list`可能是一个操作列表,指示如何处理设备上的动态分区。这些操作可能包括创建、调整大小或移动分区,这对于维护和更新设备固件至关重要。 `boot.img`是引导加载程序映像,包含了启动设备...

    On convex partitions of polygonal regions

    本文基于一篇发表于1999年的论文《On convex partitions of polygonal regions》的内容,探讨了如何将具有多边形孔洞的多边形区域分割为最少数量的凸块的问题。 #### 研究问题概述 该研究主要关注的是如何将一个...

    Theory and applications of granular labelled partitions in multi-scale decision tables

    从给定文件信息中,我们可以提炼出以下知识点: 1. 知识粒化理论:文章提出了一个与信息粒化相关的新理论,该理论利用多尺度数据进行粒计算。信息粒化是将大量数据集分解为更小、更易处理的部分的过程,它在知识...

    Functional Mock-up Interface Specification 3.0

    introduction of Clocks to more exactly control timing of events and evaluation of model partitions across FMUs, introduction of more integer types and a 32-bit float type (see Section 2.4) to ...

    g4u-ghost for unix

    g4u ("ghost for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first...

    一键去除卡米限制工具----小米全机型.zip

    在当前智能手机时代,定制化和优化系统对于很多用户而言已经是家常便饭。对于小米手机用户而言,"卡米"限制可能是一个令人头疼的问题,因为它限制了用户进行系统修改或删除预装应用的能力。为了解决这一困扰,市面上...

    WinHex_14.2_SR-3_SC

    the master boot record), if you wish to search something on several partitions of a hard disk at the same time, or if a partition is damaged or formatted with a file system unknown to Windows, so ...

    poj 3782 Equal Sum Partitions.md

    poj 3782 Equal Sum Partitions.md

    partitions:partitions(n)返回第n个分区号,最大为Matlab最大的整数功能。-matlab开发

    partitions(n)返回第n个分区号,最大为Matlab最大的整数功能。 示例:partitions(0) 返回 1,partitions(200) 返回 3972999029388。 使用的算法归因于Euler,并在Mathologer的视频中描述了该算法(此代码的灵感...

    Codeforces Global Round 7 C. Permutation Partitions

    Permutation Partitions 题目链接-C. Permutation Partitions 题目大意 给定一个 1−n 上的全排列,将这个全排列分成不相交的 k 段,定义该划分的value为各段最大值的和,求该全排列所有可能划分中 value的最大...

    linux建立MTD分区

    - `parse_mtd_partitions()`函数支持多种分区解析器类型,如命令行指定分区(`cmdlinepart`)和RedBoot分区等。 - 如果未定义`CONFIG_MTD_REDBOOT_PARTS`宏,则仅支持命令行指定分区。 5. **添加分区**: - 根据...

    A refined relation between the numbers of different parts and 1's in integer partitions

    整数分拆中关于不同部分数和1的个数之间关系的一个加细形式,赵飞燕,,一个正整数$n$的分拆指的是把$n$表示成一些正整数之和,这些正整数之间不考虑次序。依据对不同部分数按照模$m$所得的余数进行分类,我�

    Map.Framework.A.Formal.Model.of.Maps

    Chapter 3 PLR Partitions: Extending Maps to Include Point and Line Features Chapter 4 Foundational Operations for Maps Chapter 5 Constructing Map Operations Using the Fundamental Map Operations ...

    Windows读写Ext2/Ext3/Ext4文件系统4

    There are a number of evident merits of the program, which you should know. First of all,DiskInternals Linux Reader is absolutely free. Secondly, the program provides for read-only access and does not...

    An-Efficient-and-Robust-Addressing-Protocol.rar_assignment

    Autonomous addressing protocols require a distributed and self-managed mechanism to avoid address collisions in a dynamic network with fading channels, frequent partitions, and joining/leaving nodes....

Global site tag (gtag.js) - Google Analytics