hive dynamic partitions insert java.lang.OutOfMemoryError: Java heap space

动态分区问题，如果数据量大或者当动态分区大甚至只有十几个时，会出现如下异常：

2015-10-23 16:43:54,165 INFO [fetcher#10] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: assigned 20 of 34 to spark-03:13562 to fetcher#10
2015-10-23 16:43:54,166 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hive (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#9
2015-10-23 16:43:54,167 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#9
 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
 at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.OutOfMemoryError: Java heap space
 at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
 at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
 at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
 at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:304)
 at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:294)
 at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511)
 at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329)
 at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)

 

参考issue：

https://issues.apache.org/jira/browse/MAPREDUCE-6108
https://issues.apache.org/jira/browse/MAPREDUCE-6447

源码：https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java#L254

默认参数：https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

mem参考http://stackoverflow.com/questions/24070557/what-is-the-relation-between-mapreduce-map-memory-mb-and-mapred-map-child-jav

参数理解：

mapreduce.map.java.opts   -xmx配置的  heap memory  cloudera mapreduce.map.java.opts.max.heap 一般设置java.opts为memory.mb的75%
mapreduce.reduce.java.opts  -xmx配置的  heap memory  cloudera mapreduce.reduce.java.opts.max.heap 一般设置java.opts为memory.mb的75%
mapreduce.map.memory.mb  1G默认
mapreduce.reduce.memory.mb 1G默认
mapreduce.reduce.memory.totalbytes

mapreduce.reduce.shuffle.parallelcopies  shuffle开启的fetcher线程数 apache默认5，choudera默认10

mapreduce.reduce.shuffle.input.buffer.percent 默认0.7

mapreduce.reduce.shuffle.memory.limit.percent默认0.25

 如上3个参数相乘得小于1，否则将报如上错。

将mapreduce.reduce.shuffle.parallelcopies调成5，可以解决此问题。

 

另外cloudera hive hive.stats.autogather默认为true，即插入数据时会优化统计，如此在大的动态分区时load数据后会有一段很长时间的统计，且操作hive元数据表，例如每个分区的文件数，行数等等。耗时比较长时可能会timeout，需要将其设成false。

详细查看 https://cwiki.apache.org/confluence/display/Hive/StatsDev。