Hadoop下遇到OOM：unable to create new native thread

在hadoop下执行任务出现了这个异常：

Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread

         at java.lang.Thread.start0(Native Method)

         at java.lang.Thread.start(Thread.java:640)

         at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.kill(JvmManager.java:553)

         at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvmRunner(JvmManager.java:317)

         at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvm(JvmManager.java:297)

         at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.taskKilled(JvmManager.java:289)

         at org.apache.hadoop.mapred.JvmManager.taskKilled(JvmManager.java:158)

         at org.apache.hadoop.mapred.TaskRunner.kill(TaskRunner.java:801)

         at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3274)

         at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:3246)

         at org.apache.hadoop.mapred.TaskTracker.purgeTask(TaskTracker.java:2281)

         at org.apache.hadoop.mapred.TaskTracker.fatalError(TaskTracker.java:3666)

         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)

         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)

         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)

         at java.security.AccessController.doPrivileged(Native Method)

         at javax.security.auth.Subject.doAs(Subject.java:396)

         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)

         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) 

         at org.apache.hadoop.ipc.Client.call(Client.java:1107)

         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)

         at com.sun.proxy.$Proxy1.fatalError(Unknown Source)

         at org.apache.hadoop.mapred.Child.main(Child.java:310)

 

然后，为了了解这个异常，我开始了百度之旅：

也就是说在压力环境下拥有大量的线程，或者本地内存耗尽时，企图创建新的线程时抛出。

而系统能创建的线程数的计算公式如下： 
(MaxProcessMemory - JVMMemory - ReservedOsMemory) / (ThreadStackSize) = Number of threads 
MaxProcessMemory   指的是一个进程的最大内存
JVMMemory               JVM内存
ReservedOsMemory  保留的操作系统内存
ThreadStackSize        线程栈的大小

表象便是当JVM的heap size设置过大时，thread的创建数量便会减少。

因为当JVM申请分配heap size的内存的时候，会马上占有这些内存，因为这些内存是连续的。

所以会导致没有办法创建本地的线程。

 

JVM的堆和进程所需的栈都会消耗进程地址空间(address space)，而地址空间则取决于操作系统。

而进程栈和一些库也会需要这个寻址空间。

 

有2个方法可以提高创建线程的数量：
1.减少JVM的heap size；
2.减少单个线程栈的大小，在JVM启动中使用-Xss参数。目前JDK1.4中每个线程栈所大小是256K,1.5是1M.

在Linux下

查了一下man ulimit，有下面解释
-a     All current limits are reported
-c     The maximum size of core files created
-d     The maximum size of a process data segment
-f     The maximum size of files created by the shell
-l     The maximum size that may be locked into memory
-m     The maximum resident set size (has no effect on Linux)
-n     The maximum number of open file descriptors (most systems do not allow this value to be set)
-p     The pipe size in 512-byte blocks (this may not be set)
-s     The maximum stack size
-t     The maximum amount of cpu time in seconds
-u     The maximum number of processes available to a single user
-v     The maximum amount of virtual memory available to the shell

其中：只有“max user processes”会对所有application能创建总的线程数有限制

 

refer to：

http://www.iteye.com/topic/654172

http://www.iteye.com/topic/503349

http://bbs.csdn.net/topics/350248404