`
dacoolbaby
  • 浏览: 1265888 次
  • 性别: Icon_minigender_1
  • 来自: 广州
社区版块
存档分类
最新评论

Hadoop下遇到OOM:unable to create new native thread

    博客分类:
  • Java
阅读更多

在hadoop下执行任务出现了这个异常:

Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread

         at java.lang.Thread.start0(Native Method)

         at java.lang.Thread.start(Thread.java:640)

         at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.kill(JvmManager.java:553)

         at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvmRunner(JvmManager.java:317)

         at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvm(JvmManager.java:297)

         at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.taskKilled(JvmManager.java:289)

         at org.apache.hadoop.mapred.JvmManager.taskKilled(JvmManager.java:158)

         at org.apache.hadoop.mapred.TaskRunner.kill(TaskRunner.java:801)

         at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3274)

         at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:3246)

         at org.apache.hadoop.mapred.TaskTracker.purgeTask(TaskTracker.java:2281)

         at org.apache.hadoop.mapred.TaskTracker.fatalError(TaskTracker.java:3666)

         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)

         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)

         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)

         at java.security.AccessController.doPrivileged(Native Method)

         at javax.security.auth.Subject.doAs(Subject.java:396)

         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)

         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) 

         at org.apache.hadoop.ipc.Client.call(Client.java:1107)

         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)

         at com.sun.proxy.$Proxy1.fatalError(Unknown Source)

         at org.apache.hadoop.mapred.Child.main(Child.java:310)

 

然后,为了了解这个异常,我开始了百度之旅:

也就是说在压力环境下拥有大量的线程,或者本地内存耗尽时,企图创建新的线程时抛出。

而系统能创建的线程数的计算公式如下: 
(MaxProcessMemory - JVMMemory - ReservedOsMemory) / (ThreadStackSize) = Number of threads 
MaxProcessMemory   指的是一个进程的最大内存
JVMMemory               JVM内存
ReservedOsMemory  保留的操作系统内存
ThreadStackSize        线程栈的大小

表象便是当JVM的heap size设置过大时,thread的创建数量便会减少。

因为当JVM申请分配heap size的内存的时候,会马上占有这些内存,因为这些内存是连续的。

所以会导致没有办法创建本地的线程。

 

JVM的堆和进程所需的栈都会消耗进程地址空间(address space),而地址空间则取决于操作系统。

而进程栈和一些库也会需要这个寻址空间。

 

有2个方法可以提高创建线程的数量:
1.减少JVM的heap size;
2.减少单个线程栈的大小,在JVM启动中使用-Xss参数。目前JDK1.4中每个线程栈所大小是256K,1.5是1M.

在Linux下

查了一下man ulimit,有下面解释
-a     All current limits are reported
-c     The maximum size of core files created
-d     The maximum size of a process data segment
-f     The maximum size of files created by the shell
-l     The maximum size that may be locked into memory
-m     The maximum resident set size (has no effect on Linux)
-n     The maximum number of open file descriptors (most systems do not allow this value to be set)
-p     The pipe size in 512-byte blocks (this may not be set)
-s     The maximum stack size
-t     The maximum amount of cpu time in seconds
-u     The maximum number of processes available to a single user
-v     The maximum amount of virtual memory available to the shell

其中:只有“max user processes”会对所有application能创建总的线程数有限制

 

refer to:

http://www.iteye.com/topic/654172

http://www.iteye.com/topic/503349

http://bbs.csdn.net/topics/350248404

 

 

 

分享到:
评论

相关推荐

    Hadoop 启动WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using

    搭建hadoop 环境时遇到启动异常告警问题 “WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable” 上来不多说,百度收集些相关...

    Unable to load native-hadoop library for your platform...

    Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 1. 下载文件native.rar 并解压 2. 用SecureFX 上传, 替换native目录下文件,选择二进制文件上传。 3. ...

    Hadoop from the beginning: The basics

    This book is written for anyone who needs to know how to analyze data using Hadoop. It is a good book for both Hadoop beginners and those in need of advancing their Hadoop skills. The author has ...

    win10-hadoop3.0.0或其他版本 Unable to load native-hadoop library 缺失文件

    hadoop 3.0.0 或者其他版本 报错Unable to load native-hadoop library时,放到c:/system32就可以

    hadoop2.8.1 native for macOS10

    WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items 解决方法是下载hadoop源代码重新编译, 但中间需要下载安装...

    启动hadoop出现ssh: connect to host master port 22: No route to host

    当启动hadoop或者使用ssh时出现: 输入ifconfig指令发现:自己之前配置的ip:192.168.79.103地址已经发生了变化 怎么解决办法呢? 如下步骤: 重新修改静态ip 输入vi /etc/udev/rules.d/70-persistent-net.rules...

    hadoop-native-64-2.6.0.zip

    Hadoop使用过程报错:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...解决 下载过后把tar包解压在hdoop的lib/native下即可

    hadoop 2.8.0 macOS native library

    安装hadoop启动之后总有警告:Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 原因: Apache提供的hadoop本地库是32位的,而在64位的服务器上就会有问题...

    Hadoop datanode启动失败:Hadoop安装目录权限的问题

    $ sudo chown -R hadoop:hadoop /opt/hadoop-0.2.203.0 ``` 这里`/opt/hadoop-0.2.203.0`是Hadoop的具体安装路径,应根据实际情况进行调整。 2. **重新启动Hadoop服务**:修改完所有权后,需要重新启动Hadoop...

    hadoop2.8.1 native for mac

    macOS下使用hadoop2.8.1时, 执行hadoop fs 命令(如:hadoop fs -ls /tmp/input)会提示: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platfo

    hadoop-2.6.0编译好的64bit的native库

    14/01/27 10:52:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [Java HotSpot(TM) 64-Bit ...

    hadoop-native-64-2.4.0.tar

    hadoop-native-64-2.4.0.tar 解决运行hadoop 警告:WARN util.NativeCodeLoader: Unable to load native-hadoop

    hadoop lib/native

    在安装hadoop时如果遇到WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform这个错误,请下载该文件放到hadoop目录下,Mac中放到libexec目录下。

    hadoop2.6.5在centos6.7下的native包

    解决WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable问题。

    hadoop-native-64-2.7.0.tar

    解决:Unable to load native-hadoop library for your platform 问题 原因: Apache提供的hadoop本地库是32位的,而在64位的服务器上就会有问题,因此需要自己编译64位的版本。 将准备好的64位的lib包解压到已经...

    在linux中编译64位的 hadoop 2.4.1native

    用法:将hadoop-2.4.1/lib/native下所有文件删除,然后将包中的文件放入 2.自己安装搜索“编译64位的hadoop 2.4.1”参考:...

    mac hadoop2.6.0 lib/native

    mac下Hadoop native library,用于解决报错:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable。再次说明,本版本只适用于mac ...

Global site tag (gtag.js) - Google Analytics