程序开过多线程，导致hadoop作业无法运行成功

Exception in thread "Thread-20" java.io.IOException: Call to /.....:8020 failed on local exception: java.io.IOException: Couldn't set up IO streams
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)
at org.apache.hadoop.ipc.Client.call(Client.java:1075)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy2.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:120)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:239)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:204)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:98)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:117)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:115)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
Caused by: java.io.IOException: Couldn't set up IO streams
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:619)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)
at org.apache.hadoop.ipc.Client.call(Client.java:1050)
... 18 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:612)
... 21 more

 

集群所有节点增加如下配置
/etc/security/limits.conf文件中增加
username soft nproc 100000 
username   hard nproc 100000  

重启整个集群每个节点，重启hadoop集群即可