mina高并发短连接导致java.io.IOException: Too many open files解决方案

   这几天在解决一个用mina开发的高并发通信过程中产生的一个bug。

   模拟场景为：

   通过定时触发启动线程模拟高并发短连接测试，测试的服务端有2个，一个是服务有起，一个没起，客户端和服务端均在同一服务器上。执行一段时间后linux主机上通过lsof命令查看，发现有递增的文件句柄，pipe和eventpoll。

   抛出的异常如下：

  

2012-10-13 10:09:48 -org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:197)
 Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null
java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedConstructorAccessor110.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:180)
        at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:112)
        at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:93)
        at org.apache.mina.transport.socket.nio.NioSocketConnector.<init>(NioSocketConnector.java:56)
        at com.develop.webplatform.funnel.client.JobClient.sendMessage(JobClient.java:39)
        at com.develop.webplatform.funnel.client.JobClient.sendJob(JobClient.java:126)
        at com.develop.webplatform.funnel.extend.JobExecRemotelyBySocket.execJobByTask(JobExecRemotelyBySocket.java:66)
        at com.develop.webplatform.funnel.JobManager.execJobByTask(JobManager.java:27)
        at com.develop.webplatform.quartz.job.TaskJob.executeInternal(TaskJob.java:38)
        at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:223)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: org.apache.mina.core.RuntimeIoException: Failed to open a selector.
        at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:61)
        ... 15 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.initPipe(Native Method)
        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
        at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
        at java.nio.channels.Selector.open(Selector.java:209)
        at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:59)
        ... 15 more

    

    原代码中，关于客户端连接的代码如下：

   

final NioSocketConnector connector = new NioSocketConnector();
		final String[] result = new String[1];
		connector.getFilterChain().addLast("codec",
				new ProtocolCodecFilter(new ObjectSerializationCodecFactory()));
		connector.setHandler(handler);

		//设置超时
		connector.setConnectTimeoutMillis(defaultConnectTimeOut);
		ConnectFuture connectFuture = connector.connect(address);
		connectFuture.awaitUninterruptibly();	//同步，等待，直到连接完成
		if (connectFuture.isDone()) {
			if (!connectFuture.isConnected()) {	//若在指定时间内没连接成功，则抛出异常
				logger.info("fail to connect " + logInfo);

				throw new Exception();
			}
		}

       

         经过分析，导致主机文件句柄泄露的原因为，客户端发起服务端连接时，会请求系统分配相关的文件句柄，在原代码中，仅仅判断是否连接成功，而未对连接失败进行资源释放，从而造成文件句柄泄露。当总的文件句柄数超过系统设置值（ulimit -n 查看同一个进程允许的最大文件句柄数），则抛出异常“java.io.IOException: Too many open files",导致无法创建新的连接，服务器挂掉。

      更改后的代码如下：

  

final NioSocketConnector connector = new NioSocketConnector();
		final String[] result = new String[1];
		connector.getFilterChain().addLast("codec",
				new ProtocolCodecFilter(new ObjectSerializationCodecFactory()));
		connector.setHandler(handler);

		//设置超时
		connector.setConnectTimeoutMillis(defaultConnectTimeOut);
		ConnectFuture connectFuture = connector.connect(address);
		connectFuture.awaitUninterruptibly();	//同步，等待，直到连接完成
		if (connectFuture.isDone()) {
			if (!connectFuture.isConnected()) {	//若在指定时间内没连接成功，则抛出异常
				logger.info("fail to connect " + logInfo);
				connector.dispose();	//不关闭的话会运行一段时间后抛出，too many open files异常，导致无法连接

				throw new Exception();
			}
		}

 

评论

6 楼 freezingsky 2012-10-16  

感觉没说到点子上！

5 楼 snake1987 2012-10-16  

我也遇到过，并发量不高的时候有时候就直接无视了

其实有个简单的解决方法：加一个心跳


记录一下每个连接的最后一次请求的时间，超时了就处理一下，强制退出或者是怎么滴

4 楼 xmdxzyf 2012-10-16  

clovenstone 写道

login as root, check ulimit -a open file size.

by default it is usually 1024, change it to 8026 should resolve this issue. I would put this into the profile for that particular account which is running your application server session.

Remember to grant it with root access before you are able to change the open file size.

Kindly regards

Thanks for your advice.
But it can't solve the issue.Because the exception is caused in my code with error.Even though i change the open file size,the server will be broken down with increasing file size after moments.
I hava solved the issue with the means described in the blog

3 楼 CshBBrain 2012-10-16  

linux 下把一个连接 处理成一个文件句柄，高并发的服务器运行环境下肯定要把服务器允许同时打开的最大文件数量这个参数 根据需要重新配置。当然任何一台服务器都有一个性能极限，当客户端请求连接时，服务器当时已有的连接数量 达到了最大允许的连接数时，简单粗暴的处理方式 就是 拒绝连接，当然你也可以让后面的连接请求排队，当有服务器有连接释放时，在进行连接处理。

2 楼 clovenstone 2012-10-15  

login as root, check ulimit -a open file size.

by default it is usually 1024, change it to 8026 should resolve this issue. I would put this into the profile for that particular account which is running your application server session.

Remember to grant it with root access before you are able to change the open file size.

Kindly regards

1 楼 richie144 2012-10-15  

这个问题我以前在项目中也遇到过，不是资源释放的问题，释放资源了高并发也会出现这个问题，关键是控制好连接池的空闲数，才能解决根本。。。