线上tomcat 网络无法访问问题排查

线上某台机器tomcat启动无法访问,通过jstack报错。

Unable to open socket file: target process not responding or HotSpot VM not loaded

因为这个错误提示误导我以为JDK的问题，升级JDK到1.7问题依旧。

 

这个时候查看日志发现：

INFO: validateJarFile(/home/admin/tlog-proxy/target/tlog-proxy.war/WEB-INF/lib/jsp-api-2.1-glassfish-2.1.v20091210.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class

2015-12-03 02:43:53,332 org.apache.catalina.loader.WebappClassLoader validateJarFile

INFO: validateJarFile(/home/admin/tlog-proxy/target/tlog-proxy.war/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class

2015-12-03 02:44:03,987 org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deployment of web application directory /home/admin/tlog-proxy/.default/deploy/tlog-proxy.war has finished in 10,908 ms

2015-12-03 02:44:03,990 org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["http-bio-7001"]

2015-12-03 02:44:04,051 org.apache.catalina.startup.Catalina start

INFO: Server startup in 11069 ms

2015-12-03 02:44:09,258 org.apache.tomcat.util.net.JIoEndpoint$Acceptor run

SEVERE: Socket accept failed

java.net.SocketException: 打开的文件过多

        at java.net.PlainSocketImpl.socketAccept(Native Method)

        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)

        at java.net.ServerSocket.implAccept(ServerSocket.java:530)

        at java.net.ServerSocket.accept(ServerSocket.java:498)

        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)

        at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:220)

        at java.lang.Thread.run(Thread.java:745)

 

lsof | grep java | wc -l发现超过1万，正常应该只有几百。然后查看发现是抓取任务太多导致。

 

事后排查抓取任务才几百个，为啥会创建出上万个连接呢，最后排查发现HttpClient没有调用 close

 

public void close() {

if (client != null) {

client.getConnectionManager().closeIdleConnections(0, TimeUnit.SECONDS);

}

}