`
bupt04406
  • 浏览: 347702 次
  • 性别: Icon_minigender_1
  • 来自: 杭州
社区版块
存档分类
最新评论

Server端处理时间很长,Client发生SocketTimeoutException

 
阅读更多

Client端只有一个put请求,往server端写数据,server端处理时间过长,导致client端SocketTimeoutException

 

0.94版本

 

Client端发生异常SocketTimeoutException

 

12/11/20 19:03:18 WARN client.HConnectionManager$HConnectionImplementation: Failed all from region=myLittleHBaseTable,,1353401128907.2879da4a2d609943473a2421520732cb., hostname=ubuntu, port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Call to ubuntu/0:0:0:0:0:0:0:1:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:42426 remote=ubuntu/0:0:0:0:0:0:0:1:60020]
     at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
     at java.util.concurrent.FutureTask.get(FutureTask.java:83)
     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1553)
     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1376)
     at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:937)
     at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:793)
     at org.apache.hadoop.hbase.client.HTable.put(HTable.java:768)
     at updateTest.put(updateTest.java:61)
     at updateTest.main(updateTest.java:96)
Caused by: java.net.SocketTimeoutException: Call to ubuntu/0:0:0:0:0:0:0:1:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:42426 remote=ubuntu/0:0:0:0:0:0:0:1:60020]
     at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:960)
     at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:933)
     at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:147)
     at $Proxy4.multi(Unknown Source)
     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1353)
     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1351)
     at org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:210)
     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1360)
     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1348)
     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:42426 remote=ubuntu/0:0:0:0:0:0:0:1:60020]
     at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
     at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
     at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
     at java.io.FilterInputStream.read(FilterInputStream.java:116)
     at java.io.FilterInputStream.read(FilterInputStream.java:116)
     at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:318)
     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
     at java.io.DataInputStream.readInt(DataInputStream.java:370)
     at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:578)
     at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:512)

 

 

 

HBaseClient的Connection一直在等待response,然后过了60000ms后,接收到SocketTimeoutException异常

 

 

    @Override
    public void run() {
      if (LOG .isDebugEnabled())
        LOG.debug(getName() + ": starting, having connections "
            + connections.size());

      try {
        while (waitForWork()) {//wait here for work - read or close connection
          receiveResponse();
        }
      } catch (Throwable t) {
        LOG.warn("Unexpected exception receiving call responses" , t);
        markClosed( new IOException("Unexpected exception receiving call responses" , t));
      }

      close();

      if (LOG .isDebugEnabled())
        LOG.debug(getName() + ": stopped, remaining connections "
            + connections.size());
    }

 

我们可以看到在receiveResponse里面不会close掉这个connection,会把call从calls队列里面remove掉,所以导致calls队列为空,但是接着会执行waitForWork 

 

 

    /* wait till someone signals us to start reading RPC response or
     * it is idle too long, it is marked as to be closed,
     * or the client is marked as not running.
     *
     * Return true if it is time to read a response; false otherwise.
     */
    @SuppressWarnings({ "ThrowableInstanceNeverThrown" })
    protected synchronized boolean waitForWork () {
      if (calls .isEmpty() && !shouldCloseConnection.get()  && running.get())  {
        long timeout = maxIdleTime -
              (System. currentTimeMillis()-lastActivity.get());
        if (timeout>0) {
          try {
            wait(timeout);
          } catch (InterruptedException ignored) {}
        }
      }

      if (!calls .isEmpty() && !shouldCloseConnection.get() && running.get()) {
        return true ;
      } else if (shouldCloseConnection.get()) {
        return false ;
      } else if (calls.isEmpty()) { // idle connection closed or stopped
        long timeout = maxIdleTime
            - (System. currentTimeMillis() - lastActivity.get());
        LOG.info("timeout " + timeout);
        markClosed( null);
        return false ;
      } else { // get stopped but there are still pending requests
        markClosed((IOException) new IOException().initCause(
            new InterruptedException()));
        return false ;
      }
    }

waitForWork 里面会计算timeout, ( long timeout = maxIdleTime -   (System. currentTimeMillis()-lastActivity.get()); )这里的timeout得出来的结果为 负值,因为上次lastActivity的时间是在读取请求前运行的,中间一直等待server端的请求超过60000ms才超时接收到SocketTimeoutException异常才往下执行的,所以这里的 (System. currentTimeMillis()-lastActivity.get())是超过60000ms的,导致waitForWork会执行markClosed,close掉connection。
      } else if (calls.isEmpty()) { // idle connection closed or stopped
        markClosed(null);
        return false;
      }



    /* Receive a response.
     * Because only one receiver, so no synchronization on in.
     */
    protected void receiveResponse() {
      if (shouldCloseConnection.get()) {
        return;
      }
      touch();

      try {
        // See HBaseServer.Call.setResponse for where we write out the response.
        // It writes the call.id (int), a flag byte, then optionally the length
        // of the response (int) followed by data.

        // Read the call id.
        int id = in.readInt();

    }

    /** Update lastActivity with the current time. */
    protected void touch() {
      lastActivity.set(System.currentTimeMillis());
    }

client端关闭connection后,server端会抛出异常:
2012-11-20 20:54:46,825 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":60370,"call":"multi(#size=1#myLittleHBaseTable,,1353401128907.2879da4a2d609943473a2421520732cb.{\"totalColumns\":1,\"families\":{\"myLittleFamily\":[{\"timestamp\":1353473626453,\"qualifier\":\"someQualifier\",\"vlen\":14}]},\"row\":\"myLittleRow1010001\"}#), rpc version=1, client version=29, methodsFingerPrint=-56040613","client":"0:0:0:0:0:0:0:1:42779","starttimems":1353473626452,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2012-11-20 20:54:47,256 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60020: readAndProcess threw exception java.io.IOException: Connection reset by peer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
        at sun.nio.ch.IOUtil.read(IOUtil.java:175)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
        at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:2119)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1394)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:956)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:745)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:720)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)



2012-11-20 20:55:48,532 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":60252,"call":"multi(#size=1#myLittleHBaseTable,,1353401128907.2879da4a2d609943473a2421520732cb.{\"totalColumns\":1,\"families\":{\"myLittleFamily\":[{\"timestamp\":1353473688277,\"qualifier\":\"someQualifier\",\"vlen\":14}]},\"row\":\"myLittleRow1010001\"}#), rpc version=1, client version=29, methodsFingerPrint=-56040613","client":"0:0:0:0:0:0:0:1:42785","starttimems":1353473688277,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2012-11-20 20:55:48,548 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call multi(#size=1#myLittleHBaseTable,,1353401128907.2879da4a2d609943473a2421520732cb.{"totalColumns":1,"families":{"myLittleFamily":[{"timestamp":1353473688277,"qualifier":"someQualifier","vlen":14}]},"row":"myLittleRow1010001"}#), rpc version=1, client version=29, methodsFingerPrint=-56040613 from 0:0:0:0:0:0:0:1:42785: output error
2012-11-20 20:55:48,549 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020 caught a ClosedChannelException, this means that the server was processing a request but the client went away. The error message was: null


所以Server端的异常: Connection reset by peer,以及ClosedChannelException可能是因为server端处理时间过长,导致client端close掉connection,server端处理结束后再往client发送response时而出现。
还有的原因就是kill掉了client,server端处理结束后再往client发送response时而出现。


 

分享到:
评论

相关推荐

    java.net.SocketTimeoutException: Receive timed out

    2. **如何设置和处理SocketTimeoutException**:介绍如何使用`setSoTimeout()`方法设置超时时间,以及当异常发生时如何捕获和处理。 3. **EJB与Socket通信**:可能详细解释了在EJB中如何使用Socket进行网络通信,...

    AndroidStudio版登陆界面加直连Sqlserver

    综上所述,"AndroidStudio版登陆界面加直连Sqlserver"项目涵盖了Android UI设计、网络编程、数据库操作和异常处理等多方面知识,对于初学者来说,这是一个很好的实践项目,可以帮助理解Android应用与远程数据库交互...

    webhdfs-java-client-master

    WebHDFS客户端通常会处理如SocketTimeoutException、NetworkException等常见错误,并提供适当的恢复策略。 6. **测试与调试**: "webhdfs-java-client-master"项目中的测试用例可以帮助理解各种操作的正确用法,并...

    基于Java的Socket服务器,简单实现

    在提供的`SocketServerDemo`文件中,很可能包含了实现上述功能的示例代码。通过分析这个示例,你可以更深入地理解如何用Java构建一个简单的`Socket`服务器,以及如何处理客户端连接和超时。 总的来说,Java的`...

    clientSocketToDemo:演示Socket clientServer以了解套接字超时如何工作

    在"clientSocketToDemo"项目中,有两个主要部分:客户端(Client)和服务器端(Server)。客户端负责发起连接请求,而服务器端则监听特定的端口号,等待连接的到来。 1. **服务器端**:使用`ServerSocket`类创建一...

    Android超时处理

    在处理超时时,我们应捕获并处理`SocketTimeoutException`。这可能意味着网络延迟过高,或者服务器没有在规定时间内给出响应。在这种情况下,可以选择重新发起请求,或者向用户展示网络不稳定的通知。 在"TimeOut...

    java socket长连接中解决read阻塞的3个办法

    如果在指定的时间内没有接收到数据,`read()`方法会抛出一个`SocketTimeoutException`,从而强制退出阻塞状态。示例代码如下: ```java Socket socket = new Socket(host, port); socket.setSoTimeout(100); // ...

    动态设置网络延迟时间

    如果服务器没有响应,或者网络状况不佳,超过这个时间,OkHttp会抛出一个`SocketTimeoutException`。 读取超时(Read Timeout)是在连接建立后,等待服务器返回数据的时间。如果在这段时间内服务器没有发送任何数据...

    基于Scoket的CS模式聊天室

    在IT领域,Socket编程是网络通信的核心技术之一,主要用于实现客户端(Client)和服务器(Server)之间的双向通信。本文将详细解析"基于Socket的CS模式聊天室"的相关知识点,包括Socket的基本概念、CS模式的含义以及...

    java Socket编程基础代码

    Socket编程在Java中主要用于实现客户端(Client)与服务器端(Server)的通信,是Internet应用程序的基础。以下是对"java Socket编程基础代码"的详细解释。 首先,我们需要了解Socket的基本概念。Socket可以被看作...

    android HttpPost请求连接服务器端的实例

    然后,使用HttpClient发送请求,并捕获可能的异常,例如SocketTimeoutException,用于处理请求超时: ```java HttpClient httpClient = new DefaultHttpClient(); try { HttpResponse response = httpClient....

    Android端TCP通讯实现

    2. 错误处理:正确处理各种网络异常,如SocketTimeoutException、IOException等。 3. 数据编码与解码:确保发送和接收的数据格式一致,避免数据解析错误。 4. 连接管理:合理控制连接的创建与销毁,避免资源浪费。 ...

    java.net.SocketTimeoutException(解决方案).md

    常见的问题

    hbase常见错误整理3年运维经验整理

    在高并发写入情况下,客户端与NameNode之间的通信频繁,如果NameNode处理能力不足,则容易导致客户端等待时间过长,从而触发超时异常。 **解决方法** 针对此类问题,可以从以下几个方面进行优化: 1. **增加...

    java socket编程 详细的介绍,实例的教学,网站形式,不过特实用

    Socket编程在Java中主要用于实现客户端(Client)与服务器端(Server)的应用交互,例如文件传输、聊天应用、在线游戏等。本教程将提供一个详细的Java Socket编程介绍,结合实例教学,帮助开发者掌握其核心概念和...

    Java网络编程(聊天器)

    网络编程中,必须妥善处理各种可能的异常,如SocketTimeoutException、IOException等。良好的异常处理能确保程序的健壮性。 10. **关闭资源** 当通信结束后,记得关闭Socket、InputStream、OutputStream以及...

    Android Socket编程源码(与PC通讯)

    例如,服务器端可能会因为没有客户端连接而抛出`SocketTimeoutException`,客户端可能会因为网络问题无法连接到服务器。 - 为了确保程序的健壮性,需要对可能出现的异常进行捕获和处理。 4. **关闭连接** - 通信...

    异常处理问题

    如果超过这个时间,读取或写入操作将抛出SocketTimeoutException。 例如,以下代码设置了30秒的超时限制: ```java Socket socket = new Socket(); socket.connect(new InetSocketAddress("localhost", 8080), ...

Global site tag (gtag.js) - Google Analytics