Hadoop源码分析-RPC.Server

 

内部类	作用
Call	存储客户端发来的请求
Listener	监听类: 监听客户端发来的请求，内部静态类Listener.Reader: 当监听器监听到用户请求，便让Reader读取用户请求
Responder	响应RPC请求类，请求处理完毕，由Responder发送给请求客户端
Connection	连接类，真正的客户端请求读取逻辑在这个类中
Handler	请求处理类，会循环阻塞读取callQueue中的call对象，并对其进行操作

 

Server是个抽象类, 唯一抽象的方法. Server提供了一个架子, Server的具体功能, 需要具体类来完成。而具体类, 当然就是实现call方法。

/** Called for each call. */

public abstract Writable call(Class<?> protocol, Writable param, long receiveTime) throws IOException; 

 

Server.Call 和Client.Call类似, Server.Call包含了一次请求, 其中id和param的含义和Client.Call是一致的。不同点: 

connection是该Call来自的连接, 当请求处理结束时, 相应的结果会通过相同的connection, 发送给客户端。

timestamp是请求到达的时间戳, 如果请求很长时间没被处理, 对应的连接会被关闭, 客户端也就知道出错了。

response是请求处理的结果, 可能是一个Writable的串行化结果, 也可能一个异常的串行化结果。

 

Server.Connection 维护了一个来自客户端的socket连接。

它处理版本校验, 读取请求并把请求发送到请求处理线程, 接收处理结果并把结果发送给客户端。

Hadoop的Server采用了Java的NIO, 这样的话就不需要为每一个socket连接建立一个线程, 读取socket上的数据。

在Server中, 只需要一个线程, 就可以accept新的连接请求和读取socket上的数据, 这个线程, 就是Listener。

 

Server.Handler 请求处理线程一般有多个. 

Handler的run方法循环地取出一个Server.Call, 调用Server.call方法, 搜集结果并串行化, 然后将结果放入Responder队列中。

对于处理完的请求, 需要将结果写回去, 同样, 利用NIO, 只需要一个线程, 相关的逻辑在Responder里。

 

Call

  /** A call queued for handling. */
  private static class Call {
    private int id;   // the client's call id 客户端的RPC调用对象Call的id
    private Writable param;  // the parameter passed 客户端的PRC调用对象Call的参数
    private Connection connection;// connection to client 客户端连接对象,在服务端持有这个对象, 就能知道是哪个客户端连接
    private long timestamp;  // the time received when response is null; the time served when response is not null
    private ByteBuffer response; // the response for this call 当前RPC调用的响应

    public Call(int id, Writable param, Connection connection) { 
      this.id = id;
      this.param = param;
      this.connection = connection;
      this.timestamp = System.currentTimeMillis();
      this.response = null;
    }
    public void setResponse(ByteBuffer response) {
      this.response = response;
    }
  }

Server initialize and start

ipc.Server是抽象类, 抽象类不能实例化, 那么系统启动的时候, 实例化的是ipc.Server抽象类的实现类, 即ipc.RPC.Server. 即启动RPC.Server 

public class NameNode implements ClientProtocol, DatanodeProtocol, NamenodeProtocol, FSConstants,
      RefreshAuthorizationPolicyProtocol, RefreshUserMappingsProtocol {
  /** RPC server */
  private Server server;
  /** RPC server for HDFS Services communication.
  * BackupNode, Datanodes and all other services should be connecting to this server if it is configured. Clients should only go to NameNode#server */
  private Server serviceRpcServer;
  /** RPC server address */
  private InetSocketAddress serverAddress = null;
  /** RPC server for DN address */
  protected InetSocketAddress serviceRPCAddress = null;

  /** Initialize name-node. NameNode初始化 */
  private void initialize(Configuration conf) throws IOException {
    InetSocketAddress socAddr = NameNode.getAddress(conf);
    
    // create rpc server
    InetSocketAddress dnSocketAddr = getServiceRpcServerAddress(conf);
    if (dnSocketAddr != null) {
      this.serviceRpcServer = RPC.getServer(this, dnSocketAddr.getHostName(), dnSocketAddr.getPort(), 
          serviceHandlerCount, false, conf, namesystem.getDelegationTokenSecretManager());
      this.serviceRPCAddress = this.serviceRpcServer.getListenerAddress();
      setRpcServiceServerAddress(conf);
　　}
　　
    this.server = RPC.getServer(this, socAddr.getHostName(), socAddr.getPort(), handlerCount, false, conf, 
        namesystem.getDelegationTokenSecretManager());
    // The rpc-server port can be ephemeral... ensure we have the correct info
    this.serverAddress = this.server.getListenerAddress(); 
　　    
    startHttpServer(conf);
    this.server.start();  //start RPC server   
    if (serviceRpcServer != null) {
      serviceRpcServer.start();      
    }
    startTrashEmptier(conf);
  }
}

RPC.newServer()

  /** Construct a server for a protocol implementation instance listening on a port and address, with a secret manager. 
  * 构造协议实现类的服务器, 该方法的调用者NameNode实现了一系列的协议接口.
  * 以读取HDFS文件为例, 客户端发送一个RPC调用getBlockLocations()到NameNode服务端, 由NameNode进行实际的方法调用
  */
  public static Server getServer(final Object instance, final String bindAddress, final int port, final int numHandlers,
       final boolean verbose, Configuration conf, SecretManager<? extends TokenIdentifier> secretManager) {
    return new Server(instance, conf, bindAddress, port, numHandlers, verbose, secretManager);
  }

  /** An RPC Server. */
  public static class Server extends org.apache.hadoop.ipc.Server {
    /** Construct an RPC server.
     * @param instance the instance whose methods will be called 被调用的方法的实例对象
     * @param conf the configuration to use
     * @param bindAddress the address to bind on to listen for connection
     * @param port the port to listen for connections on
     * @param numHandlers the number of method handler threads to run
     * @param verbose whether each call should be logged
     */
    public Server(Object instance, Configuration conf, String bindAddress,  int port, int numHandlers, boolean verbose, 
        SecretManager<? extends TokenIdentifier> secretManager)  {
      super(bindAddress, port, Invocation.class, numHandlers, conf, 
          classNameBase(instance.getClass().getName()), secretManager); // 调用父类ipc.Server抽象类的构造方法
      this.instance = instance;
      this.verbose = verbose;
　　}
  }

ipc.Server构造方法

/** An abstract IPC service.  IPC calls take a single Writable as a parameter, and return a Writable as their value.
 * A service runs on a port and is defined by a parameter class and a value class.
 */
public abstract class Server {
  
  private String bindAddress; 
  private int port; // port we listen on
  private int handlerCount; // number of handler threads
  private int readThreads; // number of read threads
  private Class<? extends Writable> paramClass; // class of call parameters
  private int maxIdleTime;  // the maximum idle time after which a client may be disconnected
  private int thresholdIdleConnections; // the number of idle connections after which we will start cleaning up idle connections
  int maxConnectionsToNuke; // the max number of  connections to nuke during a cleanup
  protected RpcInstrumentation rpcMetrics;
  private Configuration conf;
  private SecretManager<TokenIdentifier> secretManager;
  private int maxQueueSize;
  private final int maxRespSize;
  private int socketSendBufferSize;
  private final boolean tcpNoDelay; // if T then disable Nagle's Algorithm

  volatile private boolean running = true;// true while server runs
  private BlockingQueue<Call> callQueue;// queued calls
  private List<Connection> connectionList = Collections.synchronizedList(new LinkedList<Connection>()); //maintain a list of client connections
  private Listener listener = null;//服务端监听器
  private Responder responder = null;//服务端写回客户端的响应
  private int numConnections = 0;//连接的客户端个数
  private Handler[] handlers = null;//处理类
　　
  /** Constructs a server listening on the named port and address.  
   * Parameters passed must be of the named class.  
   * The handlerCount determines the number of handler threads that will be used to process calls. */
  protected Server(String bindAddress, int port,  Class<? extends Writable> paramClass, int handlerCount, 
      Configuration conf, String serverName, SecretManager<? extends TokenIdentifier> secretManager) {
    this.bindAddress = bindAddress;
    this.conf = conf;
    this.port = port;
    this.paramClass = paramClass;
    this.handlerCount = handlerCount;
    this.socketSendBufferSize = 0;
    this.maxQueueSize = handlerCount * conf.getInt(IPC_SERVER_HANDLER_QUEUE_SIZE_KEY, IPC_SERVER_HANDLER_QUEUE_SIZE_DEFAULT);
    this.maxRespSize = conf.getInt(IPC_SERVER_RPC_MAX_RESPONSE_SIZE_KEY, IPC_SERVER_RPC_MAX_RESPONSE_SIZE_DEFAULT);
    this.readThreads = conf.getInt(IPC_SERVER_RPC_READ_THREADS_KEY, IPC_SERVER_RPC_READ_THREADS_DEFAULT);
    this.callQueue  = new LinkedBlockingQueue<Call>(maxQueueSize); 
    this.maxIdleTime = 2*conf.getInt("ipc.client.connection.maxidletime", 1000);
    this.maxConnectionsToNuke = conf.getInt("ipc.client.kill.max", 10);
    this.thresholdIdleConnections = conf.getInt("ipc.client.idlethreshold", 4000);
    this.secretManager = (SecretManager<TokenIdentifier>) secretManager;
    this.authorize = conf.getBoolean(HADOOP_SECURITY_AUTHORIZATION, false);
    this.isSecurityEnabled = UserGroupInformation.isSecurityEnabled();
    
    // Start the listener here and let it bind to the port
    listener = new Listener();
    this.port = listener.getAddress().getPort();    
    this.rpcMetrics = RpcInstrumentation.create(serverName, this.port);
    this.tcpNoDelay = conf.getBoolean("ipc.server.tcpnodelay", false);
    responder = new Responder(); // Create the responder here
    if (isSecurityEnabled) {
      SaslRpcServer.init(conf);
    }
  }

  private void closeConnection(Connection connection) {
    synchronized (connectionList) {
      if (connectionList.remove(connection))
        numConnections--;
    }
    try {
      connection.close();
    } catch (IOException e) {
    }
  }

  // NameNode在获得Server后, 会调用server.start()启动服务端. 三个对象responder,listener,handlers都是线程类, 都调用start()
  /** Starts the service.  Must be called before any calls will be handled. */
  public synchronized void start() {
    responder.start();
    listener.start();
    handlers = new Handler[handlerCount];
    for (int i = 0; i < handlerCount; i++) {
      handlers[i] = new Handler(i);
      handlers[i].start();
    }
  }
}

RPC.Server.Listener

Client端的底层通信直接采用了阻塞式IO编程，Server端采用Listener监听客户端的连接

  private static final ThreadLocal<Server> SERVER = new ThreadLocal<Server>();

  /** Returns the server instance called under or null.  May be called under #call(Writable, long)implementations, 
   * and under Writable methods of paramters and return values.  Permits applications to access the server context.*/
  public static Server get() {
    return SERVER.get();
  }

  //maintain a list of client connections 维护客户端的连接列表, 这里的Connection是Server.Connection
  private List<Connection> connectionList = Collections.synchronizedList(new LinkedList<Connection>());

  /** Listens on the socket. Creates jobs for the handler threads 监听客户端Socket连接, 为handler线程创建任务 */
  private class Listener extends Thread {
    private ServerSocketChannel acceptChannel = null; 	//the accept channel 服务端通道
    private Selector selector = null; //the selector that we use for the server 选择器(NIO)
    private Reader[] readers = null;
    private int currentReader = 0;
    private InetSocketAddress address; //the address we bind at 服务端地址
    private Random rand = new Random();
    private long lastCleanupRunTime = 0; //the last time when a cleanup connection (for idle connections) ran
    private long cleanupInterval = 10000; //the minimum interval between two cleanup runs
    private int backlogLength = conf.getInt("ipc.server.listen.queue.size", 128);
    private ExecutorService readPool; //读取池, 任务执行服务(并发)
   
    public Listener() throws IOException {
      address = new InetSocketAddress(bindAddress, port);
      acceptChannel = ServerSocketChannel.open(); // Create a new server socket 创建服务端Socket连接,
      acceptChannel.configureBlocking(false);  // and set to non blocking mode设置为非阻塞模式

      bind(acceptChannel.socket(), address, backlogLength); // Bind the server socket to the local host and port将ServerSocket绑定到本地端口
      port = acceptChannel.socket().getLocalPort(); // Could be an ephemeral port
      selector= Selector.open();  // create a selector 创建一个监听器的Selector
      readers = new Reader[readThreads];//读取线程数组
      readPool = Executors.newFixedThreadPool(readThreads);//启动多个reader线程,为了防止请求多时服务端响应延时的问题 
      for (int i = 0; i < readThreads; i++) {
        Selector readSelector = Selector.open();//每个读取线程都创建一个Selector
        Reader reader = new Reader(readSelector);
        readers[i] = reader;
        readPool.execute(reader);
      }

      acceptChannel.register(selector, SelectionKey.OP_ACCEPT); // ①Register accepts on the server socket with the selector. 注册连接事件 
      this.setName("IPC Server listener on " + port);
      this.setDaemon(true);
　　} 
　　
　　// 在启动Listener线程时listener.start(), 服务端会一直等待客户端的连接
    public void run() {
      SERVER.set(Server.this); //使用ThreadLocal本地线程,设置当前Server为当前的ThreadLocal对象
      while (running) {
        SelectionKey key = null;
        try {
          selector.select();
          Iterator<SelectionKey> iter = selector.selectedKeys().iterator();
          while (iter.hasNext()) {
            key = iter.next();
            iter.remove();
            if (key.isValid()) {
              if (key.isAcceptable())
                doAccept(key);		// ②建立连接,服务端接受客户端连接
            }
            key = null;
          }
        } catch (Exception e) {
          closeCurrentConnection(key, e);
        }
        cleanupConnections(false);
      }
      // 监听器不再监听客户端的连接,关闭通道和选择器和所有的连接对象
      synchronized (this) {
        acceptChannel.close();
        selector.close();
        selector= null;
        acceptChannel= null;
        while (!connectionList.isEmpty()) { // clean up all connections
          closeConnection(connectionList.remove(0));
        }
      }
　　}
　　
    void doAccept(SelectionKey key) throws IOException, OutOfMemoryError { //②
      Connection c = null;
      ServerSocketChannel server = (ServerSocketChannel) key.channel();
      SocketChannel channel;
      while ((channel = server.accept()) != null) {//建立连接server.accept()
        channel.configureBlocking(false);
        channel.socket().setTcpNoDelay(tcpNoDelay);
        Reader reader = getReader(); //从readers池中获得一个reader 
        try {
          reader.startAdd();//激活readSelector，设置adding为true 
          SelectionKey readKey = reader.registerChannel(channel); //③将读事件设置成兴趣事件 
          c = new Connection(readKey, channel, System.currentTimeMillis());//创建一个连接对象 
          readKey.attach(c);//将connection对象注入readKey 
          synchronized (connectionList) {
            connectionList.add(numConnections, c);
            numConnections++;
          }        
        } finally {
          reader.finishAdd();  //设置adding为false,采用notify()唤醒一个reader, 初始化Listener时启动的每个reader都使用了wait()方法等待
        } // 当reader被唤醒, reader会执行doRead()
      }
    }

    void doRead(SelectionKey key) throws InterruptedException {  //④
      int count = 0;
      Connection c = (Connection)key.attachment();
      if (c == null) {
        return;  
      }
      c.setLastContact(System.currentTimeMillis());
      
      try {
        count = c.readAndProcess();
      } catch (InterruptedException ieo) {
        throw ieo;
      } catch (Exception e) {
        count = -1; //so that the (count < 0) block is executed
      }
      if (count < 0) {
        closeConnection(c);
        c = null;
      } else {
        c.setLastContact(System.currentTimeMillis());
      }
    }   

    synchronized void doStop() {
      if (selector != null) {
        selector.wakeup();
        Thread.yield();
      }
      if (acceptChannel != null) {
        try {
          acceptChannel.socket().close();
        } catch (IOException e) {
          LOG.info(getName() + ":Exception in closing listener socket. " + e);
        }
      }
      readPool.shutdown();
    }

    // The method that will return the next reader to work with
    // Simplistic implementation of round robin for now
    Reader getReader() {
      currentReader = (currentReader + 1) % readers.length;
      return readers[currentReader];
　　}
  }

Listener.Reader

    private class Reader implements Runnable {
      private volatile boolean adding = false; //读取线程是否正在添加中,如果是,等待一秒钟
      private Selector readSelector = null; //读取线程的Selector选择器

      Reader(Selector readSelector) {
        this.readSelector = readSelector;
      }
      public void run() {
        synchronized (this) {
          while (running) {
            SelectionKey key = null;
            readSelector.select();
            while (adding) {
              this.wait(1000);
            }              

            Iterator<SelectionKey> iter = readSelector.selectedKeys().iterator();
            while (iter.hasNext()) {
              key = iter.next();
              iter.remove();
              if (key.isValid()) {
                if (key.isReadable()) {
                  doRead(key);   //④
                }
              }
              key = null;
            }
          }
        }
      }

      /**
       * This gets reader into the state that waits for the new channel to be registered with readSelector.
       * If it was waiting in select() the thread will be woken up, otherwise whenever select() is called
       * it will return even if there is nothing to read and wait  in while(adding) for finishAdd call
       */
      public void startAdd() {
        adding = true;
        readSelector.wakeup();
      }
      public synchronized SelectionKey registerChannel(SocketChannel channel) {
          return channel.register(readSelector, SelectionKey.OP_READ);  //③
      }
      public synchronized void finishAdd() {
        adding = false;
        this.notify();        
      }
    }

 

NIO通信流程

①初始化服务器时,创建监听器, 在监听器的构造方法里会创建ServerSocketChannel, 选择器, 以及多个读取线程. 并在服务端通道上注册OP_ACCEPT操作

acceptChannel.register(selector, SelectionKey.OP_ACCEPT); 

 

启动服务器会调用listener.start(), listener是个线程类, 会调用run(). 

监听器会一直监听客户端的请求, 通过监听器的Selector选择器进行轮询是否有感兴趣的事件发生(服务器感兴趣的是上面注册的接受连接事件)

 

②当客户端连接服务端, 被Selector捕获到该事件, 因为在ServerSocketChannle对OP_ACCEPT操作感兴趣,所以服务端接受了客户端的连接请求.

doAccept(SelectionKey key)

 

客户端连接到服务器, 服务端接受连接, 监听器会从读取线程池中选择一个读取线程, 委托给读取线程处理, 而不是监听器自己来处理.

 

建立SocketChannel连接, 注意不是ServerSocketChannel. (ServerSocketChannle在整个通信过程中只建立一次即服务端启动的时候)

SocketChannel channel = server.accept();

 

③往建立的SocketChannel通道注册感兴趣的OP_READ操作. 此时接收读取事件的选择器不再是监听器的, 而是读取线程的选择器

SelectionKey readKey = reader.registerChannel(channel);

channel.register(readSelector, SelectionKey.OP_READ);  //往readSelector注册感兴趣的OP_READ操作.读取线程的选择器负责轮询监听客户端的数据写入

 

同时根据(readKey和SocketChannel,当前时间)建立一个Connection注入到readKey中.

这个Connection对象是客户端和服务器的连接对象, 客户端和服务器建立连接后, 在后续的客户端写入数据过程也应该使用同一个Connection

Connection c = new Connection(readKey, channel, System.currentTimeMillis());

readKey.attach(c); //在通信过程中如果想要保存某个对象,附加在selectionKey中

 

注册读操作后,服务端的监听器的读取线程就能读取客户端传入的数据

 

④客户端开始向服务端写入数据, 读取线程Reader的选择器捕获到客户端的写入事件, 

因为读取线程注册了感兴趣的OP_READ操作,所以能够读取客户端的写入数据.

doRead(SelectionKey key)

 

读取事件的操作会根据selectionKey获得Connection, 这个Connection对象正是客户端和服务器建立连接时注入到readKey中的Connection对象

具体的读取客户端的数据的操作就在该Connection的readAndProcess方法里

connection.readAndProcess();

 

Listener.Reader的线程模型

Listener的doAccept()接受连接过程

从readers池中获得一个reader线程 

reader.startAdd(); 激活readSelector，设置adding为true --> 读线程监听客户端的数据写入,如果adding=true,表示Reader正在添加,再等待一秒钟

  将读事件设置成兴趣事件

  创建一个连接对象   

reader.finishAdd(); 设置adding为false,采用notify()唤醒一个reader, 初始化Listener时启动的每个reader都使用了wait()方法等待

将要设置读事件为兴趣事件包装在设置Reader的adding属性以及使用notify()两者之间. 是为了确保读取线程发生在设置读事件为感兴趣事件之后.

 

基于NIO的事件模型采用选择器来轮询感兴趣的事件.只要有感兴趣的操作, 选择器就会捕获进行处理. 如果没有感兴趣的事件发生则没有操作.

所以服务端接受客户端的连接和读取客户端的数据这两个操作过程发生的时刻完全是随机的. 

也就是说监听客户端连接的选择器和多个读取客户端数据的读取线程的选择器捕获事件也都是随机的.

 

但是读取客户端的数据必须保证发生在客户端连接服务器之后. 

因为如果客户端没有连接服务器, 也就不会注册读取事件OP_READ到读取线程上. 因为注册OP_READ发生在在doAccept()客户端连接服务器操作中.

 

初始化Listener时启动的每个Reader, 都会新建对应的选择器. Reader的默认字段adding=false

Selector readSelector = Selector.open();  //每个读取线程都创建一个Selector

Reader reader = new Reader(readSelector);

初始化时尽管adding=false在run()中不会执行this.wait(1000)的等待操作, 但是因为还没有客户端连接注册OP_READ事件所以选择器不会捕获该事件.

 

客户端连接服务器,服务器接受连接,在doAccept()中, 注册OP_READ到读取线程的感兴趣事件

1. 之前:  设置adding=true并激活读取线程的选择器, 注意此时读取线程的选择器进行轮询操作是不会捕获到读取事件的,因为还没注册OP_READ事件

   所以读取线程的run()如果判断adding=true, 就知道选择器关注的SocketChannel上的OP_READ事件还没注册好,需要每隔一秒钟再判断

2. 往建立的SocketChannel注册好OP_READ事件 

3. 之后: 设置adding=false并通知Reader读取线程不需要再等待下去, run()方法判断adding=false, 选择器开始轮询等待客户端的写入

 

Listener和Reader的选择器

Listener的选择器只有一个, Listener有多个Reader, 每个Reader都有自己的选择器. 

Listener的选择器来监听客户端的连接, 当监听到有一个客户端连接服务器, 就会选取一个Reader, 并往Reader的选择器注册读取操作.

这样具体的读取操作就交给了Reader进行处理. 因为Reader有多个, 所以如果有多个客户端连接并写入数据给服务器, 就可以开多个Reader同时读取.