spark2.0源码阅读剖析spark RPC框架

文章目录

与老版本区别

在Spark 0.x.x与Spark 1.x.x版本中，组件间的消息通信主要借助于Akka。但是Akka在Spark 2.0.0版本中被移除了

在Spark 1.x.x版本中，用户文件与Jar包的上传采用了由Jetty实现的HttpFileServer，但在Spark 2.0.0版本中也被废弃了，现在使用的是基于Spark内置RPC框架的NettyStreamManager。

RPC框架架构

在这里插入图片描述

TransportContext内部包含传输上下文的配置信息TransportConf和对客户端请求消息进行处理的RpcHandler。

TransportConf在创建TransportClientFactory和TransportServer时都是必须的

TransportClientFactory是RPC客户端的工厂类。

TransportServer是RPC服务端的实现。

组件详细功能说明

TransportContext：传输上下文，包含了用于创建传输服务端（TransportServer）和传输客户端工厂（TransportClientFactory）的上下文信息，并支持使用TransportChannelHandler设置Netty提供的SocketChannel的Pipeline的实现。
TransportConf：传输上下文的配置信息。
RpcHandler：对调用传输客户端（TransportClient）的sendRPC方法发送的消息进行处理的程序。
MessageEncoder：在将消息放入管道前，先对消息内容进行编码，防止管道另一端读取时丢包和解析错误。
MessageDecoder：对从管道中读取的ByteBuf进行解析，防止丢包和解析错误；
TransportFrameDecoder：对从管道中读取的ByteBuf按照数据帧进行解析；
RpcResponseCallback：RpcHandler对请求的消息处理完毕后，进行回调的接口。
TransportClientFactory：创建传输客户端（TransportClient）的传输客户端工厂类。
ClientPool：在两个对等节点间维护的关于传输客户端（TransportClient）的池子。ClientPool是TransportClientFactory的内部组件。
TransportClient：RPC框架的客户端，用于获取预先协商好的流中的连续块。TransportClient旨在允许有效传输大量数据，这些数据将被拆分成几百KB到几MB的块。当TransportClient处理从流中获取的获取的块时，实际的设置是在传输层之外完成的。sendRPC方法能够在客户端和服务端的同一水平线的通信进行这些设置。
TransportClientBootstrap：当服务端响应客户端连接时在客户端执行一次的引导程序。
TransportRequestHandler：用于处理客户端的请求并在写完块数据后返回的处理程序。
TransportResponseHandler：用于处理服务端的响应，并且对发出请求的客户端进行响应的处理程序。
TransportChannelHandler：代理由TransportRequestHandler处理的请求和由TransportResponseHandler处理的响应，并加入传输层的处理。
TransportServerBootstrap：当客户端连接到服务端时在服务端执行一次的引导程序。
TransportServer：RPC框架的服务端，提供高效的、低级别的流服务。

TransportContext

先上UML图

在这里插入图片描述

可用看到成员变量比较重要的TransportConf和rpcHandler都在里面，此外MessageEncoder，MessageDecoder也能找到。

然后比较重要的方法是createServer和createClientFactory创建服务端和客户端

public class TransportContext {
  private static final Logger logger = LoggerFactory.getLogger(TransportContext.class);

  private final TransportConf conf;
  private final RpcHandler rpcHandler;
  private final boolean closeIdleConnections;
  private static final MessageEncoder ENCODER = MessageEncoder.INSTANCE;
  private static final MessageDecoder DECODER = MessageDecoder.INSTANCE;

TransportConf

public class TransportConf {

  private final ConfigProvider conf;

  private final String module;

真实的配置由ConfigProvider提供，ConfigProvider是一个抽象类

public abstract class ConfigProvider {

  public abstract String get(String name);

  public abstract Iterable<Map.Entry<String, String>> getAll();

  public String get(String name, String defaultValue) {
    try {
      return get(name);
    } catch (NoSuchElementException e) {
      return defaultValue;
    }
  }

在这里插入图片描述

Spark通常使用SparkTransportConf创建TransportConf

object SparkTransportConf {
  private val MAX_DEFAULT_NETTY_THREADS = 8
  def fromSparkConf(_conf: SparkConf, module: String, numUsableCores: Int = 0): TransportConf = {
    val conf = _conf.clone
    val numThreads = defaultNumThreads(numUsableCores)
    conf.setIfMissing(s"spark.$module.io.serverThreads", numThreads.toString)
    conf.setIfMissing(s"spark.$module.io.clientThreads", numThreads.toString)

    new TransportConf(module, new ConfigProvider {
      override def get(name: String): String = conf.get(name)
      override def get(name: String, defaultValue: String): String = conf.get(name, defaultValue)
      override def getAll(): java.lang.Iterable[java.util.Map.Entry[String, String]] = {
        conf.getAll.toMap.asJava.entrySet()
      }
    })
  }
  private def defaultNumThreads(numUsableCores: Int): Int = {
      //如果numUsableCores小于等于0，那么线程数是系统可用处理器的数量，不过系统的内核数不可能全部用于网络传输使用，所以这里还将分配给网络传输的内核数量最多限制在8个
    val availableCores =
      if (numUsableCores > 0) numUsableCores else Runtime.getRuntime.availableProcessors()
    math.min(availableCores, MAX_DEFAULT_NETTY_THREADS)
  }
}

可用看到其实也是调用了，之前说到底sparkConf的clone方法，拷贝spark属性,并且实现了ConfigProvider的几个抽象方法。

TransportClientFactory

public TransportClientFactory createClientFactory(List<TransportClientBootstrap> bootstraps) {
  return new TransportClientFactory(this, bootstraps);
}

public TransportClientFactory createClientFactory() {
  return createClientFactory(new ArrayList<>());
}

TransportContext创建TransportClientFactory的方法

private final TransportContext context;
private final TransportConf conf;
private final List<TransportClientBootstrap> clientBootstraps;
//一个socketAddress对应一个ClientPool里面包含多个TransportClient
private final ConcurrentHashMap<SocketAddress, ClientPool> connectionPool;

/** Random number generator for picking connections between peers. */
private final Random rand;
private final int numConnectionsPerPeer;

private final Class<? extends Channel> socketChannelClass;
private EventLoopGroup workerGroup;
private PooledByteBufAllocator pooledAllocator;
private final NettyMemoryMetrics metrics;

public TransportClientFactory(
    TransportContext context,
    List<TransportClientBootstrap> clientBootstraps) {
  this.context = Preconditions.checkNotNull(context);
  this.conf = context.getConf();
  this.clientBootstraps = Lists.newArrayList(Preconditions.checkNotNull(clientBootstraps));
  this.connectionPool = new ConcurrentHashMap<>();
  this.numConnectionsPerPeer = conf.numConnectionsPerPeer();
  this.rand = new Random();

  IOMode ioMode = IOMode.valueOf(conf.ioMode());
  this.socketChannelClass = NettyUtils.getClientChannelClass(ioMode);
  this.workerGroup = NettyUtils.createEventLoop(
      ioMode,
      conf.clientThreads(),
      conf.getModuleName() + "-client");
  this.pooledAllocator = NettyUtils.createPooledByteBufAllocator(
    conf.preferDirectBufs(), false, conf.clientThreads());
  this.metrics = new NettyMemoryMetrics(
    this.pooledAllocator, conf.getModuleName() + "-client", conf);
}

TransportClientFactory的构造方法

TransportClientBootstrap

public interface TransportClientBootstrap {
  void doBootstrap(TransportClient client, Channel channel) throws RuntimeException;
}

TransportClientBootstrap是在TransportClient上执行的客户端引导程序，主要对连接建立时进行一些初始化的准备（例如验证、加密）。TransportClientBootstrap所作的操作往往是昂贵的，好在建立的连接可以重用。

在这里插入图片描述

三个实现类，我们以EncryptionDisablerBootstrap为例看看具体做了啥

private static class EncryptionDisablerBootstrap implements TransportClientBootstrap {

  @Override
  public void doBootstrap(TransportClient client, Channel channel) {
    channel.pipeline().remove(SaslEncryption.ENCRYPTION_HANDLER_NAME);
  }

}

作用是移除客户端管道中的SASL加密

创建客户端TransportClient

public class TransportClientFactory implements Closeable {

  /** A simple data structure to track the pool of clients between two peer nodes. */
  private static class ClientPool {
    TransportClient[] clients;
    Object[] locks;

    ClientPool(int size) {
      clients = new TransportClient[size];
      locks = new Object[size];
      for (int i = 0; i < size; i++) {
        locks[i] = new Object();
      }
    }
  }

ClientPool实际是由TransportClient的数组构成，而locks数组中的Object与clients数组中的TransportClient按照数组索引一一对应，通过对每个TransportClient分别采用不同的锁，降低并发情况下线程间对锁的争用，进而减少阻塞，提高并发度。

  public TransportClient createClient(String remoteHost, int remotePort)
      throws IOException, InterruptedException {
    final InetSocketAddress unresolvedAddress =
      InetSocketAddress.createUnresolved(remoteHost, remotePort);

    // Create the ClientPool if we don't have it yet.
    //判断是否存在了指定端口的客户端池
    ClientPool clientPool = connectionPool.get(unresolvedAddress);
    if (clientPool == null) {
      //numConnectionsPerPeer是transportContext中拿过来的，还记得默认是取系统cpu和8较小者
      //connectionPool
      connectionPool.putIfAbsent(unresolvedAddress, new ClientPool(numConnectionsPerPeer));
      clientPool = connectionPool.get(unresolvedAddress);
    }

    int clientIndex = rand.nextInt(numConnectionsPerPeer);
    TransportClient cachedClient = clientPool.clients[clientIndex];

    //判断client是否初始化了,且处于激活状态，更新handler访问时间并返回TransportClient
    if (cachedClient != null && cachedClient.isActive()) {
      TransportChannelHandler handler = cachedClient.getChannel().pipeline()
        .get(TransportChannelHandler.class);
      synchronized (handler) {
        handler.getResponseHandler().updateTimeOfLastRequest();
      }

      if (cachedClient.isActive()) {
        logger.trace("Returning cached connection to {}: {}",
          cachedClient.getSocketAddress(), cachedClient);
        return cachedClient;
      }
    }

    final long preResolveHost = System.nanoTime();
    final InetSocketAddress resolvedAddress = new InetSocketAddress(remoteHost, remotePort);
    final long hostResolveTimeMs = (System.nanoTime() - preResolveHost) / 1000000;
    if (hostResolveTimeMs > 2000) {
      logger.warn("DNS resolution for {} took {} ms", resolvedAddress, hostResolveTimeMs);
    } else {
      logger.trace("DNS resolution for {} took {} ms", resolvedAddress, hostResolveTimeMs);
    }

    //加锁
    synchronized (clientPool.locks[clientIndex]) {
      cachedClient = clientPool.clients[clientIndex];
      //双重验证
      if (cachedClient != null) {
        if (cachedClient.isActive()) {
          logger.trace("Returning cached connection to {}: {}", resolvedAddress, cachedClient);
          return cachedClient;
        } else {
          logger.info("Found inactive connection to {}, creating a new one.", resolvedAddress);
        }
      }
      //最后调用createClient(resolvedAddress)创建TransportClient
      clientPool.clients[clientIndex] = createClient(resolvedAddress);
      return clientPool.clients[clientIndex];
    }
  }

createClient(String remoteHost, int remotePort)方法主要做的事情是通过unresolvedAddress去connectionPool中查找地址对应的clientPool，没有则创建,再随机取其中一个client如果说客户端不为空。更新handler最后访问时间，没有则调用createClient(InetSocketAddress address)创建TransportClient。

private TransportClient createClient(InetSocketAddress address)
    throws IOException, InterruptedException {
  logger.debug("Creating new connection to {}", address);

  Bootstrap bootstrap = new Bootstrap();
  bootstrap.group(workerGroup)
    .channel(socketChannelClass)
    .option(ChannelOption.TCP_NODELAY, true)
    .option(ChannelOption.SO_KEEPALIVE, true)
    .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, conf.connectionTimeoutMs())
    .option(ChannelOption.ALLOCATOR, pooledAllocator);

  if (conf.receiveBuf() > 0) {
    bootstrap.option(ChannelOption.SO_RCVBUF, conf.receiveBuf());
  }

  if (conf.sendBuf() > 0) {
    bootstrap.option(ChannelOption.SO_SNDBUF, conf.sendBuf());
  }

  final AtomicReference<TransportClient> clientRef = new AtomicReference<>();
  final AtomicReference<Channel> channelRef = new AtomicReference<>();

  bootstrap.handler(new ChannelInitializer<SocketChannel>() {
    @Override
    public void initChannel(SocketChannel ch) {
        //客户端实际的创建地址
      TransportChannelHandler clientHandler = context.initializePipeline(ch);
      clientRef.set(clientHandler.getClient());
      channelRef.set(ch);
    }
  });

  // Connect to the remote server
  long preConnect = System.nanoTime();
  ChannelFuture cf = bootstrap.connect(address);
  if (!cf.await(conf.connectionTimeoutMs())) {
    throw new IOException(
      String.format("Connecting to %s timed out (%s ms)", address, conf.connectionTimeoutMs()));
  } else if (cf.cause() != null) {
    throw new IOException(String.format("Failed to connect to %s", address), cf.cause());
  }
    
    TransportClient client = clientRef.get();
    Channel channel = channelRef.get();
    assert client != null : "Channel future completed successfully with null client";
    long preBootstrap = System.nanoTime();
    logger.debug("Connection to {} successful, running bootstraps...", address);
    try {
        //使用clientBootstrap加工channel
        for (TransportClientBootstrap clientBootstrap : clientBootstraps) {
            clientBootstrap.doBootstrap(client, channel);
        }
    }
    return client;

createClient(InetSocketAddress address)其实就是使用nettyApi与服务端创建sockt连接clientRef存储TransportClient实例，channelRef存储socktChannel实例

public TransportChannelHandler initializePipeline(
    SocketChannel channel,
    RpcHandler channelRpcHandler) {
  try {
    TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
    channel.pipeline()
      .addLast("encoder", ENCODER)
      .addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
      .addLast("decoder", DECODER)
      //Netty的IdleStateHandler心跳机制主要是用来检测远端是否存活，如果不存活或活跃则对空闲Socket连接进行处理避免资源的浪费；
      .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
      .addLast("handler", channelHandler);
    return channelHandler;
  } catch (RuntimeException e) {
    logger.error("Error while initializing Netty pipeline", e);
    throw e;
  }
}

TransportChannelHandler clientHandler = context.initializePipeline(ch);最终netty底层的操作，ENCODER,DECODER等一些操作都在这里做的，TransportChannelHandler才是更加定制的一些操作

private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
  TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
  TransportClient client = new TransportClient(channel, responseHandler);
  TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
    rpcHandler, conf.maxChunksBeingTransferred());
    //client客户端requestHandler请求处理器,responseHandler返回处理器
  return new TransportChannelHandler(client, responseHandler, requestHandler,
    conf.connectionTimeoutMs(), closeIdleConnections);
}

上面TransportChannelHandler的创建过程，requestHandler请求处理器,responseHandler返回处理器的职责后面有机会再分析。

*TransportClient在创建时持有的都是TransportResponseHandler

TransportClient

public TransportClient(Channel channel, TransportResponseHandler handler) {
  this.channel = Preconditions.checkNotNull(channel);
  this.handler = Preconditions.checkNotNull(handler);
  this.timedOut = false;
}

TransportClient一共有五个方法用于发送请求，分别为：

fetchChunk：从远端协商好的流中请求单个块；
stream：使用流的ID，从远端获取流数据；
sendRpc：向服务端发送RPC的请求，通过At least Once Delivery原则保证请求不会丢失；
sendRpcSync：向服务端发送异步的RPC的请求，并根据指定的超时时间等待响应；
send：向服务端发送RPC的请求，但是并不期望能获取响应，因而不能保证投递的可靠性；

TransportServer

public TransportServer createServer(
    String host, int port, List<TransportServerBootstrap> bootstraps) {
  return new TransportServer(this, host, port, rpcHandler, bootstraps);
}

/** Creates a new server, binding to any available ephemeral port. */
public TransportServer createServer(List<TransportServerBootstrap> bootstraps) {
  return createServer(0, bootstraps);
}

public TransportServer createServer() {
  return createServer(0, new ArrayList<>());
}

最终都是调用TransportServer createServer(String host, int port, List bootstraps)

public TransportServer(
    TransportContext context,
    String hostToBind,
    int portToBind,
    RpcHandler appRpcHandler,
    List<TransportServerBootstrap> bootstraps) {
  this.context = context;
  this.conf = context.getConf();
  this.appRpcHandler = appRpcHandler;
  this.bootstraps = Lists.newArrayList(Preconditions.checkNotNull(bootstraps));

  boolean shouldClose = true;
  try {
    init(hostToBind, portToBind);
    shouldClose = false;
  } finally {
    if (shouldClose) {
      JavaUtils.closeQuietly(this);
    }
  }
}

TransportServer的构造器中的各个变量分别为：

context：即参数传递的TransportContext的引用；
conf：即TransportConf，这里通过调用TransportContext的getConf获取；
appRpcHandler：即RPC请求处理器RpcHandler；
bootstraps：即参数传递的TransportServerBootstrap列表；

private void init(String hostToBind, int portToBind) {

  IOMode ioMode = IOMode.valueOf(conf.ioMode());
  EventLoopGroup bossGroup = NettyUtils.createEventLoop(ioMode, 1,
    conf.getModuleName() + "-boss");
  EventLoopGroup workerGroup =  NettyUtils.createEventLoop(ioMode, conf.serverThreads(),
    conf.getModuleName() + "-server");

  PooledByteBufAllocator allocator = NettyUtils.createPooledByteBufAllocator(
    conf.preferDirectBufs(), true /* allowCache */, conf.serverThreads());

  bootstrap = new ServerBootstrap()
    .group(bossGroup, workerGroup)
    .channel(NettyUtils.getServerChannelClass(ioMode))
    .option(ChannelOption.ALLOCATOR, allocator)
    .option(ChannelOption.SO_REUSEADDR, !SystemUtils.IS_OS_WINDOWS)
    .childOption(ChannelOption.ALLOCATOR, allocator);

  this.metrics = new NettyMemoryMetrics(
    allocator, conf.getModuleName() + "-server", conf);

  if (conf.backLog() > 0) {
    bootstrap.option(ChannelOption.SO_BACKLOG, conf.backLog());
  }

  if (conf.receiveBuf() > 0) {
    bootstrap.childOption(ChannelOption.SO_RCVBUF, conf.receiveBuf());
  }

  if (conf.sendBuf() > 0) {
    bootstrap.childOption(ChannelOption.SO_SNDBUF, conf.sendBuf());
  }

  bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
    @Override
    protected void initChannel(SocketChannel ch) {
      RpcHandler rpcHandler = appRpcHandler;
      for (TransportServerBootstrap bootstrap : bootstraps) {
        rpcHandler = bootstrap.doBootstrap(ch, rpcHandler);
      }
      context.initializePipeline(ch, rpcHandler);
    }
  });

  InetSocketAddress address = hostToBind == null ?
      new InetSocketAddress(portToBind): new InetSocketAddress(hostToBind, portToBind);
  channelFuture = bootstrap.bind(address);
  channelFuture.syncUninterruptibly();

  port = ((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
  logger.debug("Shuffle server started on port: {}", port);
}

1.创建bossGroup和workerGroup（根据Netty的API文档，Netty服务端需同时创建bossGroup和workerGroup。）；
2.创建一个汇集ByteBuf但对本地线程缓存禁用的分配器；

3.调用Netty的API创建Netty的服务端根引导程序并对其进行配置；

4.为根引导程序设置管道初始化回调函数，此回调函数首先设置TransportServerBootstrap到根引导程序中，然后调用TransportContext的initializePipeline方法初始化Channel的pipeline；

5.给根引导程序绑定Socket的监听端口，最后返回监听的端口。

context.initializePipeline方法上面有说明，核心干的事ENCODER,DECODER等一些操作都在这里做的，然后加上TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);

在这里插入图片描述

TransportChannelHandler

public class TransportChannelHandler extends ChannelInboundHandlerAdapter {

其实继承了ChannelInboundHandler重点关心TransportChannelHandler实现的channelRead

@Override
public void channelRead(ChannelHandlerContext ctx, Object request) throws Exception {
  if (request instanceof RequestMessage) {
    requestHandler.handle((RequestMessage) request);
  } else if (request instanceof ResponseMessage) {
    responseHandler.handle((ResponseMessage) request);
  } else {
    ctx.fireChannelRead(request);
  }
}

当TransportChannelHandler读取到的request是RequestMessage类型时，则将此消息的处理进一步交给TransportRequestHandler，当request是ResponseMessage时，则将此消息的处理进一步交给TransportResponseHandler。

public class TransportResponseHandler extends MessageHandler<ResponseMessage> {
public class TransportRequestHandler extends MessageHandler<RequestMessage> {

TransportRequestHandler与TransportResponseHandler都继承自抽象类MessageHandler

public abstract class MessageHandler<T extends Message> {
  /** Handles the receipt of a single message. */
  public abstract void handle(T message) throws Exception;

  /** Invoked when the channel this MessageHandler is on is active. */
  public abstract void channelActive();

  /** Invoked when an exception was caught on the Channel. */
  public abstract void exceptionCaught(Throwable cause);

  /** Invoked when the channel this MessageHandler is on is inactive. */
  public abstract void channelInactive();
}

MessageHandler定义了几个方法:

handle：用于对接收到的单个消息进行处理；
channelActive：当channel激活时调用；
exceptionCaught：当捕获到channel发生异常时调用；
channelInactive：当channel非激活时调用；

public interface Message extends Encodable {
  /** Used to identify this request type. */
  Type type();

  /** An optional body for the message. */
  ManagedBuffer body();

  /** Whether to include the body of the message in the same frame as the message. */
  boolean isBodyInFrame();

Message定义

public interface Encodable {
  int encodedLength();

  void encode(ByteBuf buf);
}

实现Encodable接口的类将可以转换到一个ByteBuf中，多个对象将被存储到预先分配的单个ByteBuf，所以这里的encodedLength用于返回转换的对象数量。

在这里插入图片描述

从图看到最终的消息实现类都直接或间接的实现了RequestMessage或ResponseMessage接口，其中RequestMessage的具体实现有四种，分别是：

ChunkFetchRequest：请求获取流的单个块的序列。
RpcRequest：此消息类型由远程的RPC服务端进行处理，是一种需要服务端向客户端回复的RPC请求信息类型。
OneWayMessage：此消息也需要由远程的RPC服务端进行处理，与RpcRequest不同的是不需要服务端向客户端回复。
StreamRequest：此消息表示向远程的服务发起请求，以获取流式数据。
由于OneWayMessage 不需要响应，所以ResponseMessage的对于成功或失败状态的实现各有三种，分别是：

ChunkFetchSuccess：处理ChunkFetchRequest成功后返回的消息；
ChunkFetchFailure：处理ChunkFetchRequest失败后返回的消息；
RpcResponse：处理RpcRequest成功后返回的消息；
RpcFailure：处理RpcRequest失败后返回的消息；
StreamResponse：处理StreamRequest成功后返回的消息；
StreamFailure：处理StreamRequest失败后返回的消息；

ManagedBuffer

public abstract class ManagedBuffer {

具体属性代表的意义：

size：返回数据的字节数。
nioByteBuffer：将数据按照Nio的ByteBuffer类型返回。
createInputStream：将数据按照InputStream返回。
retain：当有新的使用者使用此视图时，增加引用此视图的引用数。
release：当有使用者不再使用此视图时，减少引用此视图的引用数。当引用数为0时释放缓冲区。
convertToNetty：将缓冲区的数据转换为Netty的对象，用来将数据写到外部。此方法返回的数据类型要么是io.netty.buffer.ByteBuf，要么是io.netty.channel.FileRegion。

Message的body方法返回的是ManagedBuffer;ManagedBuffer是一个抽象类。

具体的一些实现类。

TransportServerBootstrap

public interface TransportServerBootstrap {
  RpcHandler doBootstrap(Channel channel, RpcHandler rpcHandler);
}

TransportServerBootstrap的doBootstrap方法将对服务端的RpcHandler进行代理，接收客户端的请求。TransportServerBootstrap有SaslServerBootstrap和EncryptionCheckerBootstrap两个实现类。

public RpcHandler doBootstrap(Channel channel, RpcHandler rpcHandler) {
  return new SaslRpcHandler(conf, channel, rpcHandler, secretKeyHolder);
}

SaslServerBootstrap的doBootstrap实现

RpcHandler

public TransportChannelHandler initializePipeline(
    SocketChannel channel,
    RpcHandler channelRpcHandler) {
  try {
    TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
    channel.pipeline()
      .addLast("encoder", ENCODER)
      .addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
      .addLast("decoder", DECODER)
      .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
      // NOTE: Chunks are currently guaranteed to be returned in the order of request, but this
      // would require more logic to guarantee if this were not part of the same event loop.
      .addLast("handler", channelHandler);
    return channelHandler;
  } catch (RuntimeException e) {
    logger.error("Error while initializing Netty pipeline", e);
    throw e;
  }
}

客户端和服务端的创建都用到了TransportContext.initializePipeline而里面需要传入RpcHandler rpcHandler;

private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
  TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
  TransportClient client = new TransportClient(channel, responseHandler);
  TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
    rpcHandler, conf.maxChunksBeingTransferred());
  return new TransportChannelHandler(client, responseHandler, requestHandler,
    conf.connectionTimeoutMs(), closeIdleConnections);
}

注意到requestHandler在创建时用到了rpcHandler。其实requestHandler相当于是代理了rpcHandler。所以有必要研究一下RpcHandler。因为client只是使用到的responseHandler而它在创建时与rpcHandler无关。所以其实rpcHandler只对服务端创建有影响

public abstract class RpcHandler {
  private static final RpcResponseCallback ONE_WAY_CALLBACK = new OneWayRpcCallback();
  public abstract void receive(
      TransportClient client,
      ByteBuffer message,
      RpcResponseCallback callback);
  public StreamCallbackWithID receiveStream(
      TransportClient client,
      ByteBuffer messageHeader,
      RpcResponseCallback callback) {
    throw new UnsupportedOperationException();
  }
  public abstract StreamManager getStreamManager();
  public void receive(TransportClient client, ByteBuffer message) {
    receive(client, message, ONE_WAY_CALLBACK);
  }
  public void channelActive(TransportClient client) { }
  public void channelInactive(TransportClient client) { }
  public void exceptionCaught(Throwable cause, TransportClient client) { }
  private static class OneWayRpcCallback implements RpcResponseCallback {

    private static final Logger logger = LoggerFactory.getLogger(OneWayRpcCallback.class);

    @Override
    public void onSuccess(ByteBuffer response) {
      logger.warn("Response provided for one-way RPC.");
    }

    @Override
    public void onFailure(Throwable e) {
      logger.error("Error response provided for one-way RPC.", e);
    }
  }
}

RpcHandler是一个抽象类，各个方法的做用：

receive：这是一个抽象方法，用来接收单一的RPC消息，具体处理逻辑需要子类去实现。

channelActive：当与给定客户端相关联的channel处于活动状态时调用。
channelInactive：当与给定客户端相关联的channel处于非活动状态时调用。
exceptionCaught：当channel产生异常时调用。
getStreamManager：获取StreamManager，StreamManager可以从流中获取单个的块，因此它也包含着当前正在被TransportClient获取的流的状态。

最后讲一下requestHandler的handle方法,前面已经强调过了requestHandler是在服务端使用的

public void handle(RequestMessage request) {
  if (request instanceof ChunkFetchRequest) {
    processFetchRequest((ChunkFetchRequest) request);
  } else if (request instanceof RpcRequest) {
    processRpcRequest((RpcRequest) request);
  } else if (request instanceof OneWayMessage) {
    processOneWayMessage((OneWayMessage) request);
  } else if (request instanceof StreamRequest) {
    processStreamRequest((StreamRequest) request);
  } else if (request instanceof UploadStream) {
    processStreamUpload((UploadStream) request);
  } else {
    throw new IllegalArgumentException("Unknown request type: " + request);
  }
}

可用看到针对不同类型的请求会有不同的处理方式

private void processFetchRequest(final ChunkFetchRequest req) {
  ManagedBuffer buf;
  try {
    streamManager.checkAuthorization(reverseClient, req.streamChunkId.streamId);
    buf = streamManager.getChunk(req.streamChunkId.streamId, req.streamChunkId.chunkIndex);
  } catch (Exception e) {
    logger.error(String.format("Error opening block %s for request from %s",
      req.streamChunkId, getRemoteAddress(channel)), e);
    respond(new ChunkFetchFailure(req.streamChunkId, Throwables.getStackTraceAsString(e)));
    return;
  }

  streamManager.chunkBeingSent(req.streamChunkId.streamId);
  respond(new ChunkFetchSuccess(req.streamChunkId, buf)).addListener(future -> {
    streamManager.chunkSent(req.streamChunkId.streamId);
  });
}

处理块获取请求

1.调用StreamManager的checkAuthorization方法，校验客户端是否有权限从给定的流中读取；
2.调用StreamManager的getChunk方法，获取单个的块（块被封装为ManagedBuffer）。由于单个的流只能与单个的TCP连接相关联，因此getChunk方法不能为了某个特殊的流而并行调用；
3.将ManagedBuffer和流的块Id封装为ChunkFetchSuccess后，调用respond方法返回给客户端。

private void processRpcRequest(final RpcRequest req) {
  try {
    rpcHandler.receive(reverseClient, req.body().nioByteBuffer(), new RpcResponseCallback() {
      @Override
      public void onSuccess(ByteBuffer response) {
        respond(new RpcResponse(req.requestId, new NioManagedBuffer(response)));
      }

      @Override
      public void onFailure(Throwable e) {
        respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
      }
    });
}

处理RPC请求,将RpcRequest消息的内容体、发送消息的客户端以及一个RpcResponseCallback类型的匿名内部类作为参数传递给了RpcHandler的receive方法。这就是说真正用于处理RpcRequest消息的是RpcHandler，而非TransportRequestHandler。由于RpcHandler是抽象类，其receive方法也是抽象方法，所以具体的操作将由RpcHandler的实现了receive方法的子类来完成。所有继承RpcHandler的子类都需要在其receive方法的具体实现中回调RpcResponseCallback的onSuccess（处理成功时）或者onFailure（处理失败时）方法。从RpcResponseCallback的实现来看，无论处理结果成功还是失败，都将调用respond方法对客户端进行响应。

private void processStreamRequest(final StreamRequest req) {
  ManagedBuffer buf;
  try {
    buf = streamManager.openStream(req.streamId);
  } catch (Exception e) {
    logger.error(String.format(
      "Error opening stream %s for request from %s", req.streamId, getRemoteAddress(channel)), e);
    respond(new StreamFailure(req.streamId, Throwables.getStackTraceAsString(e)));
    return;
  }

  if (buf != null) {
    streamManager.streamBeingSent(req.streamId);
    respond(new StreamResponse(req.streamId, buf.size(), buf)).addListener(future -> {
      streamManager.streamSent(req.streamId);
    });
  } else {
    respond(new StreamFailure(req.streamId, String.format(
      "Stream '%s' was not found.", req.streamId)));
  }
}

处理流请求:

调用StreamManager的openStream方法将获取到的流数据封装为ManagedBuffer；

当成功或失败时调用respond方法向客户端响应。

private void processOneWayMessage(OneWayMessage req) {
  try {
    rpcHandler.receive(reverseClient, req.body().nioByteBuffer());
  } catch (Exception e) {
    logger.error("Error while invoking RpcHandler#receive() for one-way message.", e);
  } finally {
    req.body().release();
  }
}

处理无需回复的RPC请求processOneWayMessage方法的实现processRpcRequest非常相似，区别在于processOneWayMessage调用了ONE_WAY_CALLBACK的receive方法，因而processOneWayMessage在处理完RPC请求后不会对客户端作出响应。

private ChannelFuture respond(Encodable result) {
  SocketAddress remoteAddress = channel.remoteAddress();
  return channel.writeAndFlush(result).addListener(future -> {
    if (future.isSuccess()) {
      logger.trace("Sent result {} to client {}", result, remoteAddress);
    } else {
      logger.error(String.format("Error sending result %s to %s; closing connection",
        result, remoteAddress), future.cause());
      channel.close();
    }
  });
}

最后使用netty调用了Channel的writeAndFlush方法来响应客户端。