>
The network server for kafka. No application specific code here, just general network server stuff.
The classes Receive and Send encapsulate the incoming and outgoing transmission of bytes. A Handler
is a mapping between a Receive and a Send, and represents the users hook to add logic for mapping requests
to actual processing code. Any uncaught exceptions in the reading or writing of the transmissions will result in
the server logging an error and closing the offending socket. As a result it is the duty of the Handler
implementation to catch and serialize any application-level errors that should be sent to the client.
This slightly lower-level interface that models sending and receiving rather than requests and responses
is necessary in order to allow the send or receive to be overridden with a non-user-space writing of bytes
using FileChannel.transferTo.

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The number of queued request allowed before blocking the network threads
#queued.max.requests
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600

SocketServer

这个类上的注释阐述了kafka server的io线程模型

这个类上的注释阐述了kafka server的io线程模型
/**
* An NIO socket server. The threading model is
* 1 Acceptor thread that handles new connections
* Acceptor has N Processor threads that each have their own selector and read requests from sockets
* M Handler threads that handle requests and produce responses back to the processor threads for writing.
*/

override def run() {
startupComplete()
while (isRunning) {
try {
// setup any new connections that have been queued up
configureNewConnections()
// register any new responses for writing
processNewResponses()
poll()
processCompletedReceives()
processCompletedSends()
processDisconnected()
} catch {
// We catch all the throwables here to prevent the processor thread from exiting. We do this because
// letting a processor exit might cause a bigger impact on the broker. Usually the exceptions thrown would
// be either associated with a specific socket channel or a bad request. We just ignore the bad socket channel
// or request. This behavior might need to be reviewed if we see an exception that need the entire broker to stop.
case e: ControlThrowable => throw e
case e: Throwable =>
error("Processor got uncaught exception.", e)
}
}
debug("Closing selector - processor " + id)
swallowError(closeAll())
shutdownComplete()
}

Kafka Client端网络代码

clients包里分成主要Send、Receive、KafkaChannel和Selector四部分

Selectable是其中的网络操作的接口, Selector是具体的实现, 包括了发送请求、接收返回、建立连接、断开连接等操作。

/**
* An interface for asynchronous, multi-channel network I/O
*/
public interface Selectable {
/**
* See {@link #connect(String, InetSocketAddress, int, int) connect()}
*/
public static final int USE_DEFAULT_BUFFER_SIZE = -1;
/**
* Begin establishing a socket connection to the given address identified by the given address
* @param id The id for this connection
* @param address The address to connect to
* @param sendBufferSize The send buffer for the socket
* @param receiveBufferSize The receive buffer for the socket
* @throws IOException If we cannot begin connecting
*/
public void connect(String id, InetSocketAddress address, int sendBufferSize, int receiveBufferSize) throws IOException;
/**
* Wakeup this selector if it is blocked on I/O
*/
public void wakeup();
/**
* Close this selector
*/
public void close();
/**
* Close the connection identified by the given id
*/
public void close(String id);
/**
* Queue the given request for sending in the subsequent {@link #poll(long) poll()} calls
* @param send The request to send
*/
public void send(Send send);
/**
* Do I/O. Reads, writes, connection establishment, etc.
* @param timeout The amount of time to block if there is nothing to do
* @throws IOException
*/
public void poll(long timeout) throws IOException;
/**
* The list of sends that completed on the last {@link #poll(long) poll()} call.
*/
public List<Send> completedSends();
/**
* The list of receives that completed on the last {@link #poll(long) poll()} call.
*/
public List<NetworkReceive> completedReceives();
/**
* The connections that finished disconnecting on the last {@link #poll(long) poll()}
* call. Channel state indicates the local channel state at the time of disconnection.
*/
public Map<String, ChannelState> disconnected();
/**
* The list of connections that completed their connection on the last {@link #poll(long) poll()}
* call.
*/
public List<String> connected();
...
}

/**
* This interface models the in-progress sending of data to a destination identified by an integer id.
*/
public interface Send {
/**
* The numeric id for the destination of this send
*/
String destination();
/**
* Is this send complete?
*/
boolean completed();
/**
* Write some as-yet unwritten bytes from this send to the provided channel. It may take multiple calls for the send
* to be completely written
* @param channel The Channel to write to
* @return The number of bytes written
* @throws IOException If the write fails
*/
long writeTo(GatheringByteChannel channel) throws IOException;
/**
* Size of the send
*/
long size();
}

public interface Receive extends Closeable {
/**
* The numeric id of the source from which we are receiving data.
*/
String source();
/**
* Are we done receiving data?
*/
boolean complete();
/**
* Read bytes into this receive from the given channel
* @param channel The channel to read from
* @return The number of bytes read
* @throws IOException If the reading fails
*/
long readFrom(ScatteringByteChannel channel) throws IOException;
/**
* Do we know yet how much memory we require to fully read this
*/
boolean requiredMemoryAmountKnown();
/**
* Has the underlying memory required to complete reading been allocated yet?
*/
boolean memoryAllocated();
}

/**
* Queue up the given request for sending. Requests can only be sent on ready connections.
* @param request The request
* @param now The current timestamp
*/
void send(ClientRequest request, long now);
/**
* Do actual reads and writes from sockets.
*
* @param timeout The maximum amount of time to wait for responses in ms, must be non-negative. The implementation
* is free to use a lower value if appropriate (common reasons for this are a lower request or
* metadata update timeout)
* @param now The current time in ms
* @throws IllegalStateException If a request is sent to an unready node
*/
List<ClientResponse> poll(long timeout, long now);

关键的接口有send和poll, send方法将要发送的内容保存起来，真正的Channel读写发生在poll方法中

KafkaClient的实现类是NetworkClient。

ClientRequest中通过requetBuilder给不同类型的请求设置不同的请求内容

public final class ClientRequest {
private final String destination;
private final AbstractRequest.Builder<?> requestBuilder;
private final int correlationId;
private final String clientId;
private final long createdTimeMs;
private final boolean expectResponse;
private final RequestCompletionHandler callback;

同样的，ClientResponse也有对应各个类型不同的返回体

public class ClientResponse {
private final RequestHeader requestHeader;
private final RequestCompletionHandler callback;
private final String destination;
private final long receivedTimeMs;
private final long latencyMs;
private final boolean disconnected;
private final UnsupportedVersionException versionMismatch;
private final AbstractResponse responseBody;