Meditation: Is IOCP listener actually listening?

Meditation: Is IOCP listener actually listening?

The IOCP listener is a dedicated system thread that accepts incoming connections to SQL Server. It is a SQL Network Interface (SNI)-layer thread that listens for TCP/IP sockets or Named Pipes traffic (VIA is going away but still in the code). When a connection comes in, the listener accepts it and quickly hands it off to a SQL worker thread to process further. Think of the IOCP listener as the gate keeper – any new incoming connections go through it. A frequent confusion caused by its name is that the IOCP listener handles disk I/O. In reality its purpose in life is to handle network I/O in SQL Server. The name IOCP stands for – I/O Completion Port and indeed it does use I/O Completion ports. But I/O completion ports can be used not only for asynchronous disk I/O but also for all type of other asynchronous I/O. As the note in the I/O Completion ports article states:

“The term file handle as used here refers to a system abstraction representing an overlapped I/O endpoint, not only a file on disk. For example, it can be a network endpoint, TCP socket, named pipe, or mail slot. Any system object that supports overlapped I/O can be used. For a list of related I/O functions, see the end of this topic.”

So in summary, in SQL engine, we use the IOCP thread to handle incoming TCP socket and Named pipe traffic.

Non-Yielding IOCP Listener

If the IOCP listener thread gets stuck - you will frequently associate this with a non-yielding IOCP – then no incoming connections can be accepted. SQL Server appears effectively “hung” to the outside world, even though existing queries can be running just fine. That’s why the Scheduler Monitor thread (another system thread that checks SOS scheduler health) keeps a close watch on the IOCP listener and if it gets stuck for 15 seconds, it generates an error in the Errorlog and creates a memory dump. As a reference, the Scheduler Monitor creates a memory dump after 60 seconds for other scheduler issues. As you can see SQL is more aggressively monitoring the IOCP listener thread because it is a critical one in the system.

IOCP Listener and NUMA

There is one IOCP thread per server, if no NUMA is present. If NUMA is present, SQL Server creates one per NUMA node. Here is what mine looks like currently on an idle SQL Server.

Here is a IOCP scenario I found on MSDN blogs. BTW, for that particular blog, NumaNode::RampUp is a very specific stage in the BPool ramp up described here and I think was the root cause that scenario.

There are some great tips on debugging non-yielding IOCP listener scenarios on MSSQLWiki blog