WL#6630: Semisync separate acks collector

Objective: optimize replication delivery and the server-to-client response time.
See WL#6355 for related tasks.
O1. to improve semi-sync communication framework
The current semi-sync design is subject to Dump thread hiccups because the one
thread dispatches 2 different and quite hot task - replicate
replicated transaction events to slave and release the connection to
respond back to the client):
ack
..............
V :
Conn -> Dump ------- > IO
^ : replicate
:........:
release
Dump thread could be a real bottle neck. It can't proceed with 2nd
transaction until the 1st one has not yet been acknowledged from the
slave. Obviously the 2nd transaction propagation plus ack time,
contributing to the response time to the client, is suboptimal. It's
reasonable to claim that replication throughput is suboptimal as well
for the same reason of hiccup-prone design.
Alternatively, a special ack Collector (Col) thread could be deployed
on the master that would redistribute acks to release waiting
Connections:
Conn_2 ack
^...... Col Dump ------- > Slave
^ :
:........:

Functional Requirements
=======================
No
Non-Functional Requirements
===========================
NF-1: Reduce semisync delay by using separate threads to send and receive semisync
acknowledgement. So event streams and ACK streams can be sent and received
simultaneously.

No Interface.
No any change on user level.

ACK Receive Thread In Semisync Master
=====================================
* Start/Stop ack receive thread
The thread is controlled automatically by semisync master.
- It is started automatically when enabling semisync master through
SET rpl_semi_sync_master_enabled = ON
- It is stopped automatically when disabling semisync master through
SET rpl_semi_sync_master_enabled = OFF
* Mornitor ack receive thread
- Ack receive thread status is put in performance_schema.threads table.
Users can query its status from the table just like mornitoring other
threads.
- Column values
name: thread/semisync/Ack_receiver
type: BACKGROUND
processlist_state: A string describes its status.
1. Waiting for semi-sync slave connection.
2. Waiting for semi-sync ACK from slave .
3. Reading semi-sync ACK from slave.
* Semisync slave list
Semisync master maintains a semisync slave list. Ack receive thread
listens on all semisync slaves in the list and receive acks from them
when ack thread is on.
- Register semisync slave
Dump threads call transmit_start hook to register its slave to semisync
master. Only the slave supporting semisync slave is added in the semisync
list.
- Unregister semisync slave
Dump threads call transmit_stop to remove a semisync slave from the
semisync slave list.
Note:
The semisync slave list is maintained even when ack receive thread is
down. It helps ack thread to listen the semisync slaves connected before
semisync master is enabled.
* Share TCP connections between dump thread and ack receive thread.
- Slave IO supports only TCP connections. TCP is full-duplex connection,
Send and write can work simultaneously in separate threads.
- But NET and Vio objects are designed for synchronous, it can not be shared
between threads. So ack receive thread has its own NET and Vio which shares
sockets with the dump threads.
- select() is used to listen all semisync slaves.
* Ack_receiver Class
It encapsulates the code which controls receive thread on master
- m_status
Status of receive thread
DEFINITION:
enum status m_status;
enum status { ST_UP, ST_STOPPING, ST_DOWN };
ST_UP means ack receive thread is created and is working.
ST_DOWN means ack receive thread is destroyed.
ST_STOPPING means a user is disabling semisync master, and ack receive
thread is being destroyed.
Check stop() for detail.
- m_slaves
A slave vector which includes slaves' useful information here.
DEFINITION:
Slave_vector m_slaves
- m_mutex
m_slaves and m_status are shared between user sessions(dump threads) and
ack thread. So they should be protected by a mutex.
- start()
Start receive thread if it is down.
DEFINITION:
bool start();
LOGIC:
if m_status is ST_DOWN
{
set m_status to ST_UP.
create receive thread.
}
- stop()
Stop receive thread if it is UP.
DEFINITION:
void stop();
LOGIC:
if m_status is ST_UP
{
acquire m_mutex
set m_status to ST_STOPPING.
send a signal to wake up ack thread, it may be waiting for a signal.
wait until m_status is ST_DOWN.
release m_mutex
}
- add_slave()
Add a new semisync slave to slave list.
DEFINITION:
bool add_slave(THD *thd);
LOGIC:
initialze slave information.
acquire m_mutex
add the slave's information into m_slaves.
send a signal to ack receive thread. It may be waiting for a signal.
release m_mutex
- remove_slave()
remove a semisync slave from slave list.
DEFINITION:
void remove_slave(THD *thd)
LOGIC:
acquire m_mutex
remove thd of the slave from m_slaves.
release m_mutex
- run()
The handle function of receive thread.
DEFINITION:
void run();
LOGIC:
initialize pthread related things
while (1)
{
acquire m_mutex
if m_status is ST_STOPPING then break the loop.
wait any semisync slave to be added if slave list empty.
call select to listen on sockets, timeout is 1s.
restart and continue the loop if error or timeout happens.
receive and report acks to semisync master.
release m_mutex
}
de-initialize pthread related things
Note: Giving select a timeout makes other threads can add/remove slaves
or stop ack receive thread when there is no ack.
* ack_receive_handler
Receive thread handler
DEFINITION:
pthread_handler_t ack_receive_handler(void *arg);
LOGIC:
call ack_receiver::run().