Wednesday, February 1, 2017

TCP offload engine or TOE is a technology used in network interface cards (NIC) to offload processing of the entire TCP/IP stack to the network controller. It is primarily used with high-speed network interfaces, such as gigabit Ethernet and 10 Gigabit Ethernet, where processing overhead of the network stack becomes significant.

Usually when a packet arrives at network interface, it sends a hard interrupt to the processor. So depends up on the traffic, processor has to work more, handle the interrupts, calculate and compare the checksum etc. This will increase the load on CPU.

So a lot of modern Ethernet cards comes up with inbuilt support for checking checksum, handling errors etc. Enabling this we can reduce some load on the CPU.

First we need to check what are features enabled on the interface. They will be shown as On or Off. They may ne associated with a [fixed] parameter which says the value is fixed and we cannot change it.

[root@sysadmin ~]# ethtool --show-offload eth0
Features for eth0:
rx-checksumming: ontx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-unneeded: off [fixed]
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off [fixed]tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
loopback: off [fixed]

We can set the tx checksumming using the following command:

[root@sysadmin ~]# ethtool --offload eth0 rx on tx on

[root@sysadmin ~]# ethtool --show-offload eth0

Features for eth0:

rx-checksumming: on

tx-checksumming: on

tx-checksum-ipv4: on

tx-checksum-unneeded: off [fixed]

tx-checksum-ip-generic: off [fixed]

tx-checksum-ipv6: off [fixed]

tx-checksum-fcoe-crc: off [fixed]

tx-checksum-sctp: off [fixed]

scatter-gather: off

tx-scatter-gather: off

tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: off

tx-tcp-segmentation: off

tx-tcp-ecn-segmentation: off [fixed]

tx-tcp6-segmentation: off [fixed]

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: off [requested on]

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: off [fixed]

highdma: off [fixed]

rx-vlan-filter: off [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

fcoe-mtu: off [fixed]

loopback: off [fixed]

[root@sysadmin ~]#

We can enable scatter-gather using the following command:

[root@sysadmin ~]# ethtool --offload eth0 sg on

[root@sysadmin ~]# ethtool --show-offload eth0

Features for eth0:

rx-checksumming: on

tx-checksumming: on

tx-checksum-ipv4: on

tx-checksum-unneeded: off [fixed]

tx-checksum-ip-generic: off [fixed]

tx-checksum-ipv6: off [fixed]

tx-checksum-fcoe-crc: off [fixed]

tx-checksum-sctp: off [fixed]

scatter-gather: on

tx-scatter-gather: on

tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

tx-tcp-segmentation: on

tx-tcp-ecn-segmentation: off [fixed]

tx-tcp6-segmentation: off [fixed]

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: off [fixed]

highdma: off [fixed]

rx-vlan-filter: off [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

fcoe-mtu: off [fixed]

loopback: off [fixed]

We can eable TSO (TCP Segmentation offload) using the following command:

[root@sysadmin ~]# ethtool --offload eth0 tso on

[root@sysadmin ~]# ethtool --show-offload eth0

Features for eth0:

rx-checksumming: on

tx-checksumming: on

tx-checksum-ipv4: on

tx-checksum-unneeded: off [fixed]

tx-checksum-ip-generic: off [fixed]

tx-checksum-ipv6: off [fixed]

tx-checksum-fcoe-crc: off [fixed]

tx-checksum-sctp: off [fixed]

scatter-gather: on

tx-scatter-gather: on

tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

tx-tcp-segmentation: on

tx-tcp-ecn-segmentation: off [fixed]

tx-tcp6-segmentation: off [fixed]

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: off [fixed]

highdma: off [fixed]

rx-vlan-filter: off [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

fcoe-mtu: off [fixed]

loopback: off [fixed]

[root@sysadmin ~]#

You can enable GSO(Generic Segmentation Offload) using the following command:

Wednesday, January 25, 2017

We have seen how to do mysql master slave replication in one of our previous post. Once we setup mysql server as master with traditional replication, it will start creating bin-logs. This will go on and on, depending on the transactions happening on you database it may eat all your storage space.

One day I checked the disk usage of one of our mysql instance and found that 82% of disk is utilized!

Then I checked the size of /var/lib/mysql and found a lot of bin-log files. When I checked the master status I found that the current bin file is mysql-bin.000089. So I can delete the files till that. Or the files till the last week.

We need to keep our data safe so we do the database replication. In mysql the most common replication is Master-Slave replication. In this post we will see how we can setup a mysql server as master and another one as slave. We are using Mysql version 5.7 in this example on Centos Linux 6.8.

If the slave service is already running then you will get the following error.
ERROR 3021 (HY000): This operation cannot be performed with a running slave io thread; run STOP SLAVE IO_THREAD FOR CHANNEL '' first.

Start the slave, using the --skip-slave-start option so that replication does not start.

Or you can use the command
mysql>STOP SLAVE;

Then once the master details are set, you can start slave service.
mysql>START SLAVE;

Once the Slave is running and receiving the data from the master, you can see the slave status as follows.

Thursday, January 19, 2017

A lot of times we need to check whether its the network causing the bottleneck. There are a lot of tools like iftop to check this. nicstat is one of the nice tools to check the network performance. In this post we will see how to install and use nicstat command. This example is executed on a centos linux system.

If your system is 64bit, then you will get the following error.
[root@sysadmin nicstat-1.95]# make
gcc -O3 -m32 nicstat.c -o nicstat
In file included from /usr/include/features.h:385,
from /usr/include/stdio.h:28,
from nicstat.c:33:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
make: *** [nicstat] Error 1
[root@sysadmin nicstat-1.95]#

Then you need to install the following packages.
yum -y install glibc-devel.i686 glibc-devel

On some systems, even if you install glibc, the following error will come.

Sunday, January 15, 2017

When we use PCI network card, for eg 10g fiber NIC, it will have more than one channels. Our Server will have many CPUs also. If all the channels of the NIC is interrupting only one CPU, there will be huge stress on the single CPU and it will decrease the performance of network, server etc.

The best way to handle this distribute the interrupts to all the cpus available in the system. In this post we will see how to do it.