Thursday, July 2, 2015

zabbix_sender tool performs a partial database import when fed with too many data at once.

I am using zabbix_sender tool to inject into a zabbix database data samples related to computer network statistics.

I start with a clean zabbix database, after clearing zabbix historical data for some items, so that the database is empty. Then I load new data with zabbix sender tool, and then I analyze the data mostly in graphical mode, checking the behaviours of the systems under particular loads, using zabbix graphical interface for historical navigations, statistics, and graph overlapping. When I am done with the analysis, I again delete historical values, and load a new batch of values.

I need to manage about 500.000 records. The problem I am describing here was not evident until data processed by my scripts was below 40.000 records. Problem was not evident using non differential valued zabbix items.

Using the following command line:zabbix_sender -z 127.0.0.1 -T -i zbx-inputfile.txt

I get no errors when executing this, but older data are badly imported.

I think this is depending on data being processed by different threads of zabbix server (zabbix trappers) which can not talk each other considering that some data is to be interpreted as difference with the previous value. Differential data requires all the values to be sequentially injected and processed by a single zabbix-trapper thread.
Strangely, I did not find any mention to this potential problem, but you can not expect anything good if you are going to insert historical values in any series which is to be considered in differential mode, because data is processed before being inserted in the database, and the difference with the previous value is the actual value which goes in the database.
I tried to use -r (realtime) option in the zabbix_sender command line, but this did not give any improvement (actually I did not understand precisely what the realtime option is for).

I know that I am using Zabbix and zabbix_sender tools in a way that is probably quite far from the designer scopes, but I love this great software.

I was able to solve the issue slowing the speed at which data are sent to zabbix_sender.
In order to do this I built a filter tool which slows down a unix communication pipe.

The tool is a simple perl script called slower-pipe which sends some lines, then wait a specific time before sending the next chunk

Using the tool to slow the input feed (100 lines per chunk and 2 seconds wait) the problem vanished.
The reason for this kludge to be effective lies probably in the fact that slowing down the input data flow, a single zabbix-trapper thread is considered enough, and calculations of the differential items are correctly processed.

here is the (perl) source code for slower-pipe.pl
just copy it, save it into slower-pipe.pl and then chmod +x appropriately