Airwave GUI loading issue

Airwave hard disk was 100% utilized and GUI stopped working. cleared old files/temporary files to get some space back. However, still unable to load the GUI.

Environment : Airwave versions 8.0 and above.

We could see the below errors in the service_watcher logs related to Redis, located at /var/log/service_watcher.

bootstrap.pl(15747): Falling through to CORE::exit (15747 - Daemon::ServiceWatcher): Could not connect to Redis server at 127.0.0.1:6 379: Connection refused at /opt/airwave/local/lib/perl5/site_perl/5.10.1/Redis.pm line 423

Redis::__build_sock('Redis=HASH(0x7ffa6bd7c7a8)') called at /opt/airwave/local/lib/perl5/site_perl/5.10.1/Redis.pm line 402

Redis::__connect('Redis=HASH(0x7ffa6bd7c7a8)') called at /opt/airwave/local/lib/perl5/site_perl/5.10.1/Redis.pm line 68

Mercury::Redis::Hash::Base::get_new_redis('Mercury::Redis::Hash::ClusterStatus=HASH(0x7ffa6bccb1f8)') called at /usr/local/airwave/lib/perl/Mercury/Redis/Hash/Base.pm line 55

Mercury::Redis::Hash::Base::connect_to_redis('Mercury::Redis::Hash::ClusterStatus=HASH(0x7ffa6bccb1f8)') called at /usr/local/airwave/lib/perl/Mercury/Redis/Hash/Base.pm line 49

Mercury::Redis::Hash::Base::redis('Mercury::Redis::Hash::ClusterStatus=HASH(0x7ffa6bccb1f8)') called at /usr/local/airwave/lib/perl/Mercury/Redis/Hash/Base.pm line 88Mercury::Redis::Hash::Base::put('Mercury::Redis::Hash::ClusterStatus=HAS H(0x7ffa6bccb1f8)', 'amc.service_watcher', 'HASH(0x7ffa6bd37808)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/ServiceWatcher.pm line 246

This above error, will occur due to Redis not being able to start, If we look into redis.log, located at /var/log/redis/redis.log, we could see the below error:

[13180] 11 Nov 12:19:55.761 # Unable to set the max number of files limit to 10032 (Operation not permitted), setting the max clients configuration to 3984

Your disk is full so when redis tries to create a db file it fails because there is no space left on the disk and it creates zero sized db file. Starting redis fails even after we getting some space back, because of zero sized db file, which got created in CentOS at the following file path "/var/lib/redis/dump.rdb". This is an issue with Redis versions currently airwave has. However, In newer versions of redis this bug is fixed, if you use older version of redis simply removing dump.rdb will work. But do this if dump.rdb file size is zero, otherwise do not because you might lose data.

To fix this issue, all we need to do is, move the dump.rdb from /var/lib/redis/dump.rdb to /tmp, by using the command below:

# mv /var/lib/redis/dump.rdb /tmp/

and execute the below command to restart the redis server.

# service redis restart

Note: we need to execute the above commands, only after confirmation that we have free space on the server.