Fontconfig, the library used by freetype (and thus libavg) to manage fonts, builds a cache of font information at ~/.fontconfig. Generating this cache can take about half a minute, depending on the number of fonts you have installed. Usually, this happens only once during the first libavg install, but if libavg can't write to ~/.fontconfig, it'll regenerate the cache information on every run. The solution is to make ~/.fontconfig writeable - for instance, by deleting the directory.