Memory measurement

Methods and limitations

We use various tools to measure memory usage in JortJams. Top-level measurements
are usually obtained using
Massif, which measures heap
usage over time. JortJams also has an allocator that logs a message every time
an allocation occurs.

The Massif measurements are run on test harnesses and on the main programs.
Massif on the main programs is run as follows:

The fetcher is run under massif by itself, starting from a database
containing no data except 16 sources. The fetcher is allowed to run for a
few minutes.

The GUI is run under massif, and typical actions are done: songs played,
lists scrolled, etc.

The benchmark setup is a laptop with a Skylake-era Intel Core i3, an Intel HD
Graphics 520 GPU, and 16 GB of RAM running 64-bit Ubuntu 18.04. The graphical
environment is KDE; the window system is Xorg.

(More consistent benchmarking procedures would be a good thing, and is
something that we would like to tackle in the future.)

The measurement tools in JortJams do not consider the size of shared libraries,
which can be quite large, especially for the AppImage distribution of JortJams.
If you use (e.g.) htop or top to view JortJams GUI memory use, you may find
that the resident size of the GUI is around 90 MB, whereas here it’s reported
as ~60 MB. The resident size of the fetch process is much smaller because it
can reuse memory allocated for shared libraries.

The solution for this problem is to use fewer libraries (unfortunately
difficult due to the decision to use Qt) or link against copies of libraries
already on the system, which have a higher chance of being loaded in memory
before JortJams starts up. That may be a possibility as more Linux
distributions adopt newer Qt versions.

Fetcher memory graph

The fetcher’s memory usage hovers around 15 MB, though it is not bounded: it is
possible for very large incoming statuses to cause the fetcher to request large
amounts of memory. JortJams needs to be modified to reject very large
responses from servers; before we do that, we need to understand what “very
large” means.

GUI memory graph

The GUI uses several times more memory than the fetcher, and the memory graph
only shows part of the story. Large amounts of the GUI memory are allocated by
i965_dri.so, which is part of Intel’s GPU driver. (The benchmarks are run on
a system using an Intel HD Graphics 520 GPU.) So there’s some work to be done
here to understand these GPU driver allocations and what can be done to reduce
them.