Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

I use Mumble daily as a remote work team communication channel, so it's pretty
much open non-stop for one day at a time.

Since a few days, maybe a week, Mumble's memory usage is steadily growing to
crzay levels: right now, `top` report it's using 25% of my 12Go of memory
(resident memory shows 3g). This situation forces me to restart the client
every now and then, unfortunately at least a few times a day -- and
obviously is also a problem for overall use of resources.
I wouldn't say it started right away with the new Qt5 Mumble, but it could
have, I'm not completely sure.

It wasn't always like that, and used to work fine.

I cannot really provide credentials for the server, but I can do tests if
need be. There is no issue I can see on Mumble's stdout/stderr:

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Colomban Wendling:
> Package: mumble
> Version: 1.3.0~git20190125.440b173+dfsg-1
> Severity: important
>
> Dear Maintainer,
>
> I use Mumble daily as a remote work team communication channel, so it's pretty
> much open non-stop for one day at a time.

Understood.
I typically also leave the Mumble client open for long periods of time, but have
not tried doing this with Mumble 1.3 yet. When possible I'll try to replicate
this bug to see if I see the same behavior.

> Since a few days, maybe a week, Mumble's memory usage is steadily growing to
> crzay levels: right now, `top` report it's using 25% of my 12Go of memory
> (resident memory shows 3g). This situation forces me to restart the client
> every now and then, unfortunately at least a few times a day -- and
> obviously is also a problem for overall use of resources.
> I wouldn't say it started right away with the new Qt5 Mumble, but it could
> have, I'm not completely sure.
>
> It wasn't always like that, and used to work fine.

I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
sure what the underlying cause/issue was there, or what version of Mumble may
have fixed that bug.

Due to recent security bugs in Mumble I'm going to be preparing a version of
Mumble 1.3 with Qt4 (targeted for Stretch) -- so there will be an opportunity to
test that to see if it shows the same memory leak.

I don't personally know of a "good" way to debug memory leaks. As far as I know
this involves running the target program via a debugger like GDB and figuring
out how to watch memory allocations and frees. Unfortunately I don't have any
experience with doing that [successfully] yet.

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Le 07/02/2019 à 16:56, Chris Knadle a écrit :
> […]
>
> Just to mention: this isn't the first report of "memory leak" on Mumble -- see
> Debian bug #683827.
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683827>
> I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
> sure what the underlying cause/issue was there, or what version of Mumble may
> have fixed that bug.

The links you have there are interesting; for example
https://github.com/mumble-voip/mumble/issues/1074#issuecomment-203199996suggests that it might be due to echo canceller and other apps "messing"
with PulseAudio. I do have echo canceller enabled (that I should
actually be able to disable as I'm using push-to-talk with a headset),
and am running at least one virtual machine which could be doing
something with PulseAudio.

I'll try and do some tests next chance I get (probably next week).

> Due to recent security bugs in Mumble I'm going to be preparing a version of
> Mumble 1.3 with Qt4 (targeted for Stretch) -- so there will be an opportunity to
> test that to see if it shows the same memory leak.

I can also test if reverting to a previous version of Mumble helps; I'm
not sure which are the security issues but in my case I trust the server
so there might be no problem using an older version for testing.

> […]
>
> I don't personally know of a "good" way to debug memory leaks. As far as I know
> this involves running the target program via a debugger like GDB and figuring
> out how to watch memory allocations and frees. Unfortunately I don't have any
> experience with doing that [successfully] yet.

Valgrind's memcheck is the best I know. I'll try to see if I can run
Mumble in it and find out what I can from there -- although it often is
a pain starting from nothing, as many toolkits and apps have gazillions
of innocent leaks cluttering the results.

Thanks for the input, and I'll get back here with any data I can gather
on this.

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Colomban Wendling:

> Le 07/02/2019 à 16:56, Chris Knadle a écrit :
>> […]
>>
>> Just to mention: this isn't the first report of "memory leak" on Mumble -- see
>> Debian bug #683827.
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683827>>
>> I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
>> sure what the underlying cause/issue was there, or what version of Mumble may
>> have fixed that bug.
>
> The links you have there are interesting; for example
> https://github.com/mumble-voip/mumble/issues/1074#issuecomment-203199996> suggests that it might be due to echo canceller and other apps "messing"
> with PulseAudio. I do have echo canceller enabled (that I should
> actually be able to disable as I'm using push-to-talk with a headset),
> and am running at least one virtual machine which could be doing
> something with PulseAudio.
>
> I'll try and do some tests next chance I get (probably next week).

*nod* Okay. Yeah the echo canceler would be part of the Mumble binary, so if
the canceler is misbehaving memory-wise then that would make sense. However if
another virtual machine were causing issues with PulseAudio then that shouldn't
cause memory expansion/leaking in the Mumble client binary (AFAIK).

>> Due to recent security bugs in Mumble I'm going to be preparing a version of
>> Mumble 1.3 with Qt4 (targeted for Stretch) -- so there will be an opportunity to
>> test that to see if it shows the same memory leak.
>
> I can also test if reverting to a previous version of Mumble helps; I'm
> not sure which are the security issues but in my case I trust the server
> so there might be no problem using an older version for testing.

The two recent security issues relate to mumble-server specifically, not the
client. One security issue has been resolved, another is described in Debian
bug #919249 and on this Debian Secuirty page for CVE-2018-20743:

>> […]
>>
>> I don't personally know of a "good" way to debug memory leaks. As far as I know
>> this involves running the target program via a debugger like GDB and figuring
>> out how to watch memory allocations and frees. Unfortunately I don't have any
>> experience with doing that [successfully] yet.
>
> Valgrind's memcheck is the best I know. I'll try to see if I can run
> Mumble in it and find out what I can from there -- although it often is
> a pain starting from nothing, as many toolkits and apps have gazillions
> of innocent leaks cluttering the results.

*headsmack* I keep forgetting about Valgrind. I can try playing with that too,
assuming I can replicate the memory leak.

> Thanks for the input, and I'll get back here with any data I can gather
> on this.

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Le 08/02/2019 à 16:54, Chris Knadle a écrit :

> Colomban Wendling:
>> Le 07/02/2019 à 16:56, Chris Knadle a écrit :
>>> […]
>>>
>>> Just to mention: this isn't the first report of "memory leak" on Mumble -- see
>>> Debian bug #683827.
>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683827>>>
>>> I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
>>> sure what the underlying cause/issue was there, or what version of Mumble may
>>> have fixed that bug.
>>
>> The links you have there are interesting; for example
>> https://github.com/mumble-voip/mumble/issues/1074#issuecomment-203199996>> suggests that it might be due to echo canceller and other apps "messing"
>> with PulseAudio. I do have echo canceller enabled (that I should
>> actually be able to disable as I'm using push-to-talk with a headset),
>> and am running at least one virtual machine which could be doing
>> something with PulseAudio.
>>
>> I'll try and do some tests next chance I get (probably next week).
>
> *nod* Okay. Yeah the echo canceler would be part of the Mumble binary, so if
> the canceler is misbehaving memory-wise then that would make sense.

I tried without the echo canceller and it's behaving reasonably so far,
after 5 hours: it's using 109M resident memory, and peaked at 123M.
So it seems it's indeed the echo canceller that's leaking a lot.

> However if
> another virtual machine were causing issues with PulseAudio then that shouldn't
> cause memory expansion/leaking in the Mumble client binary (AFAIK).

IIUC from the report I linked, some software alter the input sinks in a
way that confused the echo canceller. It'd say it's still a bug on the
canceller part, but it might be triggered only on some conditions that
don't normally happen, but that some software doing their own audio
input trigger -- not quite sure.

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

> Le 08/02/2019 à 16:54, Chris Knadle a écrit :
>> Colomban Wendling:
>>> Le 07/02/2019 à 16:56, Chris Knadle a écrit :
>>>> […]
>>>>
>>>> Just to mention: this isn't the first report of "memory leak" on Mumble -- see
>>>> Debian bug #683827.
>>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683827>>>>
>>>> I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
>>>> sure what the underlying cause/issue was there, or what version of Mumble may
>>>> have fixed that bug.
>>>
>>> The links you have there are interesting; for example
>>> https://github.com/mumble-voip/mumble/issues/1074#issuecomment-203199996>>> suggests that it might be due to echo canceller and other apps "messing"
>>> with PulseAudio. I do have echo canceller enabled (that I should
>>> actually be able to disable as I'm using push-to-talk with a headset),
>>> and am running at least one virtual machine which could be doing
>>> something with PulseAudio.
>>>
>>> I'll try and do some tests next chance I get (probably next week).
>>
>> *nod* Okay. Yeah the echo canceler would be part of the Mumble binary, so if
>> the canceler is misbehaving memory-wise then that would make sense.
>
> I tried without the echo canceller and it's behaving reasonably so far,
> after 5 hours: it's using 109M resident memory, and peaked at 123M.
> So it seems it's indeed the echo canceller that's leaking a lot.

Okay. Thank you very much for testing and finding this, as it helps narrow down
where the problem is.

I'm re-titling the bug so that it will be easier for others to find when looking
to find information about it.

>> However if
>> another virtual machine were causing issues with PulseAudio then that shouldn't
>> cause memory expansion/leaking in the Mumble client binary (AFAIK).
>
> IIUC from the report I linked, some software alter the input sinks in a
> way that confused the echo canceller. It'd say it's still a bug on the
> canceller part, but it might be triggered only on some conditions that
> don't normally happen, but that some software doing their own audio
> input trigger -- not quite sure.

*nod* Hmm. So I think the theory there is that the because other VMs interact
with PulseAudio that it changes the interaction between PulseAudio and the
Mumble echo canceler such that the echo canceler uses increasing memory. It
feels like that "shouldn't happen", but then again that's the definition of a
"bug" and this is a bug, so... maybe it's possible.

I had a look at the open issues for mumble and found #3379 which is about Mumble
echo cancellation on Windows:

I'm marking this bug as related to Mumble issue #3379 upstream and adding an
entry to the upstream bug pointing to this one so that upstream can see that
this is likely affecting other OSes than Windows.