Large memory leak with ssl_peek (now partly understood)

Large memory leak with ssl_peek (now partly understood)

I've been suffering from a significant memory leak on multiple servers
running Squid 3.5 for months, but was unable to reproduce it in a test
environment. I've now figured out how to reproduce it and have done
some investigation:

When using TPROXY, Squid generates fake "CONNECT 192.0.2.1:443"
requests, using the IP address that the client connected to. At
ssl_bump step 1, we peek and Squid generates another fake "CONNECT
example.com:443" request containing the SNI from the client's SSL handshake.

At ssl_bump step 2 we splice the connection and Squid does verification
to make sure that example.com does actually resolve to 192.0.2.1. If it
doesn't, Squid is supposed to reject the connection in
ClientRequestContext::hostHeaderVerifyFailed() to prevent clients from
manipulating the SNI to bypass ACLs.

Unfortunately, when verification fails, rather than actually dropping
the client's connection, Squid just leaves the client hanging.
Eventually the client (hopefully) times out and drops the connection
itself, but the associated ClientRequestContext is never destroyed.

That is a traffic pattern that we see in the real world and is now
clearly what is triggering the leak: Apple devices make connections to
addresses within the 17.0.0.0/8 network with an SNI of
"courier.push.apple.com". courier.push.apple.com resolves to a CNAME
pointing to courier-push-apple.com.akadns.net, but
courier-push-apple.com.akadns.net doesn't exist. Since Squid can't
verify the connection, it won't allow it and after 30 seconds the client
times out. Each Apple device keeps retrying the connection, leaking a
ClientRequestContext each time, and before long we've leaked several
gigabytes of memory (on some networks I'm seeing 16GB or more of leaked
RAM over 24 hours!).

Unfortunately I'm a bit lost in the Squid code and can't quite figure
out how to gracefully terminate the connection and destroy the context.

Re: Large memory leak with ssl_peek (now partly understood)

On 08/11/2016 10:56 AM, Steve Hill wrote:

> At ssl_bump step 2 we splice the connection and Squid does verification
...
> Unfortunately, when verification fails, rather than actually dropping
> the client's connection, Squid just leaves the client hanging.

Hi Steve,

This sounds very similar to Squid bug 4508. Factory proposed a fix
for that bug, but the patch is for Squid v4. You may be able to adapt it
to v3. Testing (with any version) is very welcomed, of course:

> At ssl_bump step 2 we splice the connection and Squid does verification...> Unfortunately, when verification fails, rather than actually dropping
> the client's connection, Squid just leaves the client hanging.

Hi Steve,

This sounds very similar to Squid bug 4508. Factory proposed a fix
for that bug, but the patch is for Squid v4. You may be able to adapt it
to v3. Testing (with any version) is very welcomed, of course:

Re: Large memory leak with ssl_peek (now partly understood)

> This sounds very similar to Squid bug 4508. Factory proposed a fix
> for that bug, but the patch is for Squid v4. You may be able to adapt it
> to v3. Testing (with any version) is very welcomed, of course:

Thanks for that - I'll look into adapting and testing it.

(been chasing this bug off and on for months - hadn't spotted that there
was a bug report open for it :)