Squid 3.2.13 release notes

Squid Developers

This document contains the release notes for version 3.2 of Squid.
Squid is a WWW Cache application developed by the National Laboratory
for Applied Network Research and members of the Web Caching community.

A large number of the show-stopper bugs have been fixed along with general improvements to the IPv6 support.
While this release is not fully bug-free we believe it is ready for use in production on many systems.

Squid locates the authority-URL details available in an HTTP request as
defined by RFC 2616 and validates that all found representations are
textually equivalent. In the case of intercepted traffic the
client destination IP is also compared to the Host: authority domains
DNS entries.

When the Host: authority contradicts another authority source Squid will log
"SECURITY ALERT: Host: header forgery detected". The response will then be determined
by the
host_verify_strict
directive. Squid will respond with 409 Conflict error response when strict validation
fails and handles the request normally when strict validation succeeds or is OFF (default).

Relaying of messages which FAIL non-strict Host: validation are permitted through Squid but
only to the original destination IP the client was requesting or to explicit peers. This means
DNS lookups to locate alternative DIRECT destinations will not be done.

Known Issue: When non-strict validation fails Squid will relay the request, but can only do
so safely to the original destination IP the client was contacting. The client original
destination IP is lost when relaying to peers in a hierarchy. This means the upstream peers
are still at risk of causing same-origin bypass CVE-2009-0801 vulnerability.
Developer time is required to implement safe transit of these requests.
Please contact squid-dev if you are able to assist or sponsor the development.

The DES algorithm used by the NCSA Basic authentication helper has an
limit of 8 bytes but some implementations do not error when truncating
longer passwords down to this unsafe level.

This both significantly lowers the threshold of difficulty decrypting
captured password files and hides from users the fact that the extra bits
of their chosen long password is not being utilized.

The NCSA helper bundled with Squid will prevent passwords longer than 8
characters being sent to the DES algorithm. The MD5 hash algorithm which
supports longer than 8 character passwords is also supported by this helper
and should be used instead.

The new "workers" squid.conf option can be used to launch multiple worker
processes and utilize multiple CPU cores. The overall intent is to make
multiple workers look like one to an outside observer, while providing
knobs to customize each worker behavior if needed.

By default, all worker processes are configured identically and do what a
single Squid instance would have done. Squid.conf macro substitutions and
conditionals (see below) can be used to customize individual worker
configurations. In the paragraphs below, "can share" implies "will share by
default".

Workers can share HTTP, HTTPS, SNMP, ICP, and HTCP listening addresses.
Configuration related to ICP and HTCP clients must be adjusted to avoid
source address conflicts: Modify the IP address and/or the port used for
the protocol. Workers do not share DNS addresses by default because the OS
assigns each worker a unique DNS port.

Workers can share logs.

Workers can share caches. Memory cache is automatically shared when multiple
workers are used. Cache_dir are shared when configured with the rock
storage type. Cache_dir of other types must be adjusted to point each
disk-caching worker to its own disk area. ICP and HTCP responses are based
on the responding worker cache state.

Cache manager statistics are reported from a worker point of view, for now.
Though some reports are combined. SNMP statistics are combined across all
workers.

Startup, reconfiguration, shutdown, and log rotation are handled as for a
monolithic Squid. Abnormally terminated workers are restarted while
other workers continue serving traffic.

Squid.conf macros and conditionals

Added support for process_name and process_number macros as well as simple
if-statement conditionals in squid.conf. These features allow individual
worker customization in SMP mode. For details, search for "Conditional
configuration" and "SMP-Related Macros" sections in squid.conf.documented.

The helper multiplexer's purpose is to relieve some of the burden
Squid has when dealing with slow helpers. It does so by acting as a
middleman between squid and the actual helpers, talking to Squid via
the multiplexed concurrent variant of the helper protocol and to the
helpers via the non-concurrent variant.

Helpers are started on demand, and in theory the muxer can handle up to
1k helpers per instance. It's up to squid to decide how many helpers
to start.

The muxer knows nothing about the actual messages being passed around,
and as such can't really (yet?) compensate for broken helpers.
It is not yet able to manage dying helpers, but it will.

To configure the multiplexer add its binary name (usually /usr/share/libexec/helper-mux.pl)
in front of the name of whichever helper is being multiplexed. It takes the helper binary
path and parameters as its own command parameters. The concurrency setting already
existing in Squid is used to configure how many child helpers it may run.

Helpers which are already concurrent protocol enabled gain little benefit from the multiplexer
on most systems. However on some systems where Squid spawning helpers causes excess memory usage
the reduction in direct helper spawned by Squid can result in a great reduction in resource use.

Traditionally Squid has been configured with a fixed number of helpers and started them during
it's start and reconfigure phases. This forces the hard configuration problem of how many helpers
will be needed to be solved before starting Squid in production use.

The on-demand helpers feature allows greater flexibility and resolves this problem by allowing
maximum, initial and idle thresholds to be configured. Squid will start the initial set during
start and reconfigure phases. However over the operational use new helpers up to the maxium will
be started as load demands. The idle threshold determines how many more helpers to start if the
currently running set is not enough to handle current request loads.

The example still permits up to 200 helpers to be running at once under peak traffic loads.
But only starts 10 when Squid is initialized resulting in a faster boot up.
When client requests threaten to overload the running helpers an additional 2 will be started.

NOTE: if no startup and idle values are specified the traditional behaviour
of starting the maximum number of helpers will occur.

squid_session - ext_session_acl - Maintain a session cache of client identifiers (usually IP address).
This helper has also gone through a version update and now uses more current BerkeleyDB 4.1+ APIs.

The Surrogate extensions to HTTP protocol enable an origin web server to specify separate
cache controls for a reverse proxy acting on its behalf. Previously this was closely tied with the ESI
feature support in Squid. This release opens Surrogate support to all reverse proxies.

Reverse proxy requests sent on to the web server include the HTTP header Surrogate-Capabilities:
specifying the capabilities of the reverse proxy along with an ID which can be used to target responses with
a Surrogate-Control: HTTP header used instead of the Cache-Control: header.

The default surrogate ID is generated automatically from the Squid site-unique hostname as found by the
automatic detection or manual configuration of visible_hostname although can be configured
separately with the httpd_accel_surrogate_id option.

Security Considerations: Websites should be careful of accepting any surrogate ID.
Older releases of Squid leak the Surrogate-Control headers to external servers.
This 3.2 series of Squid will now prevent this leakage of its own ID destined responses, however it is possible
and for some uses desirable to receive external reverse-proxies Surrogate-Capabilities: headers.

NOTE: Several operating system distributions historically package Squid with a forced value of
visible_hostname localhost. If this is done on a Surrogate enabled install a manual re-configuration
is required to prevent an unacceptable surrogate ID of 'localhost' being generated.

The advanced logging modules introduced in Squid-2.7 are now available from Squid-3.2.

This feature is documented at http://wiki.squid-cache.org/Features/LogModules

The new infrastructure currently supports several different channels types (modules) ranging from
direct filesystem logging (stdio, daemon) to network logging (syslog, UDP and TCP). The daemon logging
interface allows for a custom helper to be written to process logs in real-time.

Upgrading: the access_log and cache_store_log were previously logged via what is
now called the stdio module.
This is still supported and used by default if no module is named. For best performance particularly in SMP
environments we recommend the daemon be used. The provided log_file_daemon helper
performs the traditional logging to local filesystem.

Additional to this the cache.log can now be limited to a smaller number of files stored.
Traditionally cache.log.N has been fixed at the same number of rotated files as access.log.N through the
logfile_rotate setting. The debug_options setting can now be used to configure the number
of debug cache.log files to rotate through with a rotate=N option. This is particularly useful for
logging a single cache.log at relatively high debug levels on a high-traffic system. Or one which is
required to store a long period of access.log and needs to conserve disk space.

The referer_log and useragent_log directives have been converted to built-in log formats.
These logs are now created using an access_log line with the format "referrer" or "useragent".
They also now log all client requests, if there was no Referer or User-Agent header a dash (-) is logged.

Known Issue: The TCP logging module does not recover from broken connections well.
At present it will restart the affected Squid instance if the TCP connection is broken.

In mobile environments, Squid may need to limit Squid-to-client bandwidth
available to individual users, identified by their IP addresses. The IP
address pool can be as large as a /10 IPv4 network (4 million unique IP
addresses) and even larger in IPv6 environments. On the other hand, the code
should support thousands of connections coming from a single IP (e.g.,
a child proxy).

The implementation is based on storing bandwidth-related "bucket" information
in the existing "client database" hash (client_db.cc). The old code already
assigned each client IP a single ClientInfo object, which satisfies the
client-side IP-based bandwidth pooling requirements. The old hash size is
increased to support up to 32K concurrent clients if needed.

Client-side pools are configured similarly to server-side ones, but there is
only one pool class. See client_delay_pools,
client_delay_initial_bucket_level, client_delay_parameters, and
client_delay_access in squid.conf. The client_delay_access matches the client
with delay parameters. It does not pool clients from different IP addresses
together.

Special care is taken to provide fair distribution of bandwidth among clients
sharing the same bucket (i.e., clients coming from the same IP address).
Multiple same-IP clients competing for bandwidth are queued using FIFO
algorithm. If a bucket becomes empty, the first client among those sharing
the bucket is delayed by 1 second before it can attempt to receive more
response data from Squid. This delay may need to be lowered in
high-bandwidth environments.

The Squid Cache Manager has previously only been accessible under the cache_object://
URL scheme. Which has restricted its reporting to tools which can send arbitrary
URI to the proxy.

This version of Squid now provides access through the http:// and https:// URL schemes
allowing web browsers access without having to use the cachemgr.cgi gateway and enabling
the use of HTTPS security were desired.

The cache manager is available under the path prefix /squid-internal-mgr/. For example
the URL http://example/com/squid-internal-mgr/menu will bring up the manager menu. This
means there are some configuration changes required to lock down manager access.
The manager ACL needs changing. A built-in definition is now used, equivalent
to the following regex pattern:

^(cache_object://|https?://[^/]+/squid-internal-mgr/)

The manager prefix /squid-internal-mgr/ with no action attempts to load an optional
template MGR_INDEX which may be installed amongst in the Squid error templates.
This template is not supplied with Squid but intended to be supplied by separate
cache manager applications as their front page embedding all scripts, accessors or
redirects required for their initial GUI display.

MGR_INDEX file

should contain a complete HTML page, with optional client-side scripting.

must not contain server-side scripting.

will have macro substitution performed on it using the same macros as used by the error page templates.

Version 3.2 of the CGI cache manager tool now presents XHR scripted probes to detect
proxies presenting these manager index pagess and provides direct HTTP/HTTPS web links
to those managers.

IMPORTANT: disabling this directive only allows Squid to change the
destination IP to another source indicated by Host: domain DNS or
cache_peer configuration. It does not affect Host: validation.

client_idle_pconn_timeout

Renamed from persistent_request_timeout.

cpu_affinity_map

New setting for SMP support to map Squid processes onto specific CPU cores.

connect_retries

Replacement for maximum_single_addr_tries, but instead of only applying to hosts with single addresses.
This directive applies to all hosts, extending the number of connection attempts to each IP address.

dns_packet_max

New setting to configure maximum number of bytes packet size to advertise via EDNS.
Set to "none" (the initial default) to disable EDNS large packet support.

New conditional syntax for SMP multiple-worker.
If-statements can be used to make configuration directives depend on conditions.

The else part is optional. The keywords if, else and endif
must be typed on their own lines, as if they were regular configuration directives.

logfile_daemon

Ported from 2.7. Specify the file I/O daemon helper to run for logging.

max_stale

Places an upper limit on how stale content Squid will serve from the cache if cache validation fails

memory_cache_mode

Controls which objects to keep in the memory cache (cache_mem)

'always' Keep most recently fetched objects in memory (default)
'disk' Only disk cache hits are kept in memory, which means
an object must first be cached on disk and then hit
a second time before cached in memory.
network Only objects fetched from network is kept in memory

memory_cache_shared

Controls whether the memory cache is shared among SMP workers.

Currently, entities exceeding 32KB in size cannot be shared.

server_idle_pconn_timeout

Renamed from pconn_timeout.

tproxy_uses_indirect_client

Controls whether the indirect client address found in the X-Forwarded-For
header is used for spoofing instead of the directly connected client address.
Requires both --enable-follow-x-forwarded-for and --enable-linux-netfilter

workers

Number of main Squid processes or "workers" to fork and maintain.
In SMP mode, each worker does nearly all what a single Squid daemon
does (e.g., listen on http_port and forward HTTP requests).

New stdio module to send log data directly from Squid to a disk file.
This is the historic behaviour of Squid before logging modules were introduced, and
remains the default used when no module is selected.
It is recommended to upgrade logging to the faster daemon: module.

New daemon module to send each log line as text data to a file I/O daemon handling the slow disk I/O.
New installs, or installs with no logs configured explicitly will use this module by default.

New tcp module to send each log line as text data to a TCP receiver.

New udp module to send each log line as text data to a UDP receiver.

New format referrer to log with the format previously used by referer_log directive.

New format useragent to log with the format previously used by useragent_log directive.

acl : random, urllogin

New type random. Pseudo-randomly match requests based on a configured probability.

Ported urllogin option from Squid 2.7, to match a regex pattern on the URL login field (if any).

The manager ACL requires adjustment to cover new cache manager access. So it has now been
built-in as a predefined ACL name matching URLs equivalent to the following regular expression:

^(cache_object://|https?://[^/]+/squid-internal-mgr/)

squid.conf containing the old manager definition can expect to see ACL type collisions.

auth_param

New options for Basic, Digest, NTLM, Negotiate children settings.
startup=N determines minimum number of helper processes used.
idle=N determines how many helper to retain as buffer against sudden traffic loads.
concurrency=N previously called auth_param ... concurrency as a separate option.

children-idle=N determines how many helper to retain as buffer against sudden traffic loads.

Deprecated children=N in favor of children-max=N.

http_port act-as-origin vhost no-vhost

act-as-origin ported from 2.7.
This option corrects several HTTP header issues when operating as a reverse proxy and cache.
Notably the externally visible aging of objects stored in the server-side cache.

Deprecated in favor of adaptation_send_client_ip
which applies to both ICAP and eCAP.

icap_send_client_username

Deprecated in favor of adaptation_send_username
which applies to both ICAP and eCAP.

icap_uses_indirect_client

Deprecated in favor of adaptation_uses_indirect_client
which applies to both ICAP and eCAP.

logformat

%<a Server or Peer IP address from the last server connection (next hop).

%>bs Number of HTTP-equivalent message body bytes received from the next hop.

icap::%>bs Number of message body bytes received from the ICAP server.

%sn Unique sequence number per log line. Ported from 2.7

%>eui EUI logging (EUI-48 / MAC address for IPv4, EUI-64 for IPv6).
Both EUI forms are logged in the same field. Type can be identified by length or byte delimiter.

%err_code The ID of an error response served by Squid or a similar internal error identifier

%err_detail Additional err_code-dependent error information.

%>la Rename of %la to indicate being a client connection detail.

%>lp Rename of %lp to indicate being a client connection detail.

%<p Server or Peer port number from the last server connection (next hop).

memory_pools_limit

Memory limits have been revised and corrected from 3.1.4 onwards.

Please check and update your squid.conf to use the text none for no limit instead of the old 0 (zero).

All users upgrading need to be aware that from Squid-3.3 setting this option to 0 (zero) will mean zero bytes of memory get pooled.

qos_flows

New options mark and tos and miss

tos retains the original QOS functionality of the IP header TOS field.

mark offers the same functionality, but with a netfilter mark value.

These options should be placed immediately after qos_flows.

The tos value is optional in order to maintain backwards compatability.

The preserve-miss functionality is available with the mark option and requires no kernel patching.
It does, however, require libnetfilter_conntrack.
This will be included by default if available (see the --without-netfilter-conntrack configure option for more details).

miss sets a value for a cache miss. It is available for both the tos and mark options and takes precedence over the preserve-miss feature.

range_offset_limit

Added ACL support for control over when the limit applies and when it is avoided.

refresh_pattern

New option max-stale= to provide a maximum staleness factor. Squid won't
serve objects more stale than this even if it failed to validate the object.

With an explicit empty list ="" protocol support will be built but no helpers.

With an explicit list protocol support and just those helpers will be built.

--enable-auth-digest[=HELPERS]

Specified without any parameters all helpers will be auto-built.

With an explicit empty list ="" protocol support will be built but no helpers.

With an explicit list protocol support and just those helpers will be built.

--enable-auth-negotiate

Specified without any parameters all helpers will be auto-built.

With an explicit empty list ="" protocol support will be built but no helpers.

With an explicit list protocol support and just those helpers will be built.

--enable-auth-ntlm

Specified without any parameters all helpers will be auto-built.

With an explicit empty list ="" protocol support will be built but no helpers.

With an explicit list protocol support and just those helpers will be built.

--enable-build-info

Add an additional string in the output of "squid -v".

--enable-eui

Enable Support for handling EUI operations.
This includes ARP lookups for MAC (EUI-48) addresses and the ACL arp type tests.

--enable-log-daemon-helpers

Build helpers for logging I/O.

--enable-url-rewrite-helpers

Build helpers for some basic URL-rewrite actions. For use by url_rewrite_program.
If omitted or set to =all then all bundled helpers that are able to build will be built.
If set to a specific list of helpers then only those helpers will build.
Currently one demo helper fake is provided in shell and C++ forms to demonstrate
the helper protocol usage and provide exemplar code.

--with-swapdir=PATH

Location to display in documentation for the default cache.
Updated to indicate /var/cache/squid in accordance with the filesystem layout standards.
Squid-3 no longer builds an implicit disk cache at this location, so the change is not expected
to have any effect on existing builds other than fixing some mysterious lack of core dumps.
The old /var/cache location was often non-writable which blocked core dumps creation.

--without-netfiler-conntrack

Disables the libnetfilter_conntrack library being used for the new qos_flows option mark.
default is to auto-detect the library and use where available.

Obsolete. The experimental actions enabled in 2.7 by this option have been integrated as default
actions for the rock storage type and memory caches.
The configuration option is no longer necessary and has been dropped.
NOTE: It is not yet supported by ufs, aufs, or diskd storage.