Project description

Varnish recipe for buildout

plone.recipe.varnish is a zc.buildout recipe to install Varnish. Even
though the name contains the name Plone, there is nothing Plone-specific about
this recipe: it works for non-Zope sites just as well.

which generates the VCL configuration file,
sending requests to a backend at 127.0.0.1:8081, and

varnish-script

which runs Varnish, configured to listen on 127.0.0.1:8000 for requests,
using a 512 megabyte cache.

A wrapper script for the varnish startup command is created in the bin
directory of your buildout.

Please note that the configuration generated by this recipe requires Varnish
4.0.x or later. The URL provided by the recipe points to the
latest tested Varnish release compatible with the generated configuration.

Virtual hosting

Varnish supports virtual hosting by selecting a different backend server
based on headers on the incoming request. You can configure the backends
through the backends option:

This will generate a configuration which sends all traffic for the plone.org
host to a backend server running on port 8000 while all traffic for the
plone.net host is send to port 9000.

Zope 2 hosting (with Virtual Host Monster)

If you are using Zope 2 as backend server you will need to rewrite the URL
so the Zope Virtual Host Monster (VHM) can generate correct links for links in
your pages. This can be done either by a web server such as Apache or nginx
(placed either in front or behind Varnish) but can also be done by Varnish itself.

The three options are described below.

Option 1 (rewrites after Varnish)

If generating these VHM-style URLs in a proxy behind Varnish (or if using
VHM’s ‘mapping’ feature), no extra Varnish configuration is needed.
Just make sure the backends option directs the traffic to the proxy.

Option 2 (rewrites before Varnish)

If generating these VHM-style URLs in a proxy in front of Varnish, no extra
Varnish configuration is needed as long as the original hostname is still retained
in the URL. If the hostname is not retained, you can tell Varnish to direct requests
based on the “path” instead of the hostname. For example:

This will generate a configuration which sends all traffic for any request whose
path starts with /VirtualHostBase/http/plone.org:80/Plone to a backend server
running at 127.0.0.1 on port 8000, while request paths starting with
/VirtualHostBase/http/plone.net:80/Plone are sent to port 9000.

Option 3 (rewrites within Varnish)

To have Varnish generate these VHM-style URLs, you can use the
zope2_vhm_map option.
Here is an example:

This tells us that the domain plone.org should be mapped to the location
/plone in the backend. By combining this with the information from the
backends option a varnish configuration will be generated that
maps URLs correctly.

Load Balancing

Varnish supports load balancing by configuring a director for a pool of backends.
This director sends the incoming requests that cannot be fulfilled by varnish to
backends in the pool in either random or round robin fashion. You can configure
the director via the balancer option:

[varnish-configuration]
balancer = random

This will generate a configuration which sends all traffic to the director,
which will choose a ‘random’ backend server to fulfill the request if the
content requested is not cached by varnish itself.

plone.recipe.varnish reference

The plone.recipe.varnish recipe does one or more of the following:

plone.recipe.varnish:build

compiles varnish from sources

plone.recipe.varnish:configuration

generates a VCL-configuration file

plone.recipe.varnish:script

generates a wrapper script inside your buildout that will start Varnish
with the correct configuration.

Please note that this recipe requires Varnish 4.0.x or later.

Build varnish from sources

build is based on
zc.recipe.cmmi - so all
parameters from that recipe are available here too (but rarely used). These options are available for the recipe part plone.recipe.varnish:build.

Two parameters are different/ extra:

url

Location used for download of varnish sources. Defaults to a version tested
for the selected varnish_version.

jobs

Passes the number of parallel jobs to make, defaults to 4. Adjust as
needed to your CPU resources.

varnish_version

Varnish target version. Default is 5.2. Options are:

4.0: uses 4.0.5, will stick to 4.0.x

4.1: uses 4.1.9, will stick to 4.1.x

4: uses 4.1.9, will stick to 4.x

5.0: uses 5.0.0, will stick to 5.0.x

5.1: uses 5.1.3, will stick to 5.1.x

5.2: uses 5.2.1, will stick to 5.2.x

5: uses 5.2.1, will stick to 5.x

The exact version and the default version may be changed in future release of this recipe.

VCL Configuration Generator

These options are available for the recipe part plone.recipe.varnish:configuration.

backends

Specifies the backend or backends which will process the (uncached)
requests. The syntax for backends:

[<hostname>][/<path>]:<ipaddress>:<port>

The optional hostname and path allows you to do virtual hosting.
If multiple backends are specified then each backend must include
either a hostname or path (or both) so that Varnish can direct the
matching request to the appropriate backend. Defaults to
127.0.0.1:8080.

balancer

If included and set to either random or round_robin, this option
configures varnish to load balance the servers specified by the backends
directive. Possible values: none (default), round_robin or
random.

between-bytes-timeout

If specified, this option configures the timeout (in seconds) for Varnish
waiting between bytes when receiving data from a backend. Varnish will only
wait this many seconds between bytes before giving up. A value of 0s means
this will never time out. Defaults to 60s, as per Varnish’s default
settings.

bind

Hostname and port on which Varnish will listen for requests. Defaults
to 127.0.0.1:8000.

connect-timeout

If specified, this option configures the connection timeout (in seconds)
for Varnish connecting to a backend server. Varnish will only try to
connect to a given backend for this many seconds before giving up. Defaults
to 0.4s, as per Varnish’s default settings.

cookie-whitelist

After the cookie-pass was processed this list is used to to sanitize
cookie data on the request. Cookie data to be sent to the backend includes
only cookies with the given namens. Goal is to work better with the
backend, i.e. detect if user is logged in and adjust caching to ensure no
authenticated pages get cached. Defaults are optimized for Zope2/Plone:
statusmessages __ac _ZopeId __cp

cookie-pass

This list consists of lines with a cookie-match and urlexclude in the form:
"cookiematch":"urlexcludes". If cookiematch applies for the cookiename
and the current url does not match urlexcludes, the request is passed
directly to the configured backend bypassing any caching. But if cookie
applies and url matches urlexcludes, then a lookup is forced. Defaults are
optimized for Plone, one line:
"__ac(|_(name|password|persistent))=":"\.(js|css|kss)$"

first-byte-timeout

If specified, this option configures the timeout (in seconds) for Varnish
receiving the first byte from a backend. Varnish will only wait for this
many seconds before giving up. A value of 0s means Varnish will never time
out. Defaults to 300s.

purge-hosts

Specifies hostnames or IP addresses for purge ACL. By default localhost and
the backends are allowed to purge. Additional allowed hosts are listed here.

Enable sending extra headers in responses that expose what varnish
did with the request and the cache status. Useful for debugging
cache settings and optimizations.
Possible values: on or off (default).

zope2_vhm_map

Defines a virtual host mapping for Zope servers. This is a list of
hostname:ZODB location entries which specify the location inside
Zope where the website for a virtual host lives.

zope2_vhm_port

Defines a virtual host mapping port to use in the VHM URL to send back to
clients. Useful if there is another port mapping in front of varnish, such
as haproxy. Defaults to bind port.

zope2_vhm_ssl

If specified, this maps VHM URLs to https for all requests.
Possible values: on or off (default).

zope2_vhm_ssl_port

Defines a virtual host mapping port to use in the VHM URL to send back to
clients. Useful if there is another port mapping in front of varnish, such
as haproxy. Defaults to 443.

vcl-version

Varnish VCL format version.
If not given it defaults to 4.0.

To test the generated configuration for syntactic correctness, run
varnishd -C-f./parts/varnish-configuration/varnish.vcl.

Create script to start varnish

Start varnish as a daemon or in foreground with the given settings. These options are available for the recipe part plone.recipe.varnish:script.

bind

Hostname and port on which Varnish will listen for requests. Defaults
to 127.0.0.1:8000.

build-part

References the buildout part in order to get settings from there. Defaults
to varnish-build. Set it to false in order to switch it off.

cache-location

Customise the location for the Varnish file storage. Option only applicable
when used with file or persistent cache-type options. Defaults to
using a file named storage inside the relevant parts directory
(eg parts/varnish/storage). Changing the default location can be
useful in putting the storage somewhere with quicker read speeds
(e.g. RAM disk).

cache-size

The size of the cache (limited to 2G on 32bit systems). Defaults to
256M.

cache-type

Specify the type of cache storage to use with Varnish.
Possible values: file (storage for each object is allocated from an
arena backed by a file),
malloc (storage for each object is allocated with malloc; in memory),
or persistent (experimental as at Varnish 2.1.4).
Defaults to file.

configuration-file

Path to a Varnish VCL configuration file to use. Defaults to the generated
file from the configuration-part setting.
If no configuration was generated, this setting is mandatory.

configuration-part

Names the buildout part to get settings from.
Defaults to varnish-configuration.

daemon

The file and path of the varnish daemon varnishd to use.
If not given, it looks for the build part
(see the build-part setting)
and uses its location setting plus the string /sbin/varnishd.
If there is no build part, it defaults to /usr/sbin/varnishd - the
most common place
where it’s found on many Unix systems. Adjust it if needed.

grace-healthy

Grace in the context of Varnish means delivering otherwise expired objects
when circumstances call for it. This can happen because:
(1) the backend-director selected is down, or
(2) a different thread has already made a request to the backend that’s
not yet finished.

If the backend is healthy, accept objects that are this number of seconds
old. Clients will be delivered content that is no more than number of
seconds past its TTL.

Format: number followed by a time unit: ms, s, m, h.

Defaults to None. If this is set to None the grace
feature is disabled.

grace-sick

If the backend is sick, accept objects that are this old.
See also grace-healthy.

Format: number followed by a time unit: ms, s, m, h.

Defaults to 600s. Should be greater than grace-healthy.

group

The name of the group that varnish should switch to before accepting any
request. This defaults to the main group for the specified user.

mode

Specify whether the varnish daemon should run in daemon or
foreground mode. The latter is useful when varnish is run by service
supervision tools like daemontools or runit. Defaults to daemon.

name

If specified this sets the name of the varnish instance (defaults to
the host name).

From varnishd’s manpage:

Amongst other things, this name is used to construct the name of the
directory in which varnishd keeps temporary files and persistent state.
If the specified name begins with a forward slash, it is interpreted as
the absolute path to the directory which should be used for this purpose.

runtime-parameters

Runtime parameter configuration options. The full list of available options
can be found in the manpage varnishd(1) for your version of varnish.
Examples include thread_pool_max, thread_pool_min, sess_timeout.

telnet

If specified sets the hostname and port on which Varnish will listen
for commands using its telnet interface.

script-filename

Name of the start script file in buildout:bin-directory.
Defaults to the name of this buildout part.

secret-file

In Varnish 4.X the telnet interface is no longer usable without
authentication by default. A pre shared key mechanism has been put in place
which requires both the varnish daemon and a client connection over telnet
(like the varnishadm tool) to have a shared key to authenticate. By default
if no secret-file is specified, it’s no longer possible to authenticate to
the telnet interface.

To disable this security feature (and go back to the dark Varnish 2 & 3
days) use secret-file = disabled. This is discouraged.

To enable the secret-file, give the path to a file on the filesystem that
preferably has random content and is both accessible to the varnish daemon
and a command line utility like varnishadm.

Giving secret-file the location of this file will pass on the secret to
the varnish daemon when it starts up. Afterwards you can use varnishadm
with the parameters -T host:port -S /path/to/varnish_secret to connect to
the admin telnet interface.

user

The name of the user varnish should switch to before accepting any
requests. Defaults to nobody.

Examples:

Use system varnish at /usr/sbin/varnishd, generate start script in
./bin/varnishd using a VCL-file in
./parts/varnish-configuration/varnish.vcl:

2.0a5 (2016-08-29)

Made three possible values for the varnish_version option. 4.0
(uses 4.0.3), 4.1 (uses 4.1.3), 4 (uses 4.1.3). 4.0 is the default for now.
4 is intended to be updated to 4.2.x when this is released and found
to work.
[maurits]

Fix: to disable the secret-file authentication, an empty parameter should be
passed to varnishd on startup.
[fredvd, nutjob4life]

2.0a4 (2016-02-23)

New: add option for secret-file in the script part so you can communicate to
varnish with varnishadm. See docs for usage and secret-file generation.
[fredvd]

Fix: Split at max on two ‘:’ to get a max of 3 parts as raw_backends
[jensens]

2.0a3 (2015-12-22)

re-release: 2.0a2 was a brown bag release
[jensens]

2.0a2 (2015-12-22)

Fix daemon location of script part of the recipe (/usr/bin/varnishd was
always used.
[fredvd]

Fix tests, download Varnish 4.0.3 as download.
[fredvd]

2.0a1 (2015-03-02)

refactoring and cleanup of the whole recipe and vcl generation:

skip support of varnish < v4.0, use 1.x series for older varnish support.

1.1 (2010-08-05)

1.1b1 (2010-04-25)

Updated advertised Varnish version to 2.1 and adjusted config.
[hannosch]

Correct documentation for the daemon setting and remove the default.
[hannosch]

Removed the deprecated build recipe.
[hannosch]

Added basic test infrastructure and a test for the simple buildout.
[hannosch]

Use the built-in set type instead of the deprecated sets module. This recipe
now requires at least Python 2.4.
[hannosch]

Added the ability to configure runtime parameters in the varnish runner
configuration and added information to the documentation for it.
[benliles]

Improve readability of the generated config.
[ldr]

1.0.2 (2010-01-18)

Update proposed Varnish to 2.0.6.
[hannosch]

Further documentation cleanup.
[hannosch, vincentfretin]

1.0.1 (2009-11-27)

Expose the download-url of a known-good Varnish release that works with
the configuration produced by the instance recipe.
[hannosch]

Consistently use tabs in the generated vcl file.
[hannosch]

Whitespace and documentation cleanup.
[hannosch]

1.0 (2009-08-27)

Made the vcl template build its acl purge section. At present, the vcl will
only allow purges coming from the local host. If we have multiple hosts that
are separate from localhost, any PURGE requests will be denied without this.
See http://varnish.projects.linpro.no/wiki/VCLExamplePurging
[rockdj]

Added ability to set various Varnish timeouts (connect_timeout,
first_byte_timeout, and between_bytes_timeout) from each option in the
buildout. Default values are set at Varnish defaults of 0.4s for
connect_timeout, and 60s for between_bytes_timeout. Time for
first_byte_timeout is set at 300s as per plone.recipe.varnish 1.0rc9.
[rockdj]

1.0rc11 (2009-06-27)

Reintroduced grace options. What the varnish documentation say about grace:
“varnish serves stale (but cacheable) objects while retrieving object from
backend”. The problem is “default_ttl” value is 120s (see
bin/varnishd/mgt_param.c in varnish 2.0.4). Added a special rule for
createObject url to not look up in the cache.
[vincentfretin]

Updated to refer to Varnish 2.0.4. Added a first_byte_timeout value of
300 seconds to the backend definitions. This is a new option since Varnish
2.0.3 and by default set to 60 seconds. This is arguably too low for certain
edit operations in Plone sites.
[hannosch]

1.0rc8 (2008-02-12)

Remove the custom vcl_hash from the template. Adding the Accept-Encoding
header to the cache break effectively breaks purging since nobody will
ever include those headers in a PURGE request. To make this safe we just
remove the Accept-Encoding header from all incoming requests as well.
[wichert]

1.0rc7 (2008-11-26)

Be more explicit about deprecating the :build entry point.
[wichert]

Make the :instance specifier optional: after :build has been removed
we can deprecate :instance as well.
[wichert]

1.0rc6 (2008-09-22)

Deprecate plone.recipe.varnish:build in favour of zc.recipe.cmmi: it does
not make sense to duplicate its logic here.
[wichert]

Add feature to enable verbose headers in varnish.vcl. This is primary
interesting for debugging of cache-settings. See README.txt.
[jensens]

Deal better with sources which do not have executable-bits set or
are svn exports.
[wichert]

The 1.0rc5 release was broken and has been retracted. Currently the trunk
is only usable with the Varnish 2.0-beta1 and later.
[hannosch]

1.0rc5 (2008-04-27)

Pipe is evil: it pipes the whole connection to the backend which means
varnish will no longer process any further requests if HTTP pipelining is
used. Switch to using pass instead.
[wichert]

Add a default_ttl of zero seconds to the Varnish runner to avoid a Varnish
bug with the handling of an Expires header with a date in the past.
[newbery]

Merged branches/newbery-hostnamepath.
[newbery]

We don’t need to include Accept-Encoding in the hash. Varnish takes care
of Vary negotiation already.
[newbery]

1.0rc4 (2008-03-18)

Fixed typos / whitespace.
[hannosch]

Varnish 1.1.2 is out.
[wichert]

Merged witsch-foreground-support back to trunk.
[witsch]

Use a pidfile.
[wichert]

1.0rc3 (2007-09-02)

Fixed a bug where options[“location”] was being used before it was being set.
[rocky]

Made the module name determination a little more robust during
createVarnishConfig so that recipes that specify version deps still work.
[rocky]

Do not use defaults for user and group.
[wichert]

We do need the parts: we use it for the file storage.
[wichert]

1.0rc2 (2007-08-29)

Add an option to use an existing configuration file.
[wichert]

Remove hardcoded caching for images, binaries, CSS and javascript. This
should be done by the backend server or a custom varnish configuration.
[wichert]

Add Accept-Encoding to the cache key so we can handle compressed content.
[wichert]

Test if a bin-directory exists. This allows us to compile varnish 1.0
which does not have an sbin directory.
[wichert]

1.0rc1 (2007-08-27)

Document the OSX bugfix we apply when building varnish.
[wichert]

Add a dummy update method to prevent needless recompiles.
[wichert]

Update for Varnish 1.1.1.
[wichert]

1.0b2 (2007-08-25)

When building from svn, we need to run autogen.sh.
[optilude]

Refactor the recipe: there are now separate recipes to build and configure
Varnish. This makes it possible to reconfigure varnish without having to
recompile with as well as using an already installed varnish.
[wichert]

Move the OSX patching code into a separate method.
[wichert]

Use pass for non-GET/HEAD requests. This makes a bit more sense and fixes a
login problem for Plone sites.
[wichert]

Reorganize a bit for readability.
[wichert]

Support Python 2.3 as well.
[wichert]

Make it possible to specify the user and group as well.
[wichert]

Do not create the source directory - we move the extracted source in its
place later.
[wichert]