Return of The Squid: how to build a terabyte texture cache using Squid 3 (updated)

This is a followup to my earlier post http://vex.fabulo.us/2011/01/tentacular-magic-for-sl-squidifying-your-viewer/ where I discussed how to set up a local squid web cache as an adjunct to your SL viewer’s texture cache. There were several important caveats there: First, that you could use Squid 2.7 only, since the feature we need for running an efficient cache, StoreURLRewrite wasn’t implemented in later versions and second, there were some bugs in recent viewers that resulted in some textures being cached in a broken state.

Well. Things always change, so this post is aimed squarely at the earlier problems and an attempt to find a better balance now that viewers support ten times the internal texture cache size that was previously available. So, intrepid reader, gird your loins and lets get kraken!

A New Squid

IMPORTANT NOTE: This recipe doesn’t work with version 3.4 of squid. I don’t immediately see any good reason for it, but now multiple people [thanks Lance!] have confirmed that 3.5 seems to be a minimum.

I’m going to use a copy of squid 3.5 that I downloaded from http://www.squid-cache.org/ and then compiled myself. 3.5 is relatively new, so you may or may not be able to get prebuilt binaries for your computer. As before, I’m targeting Linux (now Ubuntu 15.10), but maybe I’ll add in information at the end on other OSes as time permits.

The recipe

download squid 3.5.12:

wget http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.12.tar.gz

decide where you are going to install it. I’ve chosen /opt/squid and I’m going to keep the source tree there too.

unpack and compile

mkdir /opt/squid/src

cd /opt/squid/src

tar xvfz /tmp/squid-3.5.12.tar.gz

cd squid-3.5.12

./configure –prefix=/opt/squid

make all

make install

The New Cache

Setting up a cache using the new setup.

edit /opt/squid/etc/squid.conf

initialize the cache: /opt/squid/sbin/squid -z

run squid: /opt/squid/sbin/squid

Note: whatever user you install and run squid as must have read/write access to the cache directory (as written below, /opt/squid/var/cache/squid). If you use an installation that runs as the “squid” user, your cache directory must also be accessible to “squid.”

The New config

Following is a diff between the default squid.conf and my changes. If you are unfamiliar with the format, lines that begin with “-” are deleted from the file, “+” are added, and lines not prefixed just give context. The lines that star with “@@” indicate which lines of the before and after files are being discussed in the next section. These settings are really just a combination of lines from my previous post, the old Firestorm support post, and updates for Squid 3. The new special sauce start with “store_id_” near the end and the new script that comes in the next section.

Edit: Some people have had difficulties with the “workers 4” line. This should be considered optional – if you have a machine capable of running several worker threads at once and an internet connection fast enough to make it worthwhile, then I’d recommend it… unless of course, it doesn’t work at all when you are using it.

Note that if you want the subject-line-hinted terabyte-sized cache, you’ll need to update the cache_dir line appropriately. Something like:

cache_dir aufs /opt/squid/var/cache/squid 1000000 256 256

The New Script

The new store_id script goes here:

/opt/squid/local/bin/texture_rewrite.py

. Aside from being written in python instead of perl, it supports any number of “channels” (parallel workers), it exits gracefully, and handles a wider range of URLs. Although it really isn’t needed, I also decided to map meshes to a different store-id space so that I could manage them differently, even if I was just looking at the store-ids.

Edit: a bit of important magic is the -u argument to python, which puts python’s i/o in unbuffered mode. This is required to make sure that your script doesn’t stall out.

The New Viewer

For purposes of testing, I’m using v4.7.5.47975 of Firestorm, but other viewers should work similarly. Check out the Firestorm support page on Squid for some nice screenshots on how to set the parameters in their viewer. You might need to use a hack like my original post and set the http_proxy environment variable.

Analysis

Where’s the party, officer?

Well, the good news is that it works, both for textures and for meshes – it is clear from the logs that squid is actually effectively caching data, with lots of hits for both. It remains to be seen if it is a net win, at least for a small number of users on a single LAN. Why is this in question? Well, the bottom line is that the viewers very rarely download all the bytes associated with assets.

To consider a case that is sadly all too common in SL, think about what happens when a creator uses a 1024×1024 texture on one face of a tiny part of someone’s jewelry. If that person was a game designer, they’d be fired, but in SL the viewer needs to attempt to minimize the impact of that extremely poor choice by making a guess at how much detail is actually needed. Then, because textures are encoded more-or-less as a succession of low-to-high resolution details, the viewer will only get as many of the bytes (up to the computed “discard level”) as currently needed to render just that one bit to the screen. This is a HUGE savings for normal operations.

But when we’re caching with squid, we’ve either got the complete texture in the cache (and thus can supply whatever bytes the viewer wants handily) or we need to retrieve the whole thing from SL and start doling out the requested bytes when they show up… but the cache doesn’t stop when the viewer ceased to care – it still gets the whole thing so it doesn’t have to ask again. In theory, a cache could be smart enough to only get the required bytes, but that is a management overhead that few use cases would likely care about. It is possible that someone’s figured out how to get Squid to be that smart… and I’ll update this article if I hear about it.

Meanwhile, the bottom line is that a viewer cached in this way feels different than one that isn’t – I can’t shake the feeling that the viewer gets slightly confused by the way the cache responds – it often seems as though the viewer is satisfied with the wrong discard level for far too long, leaving you with occasional blurry textures. At one point I thought I’d identified some viewer-side cache issues and proposed some fixes, but it has been a looong time since I’ve looked at the codebase and some of my suggestions are clearly OBE, since the http client library has been updated since then.

More…

… things to try.

compare the different cache styles. ufs vs aufs but also diskd and rock. It might make sense to have a rock-based cache of small objects in front of an aufs-based cache of large objects.

The header_access lines are non-functional in the current version of these instructions, but retained because a) it doesn’t seem to do any harm (probably because the viewer http libraries no longer seem to care) and b) as documentation for future use. I believe the 3.x equivalent is reply_header_access (with the same arguments), but this too is disabled by default unless squid is compiled with –enable-http-violations. My guess based on what the setting is trying to block is that you might have better luck adding a line like:

forwarded_for delete
# or maybe
# forwarded_for transparent

It would be really interesting to figure out and execute on a cache efficiency A-B test to parameterize the end performance in various ways.

Thanks

Thanks to Lance Corrimal who worked through the above instructions, found a few bugs, and reported on various successes and failures. Dave Bell who noted that header_access is nonfunctional in squid 3.x and suggested a reminder that the user you run as needs permission to access the cache directory.