Seaside is a stateful application server and a Squeak VM is only capable of taking advantage of a single processor. When hosting a website on a multi processor server, or several servers, you need to load balance your requests across several instances of Squeak to fully utilize the hardware and handle more load than a single Squeak VM is capable of dealing with.

All serious Seaside applications that I'm aware of are front ended by Apache to offload the serving of static content like images, CSS files, JavaScript files, and static HTML while proxying only dynamic content to Seaside. Apache is an awesome platform and absolutely should be one of the tools is your toolbox; however, it can be quite challenging to setup in such a way that it makes load balancing and deploying new code seamless to your users.

I want to thank Avi Bryant for some tips about how DabbleDB handles this issue last year which gave me the basic approach and Lukas Renggli for some recent discussions that spurred me to finally spend some time on this issue and settle it for myself.

Previously I've used HAProxy, and when it became available moved to Apache's modproxybalancer module to load balance multiple Squeak VM's. I found both approaches lacking when it came to rolling out new code without being forced to blow away all existing user sessions when restarting Squeak with newly updated application code. Because of this I'd been rolling out code only during non peak hours when few users would be affected; however, I need to be able to roll out new code anytime and dynamically shift new sessions to the new servers while allowing the old VM's to remain up and running until their sessions expire and are no longer needed.

I've now abandoned modproxybalancer in favor of mod_rewrite and a few scripts which allowed me to tailor a custom solution that gives me much more control allowing seamless deployment of new code while allowing Apache to launch Squeak VM's as necessary and on the fly to handle load. The basic approach is as follows...

When a request comes in I use a rewrite condition to check for a cookie set by the Seaside telling me what server this request should be handled by.

If the cookie exists, I look up the server name in a rewrite map to find out what port that server is running on. That port feeds directly into another rewrite map that runs a bash script which uses netcat to see if the port is open or closed and returns the result.

verifyPort

If the port is open, I proxy the request directly to the server with a rewrite rule when it sees the word "open" in the result.

If the port is closed or if no cookie is found, I proxy the request on a random port chosen from the rewrite map that contains the server mappings. This script also checks to see if the port is open and return the port number immediately if it is; however, if the port isn't open, possibly because an image died, it kills any old process on that port and launches a new Squeak VM on the specified port and waits until it finds the new port open before returning the port number and allowing Apache to proxy the request.

The script kicks off a background process that runs every 60 seconds and expires any sessions that are ready for expiration and shuts the image down when it finds all sessions have expired. It also kills the UI process which isn't needed on a headless server and just wastes CPU cycles if left running.

Not that this process starts off with an immediate 60 second delay, this allows ample time for the image to startup and start receiving its first request and establish a session. If you don't start with the delay, you'd see the image startup and immediately shut back down when it found it contained no active sessions (yes, I did this a few times before figuring out what the hell was happening).

Also, this being a low priority background process, I'm not concerned with using #allSubInstances as speed isn't relevant.

When a request hits Seaside for the first time, the root component of the Seaside app checks to see if the cookie is set and has the correct value. If the cookie is missing (new request) or the cookie has the wrong value (expired request for a VM that's no longer running and timed itself out) Seaside resets the cookie to the correct value so all future requests are served by that VM.

I keep three versions of the server mappings, the active version and an A version and a B version which I can simply copy over the active version to shift traffic to a new set of VM's which Apache will launch dynamically. A simple "cp balance.confA balance.conf" instantly shifts new traffic to the ports specified in the "ALL" mapping.

balance.confB

Though the port number is part of the cookie value, I don't use that directly, it's simply a convenient way to name the app images dynamically. A user could fake a cookie value and looking it up in the mapping to find the port ensures I maintain full control over what ports images are launched on.

And finally here's a full production Apache configuration I use to invoke this setup with all the bells and whistles I currently use in my production sites...

And in my httpd.conf I specify a rewrite lock to ensure Apache doesn't have concurrency issues with any of the rewrite map scripts.

RewriteLock /var/squeak/squeak.lock

And that's it. If anyone sees any problems with this approach (Avi, Lukas, or random Apache guru) I'd appreciate a heads up. I ran it by some folks in the #apache channel in IRC and they seemed to think it was fine. I've had this in production for a little bit now and so far it's kicking ass and I'm now free to increase or decrease my pool size dynamically or shift to a whole new set of images when publishing code.