July 17th, 2005: Initial source released to the world!
September 21st, 2006: Updated for Apache 2.x API!

mod_offload is an Apache module and C program for redistributing
server load from a "base" server to one or more "offload" servers.
This was written because my site, icculus.org, was constantly slow
due to large demand for many large files. Since the webspace was
big enough to make a complete copy to another server infeasible, and
its contents changed frequently enough that I didn't want to limit my
users, I came up with this system.

mod_offload is inspired by The
Coral CDN but I wanted something more lightweight and less complex.
The Coral Cache has to solve problems I could avoid, as can most
web operators that aren't trying to mirror the whole internet.

The gist of this project is that you install the Apache module on
the base server, and the C program (as a cgi-bin or standalone daemon,
or a legacy PHP script if all else fails) on offload servers where it
will handle every request. The module on the base server will decide if
a given web request should be offloaded, and redirects the client to
one of the offload servers. The offload server gets a request from the
redirected client and sends an HTTP HEAD request to the base server, so
it can decide if it needs to (re)cache the file in question. If so, it
pulls a copy from the base server via HTTP and serves it to the client
on-the-fly. If not, it feeds the client from the cached copy. To the
users on the base server adding and changing files and the downloading
client, this is all basically transparent, beyond the base server
suddenly being less loaded once the caches start owning copies of large
and generally-unchanged files.

For those without a web server at all, the C program can be built as a
standalone daemon that provides a very basic HTTP server.

The C program is designed to take extremely little memory (as a
standalone daemon, it takes about 120 kilobytes per request it serves),
and can block "download accelerator" programs that open multiple
requests to the server.

Offload servers can block so-called "download accelerators"; at most,
X simultaneous connections from one IP address may download a given URL.
Any above that amount are denied. One IP address can download any amount
of files, and X connections on the "accelerated" URL will run at once.
Download accelerators, in modern times, usually aren't. Presumably these
are a relic of sites that throttled bandwidth per-connection...in such a
case, they were at best a way to personally game the system, but mostly,
they just turn out to be poor citizens of the Internet, as they tend to
use exponentially more server memory for long periods of time to send the
same file, without much increase in performance (actually, they require
more packets to send the same data). This method will let these people
still function without ruining the experience for everyone else.

Streaming content of data from offload server to multiple
users, even while it is still caching.

Offload servers only need a PHP script (Apache module is only
for base server), or cgi-bin access, or ability to run a
process listening on any TCP port.

Offload servers can block so-called "download accelerators".

Module supports both Apache 1.3 and 2.x APIs.

What doesn't work:

Untested on Windows. Apache module probably works there, PHP
script and C code may or may not.

Probably other things.

License:

mod_offload is licensed under the
zlib license.
This means you can statically link it into a closed-source application
and otherwise manipulate it however you like, without a warranty.
I appreciate credit when you use it, but don't require it. I find
this code useful, and hope you will, too.

Documentation:

(coming soon.)

Downloads:

(coming soon.)

Downloading with Mercurial:

mod_offload's source code may be downloaded using
Mercurial (aka: "hg").
Hg allows you to get up-to-the-minute fixes and enhancements; as a
developer works on a source tree, you can use hg to mirror that source
tree instead of waiting for an official release. Please look at the
Mercurial website
for more information on using hg, where you can also download
software for Mac OS X, Windows, and Unix systems.