Yesterday and today, wiki has had some down time.
A few days ago, I received a report that mediawiki was running out of memory and
a suggestion to increase the memory available to scripts running under PHP.
I increased the memory available to PHP from 8Mb to 12Mb. Looking at the failure
this morning, it was clear from logs that the kernel ran out of logical memory,
causing applications to close, or go catatonic.
I have responded by squeezing MySQL process memory and reducing the maximum
number of apache instances. I have also set apache so that it will not spawn
apache processes to handle peaks in demand. The rationale being that we'll have
apache and MySQL using a good deal of available resources anyway and spawning
new processes will only cause problems with further I/O and CPU loading.
I had acted on suggestions in October to increase the memory for MySQL caching
to improve mediawiki performance. I have reverted the settings in favour of
preserving more available memory. My subjective tests this afternoon show no
appreciable negative impact, and should make the system more robust by
maintaining adequate reserve memory for the system to stay up in periods of peak
demand. It is better that at times of very high demand, the odd page fails to be
delivered, than the machine being dragged down.
Given the resources we are running mediawiki on are probably adequate but
finite, and we can have very sharp spikes in demand, and the machine mediawiki
is on serves other clients, I don't see any reason to consider occasional pages
not being delivered as a fault - it is part of the design - the system is not
designed to handle any level of demand. Of course, if it becomes common, then it
is a fault.
Given that all visitors tot he domain www.openstreetmap.org are redirected to
the wiki front page, it may be an idea to take snapshots, perhaps every half
hour, of the wiki front page as the front page for openstreetmap. This will
reduce wiki load. We can use a base href tag to make sure links in the page
remain unbroken.