Joshua Bernstein said...
| The TORQUE documentation contains a nice explanation for running TORQUE
|on a large cluster. But are these ideas also pertinent to say a very
|small, say four node cluster, running, say many thousands of short lived
|jobs. Its very common in the BioIT space to have a comparitively small
|cluster, but with the many thousands of jobs lasting only a few seconds.
|Does anybody have an guidance on configuration or even source level
|changes for a high throughput, small cluster, with short lived jobs. Or
|would we expect the same changes for a large cluster to also be
|applicable to this configuration?
We meant to write up what we did, but nobody had the
time, and it wasn't all easy to reconstruct. We have
a mix of jobs ran anywhere from days (not that many
to hours (more, but still not huge numbers) to minutes
(quite a few) to seconds (lots). We have 42 queues.
Things were fins so long as we only had a couple of
hundred single CPU systems, none blazingly fast. But
as we moved into dual CPU and multi-core systems, moved
away from buying the cheapest hardware we could get,
and upgraded to a gigabit network with higher speed
backbone, we ran into massive problems.
We tried a lot of things; we can't pin down a single
change that solved "the problem", but together they
all helped a lot. I'll mention as many as I can
recall or dredge up from emails.
1) We moved the torque and maui server processes onto
a much faster, dual core system. With a small
cluster and just a few queues, this may not be
as important.
2) We bundled up small jobs to the extent we could
reasonably do so.
3) Scripts that submit many jobs are now generally
throttled by monitoring the queued jobs and not
queuing more than N at a time (we set "max_queueable"
and some submitters probably just depend on that
to help with throttling). N varies with the
queue and job type, but is always 1,000 or less.
[It can be handy to throttle job submission rate
as well; if the server is too busy handling requests
to queue up jobs, it's not actually running as many.
Again, on a small cluster this may not matter.]
4) Switch from RPP to TCP. We can't say for sure that in
the long run this helped a lot, but at the time we
tried it, it helped. Garrick (IIRC) says it shouldn't
matter, but we were getting kiled with RPP when we
expanded past a certain point. May be irrelevant
for a small cluster.
5) Tweak your torque/maui server kernel!
- Play with swappiness, pagecache, or whatever related
variables are in your kernel. We pushed ours toward
very low swappiness because we have plenty of RAM and
don't ever want to wait on the server pprocesses. We
don't run anything on this system we don't have to.
- We made the following network related changes on the
torque/maui server (2.6.9-55.0.2.ELsmp):
# Speed up NFS read performance
vm.min-readahead = 15
vm.max-readahead = 255
# release sockets faster because we use a lot of them
net.ipv4.tcp_fin_timeout = 20
# Reuse sockets as fast as possible
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
6) Run nscd. As the network and job processing sped
up, we started seeing UID, GID and host lookup errors.
We had to run nscd to cache passwd, group and host
info. We tried keeping the hosts file up to date
on the torque/maui server, and that helped, but it
was too painful in our changing environment. So we
added that to nscd. We run nscd on the server and
all client nodes, and in fact on all the Linux-based
systems that figure into anything even vaguely related
to torque. This has its own problems; the daemon is
not rock solid. We sometimes have to restart it on
every system to make sure it refreshes. [Torque and/or
maui were making insane numbers of passwd/group NIS
lookups, and for some reason insists on checking NIS
for hostnames. We've always used DNS for those.]
6) Play with every server parameter possible. One of
the biggest ones for us was "scheduler_iteration".
I don't know that we'll ever have the One True Value;
we end up readjusting it every so often. We try to
keep it so the scheduler stays busy. We also dropped
things like "node_check_rate" down quite a bit.
We also tried using routing queues to feed run queues.
That seemed to help for a while, but as we changed other
things we ended up doing away with them. You may want
to consider them, though.
We're looking at upgrading the torque/maui server again.
Sometimes maui is running 100% on a core for fairly
long periods of time.
We bumped RESERVATIONDEPTH to 100 to help assure that
high priority jobs always ran ahead of lower priority
jobs.
-Miles