<div dir="ltr">This might involve a bit of manual work, but you could do this with a root cpuset. The idea in a nutshell is to reserve a core or two for the OS, nagios, pbs_mom daemon, etc. and let jobs use the rest. The way to do this on a 12 core node would be to enable cpusets in TORQUE and in the nodes file say that it only has 11 cores (or 10 if you want to reserve 2 for these things). Then, you can either trust the OS to load balance these other processes to the unused core or you can manually make sure that these processes run under that cpuset. <div>
<br></div><div>As far as whether or not this is needed - pbs_mom should use a minimal amount of resources once a job is actually active. The amount it uses can be greater if it is a larger node and you are running lots of small jobs on the node, but even so it shouldn&#39;t be a huge amount of resources. The only things pbs_mom should need to do for a node that is already filled with jobs is send a status to pbs_server every 45 seconds by default (this can be configured) and respond to pbs_server&#39;s poll requests every 45 seconds (this is also configurable). There will be one poll request per job. I don&#39;t know how much cpu nagios uses, but typically people haven&#39;t had to use this solution except on large-scale numa systems (usually &gt; 1000 cores) which have a little better support for doing it and are often running many more jobs per node.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Sep 18, 2013 at 9:52 AM, Dave Ulrick <span dir="ltr">&lt;<a href="mailto:d-ulrick@comcast.net" target="_blank">d-ulrick@comcast.net</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
We&#39;re running TORQUE 4.2.3.1 with Moab 6.1.5 on a RHEL 6.2 Linux HPC. It<br>
has 60 compute nodes. Each has 12 CPU cores and 2 NVidia GPUs. I use<br>
Nagios to monitor various services on the cluster including the TORQUE MOM<br>
daemons.<br>
<br>
Occasionally, a user will run a job that taxes a node&#39;s resources so much<br>
that Nagios fails to get a timely response from the plugin that checks the<br>
health of the MOM so it flags the service as having a critical issue.<br>
Whenever this happens, I can usually count on the node having other issues<br>
such as a failure to accept SSH connections in a timely manner. I am<br>
concerned that we might see unpredictable problems as fallout if TORQUE<br>
and other crucial system processes are rendered unable to communicate with<br>
a node for a prolonged period of time.<br>
<br>
My theory is that processes that use a great deal of CPU and/or I/O might<br>
monopolize a node&#39;s resources so much that system processes such as the<br>
TORQUE MOM, sshd, Nagios NRPE, etc., are &quot;starved&quot; for resources. Any<br>
suggestions on how I might tune my compute nodes to prevent &quot;starvation&quot;<br>
without unduly impacting the performance of the computational jobs that<br>
are my HPC&#39;s reason for being?<br>
<br>
Thanks,<br>
Dave<br>
<span class="HOEnZb"><font color="#888888">--<br>
Dave Ulrick<br>
Email: <a href="mailto:d-ulrick@comcast.net">d-ulrick@comcast.net</a><br>
_______________________________________________<br>
torqueusers mailing list<br>
<a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>David Beer | Senior Software Engineer</div><div>Adaptive Computing</div>
</div>