The version of JBoss that is affected by this problem is 6.0.0.M2. The version that does not appear to be affected is 5.1.0.GA (I should have thought of this earlier - sorry). So perhaps the problem is something that has been changed in recent versions?

I started an instance of JBoss 6.0.0.M2 on the same box (virtual machine, that is) as the JBoss 5.1.0.GA instance. The JBoss 6 instance was affected by the scanning but the 5.1 instance was not. So that points pretty strongly to a recent change.

Thank you for confirmation of the problem. I believe that my thread dumps (attached to an earlier entry) tell essentially the same story as yours do.

I think that this is going to become a significant problem for me. I really don't want to have to restart JBoss after every scan! So I have submitted this as a bug: https://jira.jboss.org/browse/JBAS-8580

I believe that the condition variable enqueueCondition is "true" while the "poll" is returning null. What ends up happening is the "await" is returning immediately, then the queue.poll() is triggered, but returns null, then the loop cycles again. So, there's some sort of false signaling going on. Why it cycles every 1 second is because the "elapsed" eventually kicks the thread out.

Also, it would be better if the thread is interrupted to immediately return "null" rather than loop again.

@Nick: I do only a start-up of JBoss and around 10 seconds after full start the behavior can be seen. There are some internal HTTP and JNDI requests going on, since I do have applications deployed - I did not run a test with an out-of-the-box JBoss yet.

@Elias: Yes, this looks promising. I'm going to debug into that code tomorrow and let you know.

I'm suspecting there is a bug in org.jboss.threads.QueueExecutor... Suspect is this "for" loop:

I believe that the condition variable enqueueCondition is "true" while the "poll" is returning null. What ends up happening is the "await" is returning immediately, then the queue.poll() is triggered, but returns null, then the loop cycles again. So, there's some sort of false signaling going on.

There may be, however Conditions do allow for spurious wakeups which is why we re-test the condition on every iteration of the loop. So await() should return spuriously at most one time. The only possible exception would be another thread hammering on the condition variable which seems unlikely unless the queue is being flooded with tasks (in which case poll() would eventually return a task). How sure are you that you are seeing a busy wait? The profiler snapshot you give looks like it might be wallclock time (during which await() will be blocking).

If you look at the original thread stack dump, the thread statuses are WAITING, not RUNNING.

Elias Ross wrote:

Also, it would be better if the thread is interrupted to immediately return "null" rather than loop again.

Not in this case; these are thread pool worker threads which would be within their rights to simply ignore interruption, unlike application or framework threads which should take interruption as a hint to cancel the current task. The only time a thread pool would typically respect interruption is when it is being shut down. As you can see, the shutdown flag is tested in the condition loop. Interruption would awaken the waiter and the stop flag would then be read, causing the loop to terminate. If interruption were caused by an application-level interruption request, then the interrupt will be consumed and ignored by the Worker class.

The thread states are mostly waiting, but also running each for about 1 seconds per call, resulting in an overall enduring load. If you look at an individual thread you see it entering the RUNNING state for about 0.8 - 1.5 seconds then WAITING again for 30-60 seconds. With 10-20 pool-threads you get the overall load on the system.

Did a little debugging and checked the code and resolved the issue by adding the following after the if-block, which checks "if (remaining <= 0L && (allowCoreThreadTimeout || threadCount > coreThreadLimit))":

if (remaining <= 0L)
{
// JBTHR-17 if remaining is 0 or less and we did not meet the condition above,
// we do NOT want to wait a negative amount of time. Instead waiting for keepAliveTime makes most sense.
remaining = keepAliveTime;
}

Attached a patch, which I will also attach to https://jira.jboss.org/browse/JBTHR-17, including this and some visibility increasements to prevent synthetic accessor methods. (Not completely sure whether the volatiles are needed also, but this made sense to me, since they are used from different thread contextes).

Btw: Can you point me to the jboss-threads svn repository location - I did only find the sources from nexus?