I do need a separate threads for each running task, so changing the architecture isn't an option. I tried making my threadPool size equal to Runtime.getRuntime().availableProcessors() which attempted to run all 500 threads, but only let 8 (4xhyperthreading) of them execute. The other threads wouldn't surrender and let other threads have their turn. I tried putting in a wait() and notify(), but still no luck. If anyone has a simple example or some tips, I would be grateful!

Well, the design is arguably flawed. The threads implement Genetic-Programming or GP, a type of learning algorithm. Each thread analyzes advanced trends makes predictions. If the thread ever completes, the learning is lost. That said, I was hoping that sleep() would allow me to share some of the resources while one thread isn't "learning"

So the actual requirements are

how can I schedule tasks that maintain
state and run every 2 minutes, but
control how many execute at one time.

Is there any particular reason why you cannot do them concurrently sequential? That is have 8 running concurrently, and collecting results et al when they are finished. The issue is that it will still take the same amount of time if you do it with 500 threads or if you do it with 8 threads.
–
John VintMay 19 '10 at 18:34

The reality here is that it will take a longer amount of time with 500 threads then with 8 threads. With only 8 separate processors there will be a huge amount of context switching. This will cause each 2 second computation to take much much longer.
–
John VintMay 19 '10 at 18:41

2

500 threads is at least 492 too many in your case. Since you're using hyperthreading, you don't even have 8 real CPUs. I'd say use 4 threads in a round-robin configuration and you'll probably get the best performance.
–
Daniel PrydenMay 19 '10 at 19:09

1

But is it absolutely necessary they be Thread objects? You can make them Runnable and let them preserve this important internal state, then invoke run as appropriate in a number of threads that's manageable for your system. (which may be more than 4 or 8 if part of the task is blocking on Network/Database/Whatever IO.)
–
AffeMay 19 '10 at 19:31

What do you mean by, "changing the architecture isn't an option"? If you mean that you can't modify your task at all (specifically, the tasks have to loop, instead of running once, and the call to Thread.sleep() can't be removed), then "good performance isn't an option," either.

If your threads are not terminating, this is the fault of the code within the thread, not the thread pool. For more detailed help you will need to post the code that is being executed.

Also, why do you put each Thread to sleep when it is done; wouldn't it be better just to let it complete?

Additionally, I think you are misusing the thread pool by having a number of threads equal to the number of tasks you wish to execute. The point of a thread pool is to put a constraint on the number of resources used; this approach is no better than not using a thread pool at all.

Finally, you don't need to pass instances of Thread to your ExecutorService, just instances of Runnable. ExecutorService maintains its own pool of threads which loop indefinitely, pulling work off of an internal queue (the work being the Runnables you submit).

I'm not sure your code is semantically correct in how it's using a thread pool. ExecutionService creates and manages threads internally, a client should just supply an instance of Runnable, whose run() method will be executed in context of one of pooled threads. You can check my example. Also note that each running thread takes ~10Mb of system memory for the stack, and on linux the mapping of java-to-native threads is 1-to-1.

I posted my comments but this really should address your issue. You have a computation that can take 2 seconds to complete. You have many tasks (500) that you want to be completed as fast as possible. The fastest possible throughput you can achieve, assuming there is no IO and or network traffic, is with Runtime.getRuntime().availableProcessors() number of threads.

If you increase your number to 500 threads, then each task will be executing on its own thread, but the OS will schedule a thread out every so often to give to another thread. Thats 125 context switches at any given point. Each context switch will increase the amount of time for each task to run.

The big picture here is that adding more threads does NOT equal greater throughput when you are way over the number of processors.

Edit: A quick update. You dont need to sleep here. When you execute the 500 tasks with 8 processors, each task will complete in the 2 seconds, finish and the thread it was running on will then take the next task and complete that one.

This assumes CPU-bound processing. It is not correct if the threads are I/O-bound.
–
Willie WheelerMay 19 '10 at 19:51

@willie Wheeler - In the comments for the question he says he is performing computations. It is probably safe to assume that he is CPU bound. Did you read or just ding ?
–
Romain HippeauMay 19 '10 at 19:53

This schedules the Runnables every 10 SECONDS to show the behavior.
If you really need to wait a fixed amount of time AFTER processing is complete you might need to play around with which .scheduleXXX method that you need. I think fixedWait will just run it every N amount of time regardless of what the execution time is.

I do need a separate threads for each running task, so changing the architecture isn't an option.

If that is true (for example, making a call to an external blocking function), then create separate threads for them and start them. You can't create a thread pool with a limited number of threads, as a blocking function in one of threads will prevent any other runnable being put into it, and don't gain much creating a thread pool with one thread per task.

I tried making my threadPool size equal to Runtime.getRuntime().availableProcessors() which attempted to run all 500 threads, but only let 8 (4xhyperthreading) of them execute.

When you pass the Thread objects you are creating to thread pool, it only sees that they implement Runnable. Therefore it will run each Runnable to completion. Any loop which stops the run() method returning will not allow the next enqueued task to run; eg:

what behaviour is expected if a task blocks longer than the expected window for running

what behaviour is expected if a task blocks longer than the repeat rate (blocks for more than one minute)

Depending on the answers to these, some combination of ScheduledExecutorService, semaphores or mutexes can be used to co-ordinate the tasks. The simplest case is the non-blocking, non-synchronous tasks, in which case use a ScheduledExecutorService directly to run your runnables once every minute.

You can certainly find some improvement in throughput by reducing the number of threads to what the system can realistically handle. Are you open to changing the design of the thread a bit? It'll unburden the scheduler to put the sleeping ones in a queue instead of actually having hundreds of sleeping threads.

This preserves your core semantic of 'job repeats indefinitely, but waits at least one minute between executions' but now you can tune the thread pool to something the machine can handle and the ones that aren't working are in a queue instead of loitering about in the scheduler as sleeping threads. There is some wait busy behavior if nobody's actually doing anything, but I am assuming from your post that the entire purpose of the application is to run these threads and it's currently railing your processors. You may need to tune around that if room has to be made for other things :)

The OP didn't mention anything about synchronisation. This is completely off topic.
–
SimonCFeb 26 '11 at 8:47

The OP asked a way to ensure that only a fixed number of thread are "active" at a given moment, and this is a synchronization problem. May be that my solution isn't the optimal one (I like the accepted solution) by for sure it isn't off topic because it solves the problem.
–
Andrea PolciApr 20 '11 at 14:45