Hi Brian,
On 8 Apr 2009, at 18:10, Brian Granger wrote:
> The current design of the task dependency system and properties has a
> number of flaws and unfortunately, you are running into all of them.
>> But, I think I have good news. I think I have figured out a way of
> handling this that is much more robust and doesn't have the
> limitations the current approach does.
Thanks a lot for this! Very useful.
The only thing that makes me a little uncomfortable is this:
> * The task will then fail if the dependencies are not met. To get the
> task to reschedule itself, just specify a number of task retries. You
> will just need to give sufficient retries that the task will
> eventually be run on an engine that has the dependency.
Presumably, however many retries are given, there's always a
possibility that this might not happen -- I guess there's some chance
that the task will always be sent back to the same engine (I don't
think the retry system is smart enough to ensure it goes somewhere
else next time... or am I wrong?). Obviously, with enough retries the
odds of hitting a useful engine become pretty good, but it seems like
the whole system might get bogged down with tasks bouncing around all
over the place, specially when the number of tasks and engines gets
large.
Very interesting to get your feedback, anyway. I'll try implementing
something along the lines you suggest and see how well it works, and
keep an eye open for your future developments.
Cheers,
John