Make the Queue implementation replaceable

Details

Description

Queue issues are not acute, but I've just done some search, and found that it is
a recurring issue, and no enhancement has been filed. So here it is, along
with all the relevant links I've found on the issue:
The best thread I've found is this one:http://www.nabble.com/Customize-Build-Queue-td18024300.html

What if the queue ordering was separated from the algorithm(s) that determined the ordering?

More specifically, what if each job had an integer representing its priority where smaller numbers
represented higher priority and jobs with the same numeric priority were built on first come first serve
basis. All jobs with a given priority would be built and completed before any jobs with lower priority
were built.

Given this internal queue implementation, you could have
1) a field in the job configuration page allowing you to manually specify the priority
2) an extension point for zero or more algorithms to adjust multiple job priorities when a single job's
configuration is saved
3) a facility for a user to manually and temporarily "bump up" a job's priority for a single build

Separating queue ordering from the the algorithm(s) that determined the ordering allows you to keep the
queue implementation internal and straightforward, while allowing others to explore different algorithms
to suit their individual needs. Algorithms can also be composited, applying first one algorithm then
another to achieve the desired effect.

danrubel
added a comment - 2009/Oct/21 6:04 AM What if the queue ordering was separated from the algorithm(s) that determined the ordering?
More specifically, what if each job had an integer representing its priority where smaller numbers
represented higher priority and jobs with the same numeric priority were built on first come first serve
basis. All jobs with a given priority would be built and completed before any jobs with lower priority
were built.
Given this internal queue implementation, you could have
1) a field in the job configuration page allowing you to manually specify the priority
2) an extension point for zero or more algorithms to adjust multiple job priorities when a single job's
configuration is saved
3) a facility for a user to manually and temporarily "bump up" a job's priority for a single build
Separating queue ordering from the the algorithm(s) that determined the ordering allows you to keep the
queue implementation internal and straightforward, while allowing others to explore different algorithms
to suit their individual needs. Algorithms can also be composited, applying first one algorithm then
another to achieve the desired effect.

Ultimately it would be nice to have an extension point just for determining each
projects build ordering, without having to muck with the entire queue
implementation. That's the end goal.

In the interim, I can see how a real number priority could solve a number of the
problems, but it is also lacking. Notably, it is not very object oriented, so
doing things like tracking why a job was assigned a given priority is very
difficult to extract from a real number.

Composability would be nice, but I don't think it is realistic to expect to
compose arbitrary priority systems. For example, I would like to queue jobs
according to their expected run time - faster jobs go first. This maps to a
real number easily. So do user supplied priorities. But composing them seems
awfully complicated. How do I define a function from real to real that does
something meaningful with the initial priority, but still produces a queue that
orders things according to the expected run time?

Similarly for the bump up feature. I'd rather have some sort of proper
datastructure that allows for a predicate that moves a job to the top, rather
than taking the maximum current priority and adding 1 to it. With a
datastructure, the manually promoted job can be highlighted in the queue as
promoted, for example.

mdonohue
added a comment - 2009/Oct/24 12:42 PM Ultimately it would be nice to have an extension point just for determining each
projects build ordering, without having to muck with the entire queue
implementation. That's the end goal.
In the interim, I can see how a real number priority could solve a number of the
problems, but it is also lacking. Notably, it is not very object oriented, so
doing things like tracking why a job was assigned a given priority is very
difficult to extract from a real number.
Composability would be nice, but I don't think it is realistic to expect to
compose arbitrary priority systems. For example, I would like to queue jobs
according to their expected run time - faster jobs go first. This maps to a
real number easily. So do user supplied priorities. But composing them seems
awfully complicated. How do I define a function from real to real that does
something meaningful with the initial priority, but still produces a queue that
orders things according to the expected run time?
Similarly for the bump up feature. I'd rather have some sort of proper
datastructure that allows for a predicate that moves a job to the top, rather
than taking the maximum current priority and adding 1 to it. With a
datastructure, the manually promoted job can be highlighted in the queue as
promoted, for example.

You could have an extension point for providing a java.util.Comparator for
determining which job should be built first. Those could also be composed in that
if a comparator could not determine which job should be built first, then it
passes the responsibility down the chain of command. In fact the first comparator
in chain of command could check for jobs that have been manually/temporarily
bumped-up in priority by a developer waiting for a build to happen. Perhaps this
is more along the lines of what you are thinking?

danrubel
added a comment - 2009/Oct/25 9:09 PM You could have an extension point for providing a java.util.Comparator for
determining which job should be built first. Those could also be composed in that
if a comparator could not determine which job should be built first, then it
passes the responsibility down the chain of command. In fact the first comparator
in chain of command could check for jobs that have been manually/temporarily
bumped-up in priority by a developer waiting for a build to happen. Perhaps this
is more along the lines of what you are thinking?

Kohsuke Kawaguchi
added a comment - 2010/Mar/20 3:55 PM At this point enough aspects of a queue is pluggable that I consider this issue fixed.
If people need additional aspects pluggable, please open separate issues.