Class: CloudCrowd::WorkUnit

Overview

A WorkUnit is an atomic chunk of work from a job, processing a single input
through a single action. The WorkUnits are run in parallel, with each
worker daemon processing one at a time. The splitting and merging stages of
a job are each run as a single WorkUnit.

.distribute_to_nodes ⇒ Object

Attempt to send a list of WorkUnits to nodes with available capacity. A
single central server process stops the same WorkUnit from being
distributed to multiple nodes by reserving it first. The algorithm used
should be lock-free.

We reserve WorkUnits for this process in chunks of RESERVATION_LIMIT size,
and try to match them to Nodes that are capable of handling the Action.
WorkUnits get removed from the availability list when they are successfully
sent, and Nodes get removed when they are busy or have the action in
question disabled.

# File 'lib/cloud_crowd/models/work_unit.rb', line 38
def self.distribute_to_nodes
reservation = nil
loop do
# Find the available nodes, and determine what actions we're capable
# of running at the moment.
available_nodes = NodeRecord.available.to_a
available_actions = available_nodes.map {|node| node.actions }.flatten.uniq
filter = "action in (#{available_actions.map{|a| "'#{a}'"}.join(',')})"
# If there aren't any available nodes or actions don't bother doing anything.
return if available_nodes.empty? or available_actions.empty?
# Reserve a handful of available work units.
WorkUnit.cancel_reservations(reservation) if reservation
return unless reservation = WorkUnit.reserve_available(:limit => RESERVATION_LIMIT, :conditions => filter)
work_units = WorkUnit.reserved(reservation).to_a
# Round robin through the nodes and units, sending the unit if the node
# is able to process it.
while (unit = work_units.shift) and available_nodes.any? do
while node = available_nodes.shift do
if node.actions.include?(unit.action) and node.send_work_unit(unit)
available_nodes.push(node) unless node.busy?
break
end
end
work_units.push(unit) unless unit.assigned?
end
# If we still have units at this point, or we're fresh out of nodes,
# that means we're done.
return if work_units.any? || available_nodes.empty?
end
ensure
WorkUnit.cancel_reservations(reservation) if reservation
end

.find_by_worker_name(name) ⇒ Object

Look up a WorkUnit by the worker that's currently processing it.
Specified by pid@host.

#finish(result, time_taken) ⇒ Object

Mark this unit as having finished successfully. Splitting work units are
handled differently (an optimization) – they immediately fire off all of
their resulting WorkUnits for processing, without waiting for the rest of
their splitting cousins to complete.