Resource migration in a distributed computer system can be performed for performance en-
hancement and we call this activity "load balancing with resource migration". We propose
a general load balancing model with both job and resource migration and demonstrate how
this model applies to the file migration problem in distributed databases. The intractability
of this load balancing model -,.--. --. obtaining approximate solutions. In this paper, we
propose an approximation approach that partitions the distributed system into regions so
that resource migration takes place between the regions while job migration occurs within
the regions. By using a good rule for partitioning the system into regions, good approxi-
mate solutions can be obtained. We present theoretical bounds and experimental results to
indicate how our approximation approach can provide close to optimal solutions in many
instances.

A resource in a computer system is defined as any hardware or software entity required for the
execution of a user job. Examples of resources include processors, memories, interconnection
networks, system processes, data files, database relations and file servers. In a distributed
computer system, some of these resources are distributed among the various nodes in the
system. If the distribution of a resource among the nodes can vary with time, then we call
this resource a 'migratable' resource. Examples of such resources include datafiles, processes
and mobile hosts. Traditionally, the term 'load balancing' refers to the operation of dis-
tributing or redistributing the user tasks among the different nodes in a distributed system,
to achieve a desirable performance level; typical performance measures include job response
time, throughput and processor utilization. We extend this definition of load balancing to
include the operation of distributing or redistributing the migratable resources of a com-
puter system to achieve a desirable performance level. Redistribution of the user jobs and
the migratable resources among the nodes in the system are respectively referred to as job
migration and resource migration respectively. Job migration also includes process migration
which involves the migration of a task while it is still being executed.
More generally, in load balancing without resource migration, the distribution of the
resources over the nodes is fixed, and load balancing is accomplished by determining an
assignment of the resource requests to the resources such that the cost in handling these
requests will satisfy some required constraint. On the other hand, in load balancing with
resource migration, load balancing is accomplished by first determining a new distribution of
the resources over the nodes and then determining an assignment of the resource requests to
the resources under the new distribution. Obviously, load balancing with resource migration
can provide a higher opportunity for achieving a lower cost in handling the resource requests,
provided that the cost of migrating the resources is affordable. In general, load balancing
without resource migration would be performed more frequently than load balancing with
resource migration. And the latter is performed only when the pattern of resource requests
has changed dramatically and when this change is expected to last over a long period of
time.
In the literature, many load balancing models have been proposed for distributed and
multiprocessor systems. Casavant and Kuhl [5] give a survey of many of these models.
In general, load balancing models can be classified in the following four ways : (a) static
and dynamic, (b) centralized and decentralized, (c) deterministic and stochastic and (d)
models for general distributed and multiprocessor systems and those for systems with specific
topologies. Job allocation (e.g. [2, 28]) and resource allocation [6, 33] models belong to the
static category while job and resource migration models belong to the dynamic category.
In centralized load balancing models (e.g. [21, 4]) load balancing decisions are performed

at a central site. In decentralized models (e.g. [17, 18, 19, 22, 24, 26, 32]), the nodes
or processors perform load balancing using distributed strategies that rely on some local
information available to the nodes. Deterministic load balancing models usually involve
graph-theoretic (e.g. -'8]), integer programming or heuristic approaches such as those used
in distributed strategies. Stochastic load balancing models are based on Markov chains
and (e.g. queueing models (e.g. [10, 21, 32]), Bayesian decision models (e.g. [27]) or
other probabilistic approaches (e.g. [23]). Load balancing models for distributed memory
multiprocessor systems (e.g. [1, 9]) exploit the specific topologies of the interconnetion
networks. The load balancing model we propose in this paper is a centralized graph-theoretic
deterministic model that includes both job and resource migration.
First we give a few examples of applications where resource migration can also be used
for load balancing. In a distributed database or file system, file migration is performed in
order to maintain at all times, a desirable relation between the file access rates and the
distribution of file copies among the nodes. In this case, we refer to the database files
as migratable resources, the file migration activity as resource migration and the remote
file access activity as job migration. We will discuss, in detail, the example of database
file migration in a later section. Many file migration models exist in the literature (e.g.
[15, 14, 30, 31]). But these models with the exception of [15] ignore the effect of balancing
the file access requests evenly among the many copies of the file. A distributed heuristic is
proposed in [15] to determine whether to migrate a process or a file, or whether to replicate a
file and this decision is based on probablistic estimates of number of read and write accesses
to files and the utilization of various resources in the system. On the other hand, the model
we propose is a deterministic one and considers only one type of resource. Another example
of resource migration in a distributed computer system can occur when a job in one host
needs the services of a system process such as a file server, query processing program and
editor process, running on a remote host. Here, instead of sending the request to the remote
host (that is, migrating the job) and transferring the results back, the required process itself
(which is the resource) can be migrated from the remote host.
When load balancing can be achieved with both job and resource migration, first there
is a problem of determining when to migrate jobs or resources. This migration detection
problem is NP-Complete (see [31]) even in the presence of complete information regarding
the future resource requirements. Hence in many applications, these migration points are
not determined beforehand but instead the load balancing problem is considered periodically
or at time points of degradation of specific performance measures.
The other important issue in load balancing with both job and resource migration is
the problem of deciding which jobs or resource units to migrate. In general, the decision
of whether to migrate jobs or resources in a distributed computer system depends on the
following: (i) job characteristics such as resource requirements and migration cost, (ii) re-

source characteristics such as capacity, storage requirements and migration cost and (iii)
communication network characteristics such as topology and channel bandwidth. We refer
to the above problem of deciding which jobs and resource units to migrate as the general load
balancing problem with resource migration. In this problem, it is necessary to find a proper
distribution of resources and jobs among the various nodes in the system so that the desired
trade-off occurs between the job migration cost and the resource migration cost. In one
formulation of this problem, the total migration cost (of jobs and resources) is minimized.
When the resources can migrate in parallel, bottleneck cost criterion is more appropriate than
the total cost criterion for resource migration cost. In our formulation of the load balancing
problem, we use a bottleneck criterion.
Due to the intractability of the load balancing problems, determining the exact opti-
mal solutions need either exhaustive search or heuristic search procedures, all of which are
prohibitively expensive to be executed in real time. As a result, approximate solutions are
usually needed for such problems. One approach which is common to all the existing tech-
niques is to use heuristic rules for guiding the search to an approximate solution. However,
the heuristic rules for obtaining approximate solutions are generally difficult to derive in
many formulations. In this paper, we propose a new approximation approach to solve the
general load balancing problem.
In the new approach, we separate the resource migration and job migration problems.
We do this by partitioning the system (or network) into regions so that resource migration
takes place between the regions while job migration takes place only within the regions.
For the resource migration problem, we use local approximations of the network character-
istics and the total resource requirements within a region. In this problem, we attempt to
minimize the bottleneck resource migration cost but with a constraint on the job migration
costs. This resource migration problem is shown to reduce to a well-known commodity dis-
tribution problem for which efficient polynomial-time algorithms already exist. By suitable
partitioning, good approximate solutions to resource migration problem can be obtained.
Once resource migration is determined, job migration problems are solved independently for
each region. We also propose an efficient approximation algorithm for determining the job
migrations.
The organization of the paper is as follows. In the next section, we present our general
load balancing model with resource migration along with the motivation for such a model.
We also explain how this model can be used to solve a load balancing problem that arises in
the case of file migration in distributed databases. In Section 3, we mention the complexity
of the problem and explain, in detail, the proposed approximation approach. In Section
4, we present our approximation algorithm based on this approach and analyze its time
complexity. In Section 5, we present theoretical bounds for the performance of our approxi-
mation algorithm. In Section 6, we discuss the experimental results to demonstrate how our

approximation algorithm provides close to optimal solutions in many instances, especially
when an appropriate partition scheme is chosen. Finally, we summarize and present our
conclusions.

2 Proposed load balancing model with resource mi-

gration

In a general load balancing problem with resource migration, two types of costs need to be
considered in deciding whether job or resource migration needs to be performed. One is the
job migration cost and the other is the resource migration cost. Each of these costs has two
components, a fixed part and a variable part. Typically, the fixed cost is associated with
the overhead time involved in setting up the job or the resource in a new node or location
and hence is independent of the migration distance. On the other hand, the variable cost
depends on the migration distance and we assume here the use of an unit distance migration
cost. Unit distance job migration cost is defined as a measure of the time to send the job
to a remote node, unit distance away and receive the results back. Unit distance resource
migration cost is a measure of the the time to migrate a resource unit to a node, unit distance
away.
In our proposed load balancing model, we assume that the resources can migrate only
at the time when this load balancing problem is solved. On the other hand, job migration
1 occurs frequently after resource migration but only according to the resource access func-
tion determined by the load balancing model; the resource access function indicates which
resource copy should be accessed by a job from a given node. Our assumption is reasonable
in the light of the fact that the cost of resource migration is usually much higher than the
cost of job migration. The resource migration is thus necessitated by a large shift in the
resource access pattern at which time the proposed load balancing problem will be solved.
In our load balancing formulation, we determine how the resources should migrate as well
as the resource access function so as to obtain the desired trade-off between the resource
migration cost and the job migration cost. The job migration cost not only includes the
cost of communicating the resource access request to the remote node and transferring the
results back but also the processing delay at the remote site. The latter component of the
job migration cost is reduced by balancing the resource access requests among the nodes
that contain the resource copies. Either the total cost or the bottleneck cost criterion can
be used for the job migration and resource migration costs. The total cost criterion is useful
when the resources and the jobs have to be migrated one at a time as for example, when a
single broadcast bus such as Ethernet is used for migrating the software resources and the
jobs among the nodes. The bottleneck cost criterion is useful when the resources and the
1Here job migration also includes activities related to remote access of resources.

jobs can be migrated in parallel. Here, we give our formulation of the general load balancing
problem with the bottleneck cost criterion. For this, we introduce the following notations:

V
P(V)
I
I'
f: (I-

g: V

h : I-

We assume that the resource requests from all the jobs have the same processing time, inde-
pendent of the nodes. This assumption is made only to simplify the discussion. The resource
requirements of each job are specified as number of resource accesses required per unit time.
We assume that if resource requests from a node need to be processed remotely, then all
these requests are directed to the same remote node for processing. The communication cost
for a resource request remotely processed depends only on the communication distance and
is in particular, independent of the network traffic. This assumption is justified in networks
with sufficient communication bandwidth. The processing delay is however dependent on
the processing load at a node or a host and is assumed to be directly proportional to it.
The resource migration cost is denoted by R(I, f) and the job migration cost is denoted
by Q(I,g); f and g are respectively resource and job migration functions. These costs are
given as follows:

R(I, f)

Q(I,g)

max [FR + cpdf(y)y]
yEI-I'
max [F + cj.l ,) + 6(g(x))]
{xEVibx>0O}

-set of all nodes in the system
-power set of V
-set of nodes with resource units after migration
-set of nodes with resource units before migration
I') --+ migration function specifying how the resource units migrate
f(x) is the node from which the resource unit at x has migrated
I job assignment function specifying where a resource request
from a node needs to be processed
g(x) is the node that satisfies the resource access request from x
P(V) -indicates for a resource unit, the set of nodes
whose resource requests need to be processed by the resource
(h is the inverse function of g)
-resource requirements of the jobs submitted at the node x
-resource request processing capacity of a node (requests per unit time)
-communication distance from node x to node y
Z+ delay function indicating the delay in
processing a resource request at a node
-fixed costs (distance-independent) of resource and job migration respectively
-cost per unit distance for resource and job migration respectively
-desired maximum job response time

bx
q
dxy
6 : I --

FR, FJ
CR, CJ
T

Here 6(y) = -h(y) b. Note that the delay here does not represent the average delay as
q
given in a queueing model but rather gives a bound on the processing delay assuming no
particular queueing discipline. A similar delay function is employed in the processor and link
assignment problem studied in [3]. Thus the load balancing problem is one of determining I,
g and f so as to achieve a desirable combination of R(I, f) and Q(I, g). In our formulation,
we minimize R(I, f) with a constraint on Q(I, g), that is, Q(I, g) < T. In many applications,
we also have the restriction that III < II'|. The load balancing problem we wish to solve,
which we call Problem LBP, can thus be stated as follows:
Given: A network G = (V,E) with distance matrix {dy}, set I' of initial
locations of resource units, fixed cost FR and unit distance cost cR of resource
migration, fixed cost Fj and unit distance cost cj of job migration, resource
request processing capacity q and the desired job response time T.
Required: Set I of new locations for the resource units with III < I|'I, the
resource migration function f and the job migration function g such that R(I, f)
is minimized with the constraint that Q(I, g) < T.
Now we will illustrate how our proposed model can be used to address the issue of file
migration in distributed databases.

File migration in distributed databases

In a distributed relational database, relations are partitioned either vertically (grouping of
attributes) [20] or horizontally (collections of tuples) [7] into possibly overlapping fragments
which are referred to as database files. These files are distributed among the different sites
or nodes in the network. To increase the availability as well as to improve the transaction
response time, it is a common practice to replicate many of these files. The degree of
redundancy is a design choice that often depends on the cost of performing "updates" on all
the copies of a file. A transaction submitted by the user at a node translates into file access
requests which can be either queries or updates. All the file migration and allocation models
that had been proposed before (e.g. [6], [31]) assume that the file access requests are always
sent to the nearest node containing a file copy; they ignore the effect of balancing the requests
among the different file copy nodes. Our load balancing model attempts to minimize the
query response time by balancing the work load among the different file copy nodes. As the
pattern of file access requests changes, it may be necessary to migrate the file copies among
the different nodes. The file migration problem we propose involves determining where to
move the file copies to and also determining how to balance the workload among the new file
copy locations so as to minimize the query response time as well as the file migration costs.
The query response time consists of the following two components: (1) query communication
time which includes the time for sending the request and receiving the results back and (2)
query processing delay which includes the time for processing the file access request.

In formulating the file migration problem, we make the following reasonable assumptions:

1. A constant number of file copies is maintained at all times. In most applications, when
there is a constant number of file copies, the cost of updating all the file copies and
the cost of storage do not differ significantly among the different allocations of file
copies. Thus the determination of number of file copies becomes a design choice that
also needs to consider the availability of file copies in case of node failures. With this
assumption, we do not have to consider update costs in our load balancing problem.

2. Due to the first assumption, whenever a new file copy is generated at a node, some
other existing copy at another node needs to be deleted. We assume that the file
access requests queued at the node (from which a copy is deleted) are directed to a
node containing a new file copy. Thus the cost of migrating a copy from one node
to another includes the following: (i) the cost of moving the file and (ii) the cost of
redirecting the query requests already queued up at the node from which the file is
migrated to some other nodes containing the file copies.

3. All file copies can migrate in parallel.

4. Query communication delay is independent of the query traffic and is dependent only
on the communication distance. We assume that the nodes have a limited processing
capacity (expressed in file access requests per unit time) and the processing capacity is
the same for all the nodes. Moreover, we assume a query processing delay component
that is directly proportional to the query traffic directed to the node containing the
file copy.

With these assumptions, the file migration problem becomes identical in formulation to the
load balancing problem (LBP) with file copies taking the place of resource units, file access
request rate taking the place of resource requirements (i.e. b,) and query response time taking
the place of job response time.

3 Proposed approximation approach

The load balancing problem LBP can easily be shown to be NP-hard by a simple reduction
from the Dominating set problem [12]. One approach to obtaining an optimal solution is
to use heuristic search techniques such as branch-and-bound algorithms. We implemented
one such branch-and-bound algorithm. Even for reasonable size problems with 12 nodes
and 4 resource units, the branch-and-bound program took more than 2 CPU hours for some
problem instances. This is undesirable since the load balancing problem needs to be solved
in real time in almost all the applications. As a result, we seek efficient algorithms that

provide close to optimal solutions. One possible approximation approach is to use a branch-
and-bound search that terminates with an approximate solution instead of with an optimal
solution [30]. But the heuristic rules for obtaining these approximate solutions are generally
difficult to derive for constrained optimization problems such as the one we had proposed.
We propose an approximation approach that is efficient, simple to understand and yet can
provide reasonably good approximate solutions.
In the approximation approach we propose, we partition the given system into regions
so that resource migration takes place only between the regions while job migration takes
place only within the regions. The partitioning helps to achieve local approximations for the
underlying network characteristics so as to separate job and resource migration problems.
After partitioning, we first focus on resource migration only but with a view to reducing
the job migration costs. This problem will be shown to reduce to a problem known as the
bottleneck transportation problem in the Operations Research literature. From the solutions of
this problem, we can determine the number of resource units that should be transferred from
one region to another but not the exact new locations for the resource units. We give a simple
algorithm to determine the new locations for the resource units that are consistent with the
optimal solution to the transportation problem. After resource migration is determined,
we still have to determine where the remote resource access requests should be processed
so as to reduce the job migration costs. Since the resource access requests are processed
within the regions, we solve these job migration problems independently for each region. For
this job migration problem, we ignore the job communication costs and focus on balancing
the resource access requests evenly among the nodes that contain the resource units after
resource migration. First we describe how the resource migration problem is solved using
the partitioning approach. Later we address the problem of determining the job migrations.

3.1 Approximate resource migration problem among the regions

Let the set of nodes V be partitioned into m regions W1, W2,..., Wm. Later we will explain
how this partitioning can be done so as to obtain good approximate solutions. We require
that the number of regions is at most the number of resource units to guarantee that each
region has at least one resource unit after migration. Let I, I,...,I' be the corresponding
sets of nodes in the regions containing file copies before file migration; that is, If = Wj I',
for 1 < < m. Additionally, we define the following quantities:

Fj+cjS' is the average cost of transferring a resource access request to a remote node within
Wj and receiving the results back. FR + cREPk is the average cost of migrating a resource
unit from region Wj to region Wk. It is reasonable to assume that a node will not send
as well as receive a resource unit during migration. With these local approximations, the
resource migration problem between the regions is formulated as follows:

The variable zjk denotes the number of resource units that need to migrate from the region
Wj to the region Wk; zjj is the number of resource units that remain in their locations within
region Wj. We allow the possibility of some resource units not being utilized after resource
migration and hence we have the inequality in the second set of constraints. Let the quantity
Rj (TF J .S3 ) denote the minimum resource capacity (expressed in number of resource
units) needed to meet the requirements of jobs in region Wj. Also, let tjk = (FR + Ci,..
denote the cost of migrating a resource unit from the region Wj to the region Wk. Then the
approximate resource migration problem we wish to solve is formulated as follows:

The above formulation is known as the bottleneck transportation problem in the Operations
Research literature. In the general bottleneck transportation problem, there are m suppliers
each of which can supply units of a certain commodity to n destinations and there is a
shipping cost per unit of commodity for each supplier-destination pair. There is a maximum

quantity that a supplier can provide and a minimum quantity demanded by each destination.
In our formulation, the resource units are the "commodil -" and the regions we had de-
fined serve as both -uppliers" and "destin.il i. 1- Note that the bottleneck transportation
problem is infeasible if and only if L Rk > 2l 1 We will explain in Section 3.2 how
this case is addressed in our approach. In the following discussion, we will assume that the
problem is feasible.
In the above formulation of our problem, we have implicitly assumed that the regions
either supply resource units or receive resource units but do not do both. Since E'j = 0,
it follows that there always exists an optimal solution such that for a region Wj, when
Rj < Nj, region Wj serves only as a supply region and when Rj > Nj, region Wj serves only
as a destination region. Thus we can eliminate from consideration those regions Wj for which
Rj = Nj and divide the remaining regions into -,,lplly regii.,- and "destination regii.-"
For a supply region Wj, the supply quantity is given by N, Rj, while for a destination
region Wj, the demand quantity is given by Rj Nj.
Efficient algorithms (e.g. [13], [25]) had been proposed in the literature to solve m x n
bottleneck transportation problems in which there are m suppliers and n destinations. Of
these, the algorithm due to Garfinkel and Rao [13] has a polynomial time complexity given
by O((m + n )3log(mn)). The special case of m = 2 (or n = 2) can be solved optimally in
O(n) (or O(m)) time using the algorithm proposed in [29].

3.2 Reducing regional resource requirements

Now we address the case when the bottleneck transportation formulation is infeasible, that is,
when C I Rk > NA. This does not necessarily mean however that the load balancing
problem LBP is infeasible. In this case, we reduce the total resource requirement so as
to make it equal to the number of resource units available. Each region has its resource
requirements reduced with the restriction that there is at least one resource unit required in
each region after this reduction. The latter restriction is required so as to ensure that job
migration takes place only within the regions. Before we use the following procedure, we first
find the set of regions Wj for which Rj > IWj1, the number of nodes in the region. For each
region Wj in this set, let Rj
node. Then we apply the following heuristic procedure to reduce the resource requirements
further if necessary.

Procedure ReduceDemand;
(* Reduces the resource requirements for the regions so that the total requirement is equal
to p, the number of resource units available. This is done so that the reduction is somewhat
equally distributed among the regions and there is at least one resource unit in each region

We claim that the above procedure terminates with the new resource requirements R' >
R'... > R > 1 such that C1 R' = p. The termination of the procedure follows from the
fact that A strictly decreases during each iteration. To prove correctness, we easily prove
by induction the following loop invariants : (a) Y R p = A and A < (Ri 1), (b)
A > 0 implies that k' > 0 and (c) R1 > R2 > ... R, > 1 for all 1 < i < m. The procedure
has a time complexity of O(m2), since the first step has O(m log m) complexity and there
are O(m) iterations of the While loop with each iteration having O(m) complexity.
With this reduction in resource requirements, the average job response time is increased
to T' < T + maxi
certain threshold set by the system designer, we recommend the alternative of increasing the
number of resource units if possible. Since it is not desirable for the load balancing algorithm
to increase the number of resource units automatically, we will not address that issue here.
After we solve the bottleneck transportation problem, we need to determine the exact nodes
the resource units migrate to, in accordance with the optimal solution to the transportation
problem. We discuss this issue in the next section.

3.3 Determining new locations for resource units

The optimal solution to the bottleneck transportation problem determines the number of
resource units that need to migrate from one region to another. As discussed in Section 3.1,
the regions either supply excess resource units or receive deficit resource units. For those
regions for which resource requirements equal resource units available, we keep the resource

units in their original locations. For the remaining regions, we have to determine the new
locations for the resource units such that the number of units that migrate from one region
to another corresponds to the optimal solution of the bottleneck transportation problem.
We use the following greedy random assignment procedure to determine the new locations
for the resource units. In a later section, we will show how certain partitioning schemes can
guarantee that any arbitrary random assignment yields a bottleneck resource migration cost
that is close to the optimal bottleneck resource migration cost of the transportation problem.
A word on notation. For a region Wj, I' and Ij respectively denote the set of initial and
new locations of the resource units in region Wj.

Procedure FindNewResourceLocation;
(* Given the optimal solution {z1k}1
the set I of new locations of the resource units as well as the resource migration function f
using a greedy random assignment *)

This procedure takes O(p) time where p is the number of resource units initially available.
After we determine how the resource units migrate, it only remains to determine how the
resource access requests are assigned to the resource units within the regions. This problem
(which we call the job migration problem) is solved independently for each region. It is the
subject of discussion in the next section.

3.4 Determining job migrations

The problem of determining optimal job migration within each region can be posed as follows:

Note that for ease of discussion, we have absorbed the resource processing capacity q into
the resource requirements and similarly the unit distance job migration cost cj into the

inter-node distances; we have also substracted the fixed cost of job migration Fj from the
desired response time. This job migration problem is NP-hard since a special case of this
problem with equal values of dy, for all x, y V, becomes identical to the multiprocessor
scheduling problem that is already known to be NP-Complete [12]. For determining job
migration within a region, we ignore the communication costs and attempt to balance the
resource access requests evenly among the nodes that contain the resource units after re-
source migration. In other words, we attempt to determine g : V I so as to minimize
maxyi {xEv\_ (x)=y} bx. Since this problem is identical to the multiprocessor scheduling op-
timization problem, we use one of the approximation algorithms that had been proposed for
the latter problem. The heuristics behind these approximation algorithms are quite similar
to the heuristics proposed for the bin-packing problem [12]. The only difference between
the bin-packing problem and the multiprocessor scheduling problem is that the latter can
be viewed as the problem of packing a fixed number of bins as evenly as possible while
the former problem is that of packing the items using a minimum number of bins of fixed
capacity.
Before we discuss two of the heuristics proposed for the multiprocessor scheduling prob-
lem, we have a few words on bounds on the optimal value. Let T* be the optimum value
given by T* = miny T(g) = maxyEi Fx(V b(x)=y} bx. It has been proved [8] that L < T* < U
where L max ,max, b and U =max 2Ex max, bY
In the best-fit decreasing heuristic, a bin capacity of L is initially assumed and the items
(jobs) are considered in non-increasing order of their sizes. At each step, an item is attempted
to be fitted into the bin that has the smallest capacity left after the item is put into the bin;
in case more than one bin has the same smallest capacity, then the tie is broken arbitrarily.
When no bin can fit the item with the current capacity, the capacity of all the bins is
increased by the smallest amount that is necessary to fit the item into a bin. The worst-case
time complexity of the best-fit decreasing algorithm is given by O(IVI log VI + IVI||I).
In the multi-fit algorithm proposed by Coffman, Garey and Johnson [8], binary search
is employed on the interval [L, U] to arrive at the smallest bin capacity for which the first-
fit decreasing heuristic can fit all the items into the given bins without increasing their
capacities. Though the number of iterations is bounded by O(log (max, b,)), in practice it
is not allowed to exceed a predetermined constant k. The time complexity of the multi-fit
algorithm is O ( V log IV + k IV I log I |); we assume that b,'s are integers. The multi-fit
algorithm has been shown to give solutions that are no more than 6 times the optimal value
T* and can be as much as 3 times T* for some instances [11]. Friesen and Langston show
[11] that a modification of the multi-fit algorithm provides solutions that are no more than
72 times T*
We use the best-fit decreasing heuristic for our job migration problem due to its simplicity
and its potential for providing good approximate solutions on the average. The algorithm

with time complexity O(IVI log IVI + IV||II) is given below.

Procedure FindJobMigration(V, I, {b,}jv; g, {6(y)) }y);
(* Given the set of V of nodes along with their resource access request rates b,'s and the
set I of nodes that has resource units, finds a job assignment function g : V I using the
best-fit decreasing heuristic and the corresponding processing delays 6(y)'s *)
1. Sort V in non-increasing order of their request rates (b,'s). Let bl > b2 > ". > bl where
= IVI.
2. Let the capacity C'
k 1,2,. . ,p.
3. For each i = 1, 2,...,1 do steps 4-9.
4. Let J {1 0}.
5. If J # 0 then
6. find h e J such that C (L, + b;) = minkj C' (Lk + b;).
7. Dh
else
8. find 1 < h < p such that (L +b;) C = mlinl
9. C
10. We randomly assign the nodes in I to integers in the set {1,2,...,p}. For each integer
1 < i < p, if i is assigned to node u e I, then let g(x)
End FindJobMigration.

4 Approximation algorithm

In this section, we give the approximation algorithm for solving the load balancing problem
based on the ideas and procedures discussed in the last section.

Procedure Approximation algorithm;
(* Given all the necessary inputs to the LBP, determines an approximate solution (I, f, g),
where I is the set of new locations of the resource units, f is the resource migration function
and g is the job migration function. It also computes the bottleneck resource migration cost
R(I, f) and the job response time Q(I, g) *)

1. Partition the set of nodes V into m regions where m is at most equal to p, the number
of resource units initially available. Some suitable partitioning criteria are discussed
in Section 5. Let W1, W2,..., W, be the regions.

2. Compute the following quantities, the definitions of which are given in Section 3.1
(a) average communication distance S' for all 1 < j < n, (b) average communication

distance for resource migration from region Wj to region Wk for all 1 < k j < m
and (c) total resource requirements in region Wj for all 1 < j < m.

3. For each region Wj, determine the number of resource units required based the desired
response time T as follows: R, = [-Fr-c.)q I
= F cj.S')q].

4. For each region Wj for which Rj > IW 1, decrease Rj to IW1.

5. If F7' Rj > p (the number of resource units available), then apply the procedure
ReduceDemand given in Section 3.2.

7. Apply the procedure FindNewResourceLocation given in Section 3.3 to determine
the set I of new locations of the resource units as well as the resource migration function
f.

8. For each region Wj, let Ij be the set of locations of resource units in Wj after migra-
tion. For each 1 < j < m, invoke procedure FindJobMigration(TW, Ij, {Il}x^ ;
gj, {6(y))}yei), where gj : Wj Ij is the job migration function restricted to Wj. This
procedure is given in Section 3.4.

Complexity analysis: We assume that the partition information is provided by the user
and hence step 1 takes O(m) time. Step 2 takes O(n2) time where n = IV is the number
of nodes in the network. Steps 3 and 4 take O(m) time. Step 5 takes O(m2) time. Step
6 takes O(m3 log m) time if we employ the algorithm of [13] to solve the m x m bottleneck
transportation problem. Step 7 takes O(p) time where p is the number of resource units and
m < p < n. Step 8 takes O(n log n + nm) time. Step 9 takes O(n) time. Thus the worst-case
time complexity of the whole algorithm is given by O(m3 log m + n2) with the complexity of
steps 2 and 6 being dominant.

5 Performance bounds

In this section, we show how our approximation algorithm can guarantee theoretical bounds
for the solutions it provides under certain conditions that are satisfied by the partition
scheme. Before we present the results, we define the notion of an (61, 2) partition. An
(e6, 62) partition (with 61, 62 > 0) is a partition of the network into regions W 2, W2,... W,
such that the following are satisfied: (i) for each pair of regions Wj and Wk with j k,
xE < 61 for all x G Wj and y G Wk and (ii) for each region Wj, < 2 for
gk J
all x,y E Wj. Note that S' Z Z( _) whereas E'k -= II _-II) ; I' is the set
of initial locations of resource units in region Wj. Our first result relates the approximate
resource migration problem discussed in Section 3.1 to the load balancing problem LBP
defined in Section 2.

Theorem 1 For an (1, 62) partition scheme, let the approximate resource migration problem
among the regions be formulated as a (feasible) bottleneck transportation problem with desired
job response time T and let the optimal solution to that problem have E as its value. Ti, ,
our approximation algorithm provides a solution (I, f, g) to the LBP such that (i) E(1- i) <
Q1
R(I, f) < E(1 + 6) and (ii) Q(I,g) < max(2,1 + 62)T provided that 2' > maxw, bx for
all the regions Wj.

Proof: Suppose the bottleneck transportation problem has an optimal solution {zjk}
with optimal value E. Using the FindNewResourceLocation procedure discussed in
Section 3.3, we can determine a resource migration function f : I I' -- I' that is consistent
with the solution to the transportation problem. Then

The last inequality follows from the definition of an (61, 2) partition. We can prove in
a similar fashion that R(I,f) < FR + cR(1 + ei) max{(j,k)| z,>0} Efk. Since E = FR +
cRmax{(j,k)lzk>o0} E:k, we thus show that E(1 e1) < R(I,f) < E(1 + e1).
Once we determine f and I, we can determine for each region Wj, the job migration
function gj : Wj -- Ij using the FindJobMigration procedure given in Section 3.4. This
procedure uses the best-fit decreasing heuristic that can easily be shown (using an argument
identical to the one employed in [8] to show a similar bound for the multi-fit heuristic) to
give a solution with the delay 6(y) < Uj for all y G Ij. Here Uj = max 2 2 maxiEwx 3 ;

Here equation 1 follows from the definition of an (61, 2) partition scheme. Equation 2 is a
consequence of the bound (discussed above) given by the best-fit decreasing heuristic which is
used by our algorithm. Equation 3 uses the assumption that 2 > maxEw b1 and the fact
Q1
that Ij < I Wj Equation 5 follows from the fact that |I1 = C=l zkj > Rj > (T-F-cS'

Remarks: The constant max(2, (1 + 62)) used in the bound on Q(I, g) can be improved
to max ( (1+ 62)) if the modified multi-fit algorithm -.- 1 in [11] is used in the
FindJobMigration procedure instead of the best-fit decreasing fit heuristic. But we prefer
the latter due to its simplicity.
In many practical applications, the network can be naturally partitioned according to
an (61, 62) partition scheme with 61 and 62 very small and usually less than 1. Theorem 1
needs the constraint that the resource requirements at a node in a region are no more than
twice the average resource requirements within the region. We expect this constraint to be
satisfied in most instances. Next we show how a feasible solution for the LBP is related to
a feasible solution for the bottleneck transportation formulation which is the basis of our
approximate resource migration problem among the regions. The proof of this result makes
use of the following lemma.

Lemma 1 Consider a bin-packing problem in which each item x has a size s(x) EG and the
items are grouped into 1 classes SI, S2, ..., Si along with the ,'I.,. i,./ limits for these classes
given by al < a2 <
items with the restriction that for all 1 < j < 1, and for all items x G Sj, the sum of the
sizes of the items packed in the bin that contains x should not exceed aj. Tbi, ,

=

Proof: We omit the proof which works by induction on the number of classes 1. m

Now we are ready to state our second main result regarding the performance of our
algorithm.

Theorem 2 Consider an (61, 2) partition scheme with 0 < 61, 2 < 1, that also satisfies the
following : for each pair of regions Wj and Wk with j k k, (1 e2)S < (1 ci)E>k. Let
(I, f,g) be a feasible solution to the LBP with R(I, f) = E and Q(I,g) < T. Tl., there
exists a feasible solution to the bottleneck transportation problem with the desired response
time of no more than -T provided that the solution to the LBP satisfies the following
Ii 7> for all 3 1,2,...,m.

Proof: The idea behind the proof is to transform the given feasible solution for the LBP
to a feasible solution for the bottleneck transportation problem formulated with possibly
higher desired response time. Notice that a solution for the transportation problem only
needs to specify the number of resource units to move from one region to another. The
transportation problem requires that after resource migration, all the resource access requests
within a region be satisfied by the resource units that exist within the same region. Since
this requirement is not necessarily satisfied by the given solution for the LBP, we move the
resource units among the regions so as to meet this requirement. This task forms the main
ingredient of the proof.
Let (I, f, g) be the given feasible solution to the LBP that satisfies J J > 2. for some
r. Note that this requires that I Ij > 2 for all j 1,2,... m. For all pairs of regions Wj
and Wk with j : k, let z'k indicate the number of resource units that need to move from Wj
to Wk according to the solution (I, f,g). Thus

Z Z = I/jl- (6)
k#7j k#j

for all j = 1,2,..., m; here Nj is the number of resource units that are initially (before
migration) in the region Wj. For all 1 < j, k < m, we define Gjk to be the set of nodes
in region Wj that access the resource units in region Wk if resource and job migration take
place according to the solution (I, f,g). In other words, Gjk = {x E W lg(x) e Wk}. Also
let Hjk = xEG k b1. For a region Wj, we also define the following quantities:

1
,jk l= -,) q' for k /j and (7)

3 r t (8)
(rT F cj(l 62)Sj q

3jk indicates the minimum processing capacity required for a resource access request from a
node in region Wj to a node in region Wk if the desired response time is no more than rT.
Since (1 62)S < (1 e1)Ek, it is easy to see that 35 < 3>-.

For a pair of regions Wj and Wk with j z k, let

yjk = k + QkjHkj (9)

The second term in the above equation indicates the resource capacity in Wj that is used
by the nodes in region Wk if resource and job migration occur according to (I, f, g). Let Aj
indicate the excess resource capacity available in region Wj given by

Let {xjk}jEJ+,kJJ+ be an optimal solution to this transportation problem. We claim that
the following solution is feasible for the transportation problem defined in Section 3.1 for a
desirable response time of iT.
(1-2)"*

x k
R.
zjk =
0

Note that for all = 2,..., m, R
the following:

Nj X k
kJ+
Nj + x*J
k EJ+

ifj E J+ and k J+
ifj =k andj E J+
ifj = k andj J+
otherwise

u Ic Thus it is only required to show
( --,-For-c S')

> Rj, for all j j+

> Rj, for all j J+

7 [eJ+
jEJ+

By equations 14 and 15, the above inequalities are implied by the following equations:

N [Aj] > Rj, for all j J+
N,+ L-A]J > Rj, for allj J+

Thus the rest of the proof is designed to show that for all j = 1, 2,..., m, N Aj 1 > Rj.

Since the given solution to the LBP is feasible, F +cJ .)+ 6(gr(x)) < T for all nodes x. For
a node rx Gkj with k f j, dgi(1) > (1 1)E'j while for a node x G Gjj, dx,(x) > (1- c2)S;
these follow from the definition of the (61, 2) partition scheme. Thus for a node x E Gkj,
the following is true:

6(gr(x)) = E g(y)g by <
q T

cj(1
c (1

6)E'j if k
S2) S if k = j

Now we apply Lemma 1 to each region Wj. Here the resource units in the set I1 are the
bins and the items are the resource access requests. There are m classes of items Gkj where
k = 1, 2,..., with the capacity of class Gkj defined by (T- Fj cj(l e)Ek) q when
k 7 j and (T- F cj(1 )S) q when k = Then by the result of Lemma 1, we
conclude that

> (rT-Fj cj(1

> PkjH=
>' k=l

S+ j-

1)Ek) q) +rT

2)S) q

F c(1 62)S) q

The last but one inequality requires that 61, 62 < 1 and the last inequality follows from
equations 7 and 8. But for all j = 1,2,..., m, Ij, > i2, implies that

Remarks: In Theorem 2, tighter bounds are obtained by decreasing the value of r but
r must be strictly greater than 1. Smaller the value of r, higher should be the number of
resource units in each region in an optimal solution to the LBP. The feasible solution to the
bottleneck transportation problem derived in the proof of Theorem 2 has its bottleneck cost
given by the optimal value of the transportation problem formulated through equations 14
and 15. Though a good bound for this value cannot be estimated theoretically, we expect
that it will be between E(1 e) and E(1 + ei) in most cases, where E is the bottleneck cost
of the given feasible solution to the LBP.
Theorems 1 and 2 guarantee only loose accuracy bounds for our approximation algo-
rithm but we can obtain much better solutions in practice. This has been confirmed by our
experiments which will be discussed in the next section.

6 Experimental Results

We conducted experiments to evaluate the performance of our approximation algorithm and
in particular, the effectiveness of the partitioning criteria -'-,. -1. .1 in Section 5. Since we
need to compare the approximate solutions obtained by our algorithm to the optimal solu-
tions, we implemented an algorithm based on branch-and-bound search to find the optimal
solutions to the LBP. In this section, first we will explain how the branch-and-bound search
was employed in the algorithm to find optimal solutions. Then we will discuss how the ex-

periments were performed to evaluate our approximation algorithm and then interpret the
results obtained through these experiments.
Since the LBP is a constrained optimization problem, we require a two-level branch-and-
bound search, the first level for determining resource unit migrations and the second level
for determining job migrations. Due to memory limitations, we used the depth-first order
for exploring the nodes in the branch-and-bound tree at both levels. Each node in the tree
at the first level corresponds to a partial resource migration function while each node in the
tree at the second level corresponds to a partial job migration function. At the first level, a
node branches into two children nodes that correspond to the two choices of including and
excluding the migration of a resource unit between two nodes in the network. network that
does not have a resource unit. At the second level, branching of a node is based on the two
choices of including and excluding the assignment of resource access requests from a node to
a node that has a resource unit. A complete solution is one in which every resource unit is
assigned to a new location. Whenever we reach a complete solution node at the first level,
we conduct the branch-and-bound at the second level to determine a feasible assignment
of resource access requests from the nodes in the network to the new resource locations.
In a feasible assignment, the response time constraint is satisfied for every resource access
request. We terminate the search at the second level, whenever we find a feasible assignment.
In this case, the node at the first level where the second level search originated is said to be
a feasible solution node.
Pruning of the nodes in the branch-and-bound tree at both levels was based on bounding
functions that were used to estimate the lower bounds for the partial solution nodes at the
two levels. The lower bound functions are based on the bottleneck cost of partial solution and
the costs of remaining candidate edges for completing the solution. In our experiments, we
observed that approximately 511 of the nodes were pruned on the average at both levels of
the branch-and-bound tree. This indicates that the pruning power of our bounding heuristics
is good in spite of their simplicity.
For our experiments, we selected 3 networks containing 8, 12 and 15 nodes respectively.
In order to make the experiments conform with real life applications, we chose U. S. cities as
nodes and distances between them (in units of 100 miles) were used as the basis for internode
distances. The corresponding distance matrices are given in Tables 1, 2 and 3. Since the
branch-and-bound algorithm required 2 CPU hours to solve some instances of the 15 node
example, we did not include larger networks in our experimental evaluation. For each of
these networks, we evaluated our approximation algorithm under four different partitions
for various resource access rates. The intra- and inter-region metrics for these four partitions
are given in Tables 4, 5 and 6. Though the relevant characteristic of a partition is difficult
to be expressed by a single metric, we conjecture that the ratio of the average intra-region
distance to the average inter-region distance is a good measure that can be employed for this

purpose. We also determine for each partition, the smallest values of 61 and 62 for which it
is an (61, 62) partition scheme. When the access rates are large and the resource units are
scarce, another relevant characteristic of a partition is how evenly the nodes are distributed
among the different regions. We chose the four partitions for each example so as to have
variation in terms of these characteristics. For example, in the one extreme, we clustered
the nodes according to their geographic proximities while in the other extreme, we clustered
the nodes without regard to their proximities.
In each experiment of 100 trials, we used the following input data: (1) distance matrix
of the given network, (2) number of resource units, (3) average resource access request rate
with allowed deviations and (4) partition of network into regions. The number of resource
units was chosen so as to be approximately one third of the number of nodes in the network.
For simplicity, we let FR = Fj = 0, cR = cj = 1 and q = 1. In each of these trials, the
initial locations of the resource units were r,,ii1.,,'tl chosen and the resource access request
rates for the nodes were rg, I1i..,,l generated based on an uniform distribution with the given
average access rate and the allowed deviations from the average. For each such instance, an
optimal solution to the LBP was determined using the branch-and-bound approach. The
approximation algorithm of Section 4 was then employed for the given partition to determine
an approximate solution to the LBP. The bottleneck transportation problem was solved using
the algorithm of [25]. Though the branch-and-bound algorithm took as many as 2 CPU hours
to solve some instances, the approximation algorithm gave solutions almost instantaneously.
We then compared the approximate solution to the optimal solution with respect to (i)
bottleneck resource migration cost and (ii) job response time. For the branch-and-bound
(optimal) solution, we let the job response time be equal to the desired job response time
T. In order to allow consistent comparisons among the results obtained under different
partitions, we generated an identical sequence of access rates and initial resource locations
for all the partitions of a given network.
The experimental results are given in Tables 7, 8 and 9. We let the number of resource
units be respectively equal to 3,4 and 4 for the 8-node, 12-node and 15-node examples. In
these tables, column 1 indicates three different (uniform) distributions 10 2, 10 6 and
20 4 used in our experiments. Column 2 indicates the desired response time T. Column 4
indicates the average increase in response time of our approximate solutions with respect to
the desired response time. In all the 100 trials, the LBP has a solution with less than or equal
to this response time. Column 6 gives the number of trials in which the approximate solution
has a response time less than or equal to T and column 7 gives the average decrement in
response time for these trials. Column 5 indicates the average increase in bottleneck resource
migration cost of the approximate solution with respect to the optimal solution. We make
the following observations from these results:

1. As far as response time is concerned, there is a strong correlation, as expected, between

the ratio of average intra-region distance to the average inter-region distance for a
partition and its performance. Smaller this ratio, better its performance. Thus for the
8 node example, partition (d) is the best while for the 12 node and 15 node examples,
all the partitions (b), (c) and (d) give better results than partition (a).

2. When the access rates are very high and the resource units are scarce (close to number
of regions), in addition to the above measure, the evenness of the number of nodes
among the regions also plays a major role. This is because when the number of nodes
is not evenly distributed among the regions, there is a large variation in the resource
requirements among the regions that cannot be overcome by resource migration among
the regions. Note that in the 12 node example, since one of the regions must have 2
resource units, partition (b) is the best in this respect having a node distribution closer
to 6-3-3. Thus for the 12 node example, partition (b) is the best while for the 15 node
example, partition (c) is the best. For the same reason as above, partition (c) for the
8 node example, partition (d) for the 12 node example and partitions (b) and (d) for
the 15 node example give good results for small access rates but poor results for large
access rates.

3. It is encouraging to note that under reasonable response time requirements, the ap-
proximate solutions for the best partition give close to optimal response time for all
the access rates.

4. For the partitions with small ratio of intra-region to inter-region distance, the resource
migration cost increases due to the increase in average inter-region distance. Thus
there is a trade-off between the intra-region and inter-region distances that should
reflect the trade-off between job migration and resource migration. Since resource
migration occurs less frequently than job migration, smaller ratio of intra-region to
inter-region distances is always preferable.

5. In our experiments, the initial locations of the resource units are randomly generated
for each trial. This tends to give rise to resource migration between the regions in our
approximate solution at every trial. In practical applications, resource units tend to
remain within the same region for small fluctuations in the resource access rates and
hence resource migration cost will be incurred very rarely.

In summary, we recommend a partition scheme with the smallest ratio of average intra-region
to average inter-region distance and in case when the resource units are almost equal to the
number of regions, equal number of nodes per partition is desirable.

In this paper, we have proposed a load balancing model with resource migration in dis-
tributed systems that can be employed in many practical applications such as file migration
in distributed databases. To tackle the intractability of the model, we have proposed a novel
approximation approach based on partitioning the network into regions. This partitioning
approach has been shown to have a very low time complexity and yet has been demon-
strated through experiments to provide reasonably close to optimal solutions. In contrast,
the branch-and-bound algorithm that determines an optimal solution cannot be used to solve
the load balancing problems in real time.
The performance of our approximation algorithm very much depends on how the parti-
tioning is done. Our experiments demonstrate that the lower the ratio of the average intra-
region distance to the average inter-region distance, better the performance of the algorithm
in terms of job response time. Most distributed systems based on wide-area networks can
naturally be partitioned into geographical regions. For this reason, the partitioning task is
left to the system designer to specify. But if the partitioning needs to be performed by our
load balancing algorithm, we can make use of clustering algorithms proposed in the literature
(see [16] for a good review of clustering algorithms). In particular, the k-means clustering
algorithms attempt to find a partition of k regions so as to minimize the sum of the squares
of the deviations of the intra-region distances from the regional averages. Usually these
algorithms produce suboptimal solutions. Though the clustering or partitioning problem is
NP-hard in general [12], it needs to be solved only once at the time of system design and
it can remain the same so long as the network remains the same.
It is interesting to explore local search techniques that give good approximate solutions
on the average and can efficiently be parallelized. One such approach we are currently
investigating is to map the problem directly onto Boltzmann machine models which are
parallel networks with simple computing elements. This is the subject of our ongoing and
future research.
Acknowledgement : We thank Injae Hwang at the University of Florida for his invalu-
able assistance in performing the tedious experiments required for this project. We also thank
Dr. Eva Ma for providing some insight at the early stages of this project.

Ravi Varadarajan received his Bachelor of Engineering (B.E.) (Honors) degree (1978) from
the University of Madras, India and his Ph.d. (1987) degree in Computer Science from the
University of Pennsylvania, Philadelphia. Since 1987, he has been an Assistant Professor in
the Computer and Information Sciences department of the University of Florida, Gainesville.
His research interests include design of efficient sequential and parallel algorithms, parallel
architectures, distributed processing and theoretical computer science. He is a member of
the IEEE Computer Society and ACM.