Algorithm Analysis

The algorithms in this document can be run and tested individually in
NetBeans. Instead of creating a project for each, create a single project,
Miscell, with separate main classes for each.
We do not particularly need the auto-created Main class,
so delete it if you'd like.

For each of the sample programs do the following:

From the Projects window, right-click either on

the Miscell project

the Source Packages entry

the miscell package (this is most efficient)

Select New → Other and from the list select
Java Main Class (only need to go through Other once).

In the pop-up window (Name and Location), set the
Class Name something related to the algorithm of interest.
For example, you might be creating entries like this:

Class Name: LinearSearch1, LinearSearch2, ...
package: miscell

In some situations you'll want to create a simple Java Class,
not Java Main Class.

Replace the class content by the suggested content by copy/paste.

In every case you will need the import statement:

importjava.util.*;

Run the program by locating it in
Source Packages → miscell,
right-clicking and selecting Run File.

Repeated runs are easily done
by clicking the "rerun" button
in the output window.

To make repeated runs even easier, select Customize in the
drop-down below <default config>. In this window,
use the Browse button to specify the Main Class.
With this in place, you can use the "run project" button
in the menu bar.

Linear Search

For example, consider simple linear search of an integer array for a given key.
The following class representing a simple random linear search might be
something like this:

LinearSearch1

Coding

In order to create a general version which could be reused, we should
consult what Java does in the Arrays class:

Other Java types

Consider expanding the linear search algorithm to other types such as
float or String.
These two are very different because float, like int is
a primitive type and String is an Object.
With respect to float, or any of the other primitive types,
we have to write separate functions for each even though they effectively
do the same thing.

Regarding Object types, such as String,
we would want a version like this:

The problem with generating an example is how to "meaningfully"
generate an array of Strings and a search key.
A simple example (not terribly meaningful) is this:

LinearSearch3

Abstract time for linear search

Regarding the notion of "abstract time" we usually say to
"count the number of comparisons" to get a measure of the actual time.
There are several deviations
from reality:

we're overlooking any setup, like initializing the variables
found and i,
as well as other repetitive features,
like incrementing i comparing it to the size

We're ignoring the actual cost of a comparison.
The Object comparison

A[i].equals(key)

say, for String type, is not constant and is likely to be
more time-consuming than comparison of primitive types.

Nevertheless, simplifications such as this are useful for
algorithm analysis because we can more easily go on and
describe the behavior for arbitrarily large arrays.

Analysis of Linear Search

The worst case is if the element is not found or is found at the end
of the search in which case there are n comparisons.
The best case is that the thing you're looking for is in the first slot
and so there's only one comparison. We say:

the best case is 1 comparison

the worst case is n comparisons

Average case for linear search

A simple guess might be to take the average of the best and worst, getting
(n+1)/2. This answer turns out to be correct,
but we must derive it more methodically.

First of all, we need to make the assumption
that the search is successful.
Why? Consider an array of size 10, holding arbitrary integers and
the key is an arbitrary integer. Given that there are about 4 billion
32-bit integers,
the probability that our key is one of these 10 is effectively zero!
In addition, we will assume that the key is equally likely to be found
at each of the n positions.

Using this notion, the average cost is:

( total cost for finds at all positions ) / number of positions

The cost for finding at position i is (i+1), for values
i = 0 ... n-1. Therefore we derive:

1 + 2 + ⋅⋅⋅ + n

n*(n+1)/2

n + 1

average cost =

=

=

n

n

2

Language of Asymptotics

Asymptotics has to do with describing the behavior of functions,
specifically algorithmic timing functions, for arbitrarily large problem sizes.
Often a problem size can be characterized by a single parameter n,
e.g., the size of an array.

Big-O Notation

We would really like to present information about this algorithm which
ignores the constants. The "big O notation" and other complexity terminology
allows us do precisely that. We say:

T(n) = O(f(n)) or T(n) isO(f(n))
if there are positive constants C, k,
such that T(n) ≤ C * f(n) for all n ≥ k

The big-O notation is simply a convenience for expressing the relation
between an unknown "timing function", T(n), and a known
reference function, f(n).
In reality O(f(n)) is a class of functions,
of which T(n) is a member. Nevertheless, it is convenient to
be able to make statements like this

"the worst case time for such-and-such-algorithm is O(n*log(n))"

and have a rigorous basis for what you're saying.

The idea of "n ≥ k" in the definition means
eventually, i.e., if we ignore some initial finite portion.
For example, suppose

T(n) = n + 100

We can say:

T(n) ≤ 101 * n, for all n ≥ 1

but we can also say

T(n) ≤ 2 * n, for all n ≥ 100

getting a "better" asymptotic constant, C, in the sense that
it is smaller (we can always make it larger).
In either case we have proved that

n + 100 = O(n)

by finding constants C and k which make the
definition statement work.

Linear Search in big-O terms

Going back to linear search we observed that when counting comparisons:

the best case is 1
the worst case is n
the average case is (n+1)/2 = ½ n + ½

In big-O terminology, we would say this about linear search:

the best case time is O(1)
the worst case time is O(n)
the average case time is O(n)

Binary Search

Binary search searches a sorted array in the most
efficient way possible. This algorithm employs a simple example of a
divide-and-conquer
strategy in which we subdivide the problem into equal-sized "sub-problems".
The idea is simple: compare the key to the "middle" element,
if not equal, either look left or look right portion based
on whether the key is less than, or greater than, the middle element.

First Binary Search Implementation

This algorithm expresses itself most naturally in a recursive manner based
on the way I say:

search the whole array invokes search of one half or another.

In order to express the recursive nature, the parameter of the algorithm
must allow arbitrary beginnings and ends.
Our initial coding might look something like this:

The else keywords are optional because of the
return statements.
One has to keep in mind that the division used to compute mid
is integer division (i.e., fractional truncation).
As is the case with linearSearch,
the second range parameter, toIndex, is not included in the search
range. Our textbook makes, in my opinion, the unfortunate
decision to make the second parameter position in his array-based
algorithms the rightmost index, so beware.

Binary Search Visitation tree

Lay out the array positions that the algorithm would visit in a
binary tree, where the root is the middle of the array,
the left and right subtrees are generated by searches of the
left and right subarrays, respectively.
Here are the binary search trees for arrays of size 7 and 10, resp.:

Note the "left-leaning" aspect of the latter tree. This reflects the fact that
our algorithm, when it cannot split the array exactly in half, will have one
more element on the left size compared to the right side.

Algorithmic correctness proof by induction

The proof uses a form of induction called strong induction,
whereby we assume true the thing we want to prove for all values
(within a suitable range) up to that point.
Consider the execution of:
The execution of

int pos = binarySearch(A, fromIndex, toIndex, key)

We want to say that:

if pos ≥ 0 then
fromIndex ≤ pos < toIndex
and
A[pos] == key

if pos < 0 then A[i] != key, for all
fromIndex ≤ i < toIndex.

The proof is by induction on the array size, len = toIndex - fromIndex
It is a good idea to run a few examples by hand.
Again, keep in mind that computer integer division is truncated.
In mathematical terms, division of two integers with integer result:

a/b

is expressed mathematically
as the "floor" function flr, which simply truncates any
decimal part.

flr(a/b)

Base case: len = 0

This means that toIndex == fromIndex.
The key cannot be present
in the array and the algorithm indicates failure.

Inductive case: len ≥ 1

Because the array is sorted,
it is obvious that the algorithm will work so long as:

fromIndex ≤ mid < toIndex

the left (fromIndex,mid) and
right (mid+1,toIndex) ranges
both have fewer than len elements.

The even and odd cases values need to be considered separately.
Write: toIndex = fromIndex + len and compute:

Therefore, regarding big-O logarithms of all bases are equal.
In scientific computations, log(n) is understood to be
the base-10 logarithm and ln(n) the natural logarithm.
In computer language library functions, log(n) often means
the natural logarithm and base-2 logarithm is usually written with
an explicit base. For example, in Java, the function Math.log
is the natural logarithm, but we can easily write the base-2 logarithm:

staticdouble log_2(double x){returnMath.log(x)/Math.log(2);}

For our purposes, since the base-2 is preeminent, we'll drop the 2 and assume:

log(n) = log2(n)

Integer logarithms

In computational settings in which logarithms appear,
they always appear in these integer formats:

flr(log n) = the largest power of 2 so that 2power ≤ n
ceil(log n) = the smallest power of 2 so that n ≤ 2power

Binary Search Worst Case

n even => left side n/2, right side n/2 - 1
n odd => both sides have size (n-1)/2 = n/2

Let
T(n) = worst case number of comparisons in binary search of
an array of size n

In the worst case, we would end up consistently going to
the left side with n/2 elements.
Couting 1 for the middle element comparisno we get this recurrence equation
for the worst-case number of comparisons:

These order classes are upwardly inclusive, i.e.,
if T(n) = O(log n),
then of course, T(n) = O(n). We're usually
interested in the "best fit" in the sense of finding the smallest order
class to which T(n) belongs.
In order to characterize the "best fit" of an order class, we need
two other notions:

Lower bound: Ω

We say:

T(n) = Ω(f(n)) (or T(n) isΩ(f(n)))

if there are positive constants C and k, such that

T(n) ≥ C * f(n), for all n ≥ k

Exact bound: Θ

We say:

T(n) = Θ(f(n)) (or T(n) isΘ(f(n)))

if

T(n) = O(f(n)) and T(n) = Ω(f(n))

This means that there are positive constants
C1, C2 and k, such that

C1 * f(n) ≤ T(n) ≤ C2 * f(n), for all n ≥ k

The Θ concept gives the precise sense
to the notion of "order class" because it completely characterizes the
behavior of a timing function relative to a reference
function up to a constant multiple.

Order Summary

Officially there are three considerations.

O: upper bound,
meaning "we can do at least this well" up to a constant factor

Ω: lower bound, ignoring constant factors,
meaning, "we cannot expect to do better than this" up to a constant factor

Θ: characterization of the run-time behavior, up to a constant factor

Unofficially, the big-O terminology dominates the discussion
in algorithmic analysis. Authors commonly use O even
when they really mean Θ.
If the exact order class is not known it means that a
complete mathematical understanding of the run-time behavior is lacking.

Other algorithmic terminology

Asymptotic dominance of one function by another
is expressed by the little-o notation:

T(n) = o(f(n))

This means that for everyc (no matter how small),
there is a k such that

T(n) ≤ c * f(n), n ≥ k

For the most part this means the following:

limn → ∞ T(n)/f(n) = 0

Asymptotic dominance expresses the relationship of the
reference functions in the order class hierarchy above:

1 = o(log(n))
log(n) = o(n)
n = o(n2)
n2 = o(2n)
2n = o(3n)
...

The first of these relations, log(n) = o(n), is proved using
L'Hôpital's rule from calculus,
substituting a continous variable x for the integer n:

and it means:
limn → ∞ T(n)/f(n) = 1
The "wavy" equal lines suggest that these two functions are
essentially the same for large values.
For example, in a polynomial function,
we can effectively ignore all but the highest order term.
For example, if

T(n) = 100 * n + 200 * n2 + 3 * n3

then

T(n) ≈ 3 * n3

Unfortunately the Weiss textbook does not define this relation.
Asymptotic equality is, in some sense, similar to the exact bound
Θ,
except that it gives a precise order constant, which is often of interest
when you want to compare two timing functions within the same order class.
For example, let

W(n) = worst case time for linear search
A(n) = average case time for linear search

Both functions are exactly linear time and we would write:

W(n) = Θ(n)
A(n) = Θ(n)

However, the order constants are different and this is expressed using ≈:

W(n) ≈ n
A(n) ≈ ½ n

Binary Search average case

The average case timing is more complicated.
As in the case of linear search, we assume a successful search,
and that each of the array positions are equally likely to hold the search key.
We are mostly interested in getting some sense about
how much better the average case
might be, and proving that it is still logarithmic.
Technically we want to prove the lower bound:

Average binary search time(n) = Ω(log(n))

Combined with the fact that the average time can only be better
than the worst-case time, which is
O(log(n))
we can then conclude that

Average binary search time(n) = Θ(log(n))

Additionally we want to get some idea about what the order constant might be.

Counting the total number of comparisons

In order to compute the average number of comparisons,
we need to find a way to compute the total
number of comparison for all possible nodes in the positional visitation tree.
The level of a node is its distance from the root.
The root, at level 0, counts for 1 comparison.
Both of its children count for 2 comparisons each, etc.
Thus,

Total comparisons = ∑(all nodes at level i) (i+1)

In general, at
level i, if it is full, there will be 2i children,
each contributing (i+1) comparisons.

A binary tree is perfect if every level is full.
In general, the binary search position visitation
tree will not be perfect, but it can be argued that

The levels 0 to flr(log n)-1
are all full: thus a total of L = flr(log n) levels are full.

The maximum level of a node is flr(log n) (which may not be full)

We reproduce the depictions of the binary search visitation trees for arrays of size 7 and 10:

flr(log(7)) = 2

flr(log(10)) = 3

Averge comparisons lower bound

In general, if the tree is not perfect, compute the number of comparison for
the tree without the last row, i.e., with L = flr(log n) levels.
Using the same formula, we get

The problem with introducing the error checking code inside the recursive function is
that it is inefficient; only the first invocation needs to check for an invalid range,
not all the recursive calls. A better approach is to have the publicbinarySearch function call a private recursive
function like this:

Using Comparators

In many cases we want to compare objects of a class which is not innately comparable by
dictating the manner in which the objects are to be compared. This
achieved by passing an external comparator object, i.e., one which implements
this generic interface:

interface Comparator<MyClass>

The requirement of this interface is to implement the function:

int compare(MyClass first, MyClass second)

The Java API for binarySearch supports the ability
to specify user-defined comparison by passing a comparator argument:

Binary vs. Linear search

The problem with binary search is that, although the search
time is much faster, the array must be sorted for it work.
The best algorithms for sorting a random array
have a run time of O(n * log n).
So there is no advantage of binary search
over linear search if every search is on a fresh array.
Here is how we compare the two algorithms:

Linear Search

Binary Search

create array:

O(n)

O(n)

prepare array:

-

O(n*log(n))

search array:

O(n)

O(log(n))

If, using a single array, we do
O(log n) searches
then, with the Linear Search total time is
O(n * log n) which would break even
with Binary Search. To put this in perspective, if we have an
array with 1 million, or 220 entries, then, after sorting,
we would need to do roughly 20
searches for Binary Search to break even with Linear Search.

Integer exponentiation

We want to write an algorithm to compute
xn for an integer n ≥ 0. The obvious
linear-time algorithm involves repeated multiplication by x.
A more subtle version uses the binary representation of the
exponent. The basis of the algorithm is this:

Proving the correctness of an iterative algorithm is technically
harder than the correctness of a recursive algorithm,
which is more-or-less a straight-froward induction proof.

In particular, an iterative algorithm uses
variables which change state in order to control the iteration.
A proof requires that you
establish a loop invariant, which is a logical statement
expressed using the program variables so that:

it is initially true with the variable initializations

assuming that it is true at some step, it remains
true at the next step using the modified variable values

when the loop is terminated, the invariant "proves" what
you want the loop to compute.