There are lots of algorithm-analysis questions around. Many are similar, for instance those asking for a cost analysis of nested loops or divide & conquer algorithms, but most answers seem to be tailor-made.

On the other hand, the answers to another general question explain the larger picture (in particular regarding asymptotic analysis) with some examples, but not how to get your hands dirty.

Is there a structured, general method for analysing the cost of algorithms?

This is supposed to become a reference question that can be used to point beginners to; hence its broader-than-usual scope. Please take care to give general, didactically presented answers that are illustrated by at least one example but nonetheless cover many situations. Thanks!

Thanks go to the author(s) of StackEdit for making it convenient to write such long posts, and my beta readers FrankW, Juho, Gilles and Sebastian for helping me iron out a number of flaws earlier drafts had.
–
Raphael♦Apr 9 '14 at 13:04

Hey @Raphael, this is wonderful stuff. I thought I would suggest putting it together as a PDF to circulate around? This sort of thing could become a really useful reference.
–
hadsedApr 12 '14 at 4:23

1

@hadsed: Thanks, I'm glad it's useful to you! For now, I prefer that a link to this post be circulated around. However, SE user content is "licensed under cc by-sa 3.0 with attribution required" (see page footer) so anyone can create a PDF from it, as long as attribution is given.
–
Raphael♦Apr 12 '14 at 7:31

1

I am not especially competent on this, but is it normal that there is no reference to the Master theorem in any anwer?
–
babouOct 12 '14 at 14:33

1

@babou I don't known what "normal" means here. From my point of view, the Master theorem has no business being here: this is about analysing algorithms, the Master theorem is a very specific tool for solving (some) recurrences (and very roughly at that). Since the mathematics have been covered elsewhere (e.g. here) I have elected to cover only the part from algorithm to mathematics here. I give references to posts that deal with working the mathematics in my answer.
–
Raphael♦Oct 14 '14 at 9:06

3 Answers
3

Translating Code to Mathematics

Given a (more or less) formal operational semantics you can translate an algorithm's (pseudo-)code quite literally into a mathematical expression that gives you the result, provided you can manipulate the expression into a useful form. This works well for additive cost measures such as number of comparisons, swaps, statements, memory accesses, cycles some abstract machine needs, and so on.

Let's say we want to perform the usual sorting algorithm analysis, that is count the number of element comparisons (line 5). We note immediately that this quantity does not depend on the content of array A, only on its length $n$. So we can translate the (nested) for-loops quite literally into (nested) sums; the loop variable becomes the summation variable and the range carries over. We get:

Example: Swaps in Bubblesort

I'll denote by $P_{i,j}$ the subprogram that consists of lines i to j and by $C_{i,j}$ the costs for executing this subprogram (once).

Now let's say we want to count swaps, that is how often $P_{6,8}$ is executed. This is a "basic block", that is a subprogram that is always executed atomically and has some constant cost (here, $1$). Contracting such blocks is one useful simplification that we often apply without thinking or talking about it.

$A^{(i,j)}$ denotes the array's state before the $(i,j)$-th iteration of $P_{5,9}$.

Note that I use $A$ instead of $n$ as parameter; we'll soon see why. I don't add $i$ and $j$ as parameters of $C_{5,9}$ since the costs do not depend on them here (in the uniform cost model, that is); in general, they just might.

Clearly, the costs of $P_{5,9}$ depend on the content of $A$ (the values A[j] and A[j+1], specifically) so we have to account for that. Now we face a challenge: how do we "unwrap" $C_{5,9}$? Well, we can make the dependency on the content of $A$ explicit:

But can this happen, i.e. is there an $A$ for this upper bound is attained? As it turns out, yes: if we input an inversely sorted array of pairwise distinct elements, every iteration must perform a swap¹. Therefore, we have derived the exact worst-case number of swaps of Bubblesort.

This can also happen: on an array that is already sorted, Bubblesort does not execute a single swap.

The average case

Worst and best case open quite a gap. But what is the typical number of swaps? In order to answer this question, we need to define what "typical" means. In theory, we have no reason to prefer one input over another and so we usually assume a uniform distribution over all possible inputs, that is every input is equally likely. We restrict ourselves to arrays with pairwise distinct elements and thus assume the random permutation model.

Now we have to go beyond simple manipulation of sums. By looking at the algorithm, we note that every swap removes exactly one inversion in $A$ (we only ever swap neighbours³). That is, the number of swaps performed on $A$ is exactly the number of inversions $\operatorname{inv}(A)$ of $A$. Thus, we can replace the inner two sums and get

which is our final result. Note that this is exactly half the worst-case cost.

Note that the algorithm was carefully formulated so that "the last" iteration with i = n-1 of the outer loop that never does anything is not executed.

"$\mathbb{E}$" is mathematical notation for "expected value", which here is just the average.

We learn along the way that no algorithm that only swaps neighbouring elements can be asymptotically faster than Bubblesort (even on average) -- the number of inversions is a lower bound for all such algorithms. This applies to e.g. Insertion Sort and Selection Sort.

The General Method

We have seen in the example that we have to translate control structure into mathematics; I will present a typical ensemble of translation rules. We have also seen that the cost of any given subprogram may depend on the current state, that is (roughly) the current values of variables. Since the algorithm (usually) modifies the state, the general method is slightly cumbersome to notate. If you start feeling confused, I suggest you go back to the example or make up your own.

We denote with $\psi$ the current state (imagine it as a set of variable assignments). When we execute a program P starting in state $\psi$, we end up in state $\psi / \mathtt{P}$ (provided P terminates).

Individual statements

Given just a single statement S;, you assign it costs $C_S(\psi)$. This will typically be a constant function.

Expressions

If you have an expression E of the form E1 ∘ E2 (say, an arithmetic expression where ∘ may be addition or multiplication, you add up costs recursively:

with some constant costs $c_{\dots}$ for the individual statements. We assume implicitly that these do not depend on state (the values of i and x); this may or may not be true in "reality": think of overflows!

Now we have to solve this recurrence for $C_{1,4}$. We note that neither the number of iterations not the cost of the loop body depend on the value of i, so we can drop it. We are left with this recurrence:

Note again the extra constant $c_{\text{call}}$ (which might in fact depend on $\psi$!). Procedure calls are expensive due to how they are implemented on real machines, and sometimes even dominate runtime (e.g. evaluating the Fibonacci number recurrence naively).

I gloss over some semantic issues you might have with the state here. You will want to distinguish global state and such local to procedure calls. Let's just assume we pass only global state here and M gets a new local state, initialized by setting the value ofp to x. Furthermore, x may be an expression which we (usually) assume to be evaluated before passing it.

We have covered the language features you will encounter in typical pseudo code. Beware hidden costs when analysing high-level pseudo code; if in doubt, unfold. The notation may seem cumbersome and is certainly a matter of taste; the concepts listed can not be ignored, though. However, with some experience you will be able to see right away which parts of the state are relevant for which cost measure, for instance "problem size" or "number of vertices". The rest can be dropped -- this simplifies things significantly!

If you think now that this is far too complicated, be advised: it is! Deriving exact costs of algorithms in any model that is so close to real machines as to enable runtime predictions (even relative ones) is a tough endeavour. And that's not even considering caching and other nasty effects on real machines.

Therefore, algorithm analysis is often simplified to the point of being mathematically tractable. For instance, if you don't need exact costs, you can over- or underestimate at any point (for upper resp. lower bounds): reduce the set of constants, get rid of conditionals, simplify sums, and so on.

A note on asymptotic cost

What you will usually find in literature and on the webs is the "Big-Oh analysis". The proper term is asymptotic analysis which means that instead of deriving exact costs as we did in the examples, you only give costs up to a constant factor and in the limit (roughly speaking, "for big $n$").

This is (often) fair since abstract statements have some (generally unknown) costs in reality, depending on machine, operating system and other factors, and short runtimes may be dominated by the operating system setting up the process in the first place and whatnot. So you get some perturbation, anyway.

Here is how asymptotic analysis relates to this approach.

Identify dominant operations (that induce costs), that is operations that occur most often (up to constant factors). In the Bubblesort example, one possible choice is the comparison in line 5.

Alternatively, bound all constants for elementary operations by their maximum (from above) resp. their minimum (from below) and perform the usual analysis.

Perform the analysis using execution counts of this operation as cost.

When simplifying, allow estimations. Take care to only allow estimations from above if your goal is an upper bound ($O$) resp. from below if you want lower bounds ($\Omega$).

@NikosM It is out of scope here (see also the comments on the question above). Note that I link to our reference post about solving recurrences which does present Master theorem et al.
–
Raphael♦Nov 13 '14 at 10:14

Execution Counts of Statements

There is another method, championed by Donald E. Knuth in his The Art of Computer Programming series. In contrast to translating the whole algorithm into one formula, it works independently from the code's semantics on the "putting things together" side and allows to go to a lower level only when necessary, starting from an "eagle's eye" view. Every statement can be analysed independently of the rest, leading to more clear calculations. However, the technique lends itself well to rather detailed code, not so much higher-level pseudo code.

The Method

It's quite simple in principle:

Assign every statement a name/number.

Assign every statement $S_i$ some cost $C_i$.

Determine for every statement $S_i$ its number of executions $e_i$.

Compute total costs

$\qquad\displaystyle C = \sum_{i} e_i \cdot C_i$.

You can insert estimates and/or symbolic quantities at any point, weakening resp. generalising the result accordingly.

Be aware that step 3 can be arbitrarily complex. It's usually there that you have to work with (asymptotic) estimates such as "$e_{77} \in O(n \log n)$" in order to get results.

In particular, $e_8 = e_3-1$ since the every recursive call in line 8 causes a call of foo in line 3 (and one is caused by the original call from dfs). Furthermore, $e_6 = e_5 + e_7$ because the while condition has to be checked once per iteration but then once more in order to leave it.

It's clear that $A=1$. Now, during a correctness proof we would show that foo is executed exactly once per node; that is, $B = n$. But then, we iterate over every adjacency list exactly once and every edge implies two entries in total (one for each incident node); we get $C = 2m$ iterations in total. Using this, we derive the following table:

Further reading

Algorithm analysis, like theorem proving, is largely an art (e.g. there are simple programs (like Collatz problem) that we do not know how to analyze). We can convert an algorithm complexity problem to a mathematical one, as answered comprehensively by Raphael, but then in order to express a bound on the cost of an algorithm in terms of known functions, we're left to:

Use techniques we know from existing analyses, such as finding bounds based on recurrences we understand or sum/integrals we can compute.

I guess I'm not seeing how this adds anything that's useful and new, over and above other answers. The techniques are already described in other answers. This looks to me more like a comment than an answer to the question.
–
D.W.Jan 28 at 18:53

1

I daresay that the other answers prove that it's not an art. You may not be able to do it (i.e. the mathematics), and some creativity (as to how apply known mathematics) may be necessary even if you are, but that's true for any task. I assume we don't aspire to create new mathematics here. (In fact, this question resp. its answers were intended to demystify the whole process.)
–
Raphael♦Jan 28 at 18:56

@Raphael Ari is talking about coming up with a recognizable function as the bound, rather than “the number of instructions executed by the program” (which is what your answer addresses). The general case is an art — there's no algorithm that can come up with a nontrivial bound for all algorithms. The common case, however, is a set of known techniques (such as the master theorem).
–
Gilles♦Jan 28 at 20:04

@Gilles If everything for which no algorithm exists were an art, craftsmen (in particular programmers) would be paid worse.
–
Raphael♦Jan 28 at 20:28