Thursday, June 21, 2007

What We're Doing Wrong : Programming in Theory Classes

My student Adam expressed some disappointment in my first few weeks of blogging. Knowing how opinionated I am, he was hoping for more incendiary posts. I was planning to avoid controversy until after a few more weeks of building readership, but what the heck, let's see if anyone's listening.

Here's my claim: theory does untold damage to itself every year by not having programming assignments in the introductory classes on algorithms and data structures.

While I could list many reasons why having some programming is didactically a good idea for such classes, here I'm targeting the theory-self-interest point of view. Most majors come into their theory classes having primarily done programming. We like to think we're showing them the beauty of computer science by teaching them the fundamentals of mathematics and modeling that make computer science amazing. But for most students, by not connecting it to what they've previously learned -- programming -- and not explicitly showing them the practical implications of that beauty -- efficiency -- we make it seem like theory is divorced from the rest of computer science. It's an impression many of them seem to carry with them. We're not getting our message across as powerfully as we should, because of a bizarre but apparently common mindset that says theory classes should be above programming.

I urge you, put a major programming assignment or a least a few programming exercises into your algorithms/data structures classes. Theory will be better off in the long run.

20 comments:

If anyone wants to organize a public burning of CLRS, I will certainly contribute my copy. This book almost kept me away from doing theory (fortunately, after trying compilers when I first came to MIT, I decided no matter how bad theory is, it can't be worse).

The trouble is that this book, and many courses, put way too much emphasis on rigurous proofs of obvious statements, and people can easily get the impression that this is what theory is about. What we should really teach is that we're looking for brilliant ideas with a mathematical guarantee. The emphasis is on brilliant.

To provide my own biased opinion, we should begin with a class on efficiency, not "theory". The high school computer olympiads got it just right: tell them they have to write code that solves a problem instance of size N in 0.1 secons. If their algorithms runs in 2^N, they get 10% of the score, since only N<20 fits in time, if it's N^2 they get 60% since they can handle N<50k, etc... What we should teach is that 1 week of careful optimization of an N^2 algorithm will never produce something as efficient as 1 day of thinking about an NlgN algorithm. After they really really understand this idea from practical experience (in an "efficiency class"), they can take a theory course.

I'm willing to say that I personally benefited from this style of teaching. Not only did I get to keep up my programming practice while taking a theory class, the programming assignments forced me to consider many of the details of a theory that I might otherwise ignore or hand wave. It is all well and good to say that merge sort runs in O(n log n) time, but actually implementing a divide and conquer algorithm and having to consider the edge cases and what to do when you can't evenly split your input in n pieces helped me really understand what was going on and how simple choices can make a big difference. As a student of your class, I would definitely say that this was a good idea.

When the Kleinberg/Tardos textbook was written, I believe it was meant for the junior/senior level algorithms course at Cornell. I believe the prerequisites for this class are two introductory courses in programming (covering basic concepts like sorting algorithms and data structures like binary search trees) and a discrete math course. In the introductory programming course, the students are required to implement some of the algorithms and data structures. By the time the students get to the algorithms course, they assume the students have enough maturity to discuss algorithms at a high level, without implementing them. As a student at Cornell, I liked the fact that the algorithms were only discussed at a high level, since I felt previous courses had already given me the skills necessary to implement those algorithms. So I felt the textbook fit well with the curriculum. I am not sure how the textbook is used at other schools however. Are students required to take some introductory programming courses before the main algorithms course?

Is it actually true that theory instructors don't assign programming problems? I would have thought that I would be the last of the theory purists, but I've been assigning programming problems in my theory courses for several years now. But as you point out, the textbooks are sorely lacking with good implementation problems for algorithms. I try to avoid ``implement this algorithm'' type excercises, in favor of ``algorithmic experiments'' to teach some kind of general point. For example, ``Compare the DP version of this algorithm to a memoized version on the following kind of data'' or ``Find the best threshold for the base case for a divide-and-conquer algorithm''. I don't grade people's homeworks, just their presentation of the results of the experiment. What other types of programming problems are appropriate?

I agree completely with Michael. In our data structures and algorithms class, I always give an assignment, such as

http://www.santafe.edu/~moore/361/project.pdf

in which students implemented union-find to do a percolation experiment. They got to learn that a clever data structure really does make their program run a lot faster; they also learned about phase transitions, power laws, fractal dimensions, and some basic experimental procedure (in which CS undergrads are typically greatly lacking).

I liked CLRS a lot (but I didn't have it assigned as a textbook). Different students like different things and I think we need a better idea of what they like. Feedback requests after a course and maybe even after graduation should help.

At the danger of sounding fawning, your own book with Upfal on Probability and Computing contains what I consider a sterling example of a good programming exercise, the Exploratory Assignment 5.8.

In fact, I just got these tongue-in-cheek comments back from one of my more mathematically inclined students:

Second process:

I spent a good amount of time trying to find an answer based on theoretical observations, and almost convinced myself to write some code. [...]

I still believe that this question is evil: it actually requires the student to perform the suggested coding. I am still looking for a way to prove the result without introducing the ”last node is almost always a leaf” un-guessable-with-pen-and-paper hint.

I think that is exactly what we are looking for: Insight through experimentation, not merely practical verification of running time guarantees or experience in writing code with pointer manipulation and memory allocation. (Even though these things are good, too.) Of course, such exercises are at least as difficult to construct as a sleek theory question.

Good, it seems that pretty much everybody is in agreement here. My practical question then is: what kind of software or website would people recommend to teach about finite automata, context free languages and Turing machines? (Say, the first 3 chapters of Sipser.) Having to program a TM should be a rite of passage for anyone wanting to become a computer scientist, but I'm sure that my TAs can do without having to check them by hand. What are the experiences with the various options for this that are out there?

What would be great would be to develop a repository of good programming problems. Data structures problems are easy to come by (and our students get plenty of hands on practice with them) but good algorithms programming questions are trickier.

One kind of assignment that is nice to do involves analyzing the cut-over point for divide-and-conquer algrithms; i.e. don't run the recursion to the bottom - when should one switch? (Years ago Iam Munro suggested the nice problem of optimizing when to switch from Strassen's algorithm to ordinary matrix multiplication - the cross-over point is closer to 10x10 than 100x100 matrices.)

Good textbooks could serve this role as they do for conventional problems but there is no reason to wait for a new texbook.

BTW: My two favorites for pencil and paper algorithms questions are Udi Manber's old algorithms text and the recent algorithms text by Kleinberg & Tardos (my current favorite text by a wide margin). Coincidently, both texts take a very constructive approach to algorithm design as opposed to the usual catalog-based approach.

I already posted this on my own blog, but Michael suggested I copy it here as well.

- At UCI, we're on a quarter system, and there's only one quarter of required theory (more specifically, algorithms) in our undergraduate major. Ten weeks is simply not enough time to include both the theory I want to cover, and programming on top of that. We do have a separate algorithm implementation class that Dan Hirschberg teaches as an elective.

- Despite not wanting to add programming assignments to our algorithms course, I feel strongly that the students should understand that the reason for studying algorithms is for their use in programming, rather than as an obscure and abstract branch of mathematics.

- Partly for this reason, when I describe algorithms in pseudocode, I use a pseudocode syntax closely related to Python, sometimes so close that what I describe is actual runnable Python code. Why Python? Because its high level, indentation-based block structure and lack of variable declarations makes it already look very similar to pseudocode, and I think writing pseudocode in a way that is already very close to code helps the students understand the connection between theory and implementation.

- In terms of textbooks, I see the lack of connection to programming as a major weakness of CLRS. I feel that it describes the algorithms as mathematical entities that the students are supposed to find interesting per se, and doesn't really talk about implementation of algorithms or even give sufficiently concrete scenarios for the usefulness of the algorithms described. And the lack of focus on implementation causes problems for some of the description of the more advanced algorithms (e.g. where are the flow values and residual capacities stored in the network flow algorithms) making it difficult to implement them from the descriptions provided.

- The textbook I use for my undergraduate algorithms classes is "Algorithm Design" by Goodrich and Tamassia. It includes implementations in Java at the end of each chapter, which I largely ignore in the lectures but could be helpful to any students who actually read the text. But more importantly I think it does a better job of covering the connection between algorithm and implementation. And it's not as comprehensive as CLRS but there's plenty of material to fill a quarter and cover the essentials. (Disclaimer of bias: of course, one of the authors is also a colleague at UCI and a frequent coauthor.)

let us not forget that there are two "theory" areas in computer science: algorithms/complexity and languages/logics.

the algorithms area has, as michael suggests, tended to ignore implementation, but let's remember that this is not universally the case. mehlhorn's group at the mpi has long been stressing implementation (realized, for example, in leda). and blelloch's algorithms in the real world exemplifies the kind of approach michael is suggesting. i am sure there are others, but these are two good (counter)examples.

the languages/logic side of theory, which tends to be neglected in discussions such as this, has always emphasized the close (even perfect) correspondence between the theoretical concepts and their realization in running code. for example, type theory / constuctive mathematics is nothing short of a comprehensive theoretical framework for doing mathematics that has intrinisic computational content --- you can compile proofs into running code!

so, yes, michael is quite right, but the situation is not so bad as the title of the entry might suggest! for a sufficiently broad value of "we", i would assert that "we" are doing rather well at integrating theory and practice!

I thought I will agree with Michael on nearly everything, but here is an example where I disagree (if controversy is what he wants...).

I think undergraduate algorithms is one course where we have a chance to teach the students the beauty of algorithmic thinking, problem solving and proofs. There is a nice structure to think about algorithmic solutions starting from the naive solution, guided by O(), to try and design the best possible solution. Also, learn truly nontrivial ideas. I don't think we should clutter that with programming exercises.

Graduate algorithms course provides one opportunity to educate students (who will go on to do their thesis work in OS, databases, languages, HCI, networking, whatever) to put on the formal hat, seek the best possible, look for and use something beyond the greedy algorithm or the trivial NP completeness proofs that follow from formalizing one's problem in such general terms that half a dozen NP hard problems turn out to be special cases!

I once read this in the context of a debate between competitive vs . recreational youth soccer: it is foolish to think that competitive soccer need not be recreational, and recreational soccer need not be competitive.

Michael's post highlights a point Muthu made on his blog recently (too lazy to link :-)... there are two important distinctions to make: theory vs. practice and theory vs. systems. Just as it is good to infuse a programming (practice) component into theoretical courses, it is equally important to infuse sound theoretical principles (correctness arguments, time/space analyses, etc) into systems courses.

Without giving away my age, I will say that when I was an undergraduate CS major, the operating systems or networking courses had little or no theoretical component (perhaps other than a dining philosopher "argument"). I don't think things have changed much.

Over the life of CS, I feel that courses (of all kinds) and degree programs have tended to be remarkably inflexible in their structure (less so in content). The result is that we have a division of courses into "introductory" (programming, data structures, architecture...), "theoretical" (theory of comp, algorithms), and "systems" (compilers, operating systems, networks)... by abstracting the elements too much, we're doing our students an injustice: we don't show them the true nature of computer science, and how theory and practice come together to achieve great things.

I program lots of (little) stuff recreationally and professionally. LISP made sense when I took my AI course (as part of my OR master's degree work), and I grew up writing C, so that makes sense.

Trying to code and traverse a complicated tree structure in perl or python (like I've been trying lately) is a serious pain in the ass. The language semantics obscure the function that I'm trying to implement. But then, that is the challenge, right, to find a way to use the theory to develop something practical.